How to Identify and Overcome Bias in a Predictive Model

This is the first in a series of articles by Valen Analytics looking at the hurdles that insurers must overcome to effectively implement and gain value from data analytics programs.

Becoming a data-driven company in the insurance market today is essential, but it requires an understanding of the incremental adjustments necessary to keep a predictive model accurate and operating to its full potential.

Executive Summary

Brokers may be the first to notice a bias in a carrier’s predictive model—indirectly—because of the impact of the bias on their commissions, notes Kirstin Marr, president of Valen Analytics. With predictive analytics now increasingly being used by insurers and incorporated into decision-making, analytics also have a growing impact on carrier bottom lines, she says, offering some warning signs of bias and methods to overcome it.

Monitoring models in production is critical, as is knowing when they may inadvertently contain biases that could negatively impact the financial health of an insurer’s book of business. Bias can creep in from a number of sources that skew a dataset, and a change in market conditions or strategy can render a model obsolete. There are several remedies to be aware of to ensure a model matches your risk appetite.

The Warning Signs

Insurers need to consider data and analytics as a hands-on investment. Models must constantly be analyzed, refreshed and questioned to keep them operating at the highest possible efficiency. Identifying and overcoming biases built into predictive models can be challenging, but there are common trigger elements that indicate a bias and require an insurer to act.

Warning signs that help determine this include:

Large discrepancies between the model’s suggestion and current pricing for new and renewal business. If a new model is pricing a majority of new and renewal business in ways that aren’t aligned with customer or underwriter expectations, it’s possible there is a built-in bias.
Broker dissatisfaction. When a model performs in strange ways that result in unexpected swings in pricing, there is an obvious impact on broker commission. Brokers and agents on the front line tend to be the first to notice and raise a concern.
Price changes are limited to groups with a specific identifier or common trait. This could be a particular class of vehicle in commercial lines or a variance in one state vs. another. Regardless, when one of a group of policies with a common identifier is priced in strange ways, it’s possible the model has a bias on that class, line or state.

It can take a few months to identify whether shifting pricing is, in fact, the designed outcome of implementing a new model or a shortcoming. It’s often more challenging to determine in instances where an insurer has implemented predictive analytics without dedicated resources.

These are simply warning signs. Many insurers implement models precisely for the reason of shifting pricing to be more representative of actual risk. It’s important to identify whether symptoms are because of a model bias or the result of a planned shift in business strategy. It can take a few months to identify whether the change is, in fact, the designed outcome of implementing a new model or a shortcoming. It’s often more challenging to determine in instances where an insurer has implemented predictive analytics without dedicated resources, internal or external.

How Bias Occurs

Biases can occur in models for a number of reasons. In most cases, predictive models are created using anonymized information from policies that have already been generated. If, for instance, an insurer decided that any company with a poor credit score would be a bad risk, it’s possible that decision is being factored into new policies.

It’s also important to realize that credit scores can change. A model using out-of-date information might inadvertently keep insurers from targeting business within their risk appetite. Having information like a credit score can be very helpful, but it’s important to rely on a variety of factors, rather than a single detail. If this has occurred in the past, it’s possible the reliance will have a negative impact on analytics moving forward.

Niche Bias

For insurers traditionally focused on a specific product, niche biases are another concern, especially as the carrier attempts to expand into new markets. For instance, a carrier that only supports fire trucks may mistakenly believe that their model data should be applicable to ambulances. Similarly, carriers moving from one state to another may also encounter issues.

Niche biases are generally driven by the hope that applicable data to one subset of insureds will apply to another subset. This seamless transition rarely works the way insurers hope it will.

Overcoming Bias

There are a few ways to overcome a bias in a model. But taking steps to eliminate bias prior to putting a model in production is the most important. The age-old adage “an ounce of prevention is worth a pound of cure” applies since a biased model can have long-term impact to a book a business.

It’s critical to gather data from across an organization and utilize an insurer’s institutional knowledge before populating the model with external data.

Building an unbiased model requires a broader range of data than most insurers typically expect. This is both internal data, including asking underwriters for their input as models are being developed, and external, such as incorporating consortium data to fill in gaps.

Assuming a biased model made it into production, remediation requires a full effort across a number of departments to understand the impact of bias in a model and to rectify the situation. This will mean a thorough examination of every policy decision to ensure it’s priced in line with the risk it presents. Comparing loss ratios on an account level will also help to understand the impact of the bias. The policy-by-policy examination should include an in-depth look into which underwriters are most likely to incorporate the model into their pricing decisions.

This step may identify unexpected results, such as the possibility a model isn’t being used. Valen’s own 2018 Insurance Outlook found that 55 percent of front-line employees were resistant to new technologies. This means there’s a chance that a model is accurate and simply not being incorporated.

No matter how mature an insurer’s use of data is, there are many opportunities for predictive models to be built with a bias. These biases, when left unchecked, can profoundly impact an insurer’s book of business, leaving them with more risk than they want or with fewer renewals on the business they want to keep.

It’s critical to gather data from across an organization and utilize an insurer’s institutional knowledge before populating the model with external data. Insurers that believe they have a bias in a production model must perform an examination across each policy to identify the impact to their book of business and be ready to adjust the model accordingly.

Part 2: 4 Ways Execs Can Increase Underwriters’ Embrace of Predictive Analytics