Continuing our previous deeper dive into certain aspects of the Modeling Lifecycle concept, this installment is the last entry in the lifecycle—model monitoring. While monitoring is the last step in the process, it is arguably one of the more important steps since it can send a modeler back to nearly every other earlier step in the lifecycle. However, model monitoring often receives the least attention.

It also seems surprising that monitoring doesn’t get more attention. After all, with so much work devoted to building and implementing predictive models, and with models being such a critical part of an insurer’s operations, why wouldn’t a company want to know in detail how a model is performing?

Fortunately, the topic of model monitoring has been getting increased visibility and air time, including at conferences at which I was scheduled to present earlier this Spring, the cancelled iCAS Community of Practice Event and the CAS Ratemaking, Product and Modeling Seminar in New Orleans. As insurers continue to devote more time and resources to predictive analytics, similar attention must be paid to model monitoring as well.

Model monitoring is a potentially rich topic with both business and technical aspects. This particular blog entry focuses more on the case for model monitoring, covers some fundamentals, and addresses a few business considerations. A future entry will take a closer look at the tactical ways to approach monitoring.

At its core, there are a couple key pieces to model monitoring: the accuracy of the model and the business outcomes desired. Accuracy simply means that there are no issues with the data inputs going into the model and that the output is being calculated correctly. 

While not as easy as it may sound, the accuracy part of model monitoring is a more straightforward exercise. The more challenging part, which at times is as much art as science, is identifying if the model is performing as the business expected—and if not, why not.

For both of these parts, although it increases the amount of data needed and time spent, monitoring both the inputs and the outputs of the model is very important. Unfortunately, just outputs are often considered when designing a model monitoring plan. That makes it very difficult—if not impossible at times—to understand why a model is performing in an unexpected way.

There are a number of reasons why model monitoring is important. The following list includes some examples of situations that can occur where undesirable (or even disastrous) outcomes can result if a close eye is not being kept on the models:

• Data stops feeding in correctly to the model
• Changes are made elsewhere in the insurer’s system that breaks something “unrelated”
• Internal or external data shifts
• Shifts in the population of new business written or in the existing book of business that renews
• The model ages and degrades
• New business written is much different than expected
• Underwriting guidelines change which impact the business written or renewed
• The characteristics of the book of business change such as deductibles, amounts of insurance, or credit scores

Needless to say, if a solid model monitoring program is in place, the ill-effects of the above situations can be greatly mitigated or avoided.

To have a successful model monitoring program, there are three important components. A company needs to have:

• the right focus on monitoring
• the right people
• the right tools

Having the right focus means, among other things, having leadership support to devote time and resources to the work. It also includes making sure that monitoring is a high enough priority that there are people who have model monitoring as a primary responsibility, and not just something they do “if they can get to it.” 

Having the right people is more than just having enough people, however. Ideally, employees that do model monitoring are ones that have a good understanding of the models, data, and business to which the models apply. Successful model monitors are able to find the valuable aspects to pursue and are able to put the puzzle pieces together to make sense of the sometimes obscure clues that they see. 

In addition, they need to be able to clearly communicate results to others, especially those that do not know the technical details as well as they do. 

Finally, having the right tools is a key piece of a successful model monitoring program. It’s important to be able to have tools that allow for a dynamic analysis of the data—to be able to easily slice and dice the information—not just have static reports. In conjunction is the ability to easily create visualizations of the data.

Model monitoring is a critical part of the modeling lifecycle, although often times it can be a forgotten part. It is important to devote enough of the appropriate resources and attention. The next installment of this blog will dive further into of the tactical details of model monitoring. In the meantime, if you are interested in strengthening your model monitoring program, let us know! Pinnacle experts would love the opportunity to discuss with you!