Scaling to thousands of models in financial services

AI and machine learning technologies are helping financial services firm Morgan Stanley use decades of data to supplement human insight with accurate models for fraud detection and prevention, sales and marketing automation, and personalized wealth management, among others. With an AI practice that’s poised to grow, the firm is leveraging MLOps principles to scale AI and ML.¹²

“We need to be able to scale from hundreds of models to thousands,” says Shailesh Gavankar, who heads the analytics and machine learning practice in Morgan Stanley’s Wealth Management Technology department. “There are limitations to doing everything manually as long as data scientists and data analysts are working on their own ‘island’ without the ability to collaborate or share data.”

Currently, the practice is using common platforms for managing data and developing, deploying, and monitoring ML models. To build and test models, people created a sandbox with access to a centralized data lake that contains a copy of the data used in the production system, a technique that makes it easier to bring models from development into production.

In the development environment, data scientists, business analysts, and data engineers across the practice can access the same standardized data in near-real time, enabling them to efficiently and collaboratively explore, prototype, build, test, and deliver ML models. Advanced techniques mask personally identifiable information so the teams can generate insights without exposing sensitive data. “Across our AI practice, processes are built around data accuracy and privacy,” Gavankar says. “Applying the highest standards to the training system ensures that we meet data compliance and regulatory requirements.”

For good model governance, transparency, and accountability, an independent, in-house model risk management team was established. With years of experience deploying trading models, the team is responsible for assessing risk and validating the quality of ML models before they go to production. The team evaluates the accuracy of the models and works to identify sources of bias or other unintended consequences. They also review data lineage as well as plans for production monitoring and intervention should the model start to drift.

As its AI practice evolves, Morgan Stanley Wealth Management will be focusing on continuing to improve speed to market by further automating the model risk management process and integrating the sandbox and production systems. “As MLOps tools and processes enable us to operationalize models more efficiently,” Gavankar says, “we can continue to increase the number of models in production and more fully leverage AI’s ability to drive better business decisions.”

One-stop shop for model development and deployment

As AI and machine learning transform health care, health insurer Anthem, an industry leader in the use of clinical, customer-facing AI applications, is increasingly leveraging AI to reimagine and reinvent critical back-end business processes. About two years ago, the company embarked on an AI-supported journey to streamline claims management. As part of that process, leaders built a platform that consolidates model development and deployment across the enterprise.

Anthem initially built several ML models that revealed patterns in claims data, made predictions to speed processing, and identified and corrected errors. The models were successful—and leaders realized they needed to scale. “As the models began to deliver business value, we realized we needed infrastructure that could help us develop and operationalize machine learning more efficiently,” says Harsha Arcot, senior director of enterprise data science.¹³ “To address this challenge, we decided to build a single interface for all AI and ML solutions across the Anthem ecosystem.”

The company built an integrated development environment and an end-to-end platform that serves as a one-stop shop where developers and data scientists prepare and store training data, build and validate models via easy-to-use interfaces, and deploy them at scale. A feedback mechanism allows models to continuously learn and improve while a separate platform monitors the performance of production models.

Simultaneously, the company has been working on an initiative that consolidates data from seven systems into a single repository. With most of that work complete, the process of finding the data to build, train, and operationalize models is much more efficient.

The platform also provides Anthem with the flexibility to duplicate models for multiple use cases. For example, if a pipeline is already built out into the legacy claims system for a commercial use case, it can also be easily deployed on the consumer side. “It’s much more efficient than when we used to develop a model for each use case from scratch,” Arcot says.

Using the platform, Anthem data scientists have developed a number of models, including those that fast-track the processing of pre-approval claims, identify and automatically reject duplicate claims, and determine whether a medical procedure needs preauthorization. Previously, a human claims examiner or clinician needed to manually review and process all of these claims.

Arcot says the platform has dramatically increased model deployment speed. “Before we developed the platform, it took about six months to deploy very simple models,” he says. “Now we are able to develop much more complex initiatives in half the time.”