Faster iteration or tighter governance? An innovative AI/ML platform supports both.
Deloitte helped Thomson Reuters enable a stream of AI/ML capabilities within a single user interface.
The situation
Customers look to Thomson Reuters as a trusted source of content and technology. And with that understanding, the company has invested heavily in the use of automation to extract timely and important data that can help legal, tax, compliance, government, and media professionals make more confident decisions. These artificial intelligence/machine learning (AI/ML) models have been implemented across Thomson Reuters' enterprise and are driven by complex machine learning models that leverage state-of-the-art industry algorithms.
Maria Apazoglou, Thomson Reuters’ VP AI/ML & BI Platforms, had a deep understanding of the evolving AI/ML landscape and how she could apply it to help with customer success. She knew data science models can become vulnerable to performance issues (due to concept drift and data drift over time) and how imperative it becomes to assess model ethics like bias detection.
Businesses often combat this with performance modeling that requires individual data scientists to track the deterioration of models in production and then liaise with the relevant model governance teams to adjust. The number of models has grown significantly, as has the complexity of the problem statements they’re solving. That growth can increase business exposure and risk.
“As customers advance on the AI maturity curve, Deloitte observes that MLOps, ethics, explainable AI, and traceability for root-cause corrective action become focal areas for AI platform enhancements. The services we developed in collaboration with Thomson Reuters are significant steps in that direction,” noted Omer Sohail, principal, Deloitte Consulting LLP.
Apazoglou envisioned a single platform that could intentionally balance speed and governance, a platform where data scientists from across the enterprise could register specific models but leave performance tracking to an AI/ML monitoring system. The goal, she says, was to create a platform "flexible enough to accommodate different user personas, workflows, use cases, and types of models and at the same time standardize some core capabilities that then enable us to improve the way we govern, view, and trust our models."
As AI/ML models grow more complex, can monitoring them become simpler?
The solve
Apazoglou needed a formidable team to execute on this vision. Our strong relationships within Thomson Reuters and our capabilities in areas such as AI/ML, application development, and MLOps helped her see how Deloitte could become a seamless extension of her internal team. She set high-level goals for the platform with stakeholders yet maintained a laser focus on smaller details to incorporate into the innovative MLOps stack. Open communication with intended users was key. Apazoglou shared, "I've seen many situations where companies embarking on creating an enterprise platform fail to understand what the users are doing right now. What are the true gaps they have? And what can they do to get true efficiencies and improvements?"
The platform had to be use case-agnostic and scalable, with a standardized user interface (UI) that could be used by data scientists and model owners across the organization, no matter what program they'd used to create a model. With an approach based on agile execution, the team—including Deloitte solution architects and MLOps engineers—took parallel paths that enabled rapid iteration; weekly demos; and integrated, end-to-end demos for the broader Thomson Reuters user community, so stakeholder feedback could be addressed throughout the journey.
As much as possible, the platform was designed to be "clickable" to facilitate easier adoption. “We’re trying to give our data scientists a whole stream of capabilities, but the way they get this is through the same UI,” Apazoglou emphasized. The system also enables multi-account setup orchestrated by a single UI. That can mitigate scalability problems while giving Apazoglou's team the ability to collect metadata to understand how Thomson Reuters' models are being used and how healthy they are.
The entire solution was developed using various AWS services and can serve both SageMaker-supported and unsupported models through the custom design. For now, it includes three components—for model monitoring, data drift monitoring, and bias, with plans for explainability as well as improvements on the existing services down the road—and alerts relevant data scientists and stakeholders if drift is detected so they can review the data and make needed adjustments.
A platform is more useful when it’s built with users in mind.
The impact
Model governance is essential for any AI/ML model, and the fully customized monitoring application enables a smoother, more efficient process while also providing immediate visibility into how models are performing.
Thomson Reuters' data model and governance team has augmented its ability to track changes in data and identify bias in a continuous manner, and data science and product teams are better positioned for tracking performance and deterioration over time.
This can mitigate potential risks by identifying potential issues earlier and empowering timely decisions around retraining the models. For model owners and business owners, the model registry can serve as a single source of truth and a reliable resource when it comes to audit, traceability, model lift, and implications of bias and (in the future) explainability.
“Machine learning and AI can help us serve our customers while optimizing performance,” says Glenda Crisp, head of data and analytics at Thomson Reuters. “This is a key part of our strategic vision for the future.”
The initial focus for the platform has been on models used within Thomson Reuters' legal content capabilities, such as those built around classifying documents and decisions.
Apazoglou's team is moving forward with monitoring more complex models that can enable summarization of legal documents, and there are plans to expand the platform to models used internally to study customer churn and financial forecasting.
Thomson Reuters has been a pioneer in the wide adoption of responsible and trustworthy AI across its businesses, and Apazoglou is committed to continuing that practice.
“We embed AI in a lot of our products; we have AI that's quite complicated,” she noted. “Being able to iterate as fast as possible is really key for us to be able to continue to have a competitive advantage when it comes to external-facing products.”
Streamline how data scientists build AI/ML models. Let AI monitor them.
