Top Three Questions You'll Get Asked When Developing an AI-Driven Early Warning System

At Deloitte, our journey with AI-driven solutions began long before the rise of Generative AI, back when we weren't asking chatbots for the perfect soufflé recipe or insights into quantum mechanics.

Over the past five years of running our AI-driven early warning system (EWS), we’ve learned there is “no one size fits all” solution and that careful bespoke consideration is needed to make EWS fit organically into your ecosystem. Working with organizations worldwide, we've had many insightful conversations with users, enthusiasts, sceptics, and experts alike. Three questions consistently emerge, highlighting the key considerations for anyone developing or integrating an AI-driven EWS:

1. How do you achieve a forward-looking view of emerging risks?
2. How do you strike the right precision-recall balance?
3. How do you protect users from fake news?

Question 1 - How do you achieve a forward-looking view of emerging risks?

Let's rewind to the beginning. Regulators have long encouraged firms to implement mechanisms for early identification of emerging risks, enabling proactive mitigation. We embraced this challenge ourselves. In the context of credit risk, for instance, this means identifying borrowers likely to face difficulties meeting their financial obligations.

The million-dollar question is: what data sources should we use? Financial statements and loan repayment data are traditionally the first places to look as they are readily available, but they have the disadvantage of being lagging indicators of risk. Our search, therefore, turns to external data. Experience has taught us that casting a net as wide as possible yields the best results as diverse data sources increase the chances of spotting emerging risks at their genesis.

Figure 1. Predictive horizon of selected data types. 

There is a catch though: no single data source can capture every type of emerging risk: Operational challenges, legal and compliance issues, supply chain disruptions, shifting socio-political landscapes, market trends, natural disasters – the list goes on. While some of these business challenges ultimately manifest themselves in covenant breaches and financial distress, our goal is to raise the alarm much earlier than this point.

We've identified at least 300 distinct risk event types that can impact a company's ability to meet its financial obligations. By tapping into a diverse range of nearly 400,000 data sources, we achieve sufficient coverage not only to spot risks but also to cross-validate information credibility by measuring the "cluster size" of corroborating data points.

Harnessing this volume of data, especially unstructured text data from news and online media, necessitates a robust approach and Natural Language Processing (NLP) models perform the task well. They transform unstructured data into structured insights, paving the way for traditional predictive algorithms to generate risk predictions.

So, how do you achieve that coveted forward-looking view? By augmenting traditional data (financial, covenant, internal behaviour, bureau data) with news media, market, and macroeconomic data from as many diverse sources as possible. This creates a multi-dimensional picture of a company, enabling us to not only provide early warnings but also support other use cases like prospecting, KYC, and strategic decision-making.

Question 2 - How do you strike the right precision-recall balance?

Precision evaluates the quality of our predictions: "Of all the positive predictions made, how many were actually correct?" 

Recall, on the other hand, answers: "Of all the actual positives, how many did the model correctly identify?" High precision minimizes false positives, while high recall minimizes false negatives.

The dilemma lies in choosing which to prioritize. Two common approaches exist:

1. Prioritize recall: This ensures no emerging risk goes unnoticed, but it can lead to alert overload, especially for large companies, potentially overwhelming manual review processes.
2. Prioritize precision: This minimizes false positives, but risks overlooking some risks and shortening the prediction horizon.

Our experience suggests a more balanced approach: start with high recall to minimize false negatives, then gradually reduce recall as confidence in the system's output grows. This process requires nuance, adapting to different portfolios, business sectors, and geographies. Factors to consider include:

• Desired accuracy,
• Cost of false negatives vs. false positives, and
• Human capacity to address alerts.

Question 3 - How do you protect users from fake news?

Data credibility is paramount. We establish and validate it on multiple levels:

• Source level: Rigorously investigate and test each data source before production use. Social media, for example, rarely offers sufficient credibility.
• Event level: Upon identifying an emerging risk, the algorithm cross-references underlying data with other sources, measuring the "cluster size" of corroborating data points. A growing cluster size strengthens the risk signal.

Assuming the solution is now well designed, what is the secret sauce to a well-performing EWS? Humans in the loop. While alternative data, like market sentiment and media news offers valuable insights, it can also introduce noise. Human involvement is crucial for improving the system over time.

Engaging end-users adds immense value. Establishing a feedback loop allows users to provide input, which in turn allows a model to have increased weights of predictive and reliable data for each portfolio cluster and set a balanced precision-recall. This makes navigating the three complex questions we discuss in this blog a whole lot smoother.

Read about how engaging end-users improves model predictions Early Warning Systems: The Unbeatable Team of Human and AI | Deloitte UK

Risk Alert webpage: Risk Alert: The Early Warning System of the Future | Deloitte UK