Skip to main content

Model risk management

Do you know what you’re looking for?

Financial Markets Regulatory Outlook 2023

In focus

  • In the UK the Prudential Regulation Authority’s (PRA) broad-ranging proposed principles for model risk management (MRM) will be finalised, significantly expanding the scope of models subject to MRM oversight.
  • Climate risk and the use of artificial intelligence (AI)/machine learning (ML) have emerged as specific areas of supervisory concern in the EU and UK, each of which creates novel challenges from a MRM perspective. Supervisors will expect firms to be able to demonstrate that they are not beholden to “black boxes” for climate risk modelling, and to demonstrate that people, not AI/ML tools, are ultimately responsible for understanding and taking business decisions.
  • Supervisory capacity to review and approve firms’ models is a potentially significant bottleneck in many jurisdictions, creating challenges for firms looking to adjust models in response to Basel 3.1 or Solvency II revisions.

The ever expanding use of models across financial services to aid business decisions and risk management has prompted supervisory concerns around the extent to which firms appreciate and manage the possibilities that their models do not work as expected. These concerns span familiar areas, such as bank credit risk and insurance solvency modelling, but supervisors are also eyeing emerging and less well understood modelling challenges around climate risk and the use of AI/ML. 

Proposed PRA principles on MRM for banks

The PRA’s proposed principles on MRM for banks represent a significant elevation of the bar, and when finalised later this year will demand a significant programme of work to catalogue, categorise and risk-assess the models they use, and to improve governance and oversight processes. The PRA has said it expects an “Initial self-assessment” along with “prepared remediation plans” for them to comply with within a 12-month period.

MRM currently only applies to banks. However, the PRA intends to make MRM applicable to insurers once the Solvency II reforms are finalised. In the meantime, the PRA has indicated in its supervisory priorities for 2023 that insurers should consider how the MRM principles could be applied.

The PRA’s basic concern remains that many banks are not monitoring or managing effectively the aggregate risks they face due to the broad range of models they use. The overarching aim is to ensure that senior management and Boards have clear sight of the aggregate risk that models represent and receive reporting to enable them to be confident that the risk is being managed.

The main challenge for firms will stem from the PRA’s proposal for a very broad definition of what constitutes a “model”, covering any instance where a qualitative or quantitative input is subjected to a transformation that produces a qualitative or quantitative output. This captures traditional credit and market risk models, but also extends to a wide range of algorithms, estimators, heuristics, decision trees and spreadsheet-based calculators that banks may not currently define as models. Banks with regulatory approvals to use models for credit and market risk capital calculations may have hundreds of such models; beyond these categories, large banks may count their “models” under the PRA’s definition in the thousands.

The first stage of implementing the principles – subject to any changes the PRA makes to the definition of a model - will require banks to create an inventory of models and undertake a risk-classification exercise for all in-scope models. This will be challenging, not least because the technology capabilities required to support the expanded model inventories may go beyond those of existing systems. Furthermore, processes that may appear innocuous (for instance, a spreadsheet that collates inputs from three source systems and adds the figures together to form part of a finance/risk reconciliation adjustment for the monthly management accounts) may now count as a “model”. The creation of the inventory will need to be iterative to deliver the appropriate scope.

Banks already have in place model oversight and governance processes for models subject to regulatory approval, but the order of magnitude increase in the number of models within the scope of the proposed principles implies a significant resource stretch for teams that are in some cases already struggling with existing supervisory modelling requirements, including on internal ratings based (IRB) repair, hybrid mortgage models, and preparation for Basel 3.1 implementation.

Firms will need to take advantage of the risk sensitivity and proportionality built into the PRA’s proposals, to ensure that lower risk and less material models are subject to lighter touch oversight. Furthermore, not every component of a MRM process requires qualified, experienced statisticians, and firms should seek to identify aspects of their workflows that are amenable to automated solutions. The overall process may be time consuming, but it should bring transparency to decision-making and present opportunities for firms to rationalise what may be unnecessarily diverse and fragmented models across all areas of their operations.

Firms should look to develop remediation plans on the back of their implementation programmes, identifying opportunities to improve the consistency of model inputs and outputs by consolidating data sources and amalgamating models. 

EU supervisory focus

There is no EU equivalent to the PRA’s proposed principles, although EU banks face challenges of their own, continuing to deal with the findings of the Targeted Review of Internal Models (TRIM) exercise,1 the resolution of which is expected to increase risk weighted assets in affected firms by 12%, or €275 billion. The 2021 TRIM report indicated the need for extensive remediation work also. While EU banks have made improvements, this remains work in progress, with firms also having to face a similar set of economic, credit, IRB repair and Capital Requirements Directive 6/Capital Requirements Regulation 3 pressures on model resources as those listed above.

Meanwhile, the European Banking Authority (EBA) and European Central Bank (ECB) are turning their attention to the robustness of the models underlying International Financial Reporting Standards (IFRS) 9 impairments and Interest Rate Risk in the Banking Book (IRRBB), given changes in economic conditions and in particular the reversal of a decade of low interest rates. One of the EBA’s areas of concern is whether IFRS 9 and IRRBB models built during a low interest rate environment remain robust and reliable in the current higher interest rate environment. 

Climate risk modelling

Supervisory attention on the incorporation of climate factors into risk modelling is also increasing. From a model risk perspective, two issues stand out. First, the challenge of building and validating models that can meet expected standards of accuracy and reliability given incomplete climate data that may rely on interpolation, extrapolation or proxies (see more on sustainability data). Second, the potential for over-reliance on third-party solutions for elements of climate modelling, particularly in climate stress testing.

Supervisors want to ensure that firms are not dependent on black boxes and that any bias in models is understood and managed, and we expect supervisors to press firms to improve the incorporation of climate risk into their risk and stress test modelling frameworks in 2023.

External auditors also increasingly expect firms to demonstrate they have undertaken additional modelling to show that the impairment allowances held under IFRS 9 are appropriate in the context of financial risks from climate change. This additional layer of modelling, particularly if significant management judgement is incorporated, is an area where robust internal oversight and challenge should be applied. See the climate risk and the climate-nature nexus chapter for more on climate risk.

"Given that we are in the early stages of modelling climate risk, it is not surprising that firms differ in the degrees of sophistication that they exhibit in modelling. What was surprising to me was that even within a given firm there tended to be a lot of variation in how different parts of the organisation modelled things."

Anil Kashyap

FPC member2

AI/ML-based models

Model risks also arise from the incorporation of AI/ML into models across all sectors. Supervisors understand the potentially significant benefits of using AI/ML in risk assessment and management, but want to be convinced that boards and senior management understand the strengths and weaknesses that AI/ML bring to models. No matter the level of analytical sophistication AI/ML may offer, supervisors in the EU and UK expect people, and not models, to be ultimately responsible for making decisions.3 Firms making significant use of AI/ML modelling techniques are likely to need dedicated technical teams to undertake work around AI/ML model risk and validation. 

"72% of firms that responded to a 2022 joint Bank of England/Financial Conduct Authority survey reported using or developing ML applications4 "

Boards and executives of firms should be able to demonstrate that they understand the decisions that AI/ML models are intended to make and where the boundaries of those decisions are set, and this should be reflected in management information that enables model performance within those boundaries to be understood.

Both climate and AI/ML modelling present opportunities and/or requirements for firms to broaden significantly the sources of data they use for modelling: in both instances the extent to which firms have challenged the availability, quality and relevance of new types of data is likely to be questioned by supervisors.

Supervisory review of models

Resource pressures around MRM are not unique to regulated firms, with supervisors themselves facing capacity constraints in many jurisdictions. With numerous current and upcoming priority and/or mandatory model changes for which regulated firms will need supervisory permission, including around Basel 3.1 for banks and Solvency II for insurers, supervisors will need to find ways to increase the capacity of their model review processes, while not compromising on their standards. This also puts a premium on firms’ investments in their own capacity, skills and resources, in order to reduce the extent to which they will be vulnerable to deep or frequent regulatory intervention over the coming year. 

Actions for firms

Model Oversight

  • Ensure that once the expanded model universe has been identified, the Board and senior executives understand and are able to explain to supervisors the level of the firm’s reliance on models, and their models’ strengths and weaknesses.
  • Ensure that all options for taking a risk-based approach to classification and oversight of models are fully understood, and that where possible, model oversight processes take advantage of technology options to reduce reliance on scarce validation resources.


Data and AI/ML

  • Actively seek opportunities to expand sources of data, and improve the quality of data so risk models are as robust as possible, and ensure any weaknesses are understood by the Board and senior executives. Industry initiatives, particularly on climate data, may provide avenues to accelerate these efforts.
  • No matter the level of analytical sophistication AI/ML may offer, supervisors in the EU and UK expect people, and not models, to be ultimately responsible for making decisions.


Supervisory review of models

  • Ensure that where supervisory review of models is required, this is flagged as early as possible and that models are fully ready for supervisory review in accordance with agreed schedules. 

Endnotes

1 The ECB’s TRIM exercise looked at consistency of risk modelling across 65 EU banks, involved 200 on-site investigations, and resulted in some 5,800 findings

2 Speech by Anil Kashyap, Member of the Financial Policy Committee

3 EBA, Discussion paper on machine learning for IRB models, 11 November 2021 & Bank of England, CP6/22- Model risk management principles for banks, 21 June 2022

4 BoE/FCA, Machine Learning in Financial Services, October 2022

Did you find this useful?

Thanks for your feedback

If you would like to help improve Deloitte.com further, please complete a 3-minute survey