The Growing Importance of Model Risk Management: What should you focus on now to raise the bar?
The number of models being developed and used in financial institutions continues to rise and firms are becoming increasingly reliant on models to make sense of complex relationships and large volumes of information to support decision making. In some cases, decisions are made directly by models using automated systems. But in most cases, the information or output produced by models is used indirectly by the business as part of a decision-making process.
Models are a simplistic representation of a real-world process or behaviour. They are useful because they can simplify, rationalise, and accelerate information gathering and analytics as well as creating consistency and auditability of decision making. But they are always imperfect in some way and relying on them without the appropriate care exposes users to the risk of adverse outcomes. As British statistician George Box eloquently stated: “All models are wrong, but some are useful”. And even useful models can lead to adverse outcomes when used inappropriately.
Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. This risk cannot be eliminated without removing the use of models completely; however, it can be mitigated. In this article we share our views on the most important areas of model risk management (MRM) that the financial services industry in the UK should focus on to improve the mitigation of these risks.
Mitigating model risk
In our experience those organisations that manage model risk in the most effective way typically have a comprehensive and fully embedded MRM framework that is widely understood not only by the model users but also the senior leadership team.
To mitigate any risk first you must identify it. There are several steps in the development and usage of a model where risk can be introduced. A basic model lifecycle (without risk mitigation) starts with someone in the business deciding that they need a model, the model is then built, implemented, and used. Only by identifying and understanding the material (both in size and impact) risks in your organisation’s model lifecycle, can appropriate mitigation actions be developed and applied.
Model-driven organisations need to identify the key risks that arise across each step of their own model lifecycle. In organisations with a well embedded MRM framework and appropriate risk culture, the business owners of a model will take responsibility for managing these risks by ensuring appropriate controls such as independent testing, validation, and monitoring are introduced at each stage of the lifecycle.
The UK financial services industry has fallen behind its peers in its mitigation of model risk
The US has been a front runner in model risk management since the publication of SR11-7, which set the international standards that have been followed since 2011. Globally supervisors have long expected all firms using models to have an MRM framework in place, which has been implemented appropriately for how they use models. As a result, MRM has long been an integral part of regulatory reviews.
Recent supervisory inspections of applications for: internal regulatory model permissions, approaches to expected credit loss accounting under IFRS 9, and annual stress tests, highlighted that UK firms appear to be lagging international peers. As a result, the standard of MRM governance across the industry in the UK is an area of concern for the PRA. For example, the PRA identifies that model components not being integrated within a model risk framework leads to:
- ineffective risk reporting,
- gaps in compliance with internal model risk policies and procedures, and
- lack of process and controls for actions and issues management.
The PRA’s view is that the general quality of regulatory model submissions doesn’t meet its expectations. As a consequence of their concern about the quality of MRM and firms underinvestment in this risk, they are consulting with the industry to introduce higher more transparent standards through CP6/22 – read our summary here. Similar action to encourage greater focus on Model Risk is being taken by other regulators around the globe e.g., Model Management Standards and Model Management Guidance from the Central Bank of UAE.
With the release of the new UK consultation on model risk, many of our clients have conducted a gap analysis to identify where they need to build up their MRM capabilities and make enhancements to their MRM framework. Larger banks are leading the pack due to years of experience embedding models and MRM practices. Mature model-driven organisations, however, should be wary of becoming complacent, because there is still significant room for improvement. Smaller firms will have work to do with regards to model identification, enhancements to model risk documentation, and assigning roles and responsibilities of the MRM framework.
Embedding an MRM culture requires engagement from the board
While many banks already have some MRM policies and processes, we found that for those not operating effectively it tends to be because the processes were put in place to appease stakeholders rather than to truly manage and mitigate the risks to the organisation. Model Risk Management is no longer a tick box exercise that can be brushed to the side. With the introduction of CP6/22, the PRA expects a change in behaviour throughout the organisation towards model risk awareness and mind-set. This starts from the top with the board and senior management taking responsibility for the management of model risk.
The Board of directors should be aware of and understand the current and anticipated levels of model risk within the firm. Leadership is expected to:
- understand the strengths and weaknesses of key models,
- understand where models are used and their purpose,
- provide challenge to model outputs,
- understand the model risks arising from either individual models or in aggregate, and
- be kept informed on how model deficiencies are addressed.
They need to be able to demonstrate that they have a firm grasp on the sources of model risk within the organisation, in particular where there are risk classes with significant financial impacts and tail-event likelihoods that are being modelled.
We see that firms have started to conduct training sessions to prepare their leadership to meet the PRA’s expectations and to gain a deeper understanding of the range of models used across the organisation. Many banks are also reviewing their model risk reporting to ensure the right metrics that are monitored and presented to the Board. These metrics should align to model risk appetite and give a comprehensive view of the firm’s model risk.
Managing a wider model scope
The scope of models that will be subject to new MRM standards is also increasing. The PRA is broadening the definition of a model, this will bring a wider range of models into a firm’s inventory. There may be some room for debate around what should be in scope for the PRA’s model definition, particularly at the tail end where the financial and non-financial impacts are very low or negligible. A logical approach to assess the model candidate is preferred, rather than using expert judgement or a qualitative assessment – we explored this in a recent article.
It becomes important not only to identify what models are used within the firm, but also where the most material model risk lies and whether this is in line with the firms stated model risk appetite. Essentially, we can manage the model risk problem in a similar way to managing any other risk: identify, measure, report, manage, and review.
Mature banks that use internal rating-based models for regulatory capital purposes tend to have clear, documented model risk standards and procedures in place for these models. However, the standards and procedures for IFRS 9 models or other, non-credit risk, models are not necessarily designed and implemented to the same level of rigour. Furthermore, model governance structures set up for regulatory models may not be suitable for all types of models.
Inclusion of models that sit outside of regulatory and credit risk models has also prompted discussions from banks on how best to classify models into tiers based on materiality and complexity. Models that are currently deemed to have low materiality may now need to be upgraded to higher materiality, and vice versa, considering the wider model scope. Furthermore, it is relatively straightforward to compare the importance or materiality of models within a risk type, however a certain level of judgement is needed to ensure that the meaning of a material model is consistent across various model types. This usually comes down to a comparison of the potential financial impacts of the models.
More models require an independent review and challenge process that is more independent and more efficient
From the supervisor’s perspective, validation of a model by a function that is independent from the model developer is a core part of the model lifecycle. Independent validation should take an holistic approach and assess model risk across the areas of data, methodology, documentation, processes, and governance.
The expanded model scope almost certainly leads to an increase in the model validation pipeline. Firms will need to have an effective plan for managing the increased validation pipeline and utilising resources efficiently. While all models should be subject to an independent review, we have had discussions with clients to redefine the boundary where a full comprehensive review needs to be undertaken, and where a less in-depth review can be applied to a subset of models.
We notice that while most banks can identify model limitations and deficiencies as part of independent validation, the findings do not articulate what the risks are or their impact on the firm. Validation findings should clearly highlight the associated model risks and quantify the materiality where possible so that senior management can make informed decisions during the model approval process. Having the right quantitative resources is key to enable this, however this adds to the ongoing challenge the industry faces regarding sufficient resourcing in validation teams.
We have also noticed that sometimes by design but more often through the adoption of a particular culture, validation teams take on the role of final arbiter of right or wrong in a model’s development. A second line validation team that believes it is “more right” than a first line development team introduces another source of bias to the model development process. This is best mitigated by removing any approval or final decision-making responsibility from the validation team and passing it to the business, who must then take responsibility for managing the risks that have been identified by the validation team. This has the added benefit of removing from the validation team the pressure of being the organisation’s model risk goalkeeper. With this responsibility removed from the second line of defence, we observe a more efficient model approval process and shorter overall model development timelines.
It is time to understand your model risk
Considering model risk management and how it is applied across model types, we realise it is still a relatively immature discipline for risk managers compared to the likes of credit risk or market risk management. As a risk type, it also hasn’t enjoyed the same level of priority or investment.
All firms have work to do to meet the PRA’s expectations of Board and Executive understanding of aggregate model risk and model portfolio performance. Risk managers will need senior management support for the organisation to start getting a grasp of their model related risks.
Read the previous articles in this series here:
