Countdown to compliance with SS1/23 Model Risk Management: Implications for Board members
Audience:
Board members; Company Secretaries; CEOs, CFOs, CROs, CIOs, Heads of Model Risk, Heads of Model Validation, Model Owners, Heads of Internal Audit.
At a glance:
Models are used across all areas of banks, and in the assessment and management of all risk types, internal and external MI and reporting, and operational processes. In some banks, the management of overall model risk is not regarded by the PRA as sufficiently wide-ranging or robust. The PRA’s Principles for Model Risk Management (MRM, the Principles) place considerable expectations on the role of Board members in the review, approval and monitoring of the MRM framework and culture in banks. Board members should be challenging the executives in their banks and developing their own understanding in relation to MRM in a number of areas. These include:
- their understanding of the MRM framework and culture;
- understanding aggregate model risk;
- what models are used for and where they pose most material risks;
- where models do and do not work well; and
- mitigating actions in place to compensate for poor model performance.
This note sets out some questions for Board members to consider when assessing and challenging the model risks their banks face. Understanding these issues will assist them in holding the executives to account and in preparing to discuss MRM with supervisors.
Introduction
With the publication of SS 1/23, the PRA started the clock on a significant programme of work to comply with the MRM Principles by 17 May, 2024. In this note, we will examine the implications of the MRM Principles for Board members.
In publishing the Principles, one of the PRA’s ambitions is to expand and elevate the discussion of model risk within banks, in particular at Board level. The PRA’s expectations of the Board extend across all five Principles, although are most specific in relation to Principle two on Governance.
Although the PRA’s initial consultation on the Principles included all banks in scope, SS 1/23 only applies to those banks with permissions to use internal models for regulatory capital purposes. The PRA has indicated that it will review the scope of application of the Principles once it has finalised its work on the Simpler Firms Regime. In the meantime, the PRA considers that firms without internal model permissions and third-country branches “may find the proposed principles useful, and are welcome to consider them to manage model risk within their firm”.
PRA expectations
Model risk is an over-arching risk that affects a wide variety of risks across the bank and in order to ensure they have a good grasp of the risk and its mitigants Board members need reliable, timely information about model risk appetite; model risk policy and culture; the range of models in use and their strengths and weaknesses; and mitigating actions in place where models do not perform well.
The PRA expects that Boards will be involved in setting model risk appetite, understanding and monitoring model risk in the business, and in ensuring their bank has an appropriate model risk culture. While some Board members, for example the members and Chair of the Board Risk Committee, may be closer to the detail, all Board members should be familiar with each of the five MRM principles and how they affect their bank. We have set out below some areas of focus and included an annex with specific questions that all Board members should be asking and seeking answers to in the context of their review and challenge of MRM in their banks.
Ultimately, Board members should feel confident that they understand model risk and how their bank monitors, manages, and controls it. This does not mean that they need to know in detail the underpinning statistical or methodological aspects of models1 Board members should however know and be able to challenge appropriately and effectively:
The bank’s MRM framework and culture |
Board members should be able to demonstrate their understanding of:
|
What the most material models2 are used for |
Board members should be able to describe their understanding of the strengths and weaknesses of the most material models in use in the bank and know what risk they are used to assess – credit, market, fraud etc. Artificial Intelligence and/or Machine Learning models should be under particular scrutiny given both their rapid development and the interest supervisors currently have in them. |
Where the most material models work well/their target customer or transaction group |
Board members should be able to describe the key characteristics of the most material models, for example “Our most material models in the credit risk area are the suite of IRB Expected Loss models (Probability of Default (PD); Loss Given Default (LGD) and Exposure at Default (EAD)) for mortgages in England and Wales. They use both internal (account activity) and external (credit bureau, house price valuation) data to measure the likelihood of customers defaulting, and the likely loss the bank will face if this should happen. The suite of models works well for new and re-mortgage loans up to XX% LTV.” |
Where material models don’t work well |
Board members should be able to show they are aware of the limitations of material models, including instances where models have subsets of customers or transactions for which the model is less accurate; certain types of customer or transactions for which false positive or false negative3 outcomes are higher than model appetite; or where there may be “cliff effects” that mean that within a range of conditions (e.g. Central Bank base rates up to 6% and inflation below 12%) the model works, but outside that range of conditions the model cannot be relied upon. |
Mitigating actions to account for known model weaknesses |
Board members should be able to describe mitigating actions in place for weaknesses in models. These may include, inter alia, capital add-ons, additional analysis for some transaction types, enhanced monitoring of exposures or a four-eyes review process for some transactions. |
The implications or channels of effect from poor model performance |
Board members should be able to demonstrate that they understand the mechanism(s) by which poor model performance is likely to translate into impact on the bank. This may include increased risk of credit, market or operational losses, increased exposure to complaints or reputational issues, or increased risk of fines or other regulatory sanctions. |
The aggregate risk that the bank is exposed to from the use of models |
Aggregation of model risk is a challenge, given that the effects of differing models can vary significantly. Board members may need to challenge model owners and the SMF holder for MRM to explain clearly how reporting on aggregate model risk works, and ensure they have a good view of overall model risk across the bank. |
Conclusion
One of the PRA’s objectives in publishing the Principles is to elevate, expand and enhance the discussion of model risk at Board and senior executive level. Given the breadth of processes and risk types where models are used, model risk should be regarded as a risk type at the same level as credit risk or other material risks, and Board members will need to increase their focus on MRM in order to ensure that they understand, and can effectively challenge, the overall risk posed by the use of models in the business. Board members who improve their understanding of model risks at both model and aggregate levels will find that it pays off when it comes to discussing MRM with supervisors.
Annex: Some questions Board members should be able to answer about the five principles
Principle one – model inventory and classification
“Firms should have an established definition of a model that sets the scope for MRM, a model inventory and a risk-based tiering approach to categorise models to help identify and manage model risk.”
- Is the firm’s model definition broad enough to capture the range of models and model risks the PRA envisions?
- Is a model inventory in place and by how much has it grown in response to the change in model definition?
- Are differences in risk-based tiering and the materiality of models clear and understood by all Board members?
- What are the most material models in place in the bank at present? Is there a risk that any of them becomes unfit-for-purpose in the short term?
- How has model risk reporting received by the Board changed as a result of the principles?
- Is aggregate model risk reported to the Board?
Principle two – governance
“Firms should have strong governance oversight with a board that promotes an MRM culture from the top through setting clear model risk appetite. The board should approve the MRM policy and appoint an accountable individual to assume the responsibility to implement a sound MRM framework that will ensure effective MRM practices.”
- Has the Board signed off the MRM oversight policy and approach and what were the main points of debate in the process?
- Who is the SMF holder for Model Risk and how well are they executing their responsibilities?
- What is the Board’s role in ensuring model culture is appropriate and how does the Board carry it out?
- What regular MI reporting on model risk against approved risk appetite metrics does the Board receive and what is it telling the Board?
Principle three – model development and implementation
“Firms should have a robust model development process with standards for model design and implementation, model selection, and model performance measurement. Testing of data, model construct, assumptions, and model outcomes should be performed regularly in order to identify, monitor, record, and remediate model limitations and weaknesses”.
- How does the Board gain a good understanding of the model development process and keep abreast of any changes to it?
- How is the Board kept appraised of model implementation and use challenges and the work to remediate them?
- Does the Board have a good understanding of the modelling infrastructure?
Principle four – validation
“Firms should have a validation process that provides ongoing, independent, and effective challenge to model development and use. The individual or body within a firm responsible for the approval of a model should ensure that validation recommendations for remediation or redevelopment are actioned so that models are suitable for their intended purpose.”
- Has the Board been informed of, or approved if required, the firm’s policy on validation, in particular with respect to the boundary between validation and independent review4?
- What information does the Board receive about validation actions that have been outstanding beyond acceptable resolution timescales?
- Is there a channel whereby the Board or a relevant subcommittee such as Board Risk Committee can be informed of MRM concerns, independent of the model owner and/or the SMF holder?
Principle five – post model adjustments
“Firms should have established policies and procedures for the use of model risk mitigants when models are under-performing and should have procedures for the independent review of post-model adjustments.”
- What information does the Board receive about model mitigants in place for at least the most material models (e.g. post model adjustments, model use restrictions), the reasons for them and the resolution strategies in place?
- Which of the bank’s most material models currently have post model adjustments in place, why are they required, and what is the plan to remove them in the short to medium term?
______________________________________________________________________________________________
1 We use the term “model” here to include: models as currently defined by banks; models and other methodologies that are captured by the expansion in scope arising from the implementation of the PRA’s definition; and other deterministic quantitative methodologies to which banks choose to apply the Principles.
2 The specific set of models regarded as “most material" will vary from bank to bank, but will include those models that cover the most significant portfolios, risk types, and groups of customers. In most banks the IFRS9 models, IRB models covering the majority of RWAs, market risk models covering the most material risk types, and stress testing models are likely to be among the most material models.
3 There are two principal types of errors produced by models: saying something is what it isn’t (a false positive or “type one” error) or saying something isn’t what it is (a false negative or “type two” error). For some models, one type of error may be significantly more problematic than the other.
4 The principles do not require that all models are subject to a formal “validation” as that term is used in relation to models that are subject to regulatory approval. Formal validations are expected to be: independent from the model development function, undertaken by individuals with relevant qualifications and expertise; subject to formal standards in relation to statistical, data and other testing approaches etc. For less material models, the standard in the Principles is that there is an “independent review”, the specifics of which may vary from firm to firm and by model type.
