Chairs of Board Audit and Board Risk committees, CROs, CFOs, heads of model risk, model owners/sponsors, model developers, model validators, heads of internal audit.
The PRA has proposed five principles governing how firms manage the risk arising from the models they use to support decision making. The PRA’s approach will capture a potentially significant number of models that firms do not currently include in their model governance processes, and the principles require greater structure in model risk management processes as well as more active involvement by senior management, the Board, and external auditors. The time period for, and cost of, compliance will vary depending on firms’ existing model risk management approaches, but the effort and cost could be very high. This comes at a time when firms’ modelling teams are already stretched by having to deal with a large portfolio of ongoing work.
Reading time: 7-10 minutes
The PRA has issued CP 6/22, a consultation paper (CP) setting out its expectations of firms’ model risk management (MRM), along with a draft Supervisory Statement (SS) setting out more detail of the PRA’s five principles and associated guidance.
The PRA considers MRM to be a risk discipline in its own right, and the proposals set out to embed these principles, in a proportionate manner, into supervisory expectations for UK incorporated banks, building societies, and PRA-designated investment firms (collectively “firms”). Given the ongoing Solvency II review, the PRA has decided not to apply the principles to insurance firms as yet, although it notes that it may review of MRM in insurance firms once the Solvency II review is completed.
The PRA’s view is that at a time when the complexity and range of uses of models by firms are constantly increasing, it continues to see considerable shortcomings in firms’ MRM processes. The principles are intended to cover the whole model lifecycle and to apply to a broad range of model types, in that they should capture “all types of models that are used to inform key business decisions, whether developed in-house or externally (including vendor models) and models used for financial reporting purposes”.
The PRA paper proposes a very broad definition of a model: “…a model is defined as a quantitative method that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output. Input data can be quantitative and/or qualitative in nature or expert judgement-based and the output can be quantitative or qualitative.” This means that the MRM principles will capture a much wider range of models than most firms currently include in their model governance process1.
The PRA specifically includes models used for financial reporting in the scope of the paper, with an expectation that firms should “…report on the effectiveness of MRM for financial reporting to their audit committee on a regular basis…” and that “…the effectiveness of MRM for financial reporting is relevant to the auditor’s assessment of, and response to, the risk of material misstatement…”. The PRA notes that it derives considerable value from its discussions with firms’ external auditors and while it has no role in setting accounting standards it believes effective MRM for models involved in financial reporting to be important to ensuring the safety and soundness of firms.
Artificial Intelligence (AI) and Machine Learning (ML) models receive specific focus: the PRA’s view is that firms need to strengthen their MRM if they are to realise the benefits of new technology – particularly around AI and ML – safely and efficiently. The PRA notes the potential increased complexity in AI/ML models around dealing with the size and unstructured nature of the data sets, the potential lack of transparency in the underlying model algorithm, and the challenges arising from continuous learning and dynamic recalibration. The PRA asks for particular comment on whether readers of the CP believe there “…are any components of the MRM framework where the proposed principles are not sufficient to identify, manage, monitor, and control the risks associated with AI or ML models?”
The CP does note that the principles are to be applied in a proportionate manner. In particular, the PRA notes that firms qualifying for the simpler firm regime2 would not be expected to apply the principles in full.
In publishing principles, rather than detailed rules, the PRA has distinguished itself from EU regulators, whose guidelines in the area of model risk tend to be more detailed and prescriptive.
The principles are helpful to firms in that they set out the PRA’s expectations in some detail, without being overly prescriptive. They will, however, create a significant new stream of work for modelling teams already under considerable pressure.
The consultation closes on 21st October 2022 and - if implementation proceeds as currently expected - the PRA expects to publish a SS by Q1 20233, with implementation of the rules 12 months thereafter. This gives firms until Q1 2024 to undertake a self-assessment and create a remediation plan against the final SS rules. Recent experience suggests that changes from the draft to the final SS are likely to be minor, so firms with significant work to do may decide to start sooner rather than later. The self-assessment will have to be updated annually, and progress against the remediation plan will have to be reported internally on a regular basis and available to the PRA on request.
The CP is arranged around five core principles of MRM, each with a number of more detailed sub-principles. These are summarised below.
Principle 1 – Model identification and model risk classification
Firms have an established definition of a model that sets the scope for MRM, a model inventory and a risk-based tiering approach to categorise models to help identify and manage model risks.
Sub-principles:
Determining the new universe of models that is captured by the PRA’s definition of a model is likely to be a significant challenge. Even firms with a robust current model governance process may struggle to identify some processes and calculations that are captured by dint of the definition including inputs that can be qualitative, quantitative or expert-judgement driven and could result in outputs that are either quantitative or qualitative. Given the increased universe of models, designing a consistent, clear model tiering system that can be applied to what will be a wide range of model types is also likely to be difficult.
Principle 2 – Governance
Firms have strong governance oversight with a board that promotes an MRM culture from the top through setting clear model risk appetite. The board approves the MRM policy and appoints an accountable individual to assume the responsibility to implement a sound MRM framework that will ensure effective MRM practices.
Sub-principles:
Those firms that do not currently have permissions to use internal models for capital purposes are likely to face the most significant challenges with governance. Their Boards may be less familiar with modelling issues and may require considerable training before they are able to meet PRA expectations. The PRA expects MRM to be assigned to an accountable individual and that an SMF holder should be identified. The PRA notes that “…in many cases it may be that the Chief Risk Function (SMF4) is the most appropriate to fulfil this proposed expectation”, although it also notes “…the creation of an accountable individual for the framework would not relieve business risk and control functions of their responsibilities in relation to development and use of individual models within the firm.” As with any SMF function, firms should be able to articulate clearly the reason for the allocation of the accountability to the individual.
Principle 3 – Model development, implementation, and use
Firms have a robust model development process with standards for model design and implementation, model selection, and model performance measurement. Testing of data, model construct, assumptions and model outcomes are performed regularly, in order to identify, monitor, record and remediate model limitations and weaknesses.
Sub-principles:
While none of the expectations in principle 3 is new or controversial, their application to the considerably larger set of models captured by the expanded definition, along with the challenge of demonstrating compliance with requirements for testing of data, model construct, assumptions, model outcome, and validation requirements (see below) will likely prove challenging, time-consuming and resource intensive. This will particularly be the case where firms do not currently consider their approach/process/calculation to be a “model” and so they will not have captured the information required to demonstrate compliance with model development and governance policies. Retro-fitting these models to the principles is likely to be difficult and time-consuming.
Principle 4 – Independent model validation
Firms have a validation process that provides on going, independent, and effective challenge to model development and use. The individual or body within a firm responsible for the approval of a model ensures that validation recommendations for remediation or redevelopment are actioned so that models are suitable for their intended purpose.
Sub-principles:
Firms should not underestimate the impact of the point the PRA makes about the need for validation recommendations to be addressed. Although this may seem a statement of the obvious, we know that in many firms there are instances of validation actions remaining outstanding for a long time. Firms should anticipate questions from the PRA around time-to-close validation recommendations and related escalation processes. Validation teams should be able to show that they have robust processes for following up on outstanding actions, and that they are willing to escalate failure to close outstanding actions to the highest level in the firm if necessary.
Principle 5 – Model risk mitigants
Firms have established policies and procedures for the use of model risk mitigants when models are under-performing and have procedures for the independent review of post-model adjustments.
Sub-principles:
The PRA expects firms to have clearly defined metrics against which they assess models, and a clear policy for how to handle models that are not reaching required performance standards. The PRA’s expectations for PMAs are principally set out in SS 11/13. One point to bear in mind about PMAs is that they are expected to be reported to senior management in some detail4, and so the information reported to senior management will have to go through the model governance process to ensure it is accurate and complete.
The requirement for an “independent” review of PMAs is likely to mean increased responsibility for internal or external audit teams, given that model development is usually a first line activity and validation typically sits in the second line.
Firms with existing permissions to use internal models for capital purposes already have significant work in train in their modelling teams, with work to implement IRB model changes, review and revise IFRS 9 models, incorporate climate into modelling approaches for risk management and stress testing, and prepare for the implementation of Basel 3.1 already on the plan. Incorporating significant new volume into the model governance and review process risks over-stretching already scarce resource.
Compliance with the principles, particularly for smaller firms that do not have permission to use internal models for capital purposes and which need to develop and implement MRM approaches from a lower starting point, is likely to be a resource-intensive and challenging exercise.
For larger firms, with more sophisticated existing MRM capabilities, the challenge will be in adapting existing processes and policies to account for the increase in the model inventory from the revised definition, and in bringing some of the less traditional model types (such as AML risk assessment approaches) into the model governance framework without reducing the quality of work undertaken in already stretched teams.
The principles arise out of the PRA’s concern that firms’ standards in this increasingly important area of risk management are declining. In addition, there is no guidance for UK firms that sets out PRA expectations across the whole model landscape.
The PRA has provided the graphic below which sets out the way the principles map to current model types, current guidance in place and the future model-related framework. The CP is clear about the interaction between the principles and any specific supervisory statements on models: the principles are over-arching and apply to models in the aggregate. They complement, but do not supersede, existing model guidance, so firms will have to comply with the principles for all models and with specific SS as they apply to specific models.
________________________________________________
[1] A few examples of existing approaches that may not be subject to model governance at present: AML customer risk assessments; methodologies for determining whether counterparties are connected or not; pricing models; internal cost allocation methodologies; environmental risk assessments; processes for reporting adjustments; and possibly many more.
[2] See CP5/22 The Strong and Simple Framework: a definition of a Simpler-regime Firm
[3] Regulatory Initiatives Grid, May 2022, Page 26
[4] PRA SS 11/13 Internal Ratings Based (IRB) Approaches, para 19.17 (e)