Skip to main content

A principled approach: Model Risk Management in the PRA’s spotlight

Audience

Chairs of Board Audit and Board Risk committees, CROs, CFOs, heads of model risk, model owners/sponsors, model developers, model validators, heads of internal audit.
 

At a glance

The PRA has proposed five principles governing how firms manage the risk arising from the models they use to support decision making. The PRA’s approach will capture a potentially significant number of models that firms do not currently include in their model governance processes, and the principles require greater structure in model risk management processes as well as more active involvement by senior management, the Board, and external auditors. The time period for, and cost of, compliance will vary depending on firms’ existing model risk management approaches, but the effort and cost could be very high. This comes at a time when firms’ modelling teams are already stretched by having to deal with a large portfolio of ongoing work.
 

Reading time: 7-10 minutes

The PRA has issued CP 6/22, a consultation paper (CP) setting out its expectations of firms’ model risk management (MRM), along with a draft Supervisory Statement (SS) setting out more detail of the PRA’s five principles and associated guidance.
The PRA considers MRM to be a risk discipline in its own right, and the proposals set out to embed these principles, in a proportionate manner, into supervisory expectations for UK incorporated banks, building societies, and PRA-designated investment firms (collectively “firms”). Given the ongoing Solvency II review, the PRA has decided not to apply the principles to insurance firms as yet, although it notes that it may review of MRM in insurance firms once the Solvency II review is completed.

The PRA’s view is that at a time when the complexity and range of uses of models by firms are constantly increasing, it continues to see considerable shortcomings in firms’ MRM processes. The principles are intended to cover the whole model lifecycle and to apply to a broad range of model types, in that they should capture “all types of models that are used to inform key business decisions, whether developed in-house or externally (including vendor models) and models used for financial reporting purposes”.

The PRA paper proposes a very broad definition of a model: “…a model is defined as a quantitative method that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output. Input data can be quantitative and/or qualitative in nature or expert judgement-based and the output can be quantitative or qualitative.” This means that the MRM principles will capture a much wider range of models than most firms currently include in their model governance process1.

The PRA specifically includes models used for financial reporting in the scope of the paper, with an expectation that firms should “…report on the effectiveness of MRM for financial reporting to their audit committee on a regular basis…” and that “…the effectiveness of MRM for financial reporting is relevant to the auditor’s assessment of, and response to, the risk of material misstatement…”. The PRA notes that it derives considerable value from its discussions with firms’ external auditors and while it has no role in setting accounting standards it believes effective MRM for models involved in financial reporting to be important to ensuring the safety and soundness of firms.

Artificial Intelligence (AI) and Machine Learning (ML) models receive specific focus: the PRA’s view is that firms need to strengthen their MRM if they are to realise the benefits of new technology – particularly around AI and ML – safely and efficiently. The PRA notes the potential increased complexity in AI/ML models around dealing with the size and unstructured nature of the data sets, the potential lack of transparency in the underlying model algorithm, and the challenges arising from continuous learning and dynamic recalibration. The PRA asks for particular comment on whether readers of the CP believe there “…are any components of the MRM framework where the proposed principles are not sufficient to identify, manage, monitor, and control the risks associated with AI or ML models?”

The CP does note that the principles are to be applied in a proportionate manner. In particular, the PRA notes that firms qualifying for the simpler firm regime2 would not be expected to apply the principles in full.

In publishing principles, rather than detailed rules, the PRA has distinguished itself from EU regulators, whose guidelines in the area of model risk tend to be more detailed and prescriptive.

The principles are helpful to firms in that they set out the PRA’s expectations in some detail, without being overly prescriptive. They will, however, create a significant new stream of work for modelling teams already under considerable pressure.

The consultation closes on 21st October 2022 and - if implementation proceeds as currently expected - the PRA expects to publish a SS by Q1 20233, with implementation of the rules 12 months thereafter. This gives firms until Q1 2024 to undertake a self-assessment and create a remediation plan against the final SS rules. Recent experience suggests that changes from the draft to the final SS are likely to be minor, so firms with significant work to do may decide to start sooner rather than later. The self-assessment will have to be updated annually, and progress against the remediation plan will have to be reported internally on a regular basis and available to the PRA on request.
 

The Principles

The CP is arranged around five core principles of MRM, each with a number of more detailed sub-principles. These are summarised below.

Principle 1 – Model identification and model risk classification

Firms have an established definition of a model that sets the scope for MRM, a model inventory and a risk-based tiering approach to categorise models to help identify and manage model risks.

Sub-principles:

  • Model definition should follow the PRA’s definition but should allow for other “…material deterministic methods such as decision-based rules or algorithms.” to be subject to MRM, even if they do not meet the PRA definition.
  • Model inventory should be held centrally and contain information on model purpose and use; model simplifications and limitations; validation findings; and governance details.
  • Model tiering should be consistent, firm-wide, and consider both size and complexity in assigning materiality levels to models. Complexity may arise from model methodology, data availability, implementation approach, or the frequency of use or importance of the output to business decisions.

Determining the new universe of models that is captured by the PRA’s definition of a model is likely to be a significant challenge. Even firms with a robust current model governance process may struggle to identify some processes and calculations that are captured by dint of the definition including inputs that can be qualitative, quantitative or expert-judgement driven and could result in outputs that are either quantitative or qualitative. Given the increased universe of models, designing a consistent, clear model tiering system that can be applied to what will be a wide range of model types is also likely to be difficult.

 

Principle 2 – Governance

Firms have strong governance oversight with a board that promotes an MRM culture from the top through setting clear model risk appetite. The board approves the MRM policy and appoints an accountable individual to assume the responsibility to implement a sound MRM framework that will ensure effective MRM practices.

Sub-principles:

  • The Board of directors should be actively involved in MRM, setting model risk appetite and ensuring that MRM is effective and proportionate to the size of the firm.
    An appropriate SMF holder should be identified to ensure that MRM is included in accountabilities at the highest level in the firm.
  • Policies and procedures should be in place covering the MRM framework, interaction with other elements of the risk framework, all aspects of the model lifecycle, model tiering, model development, model validation, model usage, data quality, model performance monitoring, model risk mitigants, and model approval and change.
  • Roles and responsibilities should be clearly documented and assigned to staff with appropriate skills and experience. There should be clear separation of development and validation duties.
  • Internal Audit should periodically assess the effectiveness and operation of the MRM framework.
  • Use of externally developed models should operate in line with PRA SS 2/21 on outsourcing and third-party risk management and the overall principle that the firm remains accountable for the use of models, even if the model is provided by a third party.

Those firms that do not currently have permissions to use internal models for capital purposes are likely to face the most significant challenges with governance. Their Boards may be less familiar with modelling issues and may require considerable training before they are able to meet PRA expectations. The PRA expects MRM to be assigned to an accountable individual and that an SMF holder should be identified. The PRA notes that “…in many cases it may be that the Chief Risk Function (SMF4) is the most appropriate to fulfil this proposed expectation”, although it also notes “…the creation of an accountable individual for the framework would not relieve business risk and control functions of their responsibilities in relation to development and use of individual models within the firm.” As with any SMF function, firms should be able to articulate clearly the reason for the allocation of the accountability to the individual.
 

Principle 3 – Model development, implementation, and use

Firms have a robust model development process with standards for model design and implementation, model selection, and model performance measurement. Testing of data, model construct, assumptions and model outcomes are performed regularly, in order to identify, monitor, record and remediate model limitations and weaknesses.

Sub-principles:

  • Model purpose and design should be clearly stated and set out in advance of model development work commencing.Model design should be suitable for the intended use.
  • Modelling techniques should be consistent with published research, or generally accepted industry practice where appropriate. Alternative approaches should be assessed where possible.
  • Data used for models should be suitable, consistent with the selected modelling approach and representative of the population for which the model is intended. Model development should ensure there is no bias in the data used, and where data adjustments are applied, or external data are used, this should be clearly documented and an assessment made to determine whether model materiality (tiering) should be adjusted.
  • Model development testing should be undertaken to confirm that the model works as intended and development testing should identify the conditions where model performance is expected to be acceptable.
  • Model adjustments and the application of expert judgement in models should be clearly justified and documented in the model documentation and in the model inventory. The use of conservatism to account for model uncertainties should be clearly understood and agreed by the business owners and users of the model.
  • Model documentation should be comprehensive, up to date, and allow an independent third party to understand how the model operates and its limitations. Documentation should include:
  • the use of data;
  • the choice of methodology;
  • performance testing; and
  • limitations and the use of expert judgement, if any.
  • Models should be implemented in systems that are fit for purpose for hosting models and capturing the data associated with them. Findings associated with implementation testing should be documented.

While none of the expectations in principle 3 is new or controversial, their application to the considerably larger set of models captured by the expanded definition, along with the challenge of demonstrating compliance with requirements for testing of data, model construct, assumptions, model outcome, and validation requirements (see below) will likely prove challenging, time-consuming and resource intensive. This will particularly be the case where firms do not currently consider their approach/process/calculation to be a “model” and so they will not have captured the information required to demonstrate compliance with model development and governance policies. Retro-fitting these models to the principles is likely to be difficult and time-consuming.
 

Principle 4 – Independent model validation

Firms have a validation process that provides on going, independent, and effective challenge to model development and use. The individual or body within a firm responsible for the approval of a model ensures that validation recommendations for remediation or redevelopment are actioned so that models are suitable for their intended purpose.

Sub-principles:

  • Validation teams should be independent from model developers and model owners.
  • Validation teams should independently review and periodically re-validate models and provide reports through the model governance process.
  • Validation teams should provide independent, unbiased views on the suitability of models for their intended use, the design of models, the accuracy and relevance of development data.
  • Model owners are responsible for model performance: Validation teams should share responsibility for model performance monitoring (ensuring parameter estimates are appropriate, checking and challenging model assumptions, ensuring models remain appropriate for the intended use, ensuring models are used within agreed model boundaries);
  • process verification (ensuring model inputs are appropriate, ensuring that the model that was implemented in systems is the model that was approved).
    Validation should have access to the Board/Board committees to escalate concerns if necessary.

Firms should not underestimate the impact of the point the PRA makes about the need for validation recommendations to be addressed. Although this may seem a statement of the obvious, we know that in many firms there are instances of validation actions remaining outstanding for a long time. Firms should anticipate questions from the PRA around time-to-close validation recommendations and related escalation processes. Validation teams should be able to show that they have robust processes for following up on outstanding actions, and that they are willing to escalate failure to close outstanding actions to the highest level in the firm if necessary.
 

Principle 5 – Model risk mitigants

Firms have established policies and procedures for the use of model risk mitigants when models are under-performing and have procedures for the independent review of post-model adjustments.

Sub-principles:

  • Post-model adjustments (PMA) should be subject to a policy and process that is part of the governance process, with approval of PMAs subject to the same approval as the underlying model. In particular:
  • PMAs may vary in specific approach for different model types, but the overall outcome should be consistent;
  • PMAs should be transparent, robustly documented, and subject to independent review commensurate to their materiality; and
  • firms should consider whether, where a PMA is material, or if multiple PMAs are in place, is an indication of an underlying flaw in model design or implementation and consider remedial action if it is.
  • Restrictions on model use should be applied where models are not performing or where model reporting indicates a significant breach has or is likely to occur.
  • Exceptions and escalations should be formally considered as part of the model policy suite. Exceptions should be temporary, and subject to PMAs if appropriate.

The PRA expects firms to have clearly defined metrics against which they assess models, and a clear policy for how to handle models that are not reaching required performance standards. The PRA’s expectations for PMAs are principally set out in SS 11/13. One point to bear in mind about PMAs is that they are expected to be reported to senior management in some detail4, and so the information reported to senior management will have to go through the model governance process to ensure it is accurate and complete.

The requirement for an “independent” review of PMAs is likely to mean increased responsibility for internal or external audit teams, given that model development is usually a first line activity and validation typically sits in the second line.
 

Implications for firms

Firms with existing permissions to use internal models for capital purposes already have significant work in train in their modelling teams, with work to implement IRB model changes, review and revise IFRS 9 models, incorporate climate into modelling approaches for risk management and stress testing, and prepare for the implementation of Basel 3.1 already on the plan. Incorporating significant new volume into the model governance and review process risks over-stretching already scarce resource.

Compliance with the principles, particularly for smaller firms that do not have permission to use internal models for capital purposes and which need to develop and implement MRM approaches from a lower starting point, is likely to be a resource-intensive and challenging exercise.

For larger firms, with more sophisticated existing MRM capabilities, the challenge will be in adapting existing processes and policies to account for the increase in the model inventory from the revised definition, and in bringing some of the less traditional model types (such as AML risk assessment approaches) into the model governance framework without reducing the quality of work undertaken in already stretched teams.

The principles arise out of the PRA’s concern that firms’ standards in this increasingly important area of risk management are declining. In addition, there is no guidance for UK firms that sets out PRA expectations across the whole model landscape.

The PRA has provided the graphic below which sets out the way the principles map to current model types, current guidance in place and the future model-related framework. The CP is clear about the interaction between the principles and any specific supervisory statements on models: the principles are over-arching and apply to models in the aggregate. They complement, but do not supersede, existing model guidance, so firms will have to comply with the principles for all models and with specific SS as they apply to specific models.

________________________________________________

References

[1] A few examples of existing approaches that may not be subject to model governance at present: AML customer risk assessments; methodologies for determining whether counterparties are connected or not; pricing models; internal cost allocation methodologies; environmental risk assessments; processes for reporting adjustments; and possibly many more.

[2] See CP5/22 The Strong and Simple Framework: a definition of a Simpler-regime Firm

[3] Regulatory Initiatives Grid, May 2022, Page 26

[4] PRA SS 11/13 Internal Ratings Based (IRB) Approaches, para 19.17 (e) 

Our thinking