If AI (and LLMs in particular) are becoming so smart through learning, why do we need to be concerned about validation? The most obvious answer is that the vast amount of data required to train such models is being scraped from the internet with often little or no attribution or permission. AI software and applications use state-of-the-art machine learning models and techniques (sometimes public but increasingly private) to consume such large-scale data to identify features and train models to provide a stream of novel insights through new capabilities. Therefore, the approach to performing quality-assured validation for AI models more widely assumes critical importance as a research topic from both an academic and commercial perspective.

Validation of pricing and risk management models in finance has a well understood methodology and generally accepted best practices. In contrast, LLMs have already attracted considerable media coverage on issues of bias (e.g., race, gender, age, geography) and toxicity (e.g., abuse and offense), making it challenging to validate such models in the face of an increasingly rapid and ever-widening spread of AI. From a model validation perspective, the first challenge is the sheer volume of data required to train and test AI models such as LLMs is frequently several orders of magnitude greater than required by traditional models. For example, to give a sense of scale, OpenAI’s ChatGPT-4 was trained on 100 trillion parameters, which is orders of magnitude times more data than will typically be used to develop and test traditional risk models. To put this in context, LLMs are trained on about 10,000 times more data than a human ever sees. A consequence of this greater size and dimensionality is that classical asymptotic theory can often fail to provide useful predictions and standard statistical techniques can break down, making validation by traditional means infeasible.

Second, the high degree complexity in the algorithms underlying AI models and LLMs (in particular), requires new validation approaches and techniques capable of providing insight into how and why new results are produced. This leads to the third and perhaps most challenging validation issue,  results are emerging from LLMs that are proving difficult to explain with current methods. Emergent behaviour occurs when quantitative changes in a system result in qualitative changes in behaviour. This is increasingly being observed when an ability is not present in smaller LLM models but is present in larger LLM models. LLM models have been scaled in three directions, namely, amount of computation, number of model parameters and size of the training dataset. Emergence leads to unpredictability and appears to increase with scaling, making it difficult for researchers to anticipate the consequences of widespread LLM use. Interestingly, in about 5% of the tasks, researchers have found what they called “breakthroughs,” in the form of rapid, dramatic jumps in performance at some threshold scale with the threshold varying based on the task and model.  In other words, scaling laws do not appear to work as predictors of emergent abilities, making it difficult to design model validation approaches.

Proprietary LLMs like OpenAI’s ChatGPT-4 are, to date,  the most prominent examples of AI in the public imagination. However, improvements in model training like low-rank adaption (LoRA) are making comparable performance accessible to open-source alternatives like LLaMA, from Meta (Large Language Model Meta AI). Which framework will come to dominate so-called ‘foundational’ AI models, or if they coexist, is an open question.

Mökander et al (2023)¹ helpfully describe a three-tiered approach to auditing/validating LLMs: model governance, and performance at the model and application-specific levels. The stage that is closest to the end-user, the application specific stage, essentially involves mapping the range of possible outputs to the expected range of inputs. These might be checked for accuracy, bias, and compliance to sector-specific regulations. The complexity of the models means that small changes to the inputs may lead to qualitatively different outputs, creating a vast possible response space from a single query. With a closed model, potentially without the access and control required to ensure reproducibility, the complexity of inputs and applications that can practically be validated is limited.

Model-level audit or validation is a more general exercise in the robustness, performance, truthfulness, and information security of the underlying LLM, unrestricted to a specific application. The transparency of this process with an open-source model is assured. There is also a lack of a profit motive to maximize published performance and the ‘wisdom of crowds’ to identify and fix bugs. However, open access provides equal opportunity for good and bad actors to find flaws, which may be an important consideration for use in finance. Fragmentation is also a concern with open models. Different versions may exist simultaneously, and validation may need to consider variability and back-compatibility.

The challenges outlined above make it clear that whilst AI models have validation requirements that share common features with those used for traditional models, it is not sufficient to simply limit validation of AI models to using only traditional model validation tools. Instead, it is important to apply extra scrutiny and effort for AI models to validate the entire model life-cycle process, beginning with data preparation for training and testing to improve model performance, through to infrastructural activities around AI model building (such as feature engineering and parameter optimization). Figure 1 identifies the five additional planks of a validation and quality assurance strategy required for AI models, as helpfully identified by Tao et al².