The Model Validation Playbook for GenAI: Lessons from Banking

Introduction

Let’s start with a recent, increasingly common scenario in the Risk Management department of large banks. Let’s say a risk model validator at a large bank opens a submission. The model is an AI assistant that reads a borrower’s financial statements, pulls relevant third-party research, and drafts the first version of a credit memo. It saves analysts several hours a week, and obviously the business wants this AI model to go live next quarter.

She opens the standard validation template to start the review process. This template has been refined over a decade of regulatory examinations and worked on every scorecard, every loss forecasting model, every pricing engine she has reviewed. She reaches the first question:

“Provide the development sample.“

But there is no development sample. This gen AI model was trained on a corpus nobody at the bank has seen and by a vendor who won’t describe it. And that is only the first question from the remaining ninety. Model risk management was never designed for generative AI in banking.

So, this is roughly where every model risk function in the banking/financial industry currently finds itself. An effective challenge on a model we cannot retrain, whose training data we cannot see, looks different from an effective challenge on a traditional scorecard. The craft shifts from replication to test design.

Why this framework matters beyond banking

The core challenge described in this article, i.e., how to validate a system you cannot fully inspect, is now a problem for anyone deploying generative AI in a serious production context. Whether you are building a medical summarisation tool, a legal research assistant, or a customer-facing chatbot, the same questions apply: What does ‘good’ look like when there is no ground truth? How do you catch confident errors before they reach a user?

The framework that follows in this article, based on risk tiering, outcome-based evaluation, robustness testing, and monitoring for silent drift, was built for banks, but it is directly transferable to any domain where the cost of being wrong matters more than the cost of being slow.

What model risk management in banking actually does

If you work in data science outside banking, this discipline may be unfamiliar. So let’s set up the context properly.

Banks run on mostly traditional statistical predictive models. These models decide who gets credit and at what price. Models set how much capital the institution must hold against its loan book. Models forecast losses under hypothetical recessions, value illiquid positions, flag suspicious transactions, and determine reserves that flow directly into published financial statements. When one of these is wrong, the consequences are not an unhappy user; they are mispriced risk, understated reserves, regulatory findings/penalty, and occasionally a very large loss.

The industry learned this expensively. Credit models that assumed house prices don’t fall nationally contributed materially to the 2008 crisis. A revised risk model at one bank in 2012 understated exposure so badly that a trading loss ran into billions before anyone caught it. Regulators responded by formalising the discipline: US supervisory guidance issued in 2011 (known to everyone in the field as SR 11-7) defined model risk as the potential for adverse consequences from decisions based on incorrect or misused model output, and required banks to manage it deliberately.

The EU AI Act codifies a similar expectation for high-risk AI systems used in creditworthiness assessments, pricing, or essential banking services. Its core obligations on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and accuracy/robustness map closely onto SR 11-7’s conceptual soundness, outcomes analysis, and ongoing monitoring. For global banks, one validation framework can be structured to satisfy both regimes, but the AI Act adds explicit requirements around fundamental rights impact assessments and post-market monitoring that extend the second line’s traditional scope.

The model risk management structure is remarkably consistent across large institutions:

The second line is the part this article is about. A validator doesn’t just check arithmetic. They ask whether the modelling approach was conceptually appropriate, whether the data supported it, whether the output actually performs, whether the production implementation matches what was approved, and whether the people using the output understand its limits. Nothing goes live without their sign-off, and everything gets re-examined periodically.

Three things anchor that review, and they have been stable for over a decade: conceptual soundness (is the approach defensible?), outcomes analysis (does the output hold up when tested?), and ongoing monitoring (is it still working now?).

Why Generative AI Breaks Traditional Model Validation

Generative AI has arrived in banks faster than any modelling technology in recent memory, and not in a specific shape. It can be complaint summarisation, policy lookup, research retrieval, first drafts of credit memos, internal documentation, literally anything.

These models are attractive because they directly influence cost, but they can also be risky. Because they sit close to customers and close to credit decisions. These are exactly the places where a regulated institution has the least appetite for a wrong answer.

And the validation apparatus that existed to prevent this risk no longer fits. Every question on the template assumes properties these systems don’t have.

1. Five Structural Breaks in Gen AI Model Validation

The first row has a direct procedural consequence. The unit of validation is the system, not the model. Two use cases sitting on the same base model are two separate validations.

The second row inverts an instinct worth naming. A system that answers correctly 95% of the time and produces a confidently wrong answer 5% of the time is a different object from a model with a 5% error rate on a known error distribution. Fluency is uncorrelated with correctness in a way no calibrated PD model has ever been.

Because these structural breaks make blanket validation impossible, risk tiering becomes the lever that determines how much evidence is enough.

2. Risk Tiering for Generative AI: A Practical Framework

Everything downstream scales off the risk tiering, so this is where a validator focuses the most. The classical drivers still apply – business reliance, impact of error, complexity, control feasibility. Two more matter specifically here.

How far does the output travel?

Does it just produce text, or does it also act? A system that drafts something a person reads has low risk. A system that calls tools, writes to systems of record or triggers downstream processes has an entirely different risk profile.

3. The three questions an AI Model Risk Assessment report answers

The questions are the same ones we have always asked. What counts as evidence has changed completely.

4. Before we look at the output

A. Should this be a language model at all?

The cheapest and most valuable challenge available to the second line. A surprising number of proposed use cases are deterministic problems where a generative model introduces variance to solve something a parser handles exactly. If a simpler method gets 90% of the value with none of the fabrication, that comparison belongs in the report. Complexity has to earn its place.

B. What exactly is the system?

Ask for a component map. Then ask which components the institution actually controls.

In production, retrieval quality and prompt construction usually dominate performance. The base model is often the least important source of variance, which is fortunate, because it’s the part you can’t inspect.

C. What went into it?

The specification is not “we used a foundation model.” Each customisation choice needs a justification:

D. What can we honestly explain?

Traditional model explanation tools don’t help much here. In practice, two approaches work. The first is claim-level attribution, i.e., go through each factual statement in the model’s output and check whether it traces back to a specific passage in the source material. This is achievable and useful. The second one is fully characterising how the system behaves across every possible input. It is not achievable.

The practical substitute is behavioural testing over a structured set of cases. You deliberately test the system across many controlled input types, observe how it behaves, and then generalise carefully from those observations. You are profiling the system, not proving it.

5. Outcome Analysis for LLM Risk: Metrics for Hallucination and Groundedness

The outcome analysis is the most crucial step. This is where we test whether the system does what it claims and where it fails.

A. Metrics depend on the task

The starting point is mapping the use case to the right evaluation dimensions.

These high-level dimensions are a starting point. In practice, a validation report needs to score every output on explicit, separately reported dimensions. Collapsing them into a single quality score hides the specific ways a system can fail.

B. The full dimension set

The first group answers a simple question: Is the output factually reliable?

Correctness and grounding

These first three often collapse into one number more than they should. A response can be perfectly grounded in an out-of-date document. So, it may be grounded but untrue given the context. A response can be true but ungrounded, which is unusable in a regulated workflow because the reviewer cannot verify it. In credit and compliance work, groundedness is usually the binding constraint, because verifiability is the control.

The second group answers a different question: Is the output usable and safe in a regulated workflow?

Compliance and expression

C. Robustness

Perturb the input in ways that preserve meaning, and check the output doesn’t move:

Reordering the evidence is a high-value test on this list. If the answer changes when nothing about the evidence changed, we have found a real weakness.

D. Where it fails, not just how often

Aggregate scores hide localised failure. We can embed the input population, cluster it, and score per cluster. Weak segments are usually semantically coherent, e.g., a document type, a product, a phrasing style. Once identified, we can scope them out or fix them.

E. Hallucination Detection at Scale: Three Proven Methods

Three ways to detect hallucination.

F. Judge the judge

Most processes use a second language model to score against the rubric. That judge is itself a model producing an output that informs a control decision.

Adopting automated scoring without validating the scorer moves model risk rather than reducing it.

6. Safeguards for Generative AI in Banking

We cannot make a foundation model stop fabricating. However, we can build a system that catches or bounds fabrication.

7. Ongoing Monitoring for Generative AI in Banking

A monitoring plan needs a defensible frequency and testing plan. Tests should include indicators mapped to actual risks, thresholds that trigger a breach, a decision rule when indicators disagree, and a pre-agreed action plan.

That last one is the most informative and the least collected. An override rate that falls steadily over six months is not evidence the model improved.

8. Conclusion: The Future of Model Risk Management in the AI Era

A valuable model risk review should include adversarial case sets that break the system. For example, an ambiguous document, a query that leads to the fabrication of a citation, or an edge case the developer had never encountered because the normal situation had worked previously.

Two fundamental principles, both predating generative AI and indeed being more acute because of it.

No model is absolutely valid. Validation only proves that a model is suitable for a specific purpose under certain conditions, not that it is correct in all cases. With generative systems, this point matters more. The only things that limit the otherwise unlimited output space are the purpose and the conditions.

More complexity is not always better. The same applies to the governance built around them. We don’t need a hundred-page report on a low-materiality AI assistant.

Generative AI doesn’t make model risk management obsolete. In fact, it makes it more essential. The second line’s job shifts from replication to test design, from single-model evaluation to system-level assurance. The questions are the same; the evidence has changed.

Further reading

• SR 11-7 and SR 26-02 Supervisory Guidance on Model Risk Management

• NIST AI 600-1, Generative AI Profile, generative-specific risk categories mapped to the AI Risk Management Framework, including confabulation, data privacy, information security and human-AI configuration

• EU AI Act – the obligations side of the same problem

• RAGAS (Es et al.) – reference-free evaluation of retrieval-augmented systems

• SelfCheckGPT (Manakul et al.) – sampling-based fabrication detection without an external source

• Chain-of-Verification (Dhuliawala et al.) – verification-question approach to reducing fabrication

• Semantic Uncertainty (Kuhn, Gal and Farquhar) – clustering generations by meaning rather than wording

• CheckList (Ribeiro et al.) – behavioural testing for language systems