The rejection arrived as a four-line email. Maria, 34, had banked with the same branch for nine years, earned £48,000 as a radiographer, and repaid a previous £6,000 loan without missing an instalment. The system said no to her £12,000 application anyway.
Nobody at the bank could tell her why. The decision came from a gradient-boosted model with 400 trees trained on 3.1 million historical applications, and the analyst who built it was on annual leave.
What follows is the sequence we used that afternoon to get a defensible answer, including the numbers it produced. It is not a theory piece about opacity. It assumes you have a model you cannot read and a decision you have to justify. Budget 90 minutes.
Step 1: Choose the question before you choose the tool
“Explain the model” is not a task. It is four tasks wearing one coat, and each needs different machinery:
- Global behaviour: across 500,000 applicants, which inputs move the score at all?
- Local decisions: why did Maria score 0.41 default probability instead of 0.09?
- Counterfactuals: what would have produced a yes?
- Group fairness: do two applicants with identical finances get different answers because of something unmeasured?
Complaints teams want the middle two. Data scientists default to building the first. That mismatch burns more explainability budget than any technical limitation, and it is why automated decision systems end up shaping access to credit, insurance and jobs with nobody able to account for a single case. It is a pattern examined closely in the hidden decisions blackbox AI makes about your life.
Step 2: Check the model deserves the effort
Before spending a day on explanations, look at calibration and at holdout performance from the last six months, not the training report. A classifier at 0.64 AUC on stale data will produce explanations that accurately describe a broken process. So will a model that predicts 30% default risk on cases that historically default 12% of the time.
Explaining a badly calibrated model is arguably worse than explaining nothing, because the explanation inherits the score’s false authority. Teams discover this after publishing a feature-importance chart for a model that should simply have been retrained, which is a good reason to understand what you are actually uncovering about the hidden logic inside a decision model before you put it in front of a regulator.
Step 3: Map the global behaviour first
Two techniques cover most of what you need, and they take about twenty minutes combined.
Permutation importance, five lines of Python
Shuffle one column at a time in a holdout set and watch the error move. With scikit-learn that is permutation_importance(model, X_holdout, y_holdout, n_repeats=10, random_state=0), then sort the means.
On this lending model, shuffling debt_to_income cost 0.21 AUC. Shuffling credit_history_length cost 0.04. Two of twenty-eight features carried most of the signal, which told the risk team immediately where to focus its documentation effort.
Read a SHAP summary plot with one question
Do not read it like a leaderboard. Read it looking for clusters and thresholds. In this model, almost every applicant with three or more late payments in the previous 18 months landed on the negative side of the plot, regardless of income or tenure. That is a rule the model invented. Nobody wrote it into the policy document, and no human had reviewed it.
A caveat you will hit within a week: correlated inputs split the credit between them. Late payments and returned direct debits each look half-important when they are really one behaviour measured twice.
Step 4: Explain the single decision
For Maria, a SHAP waterfall gave the whole story in six bars, expressed in probability terms:
- base rate for the portfolio: 0.09
- debt-to-income of 0.62: +0.19
- three late payments in 18 months: +0.11
- eleven credit enquiries in six months: +0.07
- nine-year account tenure: -0.04
- stable salaried income: -0.01
Total: 0.41, comfortably over the 0.25 cut-off. The two-line version for a complaints handler reads: “Declined primarily on affordability ratio and recent repayment history; long tenure and stable income counted in the applicant’s favour but did not offset it.” That sentence is worth more than the chart, because it can be read aloud to a customer without a translation layer.
Step 5: Turn the explanation into a counterfactual
People rarely want to know why. They want to know what to change. Generate counterfactuals with a library such as DiCE, or with a constrained random search around the applicant’s real profile, and enforce two rules from the start.
- Only suggest features the person can act on. Age, postcode and account tenure are off limits.
- Keep the number of changes to two or three, or the advice turns into fiction.
The unconstrained run suggested Maria increase her income by £22,000, which is not advice. With constraints applied, the answer became: clear two of the three late payments through a payment plan and bring the card utilisation behind that 0.62 ratio below 0.48, and the model flips to approve. She did the second one, and her appeal succeeded six weeks later.
Step 6: Attack your own explanation
Explanations are model outputs, and they fail like any other output. Four checks catch most of it:
- Rerun with a different seed. If your top three features reshuffle, the story is fragile and probably reflects correlated inputs.
- Retrain without the top feature. If accuracy barely drops, that feature may be standing in for something else entirely.
- Change the imputation. Fill missing values a second way. Explanations that swing wildly are describing your preprocessing, not the applicant.
- Compare two models. Three architectures can score within 0.01 AUC of each other and rank features in completely different orders. An explanation belongs to a model, not to the problem.
Step 7: Write it down in a form that survives review
An explanation living in a notebook will not help anyone in eighteen months. Record the model version and hash, the training window, the explainer library and its version, the random seed, the holdout slice used for any worked example, and one plain-English paragraph per decision type. Reviewers read the paragraph first and the appendix second, if at all.
If a model card or audit summary is going to be shared outside the team, consider tracking where it travels. Watermarking text in Python gives you a way to trace a leaked copy back to whoever received it, which matters more than usual when the document describes how a lending model treats protected groups.
When the vendor will not hand over the weights
Plenty of systems are black boxes by contract rather than by architecture. The workaround is probing. Send 20,000 synthetic applicants through the vendor’s API, staying inside its acceptable-use terms, then fit a shallow surrogate model to the responses. If a depth-four decision tree reproduces 94% of the vendor’s decisions, you have an approximation good enough to reason about, argue with and monitor.
Do not send real customer records to a third party for this. Run the surrogate pipeline on your own hardware; a small local model handled the labelling and clustering steps in our case, and this walkthrough on deploying local agents with LFM2.5-2.6B covers the setup if you want the same pattern.
The 90-minute version
- 0-10 min: write down the exact question you are answering, in one sentence
- 10-25 min: calibration check and holdout AUC
- 25-45 min: permutation importance, then SHAP summary for thresholds
- 45-65 min: waterfall for the case in front of you
- 65-80 min: constrained counterfactuals, human-readable
- 80-90 min: rerun with a new seed, then write the two-line reason
What all of this costs
KernelSHAP on a single case took roughly 40 milliseconds against this model. Fine for appeals, useless inside a 10-millisecond payment authorisation path, which is why the honest answer to “can we explain everything in real time” is usually no. Explanation artifacts grow quickly as well: storing contributions for 500,000 monthly decisions adds real storage and real privacy surface, since a SHAP matrix is a detailed financial profile of a named person. Monitoring costs more, because a model’s explanations drift before its accuracy does, and by the time the AUC moves the behaviour has been off for months.
None of this makes the model transparent. Maria’s bank still cannot describe 400 trees in a way a customer would recognise. What changed is everything around them: every declined application now generates a two-line reason code, every appeal carries a counterfactual, and the risk team has a number for how much the model leans on debt-to-income rather than a hunch. The box stayed shut. At least now somebody stands outside it with a torch.