A credit team I worked with had a model that declined 28% of applicants. The vendor’s one-pager said it “weighs over 400 signals.” That is not an explanation, it is a shrug. After six weeks of poking at the thing by hand, they had narrowed it to three inputs doing most of the work, and one of them was a postcode quietly standing in for ethnicity. Nobody read a single line of the model’s internals. They just asked it the right questions in the right order.
That is the whole method. A blackbox AI system will never explain itself, but it will answer honestly if you probe it properly. Here is the process I use: seven steps, one afternoon for the first pass, and a standing fifteen-minute check after that.
Step 0: Pick One Decision, Not the Whole Model
You cannot understand a blackbox AI model in general. You can only understand it on a specific input. Name one decision the system makes: a loan refusal, a fraud flag, a ticket routed to a human, a resume that never reaches a recruiter.
Narrow is the point. These systems quietly shape mortgages, hiring, and insurance payouts, which is the territory covered in a wider examination of the hidden decisions that run your life. Your job is smaller and far more useful: explain one of those decisions end to end.
Step 1: Freeze the Version and Write the Question Down
Pin the model version, the date, the endpoint, and the temperature. Providers ship silent updates, and a probe set run against last month’s build tells you nothing about this month’s. If you are testing an API, save the exact model string, not just the product name.
Then write your question in one sentence. “For applicants scoring above 700, what pushes a decision from approve to decline?” A vague question produces vague probes, and you will spend three hours generating charts nobody can act on.
Step 2: Build a 200-Row Probe Set by Hand
Synthetic data only. Do not paste customer records into anything. Two hundred rows with around a dozen fields takes about 90 minutes to write, and that set will carry you for a year. Here is what goes in:
- Baselines: five plausible, ordinary cases modelled on the shape of real ones, with identifiers scrambled.
- Twins: the same row with exactly one field changed. Tenure of 11 months becomes 12. Income of 3,999 becomes 4,000.
- Boundaries: the values either side of every documented threshold, plus 0, -1, and 999999.
- Broken rows: a blank field, an empty string, a 190-year-old applicant, an emoji where a surname should be.
- Adversarial rows: two rows that differ only by a name or postcode that logically should not carry any weight at all.
If 200 rows feels excessive, start at 40. Structure matters more than volume, and this 90-minute interrogation walkthrough shows how far you can get in a single sitting.
Step 3: Flip One Variable at a Time
Run the twins in pairs. Change tenure, hold everything else steady, compare the two outputs. Then change income, hold tenure. One variable per run, or you learn nothing.
A real example from a support routing model: a base ticket went to Tier 1. We changed the customer’s plan from “Pro” to “Business” and the identical ticket jumped to Tier 3. Nothing in the documentation listed plan tier as a routing input. Two probes, ninety seconds, one uncomfortable conversation with the vendor.
Step 4: Score the Flip Rate, Not Your Gut
Count how many rows change their output when you alter something that logically should not matter. If 40 of 200 rows flip, your instability rate is 20%, and that is a number you can put in a slide. Re-run each row three times at temperature 0 as well. Anything above roughly 5% variation between identical runs is nondeterminism, which means every “why did this happen” investigation has a coin-flip floor underneath it.
The rejection-reason trap
Language models generate explanations that read beautifully and frequently bear no relation to the features actually driving the score. Test it directly: submit the same case twice with the list of available reason codes in a different order. If the stated reason changes while the decision stays fixed, the explanation is decoration. Ship that to a customer and you have a compliance problem, not a UX one.
Step 5: Push the Edges, Where Blackbox AI Gets Weird
Boundary rows are where the interesting failures live. One routing model sent “password reset” tickets to the billing queue whenever the customer’s plan field was empty, a quirk invisible in any normal traffic sample and a one-line fix once observed. Missing values, zeros, and negative numbers account for a disproportionate share of nonsense output. Probe them before your users do.
Step 6: Log Everything So You Can Rerun It
Keep a plain CSV with five columns: input, output, timestamp, model version, notes. No dashboard required. The payoff arrives on the second run, when a provider update lands and you can prove the decline rate shifted six points overnight because last month’s answers are sitting in a file.
If any model output leaves your building as customer-facing writing, provenance is worth setting up at the same time. Watermarking text in Python gives you a way to trace a paragraph back to the run that produced it.
When You Can Open the Box Instead
Probing is the workaround for hidden weights. Sometimes the faster route is to stop using a blackbox for that particular job. Small open-weight models running on your own hardware hand you logits, token probabilities, and attention maps, and for narrow tasks like classification or field extraction the accuracy gap is often smaller than teams expect. That is the trade behind deploying local agents on a compact open model rather than calling an opaque endpoint for every decision.
What the Findings Usually Change
Three outcomes, in rough order of frequency. You fix the input: the postcode proxy gets removed, accuracy dips 0.4 points, and complaints fall by a fifth. You move the threshold: the model was fine, the cut-off was set in 2019 and never revisited. Or you replace the model, which is rarer than vendors fear and justified only when the same flaw shows up in every configuration you try. There is a longer discussion of the hidden logic behind high-stakes automated decisions if you need that context for the argument.
Turn the Probe Set Into a Habit
The first afternoon is the expensive part. After that the maintenance is trivial:
- Rerun the full probe set after every model or version change. Fifteen minutes, no meeting needed.
- Add a row whenever a complaint arrives. The complaint is a free probe you did not have to invent.
- Share your flip rate with the vendor. Nothing accelerates a fix like a 20% instability figure in an email.
- Keep the version log. When somebody asks in eight months why the decline rate moved, you will have the answer and they will not.