Picture a support inbox for a bank. Every message that comes in needs to be sorted into a category, a lost card, a refund request, a wrong charge, and sent to the right team.
A bank support analyst routes incoming customer messages about lost cards, refunds, and incorrect charges to separate support teams.
Now picture that sorting job handed to an AI model instead of a person. The model reads the message and hands back a short note in a fixed format, a little like a form with the same boxes every time, so the rest of the program can read it automatically and decide what to do next. No human has to interpret free text.
That fixed format is usually a small block of computer readable text called JSON, short for JavaScript Object Notation. Think of it as labeled boxes on a form. One box called intent holds the category. Another called priority holds how urgent it is.
The program reading the model’s answer does not understand English. It looks for those exact boxes, spelled exactly the way it expects, every single time. If a box goes missing, or a label is spelled slightly differently than the program expects, the program has no way to notice on its own. It just quietly stops working for that one message, while everything on the surface still looks fine.
AI companies release new model versions constantly, and deciding which LLM to use for a given job usually comes down to one number from the AI testing teams already run, how often it picks the right category. That single score can go up while something else, the exact shape of the reply, quietly gets worse, and a rising average has no way to warn you.
With plain prompting, you simply write:
“Return your answer as JSON.”
The model may still return:
Sure, here is the JSON:{"intent": "request_refund"}
That extra sentence can break code that expects JSON only. The model may also leave out a field or use a value the program does not expect.
Structured Outputs is a stricter feature that makes the model follow a predefined JSON structure, such as requiring intent, priority, and needs_human. It can prevent many formatting problems, but the application still needs to check whether the values and decision are correct.
I ran a real LLM regression test on one small, real application, instead of trusting the accuracy number alone. I built a support triage assistant, gave it 47 real customer messages from a public banking dataset, and ran the exact same messages through three real versions of an OpenAI model, an older one, the one I am treating as the model currently in production, and a newer candidate being considered as a replacement.
I expected the newer model to choose the correct category more often, but I also worried that it might occasionally ignore the exact format the program requires. Instead, the newer model followed the format every time. The production model made the formatting mistake. It did so quietly, on every refund question in the sample, spelling one label Request_refund with a capital R instead of the lowercase request_refund the rest of the system expects.
A human reading the reply would call it correct. A program matching labels exactly would silently drop every one of those tickets.
That is the problem this article is about: a model can sound correct to a person and still be wrong for the software that uses its answer.
A schematic diagram showing a bar chart of overall accuracy rising from an old model to a new model on the left, next to a paired comparison on the right where the old model answers one test case correctly and the new model answers the same case wrong, labeled as a negative flip
What this project builds, and why it uses Weave
Before writing any code, it helps to have one clear picture of what gets built and how its pieces fit together.
This project does five things:
Weave records what happens when the application runs: the question, the instructions, the model, the response, and the timing.
The instructions given to the model are saved with a version number, so older and newer instructions can be compared.
The real customer questions are saved as a test dataset, so every model answers the same examples.
A strict checker tests each response for exact requirements, such as valid JSON, required fields, allowed labels, and the correct category.
A second AI model reads each response and gives it a quality score, more like a human reviewer would.
The strict checker looks for exact machine requirements. The second AI judge evaluates the answer more like a reader. Using both helps reveal problems that either checker might miss.
Each of those ideas gets explained properly as it comes up. For now, start with Weave itself, since everything else in this article is recorded inside it.
Weave is a tool from Weights & Biases (W&B) for watching what an AI application actually does while it runs. Add one line, @weave.op(), above any Python function, and every single call to that function gets saved automatically, the exact text that went in, the exact text that came back, and how long it took.
Weave calls one of these saved records a trace, and it stores every trace in a project you can open and browse in a web page, the same way a photo app keeps a timeline of every photo you take.
A trace is not only useful for debugging a broken run after the fact. Once an application has been answering real questions for a while, its saved traces are also a ready made source of real examples, which matters later in this article, since the same 47 real questions that trace the application also become the dataset it gets tested against.
The application itself is deliberately small, one function, triage_message(text, model, prompt_ref), that reads one real customer message and asks a model to answer with a JSON object shaped like this:
Four boxes, every time. intent names the category. priority is low, medium, or high. needs_human is true or false, and it decides whether the message gets escalated to a person instead of handled automatically. reply is the short message the customer actually sees.
Only two things change across the rest of this article: which model answers, and which version of the instructions or the grading rules is active. The application logic itself never changes, which is what makes the comparisons later in this article fair.
One choice about how the model gets asked matters enough to explain now. The request to OpenAI uses plain prompted JSON, meaning the model is simply told in words to reply in this shape. It does not use OpenAI’s stricter Structured Outputs feature, the one already mentioned above, which can force a model’s answer into a fixed shape by construction.
That is deliberate, not an oversight. Using the strict feature here would have hidden some of the very failures this article is built to look for, an invalid response, extra text wrapped around the JSON, or a mislabeled field. Later in the article, once you have seen what actually broke, there is an honest look at exactly which of those failures the strict feature would and would not have caught.
The real customer messages come from BANKING77, a public dataset from a 2020 research paper by Iñigo Casanueva and coauthors at PolyAI (CC BY 4.0 license). It contains 13,083 real banking customer service questions, each labeled by hand with one of 77 fine grained categories, a card that never arrived, a refund that never showed up, a payment the customer does not recognize, and so on.
Fine grained means many of those 77 categories sound close enough to genuinely confuse a model, which is exactly the property that makes this dataset useful here. A model that can only tell the easy cases apart is not being tested very hard.
Setup
This project was written and run with Python 3.11 and Weave:
You need an OpenAI application programming interface (API) key, and a free Weights & Biases account for Weave. Run wandb login once in the activated environment, or set a WANDB_API_KEY environment variable. Save an OPENAI_API_KEY the same way, either as an environment variable or in a .env file next to the script below.
One small version note. This project was run against weave==0.52.40. Weave printed a notice on every run saying that exact version had been recalled over a technical issue and recommending an upgrade. The recall did not change anything in the results here, but install the current release instead of pinning an old one, pip install -U weave, unless you have a specific reason not to.
The complete script
Everything in this article, the traced application, the two versions of its instructions, the dataset, the strict rule based grader, the AI grader, and the model comparison, lives in one script. Save it as banking77_regression.py:
"""BANKING77 structured output regression testing.Traces a small banking support triage application with Weave, versions itssystem prompt, turns real BANKING77 questions into a Weave Dataset, buildsa graded LLM judge next to a deterministic contract scorer (valid JSON,required fields, allowed values, correct intent, correct escalation), runsa Weave Evaluation across three OpenAI models, validates the judge againstthe deterministic scorer, and checks whether a candidate model that wins onintent accuracy still respects the output contract the application dependson.Deliberately does NOT use response_format json_object or strict StructuredOutputs. Plain prompted JSON is the point, it is what makes invalid JSON,prose wrapped around JSON, and field value drift reachable failure modes.Run modes, in order: fetch fetch and save the real BANKING77 test questions used smoke smoke test prompt v1 against the candidate model dataset publish prompt v1/v2 and the Weave Dataset evaluate run the Weave Evaluation for all three models (judge v1) judge_check compare the judge scores to the deterministic scores refine_judge publish an improved judge prompt and rerun the evaluation judge_check_v2 compare the refined judge scores to the deterministic scores contract compare production vs candidate on accuracy vs contract sorted_diffs sort two models by absolute judge score difference"""import argparseimport asyncioimport csvimport ioimport jsonimport reimport statisticsimport timefrom pathlib import Pathimport requestsimport weavefrom dotenv import load_dotenvfrom openai import OpenAI, RateLimitErrorROOT = Path(__file__).resolve().parentOUTPUTS_DIR = ROOT / "outputs"OUTPUTS_DIR.mkdir(exist_ok=True)load_dotenv()PROJECT = "wb-authors/model-regression-tests-weave" # replace with your own W&B entity/projectPROMPT_NAME = "banking77-solver-prompt"JUDGE_PROMPT_NAME = "banking77-judge-prompt"DATASET_NAME = "banking77-eval-set"client = OpenAI()def call_with_retry(fn, *args, max_attempts=5, **kwargs): """Weave's Evaluation runs every row concurrently, which can burst past a shared organization tokens per minute limit on the judge model even though each individual call is small. Retries with backoff instead of letting a rate limit error drop that row's score.""" for attempt in range(max_attempts): try: return fn(*args, **kwargs) except RateLimitError: if attempt == max_attempts - 1: raise time.sleep(2 ** attempt)BANKING77_TEST_CSV_URL = ( "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/" "master/banking_data/test.csv")# Escalation policy, written before looking at any model output. Intents# that involve fraud, loss, blocked access, unrecognized money movement, or# compliance checks require a human. Everything else is routine.ESCALATE_INTENTS = { "lost_or_stolen_card", "lost_or_stolen_phone", "compromised_card", "card_payment_not_recognised", "direct_debit_payment_not_recognised", "cash_withdrawal_not_recognised", "unable_to_verify_identity", "verify_source_of_funds", "pin_blocked", "transaction_charged_twice",}# The 20 intents sampled for this article's dataset, chosen as five# confusable clusters so models have real room to disagree, not an# arbitrary slice of the 77 categories.SELECTED_INTENTS = { "card_arrival": 3, "card_not_working": 3, "lost_or_stolen_card": 3, "declined_transfer": 2, "failed_transfer": 2, "transfer_not_received_by_recipient": 2, "pending_transfer": 2, "request_refund": 3, "Refund_not_showing_up": 3, "card_payment_not_recognised": 3, "direct_debit_payment_not_recognised": 2, "cash_withdrawal_not_recognised": 2, "verify_my_identity": 2, "why_verify_identity": 2, "unable_to_verify_identity": 3, "verify_source_of_funds": 2, "compromised_card": 2, "lost_or_stolen_phone": 2, "pin_blocked": 2, "transaction_charged_twice": 2,}PROMPT_V1 = ( "You are a banking support triage assistant. A customer will send you " "one message. Read it and reply with a JSON object with exactly these " "four fields, intent, priority, needs_human, reply. intent must be one " "of the allowed banking intent labels given below, copied exactly. " "priority must be one of low, medium, high. needs_human must be true " "or false. reply is a short, natural reply to the customer.nn" "Allowed intent labels: {intents}")# v2 fixes a real gap found in the v1 smoke test, outputs/# banking77_smoke_test_v1.json. v1 never told the model the escalation# policy, so it guessed at needs_human/priority using its own judgment and# got escalation_correct on only 2 of 6 smoke examples even though intent# and JSON format were both already perfect. v2 states the policy# explicitly. Re-run on the same 6 examples afterward, 6 of 6 correct.PROMPT_V2 = ( "You are a banking support triage assistant. A customer will send you " "one message. Read it and reply with a JSON object with exactly these " "four fields, intent, priority, needs_human, reply. intent must be one " "of the allowed banking intent labels given below, copied exactly. " "priority must be one of low, medium, high. needs_human must be true " "or false. reply is a short, natural reply to the customer.nn" "Escalation policy, apply it exactly. If the intent is one of these, " "set needs_human to true and priority to high, regardless of how the " "message is worded, card_payment_not_recognised, " "cash_withdrawal_not_recognised, compromised_card, " "direct_debit_payment_not_recognised, lost_or_stolen_card, " "lost_or_stolen_phone, pin_blocked, transaction_charged_twice, " "unable_to_verify_identity, verify_source_of_funds. For every other " "intent, set needs_human to false and priority to low or medium, " "never high, even if the customer sounds frustrated or the situation " "sounds urgent.nn" "Allowed intent labels: {intents}")JUDGE_PROMPT_V1 = """You are grading one model's response to a banking customer support message.You will see the customer's message, the correct intent label, the correctescalation policy for that intent (whether it should escalate to a humanwith high priority, or not escalate and not use high priority), and themodel's raw response.Apply this hard ceiling before anything else. If the response is not validJSON, is missing any of the four required fields (intent, priority,needs_human, reply), or uses an intent label that is not a real bankingintent, the score must be 0 to 2, regardless of how good the reply textsounds. A broken output cannot be used by the downstream code that expectsthis exact shape.If the output is valid and complete, apply this next ceiling. If theintent is wrong, or the escalation decision (needs_human and priority)does not match the stated policy, the score must be 3 to 5, regardless ofreply quality.Only if the output is valid, complete, has the correct intent, and followsthe escalation policy correctly, score the reply text itself from 6 to 10:- 6 to 8: the reply is generic, or only loosely addresses the customer's specific message- 9 to 10: the reply is clear, specific to the customer's actual message, and appropriately tonedReturn your grade as strict JSON with this exact shape and nothing else:{"score": , "reasoning": ""}"""# v2 fixes a real miscalibration found during judge validation, see# banking77_judge_check_v1.json. The judge read the policy phrase "not# escalate, and not use high priority" as if it meant one specific# priority value was required, and penalized responses that used medium# instead of low even though both are valid non escalating priorities.# This produced 4 to 6 false disagreements per model. v2 states the rule# exactly (low and medium are both correct) and disagreements dropped to# 0 for two of the three models. The disagreements that remained for the# production model afterward were a different, real, non hypothetical# issue described in the article, not a judge bug.JUDGE_PROMPT_V2 = """You are grading one model's response to a banking customer support message.You will see the customer's message, the correct intent label, the correctescalation policy for that intent (whether it should escalate to a humanwith high priority, or not escalate and not use high priority), and themodel's raw response.Apply this hard ceiling before anything else. If the response is not validJSON, is missing any of the four required fields (intent, priority,needs_human, reply), or uses an intent label that is not a real bankingintent, the score must be 0 to 2, regardless of how good the reply textsounds. A broken output cannot be used by the downstream code that expectsthis exact shape.If the output is valid and complete, apply this next ceiling. If theintent is wrong, the score must be 3 to 5. Separately, check theescalation decision against the stated policy exactly as follows. If thepolicy says escalate, needs_human must be true and priority must beexactly high, anything else is a mismatch. If the policy says do notescalate, needs_human must be false and priority must not be high, butlow and medium are BOTH fully correct values for a non escalating case,there is no single required value between them, and choosing mediuminstead of low is not a mismatch and must not be scored as one. Only awrong needs_human value or a priority of high on a non escalating casecounts as an escalation mismatch. If the intent is right but theescalation mismatches by this exact definition, the score must be 3 to 5.Only if the output is valid, complete, has the correct intent, and followsthe escalation policy correctly by the exact definition above, score thereply text itself from 6 to 10:- 6 to 8: the reply is generic, or only loosely addresses the customer's specific message- 9 to 10: the reply is clear, specific to the customer's actual message, and appropriately tonedReturn your grade as strict JSON with this exact shape and nothing else:{"score": , "reasoning": ""}"""MODEL_SLOTS = { "older": "gpt-4o-mini", "production": "gpt-4.1-mini", "candidate": "gpt-5-mini",}JUDGE_MODEL = "gpt-4.1"# ---------------------------------------------------------------------------# Data fetch# ---------------------------------------------------------------------------def fetch_all_rows(): r = requests.get(BANKING77_TEST_CSV_URL, timeout=30) r.raise_for_status() reader = csv.DictReader(io.StringIO(r.text)) return list(reader)def fetch_examples(): rows = fetch_all_rows() all_categories = sorted(set(row["category"] for row in rows)) by_cat = {} for row in rows: by_cat.setdefault(row["category"], []).append(row["text"]) selected = [] qid = 0 for cat, n in SELECTED_INTENTS.items(): texts = by_cat.get(cat, []) for text in texts[:n]: qid += 1 selected.append({ "question_id": f"b77-{qid:03d}", "text": text, "ground_truth_intent": cat, "ground_truth_escalate": cat in ESCALATE_INTENTS, }) out = {"all_categories": all_categories, "selected": selected} out_path = OUTPUTS_DIR / "banking77_examples_selected.json" with open(out_path, "w") as f: json.dump(out, f, indent=2) print(f"Saved {len(selected)} examples across {len(SELECTED_INTENTS)} intents to {out_path}") return outdef load_examples(): return json.load(open(OUTPUTS_DIR / "banking77_examples_selected.json"))ALLOWED_PRIORITIES = {"low", "medium", "high"}# ---------------------------------------------------------------------------# The traced application# ---------------------------------------------------------------------------@weave.op()def triage_message(text: str, model: str, prompt_ref: str) -> str: """The one small application traced throughout this article.""" prompt = weave.ref(prompt_ref).get() response = call_with_retry( client.chat.completions.create, model=model, messages=[ {"role": "system", "content": prompt.content}, {"role": "user", "content": text}, ], max_completion_tokens=1000, ) return response.choices[0].message.content or ""class Banking77Solver(weave.Model): model_name: str prompt_ref: str @weave.op() def predict(self, text: str) -> str: return triage_message(text, self.model_name, self.prompt_ref)# ---------------------------------------------------------------------------# Deterministic contract scorer# ---------------------------------------------------------------------------def parse_json_response(raw: str): """Try three real strategies a plain text pipeline would try, in order, and record which one worked.""" text = raw.strip() try: return json.loads(text), "direct" except json.JSONDecodeError: pass fenced = re.search(r"```(?:json)?s*({.*?})s*```", text, re.DOTALL) if fenced: try: return json.loads(fenced.group(1)), "fenced" except json.JSONDecodeError: pass first = text.find("{") last = text.rfind("}") if first != -1 and last != -1 and last > first: try: return json.loads(text[first:last + 1]), "extracted_braces" except json.JSONDecodeError: pass return None, "unparsable"@weave.op()def contract_scorer(question_id: str, ground_truth_intent: str, ground_truth_escalate: bool, all_categories: list, output: str) -> dict: parsed, parse_method = parse_json_response(output) result = { "question_id": question_id, "valid_json": parsed is not None, "parse_method": parse_method, "has_required_fields": False, "intent_allowed": False, "intent_correct": False, "priority_allowed": False, "escalation_correct": False, "parsed": parsed, } if parsed is None or not isinstance(parsed, dict): return result required = {"intent", "priority", "needs_human", "reply"} result["has_required_fields"] = required.issubset(parsed.keys()) intent = parsed.get("intent") if isinstance(intent, str): result["intent_allowed"] = intent in all_categories result["intent_correct"] = intent == ground_truth_intent priority = parsed.get("priority") if isinstance(priority, str): result["priority_allowed"] = priority.lower() in ALLOWED_PRIORITIES needs_human = parsed.get("needs_human") if isinstance(needs_human, bool): result["escalation_correct"] = ( needs_human == ground_truth_escalate and (not ground_truth_escalate or (isinstance(priority, str) and priority.lower() == "high")) and (ground_truth_escalate or not (isinstance(priority, str) and priority.lower() == "high")) ) return result# ---------------------------------------------------------------------------# Graded LLM judge, rubric filled in after reading real smoke test output# ---------------------------------------------------------------------------class LLMJudge(weave.Scorer): judge_model: str judge_prompt_ref: str @weave.op() def score(self, question_id: str, text: str, ground_truth_intent: str, ground_truth_escalate: bool, output: str) -> dict: judge_prompt = weave.ref(self.judge_prompt_ref).get() policy_note = ( "escalate to a human, meaning needs_human must be true and " "priority must be exactly high" if ground_truth_escalate else "not escalate, meaning needs_human must be false and priority " "must not be high, but either low or medium is correct, there " "is no single required value between the two" ) user_content = ( f"Customer message:n{text}nn" f"Correct intent label: {ground_truth_intent}n" f"Correct escalation policy for this intent: should {policy_note}.nn" f"Model's raw response:n{output}" ) response = call_with_retry( client.chat.completions.create, model=self.judge_model, messages=[ {"role": "system", "content": judge_prompt.content}, {"role": "user", "content": user_content}, ], max_completion_tokens=500, response_format={"type": "json_object"}, ) raw = response.choices[0].message.content or "{}" try: parsed = json.loads(raw) score = int(parsed.get("score", 0)) reasoning = str(parsed.get("reasoning", "")) except (json.JSONDecodeError, ValueError): score = 0 reasoning = f"judge returned unparsable output: {raw[:200]}" return {"question_id": question_id, "judge_score": score, "judge_reasoning": reasoning}# ---------------------------------------------------------------------------# Smoke test, run before the judge rubric is written# ---------------------------------------------------------------------------def smoke_test_step(): weave.init(PROJECT) data = load_examples() all_categories = data["all_categories"] v1_ref = weave.publish( weave.StringPrompt(PROMPT_V1.format(intents=", ".join(all_categories))), name=PROMPT_NAME, ) print("prompt v1:", v1_ref.uri()) smoke = [] for ex in data["selected"][:6]: output = triage_message(ex["text"], MODEL_SLOTS["candidate"], v1_ref.uri()) scored = contract_scorer(ex["question_id"], ex["ground_truth_intent"], ex["ground_truth_escalate"], all_categories, output) smoke.append({"question_id": ex["question_id"], "text": ex["text"], "raw_output": output, "scored": scored}) print(ex["question_id"], "valid_json:", scored["valid_json"], "escalation_correct:", scored["escalation_correct"]) with open(OUTPUTS_DIR / "banking77_smoke_test_v1.json", "w") as f: json.dump(smoke, f, indent=2) with open(OUTPUTS_DIR / "banking77_prompt_v1_ref.json", "w") as f: json.dump({"prompt_v1": v1_ref.uri()}, f, indent=2)# ---------------------------------------------------------------------------# Dataset + prompt publishing, v2 filled in after inspecting the smoke test# ---------------------------------------------------------------------------def build_dataset_step(): weave.init(PROJECT) data = load_examples() all_categories = data["all_categories"] v2_ref = weave.publish( weave.StringPrompt(PROMPT_V2.format(intents=", ".join(all_categories))), name=PROMPT_NAME, ) print("prompt v2:", v2_ref.uri()) rows = [ { "question_id": ex["question_id"], "text": ex["text"], "ground_truth_intent": ex["ground_truth_intent"], "ground_truth_escalate": ex["ground_truth_escalate"], "all_categories": all_categories, "prompt_version": "v2", } for ex in data["selected"] ] dataset = weave.Dataset(name=DATASET_NAME, rows=rows) dataset_ref = weave.publish(dataset, name=DATASET_NAME) print("dataset:", dataset_ref.uri()) v1_ref = json.load(open(OUTPUTS_DIR / "banking77_prompt_v1_ref.json"))["prompt_v1"] with open(OUTPUTS_DIR / "banking77_refs.json", "w") as f: json.dump({"prompt_v1": v1_ref, "prompt_v2": v2_ref.uri(), "dataset": dataset_ref.uri()}, f, indent=2)# ---------------------------------------------------------------------------# Evaluation# ---------------------------------------------------------------------------def get_refs(): return json.load(open(OUTPUTS_DIR / "banking77_refs.json"))def collect_run_rows(client_obj, evaluate_call, examples_by_text): client_obj.flush() descendants = list(client_obj.get_calls(filter={"trace_ids": [evaluate_call.trace_id]})) by_question_id = {} for call in descendants: trace_name = (call.summary or {}).get("weave", {}).get("trace_name", "") if trace_name == "triage_message": q_text = call.inputs.get("text") ex = examples_by_text.get(q_text) if ex: by_question_id.setdefault(ex["question_id"], {})["output"] = call.output elif trace_name == "contract_scorer": out = call.output or {} qid = out.get("question_id") if qid: by_question_id.setdefault(qid, {})["contract"] = out elif trace_name == "LLMJudge.score": out = call.output or {} qid = out.get("question_id") if qid: by_question_id.setdefault(qid, {})["judge"] = out return by_question_idasync def run_evaluation_for_judge(judge_prompt_ref: str, output_suffix: str = ""): refs = get_refs() weave_client = weave.init(PROJECT) data = load_examples() examples_by_text = {ex["text"]: ex for ex in data["selected"]} dataset = weave.ref(refs["dataset"]).get() judge = LLMJudge(judge_model=JUDGE_MODEL, judge_prompt_ref=judge_prompt_ref) evaluation = weave.Evaluation( name="banking77-model-comparison", dataset=dataset, scorers=[contract_scorer, judge], ) all_results = {} for slot, model_name in MODEL_SLOTS.items(): solver = Banking77Solver(name=f"banking77-solver-{slot}", model_name=model_name, prompt_ref=refs["prompt_v2"]) print(f"Evaluating {slot} ({model_name})...") summary = await evaluation.evaluate(solver) evaluate_calls = list(evaluation.get_evaluate_calls()) latest_call = evaluate_calls[-1] rows = collect_run_rows(weave_client, latest_call, examples_by_text) all_results[slot] = {"model": model_name, "summary": summary, "evaluate_call_url": f"https://wandb.ai/{PROJECT}/r/call/{latest_call.id}", "rows": rows} print(f" summary: {json.dumps(summary, default=str)[:400]}") out_path = OUTPUTS_DIR / f"banking77_evaluation_results{output_suffix}.json" with open(out_path, "w") as f: json.dump(all_results, f, indent=2, default=str) print("Saved", out_path) return all_resultsdef evaluate_step(): weave.init(PROJECT) judge_prompt_ref = weave.publish(weave.StringPrompt(JUDGE_PROMPT_V1), name=JUDGE_PROMPT_NAME).uri() with open(OUTPUTS_DIR / "banking77_judge_refs.json", "w") as f: json.dump({"judge_prompt_v1": judge_prompt_ref}, f, indent=2) asyncio.run(run_evaluation_for_judge(judge_prompt_ref, output_suffix="_v1"))def refine_judge_step(): weave.init(PROJECT) judge_v2_ref = weave.publish(weave.StringPrompt(JUDGE_PROMPT_V2), name=JUDGE_PROMPT_NAME).uri() judge_refs = json.load(open(OUTPUTS_DIR / "banking77_judge_refs.json")) judge_refs["judge_prompt_v2"] = judge_v2_ref with open(OUTPUTS_DIR / "banking77_judge_refs.json", "w") as f: json.dump(judge_refs, f, indent=2) asyncio.run(run_evaluation_for_judge(judge_v2_ref, output_suffix="_v2"))# ---------------------------------------------------------------------------# Analysis# ---------------------------------------------------------------------------def judge_check_step(suffix="_v1"): results = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) report = {} for slot, data in results.items(): rows = data["rows"] pairs = [(r["judge"]["judge_score"], r["contract"]) for r in rows.values() if "judge" in r and "contract" in r] judge_scores = [p[0] for p in pairs] contract_clean = lambda c: ( c["valid_json"] and c["has_required_fields"] and c["intent_allowed"] and c["intent_correct"] and c["priority_allowed"] and c["escalation_correct"] ) disagreements = [ (qid, r["judge"]["judge_score"], r["contract"]) for qid, r in rows.items() if "judge" in r and "contract" in r and ((r["judge"]["judge_score"] >= 6) != contract_clean(r["contract"])) ] report[slot] = { "n": len(pairs), "mean_judge_score": statistics.mean(judge_scores) if judge_scores else 0, "valid_json_rate": statistics.mean([1 if r["contract"]["valid_json"] else 0 for r in rows.values() if "contract" in r]), "has_required_fields_rate": statistics.mean([1 if r["contract"]["has_required_fields"] else 0 for r in rows.values() if "contract" in r]), "intent_allowed_rate": statistics.mean([1 if r["contract"]["intent_allowed"] else 0 for r in rows.values() if "contract" in r]), "intent_correct_rate": statistics.mean([1 if r["contract"]["intent_correct"] else 0 for r in rows.values() if "contract" in r]), "priority_allowed_rate": statistics.mean([1 if r["contract"]["priority_allowed"] else 0 for r in rows.values() if "contract" in r]), "escalation_correct_rate": statistics.mean([1 if r["contract"]["escalation_correct"] else 0 for r in rows.values() if "contract" in r]), "disagreement_count": len(disagreements), "disagreements": disagreements, } out_path = OUTPUTS_DIR / f"banking77_judge_check{suffix}.json" with open(out_path, "w") as f: json.dump(report, f, indent=2, default=str) print(json.dumps(report, indent=2, default=str)) return reportdef contract_step(suffix="_v2"): results = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) prod_rows, cand_rows = results["production"]["rows"], results["candidate"]["rows"] diffs = [] for qid, prod_row in prod_rows.items(): cand_row = cand_rows.get(qid) if not cand_row or "contract" not in prod_row or "contract" not in cand_row: continue pc, cc = prod_row["contract"], cand_row["contract"] diffs.append({ "question_id": qid, "production_intent_correct": pc["intent_correct"], "candidate_intent_correct": cc["intent_correct"], "production_contract_clean": all([pc["valid_json"], pc["has_required_fields"], pc["intent_allowed"], pc["priority_allowed"], pc["escalation_correct"]]), "candidate_contract_clean": all([cc["valid_json"], cc["has_required_fields"], cc["intent_allowed"], cc["priority_allowed"], cc["escalation_correct"]]), }) out_path = OUTPUTS_DIR / "banking77_contract_diffs.json" with open(out_path, "w") as f: json.dump(diffs, f, indent=2) print(json.dumps(diffs, indent=2)) return diffsdef sorted_diffs_step(suffix="_v2", model_a="older", model_b="candidate"): results = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) a_rows, b_rows = results[model_a]["rows"], results[model_b]["rows"] diffs = [] for qid, a_row in a_rows.items(): b_row = b_rows.get(qid) if not b_row or "judge" not in a_row or "judge" not in b_row: continue a_score, b_score = a_row["judge"]["judge_score"], b_row["judge"]["judge_score"] diffs.append({"question_id": qid, f"{model_a}_score": a_score, f"{model_b}_score": b_score, "abs_diff": abs(b_score - a_score), "diff": b_score - a_score}) diffs.sort(key=lambda d: -d["abs_diff"]) out_path = OUTPUTS_DIR / f"banking77_sorted_diffs_{model_a}_vs_{model_b}.json" with open(out_path, "w") as f: json.dump(diffs, f, indent=2) print(json.dumps(diffs[:8], indent=2)) return diffsif __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--mode", required=True) args = parser.parse_args() if args.mode == "fetch": fetch_examples() elif args.mode == "smoke": smoke_test_step() elif args.mode == "dataset": build_dataset_step() elif args.mode == "evaluate": evaluate_step() elif args.mode == "judge_check": judge_check_step("_v1") elif args.mode == "refine_judge": refine_judge_step() elif args.mode == "judge_check_v2": judge_check_step("_v2") elif args.mode == "contract": contract_step("_v2") elif args.mode == "sorted_diffs": sorted_diffs_step() else: raise SystemExit(f"unknown mode {args.mode}")
Run the steps in order, each one building on the outputs of the last:
fetch downloads the BANKING77 test questions used in the article.
smoke runs the first prompt on six questions so we can catch obvious problems before the full evaluation.
dataset saves the improved prompt and publishes the reusable Weave Dataset.
evaluate runs all three models against the same 47 questions and records the outputs and scores.
judge_check compares the AI judge with the strict checker.
refine_judge publishes a clearer judging rubric and runs the evaluation again.
judge_check_v2 checks whether the revised judge now agrees with the strict checker.
contract compares the production stand in and the candidate on exact output requirements.
sorted_diffs sorts model score differences so the largest changes are easy to inspect first.
The rest of this article explains what those runs produced, in plain terms, using the real output saved along the way.
Why the instructions needed a second version
The smoke step exists for a reason worth explaining before anything else. Before trusting one set of instructions with 47 real customer messages across three models, run it on a small handful first and actually read what comes back.
That first attempt, PROMPT_V1, told the model the JSON shape and the allowed category labels, but it never told the model the rule for deciding needs_human and priority. It left that judgment call entirely up to the model.
On six smoke test examples, the JSON formatting and the category choice were already perfect, 6 out of 6. The escalation decision was correct on only 2 out of 6. The failures were not random guesses either.
On the message “I still have not received my new card, I ordered over a week ago,” the correct category, a card that has not arrived, is meant to be handled routinely under the rule this project defines. The model answered that it needed a human right away and marked it medium priority.
That is a perfectly reasonable read of the words, the message does sound a little frustrated. It is also the wrong answer for a program that needs one fixed rule applied the same way every time, not a rule that shifts depending on tone.
PROMPT_V2 fixes this by spelling the rule out directly. It names exactly which categories require a person and high priority, and states that every other category must use low or medium, never high, no matter how the message sounds.
Rerun on the same six examples, the escalation decision was correct on 6 out of 6, with the JSON formatting and category choice unchanged. Both versions of the instructions stayed saved in Weave under the same name, a small prompt registry that a reader can open and compare side by side, not a claim to take on faith.
Turning real production traces into an LLM eval dataset
A Weave Dataset is a saved, versioned list of rows, and once it exists, a Weave Evaluation can run any application against every row and grade what comes back. This project’s dataset holds 47 real BANKING77 questions across 20 of the dataset’s 77 real categories.
Those 20 were not picked at random. They form five groups of categories that sound close enough to genuinely confuse a model, card problems, transfer problems, refund problems, unrecognized payments, and identity checks, plus a few standalone security and billing categories.
23 of the 47 questions are meant to escalate to a person under the fixed rule, and 24 are not, a deliberately even split. Each row carries the real question text, the real correct category, whether it should escalate, and the full list of 77 valid category labels the model is allowed to choose from.
Two different graders, checking two different things
Every answer in this project gets graded twice, by two very different kinds of checker, and the difference between them matters for everything that follows. One checker, contract_scorer, follows a fixed rule with no room for interpretation, the same way a form processing machine either finds a barcode in the right spot or does not.
It parses the raw text as JSON, tries a couple of common fallback methods if the first attempt fails, and then checks a short list of yes or no questions.
Are all four fields present? Is the category one of the 77 real allowed labels? Does it match the correct answer? Is the priority an allowed value? Does the escalation decision match the rule?
None of that requires judgment. A field is either there or it is not.
The second checker is a graded AI judge, a separate model call that reads the customer’s message, the correct answer, the escalation rule, and the first model’s raw response, then hands back a score from 0 to 10 with a short explanation, closer to a second person reading the reply and forming an opinion.
Its scoring rules were written only after reading real responses from the smoke test, not guessed in advance, and they follow the same priorities as the strict checker on purpose.
Broken output or a disallowed category caps the score at 2. A wrong category or a broken escalation rule caps it at 5. Only a response that is valid, correctly labeled, and correctly escalated gets judged on how good the actual reply text is.
The judge runs on a separate model, gpt-4.1, one that is not any of the three models being compared, so it is never grading its own family’s work.
Comparing three real LLM model versions
A Weave Evaluation ties the dataset, the application, and both graders together in one run. Three model choices stand in for a real AI model selection decision a team might face:
gpt-4o-mini, standing in for an older model still running in some legacy code path.
gpt-4.1-mini, treated here as the model currently in production.
gpt-5-mini, the newer model a team is considering deploying in its place.
Running the full comparison, using the corrected grading rules from the next section, produced this real result, computed from all 47 questions per model:
what was checked
older (gpt-4o-mini)
production (gpt-4.1-mini)
candidate (gpt-5-mini)
valid JSON
1.000
1.000
1.000
all four fields present
1.000
1.000
1.000
category label allowed
1.000
0.936
1.000
category correct
0.723
0.766
0.915
priority value allowed
1.000
1.000
1.000
escalation decision correct
0.957
0.957
0.979
average AI judge score (0 to 10)
6.57
7.79
9.34
average reply time (seconds)
2.13
3.11
8.86
A number close to 1.000 means nearly every one of the 47 answers passed that check.
Annotated Weave native metrics radar chart comparing the candidate and production models across every check at once, with a callout warning that on Latency, Total Tokens, and Cost, smaller is better, unlike the other axes
Every one of these runs is visible and comparable in Weave’s own dashboard.
Annotated Weave Evaluations tab listing several real evaluation runs for the banking triage solver, with callouts marking a warning icon from a rate limit incident, a clean run, and the escalation_correct and has_required_fields contract check columns
Checking whether the AI judge can actually be trusted
Before trusting any score an AI judge hands out, it helps to check its work against something that cannot be argued with, which is exactly what the strict rule based checker is for. The first version of the judge’s scoring rules, run against all three models, disagreed with the strict checker 4 times on the older model, 3 times on production, and 6 times on the candidate.
Every one of those disagreements had the same shape. The judge scored a response as only partly correct even though the strict checker said the category and the escalation decision were both right.
Reading the judge’s own written explanations showed exactly why. The judge had read the rule “not escalate, and not use high priority” as if it meant one single specific priority value was required, and it was marking a response wrong just for choosing medium instead of low, even though the rule never asked for one specific value between the two.
That is a real, fixable misreading, not a vague sense that something was off.
The corrected version of the rules, JUDGE_PROMPT_V2, states plainly that low and medium are both correct for a routine case, and neither counts as a mismatch. Rerunning the same 47 questions per model against the corrected judge dropped the disagreements to zero for the older model and the candidate model.
Production still showed 4 disagreements afterward, and it would have been easy to assume the judge simply needed one more fix. It did not. Reading those four cases one by one turned up something else entirely, which is the actual point of the next section.
Sorting by the size of the disagreement instead of reading every answer
Forty seven questions across three models adds up to 141 separate graded answers, more than anyone wants to read line by line. Sorting by how far apart two models’ scores are turns that pile into a short list, worth starting with the biggest disagreements and working down from there.
Sorting the older and candidate models this way puts several perfect swings at the top, the older model scoring 3 out of 10, the candidate scoring 10 out of 10, on the exact same question.
One of them shows the pattern clearly. For the message “How do I locate my card?”, the correct category is a card that has not arrived yet. The older model, gpt-4o-mini, answered with a different, real category about linking a card, a genuinely plausible misreading if you are not holding the full list of 77 labels in front of you, the word “locate” does sound a little like a linking question out of context.
The newer model, gpt-5-mini, answered correctly. Nothing about this pair has anything to do with formatting. Both answers were valid JSON with all four fields present.
This difference is about which model actually understood the message correctly, on a dataset built specifically to include categories that sound alike.
What actually happened when production was tested against the candidate
This is the comparison the whole project was really built for. Treat gpt-4.1-mini as the model currently in production, and gpt-5-mini as the candidate being considered to replace it, then check every measurement, not only the overall judge score.
The candidate matched or beat production on every single one, including every formatting check. Neither model ever produced invalid JSON or left out a field, both were perfect there.
But the row for allowed category labels tells a different story. Production scored 0.936. Both other models scored a perfect 1.000. Production is the one with a real formatting problem here, not the candidate.
Annotated Weave native Compare evaluations view for the candidate and production models, with callouts marking which color is which model, the two metrics that tie perfectly, and the two gaps Weave itself flags in red
The cause is specific, and it repeats identically across every refund question in the sample. On three separate real customer messages, all asking about a refund, gpt-4.1-mini answered with the category spelled "Request_refund", a capital R.
The real BANKING77 label, and the value written in every row of the dataset, is lowercase, request_refund. A program checking that label the way real software actually does, an exact match, would silently fail to route every single one of these tickets, even though the reply text underneath reads just fine.
Customer message: Can I receive a refund for my item?gpt-4.1-mini's raw response:{"intent": "Request_refund", "priority": "medium", "needs_human": false, "reply": "Please provide the details of the transaction so I can assist you with the refund process."}
Annotated Weave trace detail for this exact question, with callouts marking the model’s raw output containing Request_refund with a capital R, the fields that passed, and the two fields that failed
The AI judge gave that response a 9, and its own written explanation said plainly, “the correct intent (case sensitivity is not penalized),” naming the exact thing it was choosing to ignore. The strict checker disagreed, correctly, because "Request_refund" simply is not one of the 77 real category labels at all, and an exact match check is precisely what routing code in a real system actually runs.
The same capital R habit showed up on two other real refund questions in the sample, not only this one, and both gpt-4o-mini and gpt-5-mini wrote the correct lowercase label on all three.
This one specific habit is a real, already shipping formatting bug in the model currently in production. The candidate did not introduce it. It was only visible at all because something else existed to compare it against.
This deserves to be said plainly, since the honest result matters more than a tidy one. This project did not find the pattern it set out looking for. The candidate never broke a rule production was following correctly.
The project still earned its cost, because a test built to catch a new problem caught an old one instead, on a bug a person skimming the reply would never notice, since the reply itself reads as completely correct.
One more real case is worth including precisely because it complicates the story instead of wrapping it up neatly. For the message “Where can I view my PIN?”, the correct category should have triggered escalation.
All three models, including the newer one, answered with a different category about changing a PIN, and marked it as not needing a person, a reasonable sounding guess that misses the point. It is the only question in the whole sample where the newer model got both the category and the escalation decision wrong at once.
Reading the message again, it genuinely reads more like a request to view or change a PIN than a report of one being blocked, which is worth treating as a possible labeling question in the original dataset, not only a shared model mistake.
A similar case turned up inside the four leftover judge disagreements on production. One customer described a declined card purchase, and the dataset’s own answer for that question was a category about a declined transfer, while gpt-4.1-mini answered with a different, real, defensible category about a declined card payment.
Public datasets are built by people, and their labels are not beyond question. An honest project says so when it finds a case like that, instead of quietly counting it as one more model mistake.
The full loop, and what it does not prove
Put together end to end, this project is one repeatable loop for regression testing an LLM before it replaces one already in production. Trace a small real application with Weave. Fix its instructions once a real smoke test finds a real gap.
Turn real traces into a dataset. Build two graders, one strict and one that reads for meaning, and check them against each other. Run a full comparison across three models.
Sort the results by how much they disagree instead of reading every row. Finally, run the one comparison a real deployment decision actually depends on, the model already live against the one being considered to replace it.
One honest question remains open, and it is worth sitting with rather than resolving too neatly. The AI judge read straight past the capital letter difference in Request_refund because it was grading for meaning, and the strict checker caught it because it was not. That gap, a judge that reads more kindly than the exact rule a real system depends on, is close to unavoidable for any grader built to read like a person.
If a project only had an AI judge, with no strict rule based checker running alongside it, how would anyone ever catch a bug like this one, an answer that looks obviously correct and is silently, mechanically wrong underneath?
What this specific project did prove is narrower than a verdict on which model is better in general, and more useful because of it. On one small application, across 47 real customer messages, the newer model never lost to the one already running in production, on formatting or on accuracy.
The most useful thing this project found was not really about the future model being considered at all. It was about the one already live, and the only reason to see it was building something to compare it against.
Sources
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić, Efficient Intent Detection with Dual Sentence Encoders, Proceedings of the 2nd Workshop on Natural Language Processing (NLP) for Conversational AI, Association for Computational Linguistics (ACL), 2020. Introduces the real BANKING77 dataset used throughout this article.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto, Positive-Congruent Training: Towards Regression-Free Model Updates, Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Introduces the negative flip, the finding that motivated this project’s original hypothesis.
OpenAI, Introducing Structured Outputs in the API, official product announcement. The source for this article’s opening claim about the unreliability of plain prompted JSON.
