如何给你的 AI 流水线设计一套可迁移的「考试」

So far this series has been about giving my order-reading LLM an exam.

Some of you have been reading it thinking: "Mine isn't orders, it's meeting-minutes summarization." "I'm using it for email triage."

Good news — the exam is built to be stolen. Only two things need swapping: the reference data you match against (mine: a product catalog) and the worst accident (mine: wrong goods loaded onto a truck). Let's go in order.

Step 1 · Write down the worst first

Before writing a single test question, write this:

When this AI is wrong, which of the consequences cannot be undone?

For my order program it was "the wrong goods get loaded onto a truck." What is it for yours? A few examples —

auto-reply email AI    a wrong answer goes out to the customer — you can't unsend it
meeting-minutes AI     writes "decided" on something undecided — once shared, work proceeds on it
research AI            an invented number enters the report — reports travel upward
data-cleanup AI        overwrites the original — no backup, no recovery

You can see the pattern. Sent, deleted, escalated — these don't come back. That's your truck.

Step 2 · Build the grade table

Once the worst is written down, the grades fall out on their own. One criterion — can a human undo it?

FATAL      cannot be undone              sent, deleted, reported, charged
RISKY      confirmed something ambiguous  right this time, fatal next time
MISSED     dropped something              a human can still catch it
HARMLESS   over-asks "please confirm"     just slower

The only line you have to define yourself is the first one. The other three read the same in any project.

And take the principle with you as-is: a wrong confirmation is worse than no confirmation.

Step 3 · Plant the traps

Exam questions come from traps, not from normal cases. There are four kinds, and every domain has all four.

Confusable pairs — mine was clear tape 48mm vs 60mm. In meeting minutes: two attendees named Kim, one an associate, one a manager. In email: reply vs forward. Find the pairs in your data that are similar enough to confuse, and deliberately put both in.

Plausible non-targets — "What are the specs on the 250 shipping box?" contains a product name but is not an order. In minutes: "let's decide that next time" (not a decision). In email: a promotional mail (not something to answer). Put in things that only look like what the AI is supposed to catch.

Mid-message reversals — "5 boxes please — no wait, make it 3." In minutes: "let's go with A → actually B is better." Questions that are wrong if you only read the first half.

Accidents after learning — mandatory if your system has memory. A situation where fresh information must beat the remembered value ("tape 60"), and a situation where memory must not change the verdict (a question is still a question).

How many questions? The metric isn't a count — it's your accident list from Step 1. At least one question per accident type, built to cause exactly that accident. From my list:

inquiry mistaken for an order → goods nobody ordered ship   → "What are the specs on the 250 shipping box?"
change/cancel mistaken for a new order → ships twice        → "I ordered 5 boxes — please send only 3"
unit misread → quantity ships at 50x                        → "250 5"

Five accidents on your list means five questions minimum. Mine grew to 29 because I kept adding variants per accident; 10 is plenty to start.

Add a few normal, well-behaved cases at the end — only a few. Models rarely fumble the easy ones, so normal cases mostly catch nothing. The exam's job is not to watch the model succeed. It's to find where the accidents are.

Step 4 · Write the answer key — then doubt it

For each question, write the expected answer. And here comes the lesson of the first post:

Your answer key will be wrong. I was wrong three times in the key and twice in the grader.

The correct answer to an ambiguous question is "needs confirmation" — not a plausible-looking verdict

When the model answers differently from your key, re-read the data before docking points. The model may be right. But if your fix turns a "needs confirmation" into a confident verdict, be suspicious of the fix first

For every question, record in advance: "does the reference data alone pin the answer down to exactly one?" When the model and the key later disagree, that flag is the referee

The third rule by example — "tape 60, 2 boxes": the catalog has a 60mm, so the data pins the answer. If the model answers differently here, the model is wrong; the key stands. "Clear tape, 2 boxes": no width given, the data cannot pin it. The correct answer was "needs confirmation" from the start — if I wrote anything else in the key, the key is what's broken.

A reader of the first post talked me into this flag. Bonus effect: recording it forces you to open the reference data for every question while you're still writing the exam — which catches answer-key mistakes at authoring time.

Step 5 · Grade, save, decide

Count results by grade, not by score. And save the model's raw answer sheets to files — grading criteria keep changing, and with the raw sheets you can re-grade without calling the model again.

Everything passed? Then write the three-line ledger. It's you being honest with yourself about what the exam guarantees and what it doesn't.

VERIFIED       one question per accident type, and FATAL held at zero
NOT VERIFIED   production data has never been through it — I invented every sentence myself
GUARDED        on anything uncertain it must not confirm — it hands off to a human

What justifies shipping is not line one. It's lines one and three combined. The exam score covers only what I could imagine (line two is that confession), and the sentences I couldn't imagine land on line three's guard. Without line three, an exam score is just a number — the first unimagined sentence causes the accident.

The whole thing on one card

Write down the accidents that can't be undone — that's your truck

Build the grade table — only the first line is yours to define

Plant the four traps — confusable pairs, plausible non-targets, mid-message reversals, after-learning

Write the answer key, then doubt the answer key

Grade by severity, save raw sheets, write the three-line ledger

The exam you may steal. The answer key you have to write yourself.

And that answer key will be wrong. Catching it is what the exam is for.

P.S. The test runner and grader that run exactly this structure are public → github.com/ramses203/llm-test-harness