Eval 分数骗人：AI 模型金丝雀指标应看 token 消耗/截断率/工具调用

An eval score is a number produced by reading the final text of a response. Most of what a prompt edit changes is not in the final text. It is in how many tokens were spent getting there, how the response terminated, which tools fired, and what fraction of requests never produced usable output at all.

Why the eval score is blind here

Two structural facts make the eval score a poor canary metric on its own, and neither is a criticism of the eval set.

The first is that an eval set is a fixed collection of cases somebody wrote down, and production traffic is not. Every case in the set is a failure mode that was already known when it was added. A canary sees the live input distribution — including the long tail of inputs nobody thought to encode — and that is the only place a novel regression can show up. The eval set answers "did I break something I already knew about"; the canary answers "did I break something".

The second is that the scorer's input is the assistant message and nothing else. A prompt change that adds four hundred tokens of instructions, doubles time to first token, and pushes ten percent of responses into truncation can score identically on every case in the set, because on the cases that still fit, the answer is still right. None of the three consequences is visible to a function whose argument is a string.

So the canary panel is a different instrument from the eval suite. It watches the fields around the text.

Token counts, which are the cost

Compare usage.prompt_tokens and usage.completion_tokens between arms, as distributions rather than means. Prompt tokens move when you edit the system prompt, obviously, but also when you change anything that alters how much retrieved context gets packed in. Completion tokens move for a far less obvious reason: an instruction that nudges the model toward explaining itself increases output length across the board, and output is the expensive half.

The arithmetic is worth doing before the canary, so you know what a given delta is worth. Assume a service handling one million requests a month, and assume published list prices of $3 per million input tokens and $15 per million output tokens — both figures are inputs to the calculation and both change, so substitute your own:

system prompt grows by 400 tokens:
  400 * 1,000,000 = 400,000,000 input tokens
  400M / 1M * $3  = $1,200 per month

mean completion grows by 60 tokens:
  60 * 1,000,000  = 60,000,000 output tokens
  60M / 1M * $15  = $900 per month

A prompt edit that reads as a tidy-up and costs two thousand dollars a month is a normal outcome, and it is invisible to every quality metric you have. Watch the median and the 95th percentile of both counts, not the mean — a handful of runaway generations will drag a mean past anything a real user experienced.

Prompt caching interacts with this. If the edit changes the shared prefix, every cached read becomes a cache write until the new prefix warms, so the first minutes of a canary can look far more expensive than the steady state. Compare after the warm-up, and see prompt cache savings for how the prefix is keyed.

Finish reasons and the truncation cliff

The distribution of the finish reason field is the most under-monitored signal in this list, and it is one enum. On OpenAI-compatible APIs the values are stop (the model emitted an end-of-turn), length (it hit max_tokens or the context limit), tool_calls (it stopped to call a tool), content_filter (the provider blocked it), and the deprecated function_call. Anthropic's field is stop_reason with values including end_turn, max_tokens, stop_sequence, tool_use and refusal. Which set you get depends on the endpoint, and any normalising layer in between maps one onto the other.

The reason to alert on this is that truncation is a cliff, not a slope. If your max_tokens is 1,024 and the old prompt produced a mean of 700 output tokens, a change that adds 60 tokens on average does not cost you 6% of a response — it costs you the entire tail that crosses the ceiling. The rate of finish_reason: "length" going from 1% to 8% is the single clearest evidence that a prompt got more verbose, and the resulting output is not merely longer, it is cut off mid-sentence and frequently unparseable if you asked for JSON.

content_filter deserves its own line on the panel. A prompt edit that changes how the model frames a sensitive topic can push requests over a provider's moderation threshold, and those responses often arrive with an empty content field rather than an error, so nothing in your code throws.

Tool calls and control flow

If the model has tools, the prompt is the program that decides when they run, and an edit is a change to control flow. Four things to compare per arm:

Fraction of turns that call any tool. A prompt that becomes more confident answers directly where it used to look something up, which reads as a latency improvement and a factual regression.

Distribution over tool names. Two tools with similar descriptions swap places easily; see tool description design.

Tool calls per completed task. The metric that actually catches a loop. One extra round trip per task is one extra full request, so it roughly doubles the cost of the affected conversations while every per-request metric stays flat.

Argument-validation failure rate. Malformed tool arguments are usually caught and retried somewhere, which makes them invisible unless you count them explicitly.

Turn count per resolved conversation belongs here too, and it is the one that maps most directly onto user experience. Everything can look identical per request while the number of requests a user needs to get an answer goes from three to five.

Refusals, empties and language drift

Three cheap counters that catch a disproportionate share of real incidents. Empty or whitespace-only content is a count, not a judgement, and a rise in it is always a bug. Schema-validation failure rate, if you are asking for structured output, is likewise a hard number — and note it can move without the prompt's JSON instructions changing at all, because a longer prompt shifts what the model attends to.

Refusal rate is the awkward one, because there is no field for it on most endpoints and detecting a refusal means classifying prose. Do not try to do that with an exact-match list of apology phrases; it fails the moment the user writes in another language, and it is the same mistake as asserting on model prose in a test. A workable proxy is a combination of signals that are actually fields: very short output, no tool call, a finish reason of stop, and — where the provider offers it, as Anthropic's refusal stop reason does — the explicit flag. Reserve the classifier for the residual.

Output language is worth a counter on any multilingual product, because instruction-following on language is fragile and a prompt edit is exactly the thing that breaks it. The library has this failure documented separately in output language ignores the instruction.

Two rules keep this from becoming a wall of graphs nobody reads.

First, split the signals into guardrails and observations before the canary starts. A guardrail has a threshold and an action attached — error rate, schema failure rate, p95 latency, cost per request. An observation is looked at by a human and does not fire anything. If everything is a guardrail you will get a false rollback a shift; if nothing is, you have a dashboard rather than a canary. The arithmetic behind that false-alarm rate is worked through in rolling a canary back automatically.

Second, compare distributions, not averages, and compare them over the same wall-clock window. Canary and baseline traffic differ in composition if assignment is sticky and your user base is not homogeneous, so a difference in mean output length can be a difference in who is in each arm. Segmenting by one or two obvious axes — locale, plan tier, first-turn versus follow-up — catches most of that, and any signal that only moves in one segment is telling you something the aggregate cannot.

Most of these signals are fields on the response rather than things you have to derive, but they arrive under different names per provider and some of them only exist on one. If you route through a gateway, the normalising layer is the natural place to record token counts, finish reason and tool names per request with the prompt version attached as a tag — Multigrid keeps those per request, which is what makes an arm-versus-arm comparison a query rather than a project.