AI编程助手在给自己的作业打分

Here's a failure mode I keep running into, and I don't think it gets enough attention relative to how expensive it is.

An AI coding agent finishes a task. It tells you what it did. "Ran the test suite, all passing. Updated the config. Cleaned up the temp files."

You have essentially no way to check that, short of reading the entire session transcript line by line. And the transcript is thousands of lines, so nobody does.

The agent is the only witness to its own behavior, and it's also the party with an interest in the summary sounding successful. That's not a claim about intent. It's just a structural problem: the model producing the summary is the same model that produced the work, working from the same context, with the same blind spots.

Where the gap actually opens

Claims that don't match execution. "All tests pass" when the test command errored out and the agent read the error as noise. "Migration applied" when it wrote the file but never ran it. These aren't hallucinations exactly, they're optimistic summaries of ambiguous outcomes.

Secrets in the transcript. An agent runs cat .env once to check something. That key is now sitting in the session log, in plaintext, on disk, indefinitely. Nobody ever goes back and cleans it.

Irreversible actions you didn't clock. In a long session, rm, force pushes, dropped tables, and sent requests scroll past. The summary at the end will not enumerate them. If you need to know exactly what can't be undone, the summary is the wrong artifact.

Instructions quietly dropped. You wrote rules in your CLAUDE.md or your system prompt. Twenty thousand tokens later, adherence has drifted. The agent isn't lying about following them; it doesn't have a reliable view of its own compliance.

Cost with no attribution. You know the session cost money. You don't know which step burned it.

The approach: audit the transcript, not the summary

The transcript is the ground truth. It's a complete record of what actually happened, and unlike the summary it wasn't written to be reassuring. It's just too long for a human to read.

So the fix is unglamorous: parse the transcript, look for specific things, report them. Deterministic checks against a file that already exists on your machine.

I've been building small tools for this, each answering one question:

• Did the agent's claims match what actually executed?
• Did it leak secrets into the transcript?
• What irreversible actions did it take?
• Did it actually follow the rules you gave it?
• Where did the tokens and the money go?

They're Python CLIs, zero dependencies, and they run against a finished session offline. No API calls, no keys, nothing uploaded. That last part matters: a tool built to find secrets in your transcript would be an absurd thing to ship if it had to send that transcript anywhere.

They're open source at github.com/0xelitesystem.

Why deterministic beats "ask another model"

The obvious alternative is to have a second model read the transcript and evaluate the first one. I don't think that works well for this, for two reasons.

You've added a second thing that can be wrong, and you now need a way to check it. That regress doesn't terminate anywhere useful.

And the questions worth asking are mostly not judgment calls. "Did a command that force-pushes appear in this session" is a string match, not an opinion. "Does this transcript contain something shaped like an API key" is a pattern match. Deterministic checks give you the same answer every time, run offline, cost nothing, and are auditable themselves. Use a model for the parts that genuinely need judgment, and don't use one for the parts that don't.

If you take one thing from this

Whatever tooling you use, adopt the habit: after any unattended agent session that touched anything real, check the transcript rather than the summary. Even manually. Even just grepping for destructive commands and for anything that looks like a credential.

The general principle is one I keep coming back to across everything I build. Prefer the system that lets you verify it over the system that asks you to trust it. It's usually less impressive and it's almost always what you want at 2am when something has gone wrong.