我用他的代码测量他的 App：审计互证的工程复盘

This is part five of a series about pointing an append-only audit log at things that count tokens. Parts one through four found accounting defects in other people's trackers and one expensive hole in my own routing. This one is about what happens after the report lands, because this time the audit ran in both directions: the maintainer shipped the fix in three days, and his closing comment contained two findings aimed back at my report. One of them caught a claim I had published without its scope. My re-measurement then caught the mechanism he had guessed for it.

The target is Clawdmeter, a desktop app that shows your live Claude Code usage with a pixel mascot. weltern announced the cross-platform release on r/ClaudeAI; I pointed my harness at its token path the same day and filed #21: every transcript-derived number in the app read about 2.34× high.

Same class as part three, so one paragraph and no re-litigation. Claude Code writes one assistant message as several JSONL records, one per content block, and each record repeats the same message.id with the same usage object. src/transcript.py summed per record in three places: the token counts beside the 5h/7d bars, everything the Stats page prices, and the per-session totals. Driving those three functions over a synthetic corpus with a known-exact manifest: 2.34×, 2.34×, 2.37×. Both record-level paths emitted 1,249 events for 540 messages. The percentage bars were never affected, since those come from rate-limit headers, and the overage figure comes from the OAuth endpoint. The report said so, because a fair report names what is not broken.

Running his code instead of mine

The part of the method worth stealing. Nothing in the report reproduces Clawdmeter's logic, models it, or reimplements it. The harness imports the vendor's own transcript.py and calls his functions. The app is a Qt desktop program, and the token path doesn't need Qt, so the whole trick is a twelve-line stub:

pkg = types.ModuleType("PySide6")
qtcore = types.ModuleType("PySide6.QtCore")
qtcore.QObject = ...   # three no-op classes
sys.modules["PySide6"] = pkg
sys.modules["PySide6.QtCore"] = qtcore

sys.path.insert(0, str(clawdmeter_clone / "src"))
import transcript as T          # his module, unmodified
T._file_token_events(path)      # his functions, his numbers

Why bother, when reading the code already showed the defect? Because of the dispute it removes. A reimplementation invites "your model of my code is wrong", and that argument can run for days. Numbers that came out of the vendor's own functions leave exactly one question open, whether the input corpus is fair, and a synthetic corpus with a by-construction manifest answers that in the same breath. The report gets to say "your functions produce 2,592,168 where the manifest says 1,108,697" instead of "I believe your code over-counts".

It also makes reproduction nearly free: git clone, one script, about a minute, a fake $HOME, no PySide6 install, nothing real touched.

Filed August 5. Confirmed August 6: "Thank you again for this report, I've confirmed the bug and will be patching it in an upcoming hotfix release." Shipped August 8 as v3.0.1, with a release note that opens the way I suggested it should, because a correct fix that halves every visible number looks exactly like a regression: "Your token and value numbers will drop by roughly 2.5×. That is the fix, not a regression."

One detail I want to be flat about: he never ran my harness. He reproduced independently, on his own corpus, with his own tooling. That is not the fixture failing at its job. The fixture's job is to make the claim cheap to check and expensive to dispute; his choice to verify it his own way instead is a stronger confirmation than an exit code, and the diagnosis still arrived pre-named, which is what made three days possible. A reproduction script is an offer, not a toll booth.

The fix report audited me back

His closing comment is the reason this post exists. It opens with "Your report was accurate and the diagnosis was right", and then does something I had not seen a maintainer do in five repos: it treats the bug report itself as a measurable claim, measures it, and reports back where it breaks. "Two things turned up that are worth passing back, since you audit other trackers."

His corpus came out worse than mine: 44,196 usage-bearing records across 18,585 distinct message ids, 73.0% on more than one record, work tokens 2.575× high. And then the two findings.

First: the repeated usage objects are not always identical. My report said that on my corpus, 100.0% of duplicated messages carried byte-identical usage on every record. On his corpus, 3,799 groups differ, and in every one output_tokens is a running total: early records carry a partial count, the last carries the final figure. His example message reads 5, 5, 5, 328 across four records. That makes the choice of which record to keep load-bearing. A per-bucket max is correct; keep-first would have reported that message as 5 output tokens instead of 328.

Second: collapsing per file is not enough. 1,094 of his message ids appear in more than one transcript file, because resuming a session replays its records verbatim into a new file. Deduping within each file still left 1.095× inflation on his tree. The collapse has to live in the account-wide aggregator, and a replayed copy can carry an all-zero usage block where the original carries the real numbers, so a record-level dedup placed in front of the message collapse can keep the wrong copy. Folding every record into its message under a per-bucket max sidesteps the whole trap.

He also turned my one-line invariant, usage events should equal distinct message ids, into a regression test, added six more, and ran nine mutations of the fix against them. Nine of nine caught. That last number is a stronger claim than any test count, and it is the part I would not have thought to ask for.

The split is a field, not a version

Finding one corrects something I wrote, so I owed it a measurement rather than a thank-you. My "100.0% byte-identical" was true and under-scoped: it was true of the main conversation path, and I did not say so. On my corpus, regrouped:

Not one main-path group differs, across 63 days and 1,521 transcripts. Every differing group has a sidechain record in it. The discriminator is isSidechain, not the Claude Code version he suspected — and the reason I can say that with some force is that the version hypothesis makes a testable prediction and the field hypothesis makes a different one. If the shape tracked the writer version, main-path groups would differ on some machines. They never do here. His Linux and macOS corpora "still showing the byte-identical shape" is what a main-path-heavy tree looks like; an agent-heavy tree shows his 3,799.

His running-total observation, meanwhile, holds without exception on my side: 12,709 differing groups, 100% non-decreasing, 100% with the last record carrying the max. He was right about the shape and wrong about the cause. I was right about my corpus and wrong to publish the number without its denominator. Neither error survived a second corpus, which is the whole argument for sending measurements instead of opinions.

The artifact I nearly handed him

One warning went into my reply because it is the check anyone would write next, and it lies. Group repeated records by message.id alone and my corpus reports 560 non-monotonic groups, output counts that go down mid-message. Every one is fake: records from unrelated files spliced into one sequence by the sort. Group per (file, message.id) and all 560 vanish. I know the trap is real because I fell into it on a different tracker and reported the bad number before catching it.

Two more numbers from the re-measurement, since they size his findings on an agent-heavy tree. Keep-first against per-bucket max at corpus scale: 20,740,608 output tokens against 38,533,663. Keep-first loses 46.2% of output, so max is not a tiebreak, it is most of the number. And his all-zero replay trap does not reproduce here at all, zero groups in 1,521 transcripts, which means you cannot harvest a test fixture for it from a tree like mine. His synthetic fixture for that case is load-bearing. The script is duplicate_usage_shape.py, read-only, stdlib only, and it prints the artifact count next to the real one so the next person doesn't rediscover the splice.

Red, green, and an API change that proves the point

Yesterday I re-ran the harness against both tags. On v3.0.0 it is red the way the report said: 2.338×, 2.338×, 2.373× on the three call sites, exit 1. On v3.0.1 all three land exact, 1,108,697 and 732,191 to the digit, 540 rows for 540 messages, exit 0.

The check needed one adaptation, and the adaptation is itself the verification. _file_token_events no longer returns something you can sum: it now emits per-record events carrying both a record key and a message key, uncollapsed, with a docstring that says the collapse deliberately belongs to the account-wide caller. The fix is not a patch over the three sites; it is the architecture his second finding argued for, folded under the per-bucket max his first finding argued for. The code now contains the whole conversation.

Counterweights I owe you

My 100.0% shipped without its scope, and the fixture didn't stop it. The harness was green, the totals were exact, and the prose still over-claimed. A fixture pins code; nothing pins your sentences except someone else's corpus.

He never ran the harness, so this series' fastest fix owes nothing to the reproduction being runnable. What earned the three days was the diagnosis being named and the numbers being exact. I keep shipping the fixture anyway, but I should be honest that its value here was rhetorical, not operational.

The isSidechain split is established on one corpus: mine. His 3,799 differing groups were not regrouped under the split, so "every differing group has a sidechain record in it" is a claim about my tree that his data could still complicate.

I do not know why Claude Code writes sidechain usage as a running total and main-path usage finished. The field predicts the shape; the mechanism behind the field is unmeasured.

Part three argued response time tracks queue depth, and a solo maintainer answering in a day fits that line suspiciously well. One data point cannot separate "small queue" from "this particular maintainer", and the 30k-star PR from part three is still open, which proves the thesis only if you already believe it.

His version hypothesis was reasonable on his data. It took a second corpus to kill it, exactly as it took his corpus to kill my 100.0%. One corpus per claimant is how both of us over-generalized, and there is no reason to believe I have stopped.

What I would generalize

Run their code, not your model of it. A GUI is not a reason to reimplement a parser; a twelve-line stub deletes the entire "you misread my code" branch of the argument, and what is left is arithmetic.

A fix report is a dataset. The closing comment on a bug you filed can contain more measurement than the bug did. Read it the way you read a transcript, and answer it with numbers, because the person who just fixed your finding is the best-calibrated reviewer your method will ever get.

Publish invariants with their denominator. "100.0%" was true of a population I forgot to name. The correction cost a table; leaving it unscoped would have cost every future reader who audits an agent-heavy tree.

When two corpora disagree, look for a field before you blame a version. A version split and a field split predict different shapes. Ask what each hypothesis forbids, then check for the forbidden thing. Zero differing main-path groups is the kind of zero that settles arguments.

Collapse under per-bucket max. It is correct when duplicates are identical, correct when they are running totals, and immune to the all-zero replay copy. Keep-first was quietly losing 46.2% of output on my tree.

Group per (file, id), or the sort will manufacture your finding for you. My 560 phantom groups came from splicing files together before grouping. Print the artifact count next to the real one so the check that lies gets caught by the check that doesn't.

All five parts run on TraceGuard's routing_audit, an append-only, message.id-keyed ingest of Claude Code transcripts into SQLite. Apache-2.0, pip install traceguard. The Clawdmeter harness and the shape-measurement script live in usage-tracker-audit/clawdmeter-dedup; the corpus generator builds a fake $HOME and touches nothing real.

The series scoreboard, since part four's count is stale: seven shipped fixes across four repos — three in splitrail, two in tokscale (v4.9.0 and v4.11.0), Clawdmeter v3.0.1, and viberank's high-water-mark fix, deployed. The eighth is still an open PR behind a 135-deep queue, which part three already explained.

The thing this part adds to the series is not a seventh notch. It is that the method survived contact with a maintainer who measured back, and got better for it: per-bucket max and per-(file, id) grouping are now in my collapse rules because his corpus broke my claim, and the version hypothesis died because mine broke his. Audit is not a thing you do to a codebase. Done right, it is a thing two corpora do to each other.