8.0
热点
AI SCORE
编程提效2026-08-07 21:29
JSON提示词Runner bug验证：200次重跑结论依然显著
dev.to · AI#AI评测#提示词工程#开源
Editor brief · 编辑速览
开发者发现实验Runner的bug并修正后，用200次云端重跑验证了原有效应仍然成立，附完整代码和原始数据。
以下是对这篇英文文章的完整中文翻译：
The corrected Qwen estimate below remains valid for this frozen model and workload, but it is not a portable optimizer result. A fresh Llama replication reversed direction. After an external review found a numeric-language mismatch in that Llama control, I ran one preregistered exact canonical correction. The canonical string control still scored 92/150 versus 82/150 for the immutable integer treatment, with a paired interval of [-12.7, -0.7] points. See the complete correction decision.
我有一个自己想要相信的结果。
将面向模型的 JSON 答案从带符号的数字字符串改为原生整数，然后确定性将其转换回调用者的字符串契约，在 Qwen2.5-7B 上似乎恢复了 14.3 个百分点的约束准确率。
然后一次审计发现了实验运行器中的一个真正问题。
历史 Outlines 路径可能在 Qwen 聊天模板已经应用后才收到 prompt，而 Outlines Transformers 适配器可能再次应用该模板。接受的 artifacts 保持了完整性和来源有效性，但对照组和处理组的证据无法再支持架构决策，除非进行一次修正后的配对运行。
因此我冻结了一个新协议，修正了比较的双方，跑了 200 次新的云端生成，下载了每一行数据，并独立验证了数据束。
效果在方向和实际规模上都存活了下来：
契约有效的正确率从 18/49 提升到 24/49，配对增益为 12.2 个百分点，而最终外部模式有效性保持在 100%。
但不确定性很重要。精确的自举区间是 [0.0, 26.5] 个百分点，精确 McNenar p = 0.146。这是下一阶段的有力证据，而非通用解决方案的证明。
完整的代码、协议、原始行、哈希、追踪、验证报告和决策已在仓库中公开。
Vaibhav701161 / constrained-decoding-lab
A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.
Results | Alignment result | Corrected replication | Paired evidence | Study design | Reproduction | Evidence | Public Kaggle artifacts | Progress presentation | Technical articles | Limitations
Progress presentation
Watch this presentation for an overview of the project's progress, key findings and results to date.
Constrained decoding solved the formatting problem, but it did not preserve all of the model's recoverable mathematical accuracy. On Qwen2.5-7B, prompt-only JSON achieved 79.6% recoverable accuracy and 0% schema compliance. Outlines and XGrammar each achieved 61.2% recoverable accuracy and 100% schema compliance. The paired semantic effect was -18.4 percentage points for both backends (exact McNemar p = 0.003906).
这是对约束解码的受控研究，artifact 验证了 JSON prompting、语法约束解码和输出字段顺序如何影响数学准确率和模式合规性。
进度展示
观看此演示文稿以了解项目进展、关键发现和迄今结果的概述。
约束解码解决了格式问题，但没有保留模型全部可恢复的数学准确率。在 Qwen2.5-7B 上，仅 prompt 的 JSON 实现了 79.6% 的可恢复准确率和 0% 的模式合规性。Outlines 和 XGrammar 都实现了 61.2% 的可恢复准确率和 100% 的模式合规性。配对语义效应为两个后端均为 -18.4 个百分点（精确 McNemar p = 0.003906）。
这不是说约束解码普遍有害的声明。这是一个受控……
Suppose an API requires this external object:
{
"reasoning": "...",
"answer": "18000"
}
The answer must be a canonical signed-integer string. The model, however, may be better aligned with a native JSON integer:
{
"reasoning": "...",
"answer": 18000
}
The proposed compiler does not weaken the caller's contract. It creates a separate model-facing contract, generates once under that grammar, applies an exact inverse transformation, and validates the final object against the original schema.
假设一个 API 要求这个外部对象：
{
"reasoning": "...",
"answer": "18000"
}
答案必须是规范的有符号整数字符串。然而，模型可能与原生 JSON 整数对齐得更好：
{
"reasoning": "...",
"answer": 18000
}
提议的编译器不会削弱调用者的契约。它创建一个独立的面向模型的契约，在该语法下生成一次，应用精确的逆转换，并根据原始模式验证最终对象。
external schema
|
v
contract IR and applicability checks
|
v
model-facing schema plus replayable transform plan
|
v
one constrained generation
|
v
deterministic reverse transduction
|
v
original external-schema validation
For this experiment, the only semantic transform is:
external canonical signed-integer string
-> internal JSON integer
-> arbitrary-precision base-10 stringification
-> external validation
There is no second model call, sign repair, rounding, regex guessing, or fallback coercion. A boolean is rejected even though Python treats it as an integer subtype. Lexical values such as "+26", "00026", "26.0", and "2.6e1" are not guessed into the accepted language. Unsupported cases fail closed.
对于这个实验，唯一的语义转换是：
外部规范有符号整数字符串
-> 内部 JSON 整数
-> 任意精度 base-10 字符串化
-> 外部验证
没有第二次模型调用、符号修复、舍入、正则猜测或回退强制转换。布尔值会被拒绝，即使 Python 将其视为整数子类型。诸如 "+26"、"00026"、"26.0" 和 "2.6e1" 这样的词法值不会被猜入接受的语言。不支持的用例会闭式失败。
The audit identified three separate defects or inconsistencies.
For string inputs, Outlines owns chat templating through its Transformers adapter. The runner had passed an already formatted chat string into that layer. A Qwen prompt could therefore be wrapped twice.
The corrected path passes the raw project prompt to Outlines. Direct Transformers and XGrammar generation use the same shared formatting helper exactly once. Tests compare effective token IDs and prove that the nested form is different.
Direct generation and XGrammar counted generated tensor IDs. Outlines counted the visible text after decoding. A backend-only stop token could make those metrics incomparable.
The corrected metric retokenizes visible generated content without special tokens for every backend. This is a measurement fix, not a model behavior change.
Permissive JSON whitespace loops add legal token paths and can create avoidable stalls. The corrected runner pins compact separators in XGrammar and an empty whitespace pattern in Outlines.
Real pinned-backend tests verify that compact valid JSON is accepted and a long whitespace prefix is rejected.
审计发现了三个独立的缺陷或不一致。
1. 双重聊天模板风险
对于字符串输入，Outlines 通过其 Transformers 适配器拥有聊天模板化。运行器已将一个已经格式化好的聊天字符串传递到该层。因此 Qwen prompt 可能被包装两次。
修正后的路径将原始项目 prompt 传递给 Outlines。Direct Transformers 和 XGrammar 生成都恰好一次使用相同的共享格式化辅助函数。测试比较有效的 token ID，并证明嵌套形式是不同的。
2. 后端相关的生成 token 计数
Direct generation 和 XGrammar 计算生成的 tensor ID。Outlines 计算解码后的可见文本。后端专用的 stop token 可能使这些指标不可比。
修正后的指标对每个后端的可见生成内容进行重新 token 化，不含特殊 token。这是一个测量修复，而不是模型行为变更。
3. 非规范空白策略
宽松的 JSON 空白循环会增加合法 token 路径并可能造成可避免的停顿。修正后的运行器在 XGrammar 中固定紧凑分隔符，在 Outlines 中使用空空白模式。
真正的固定后端测试验证紧凑的有效 JSON 被接受，长空白前缀被拒绝。
这些修正改变了实验系统。将修正后的处理组与历史对照组进行比较会混用运行器版本，因此我重新运行了双方表示。
The protocol was frozen before the cloud run
The corrected question was narrow:
Under greedy FP32 decoding on Qwen2.5-7B-Instruct, does a model-facing JSON integer improve contract-valid correctness over a signed numeric string when both sides use the same corrected source and the final object must satisfy the unchanged external contract?
The frozen design used:
修正的问题是狭义的：
在 Qwen2.5-7B-Instruct 的贪婪 FP32 解码下，当双方使用相同的修正源且最终对象必须满足不变的外部契约时，面向模型的 JSON 整数是否比带符号数字字符串提高契约有效的正确率？
冻结的设计使用：
The job first ran the same five deterministic items in all four conditions. Expansion to 50 was allowed only if row identity, prompt equivalence, schema validity, run signatures, nonblank output, and cap status passed. The canary reused the same result files, so it did not create duplicate research rows.
任务首先在所有四种条件下运行相同的五个确定性项。只有在行标识、prompt 等价性、模式有效性、运行签名、非空输出和容量状态都通过后才扩展到 50 个。canary 重用了相同的结果文件，因此没有创建重复的研究行。
The Green rule required at least five points of recovery, 100% final external validity, paired wins exceeding losses, no systematic replacement failure, and evidence consistent with a representation-boundary mechanism.
Green 规则要求至少五点的恢复、100% 的最终外部有效性、配对胜利超过失败、无系统性替换失败，以及与表示边界机制一致的证据。
Importantly, semantic correctness was not part of the canary. Stopping or expanding based on the first five answers would have biased the confirmation.
重要的是，语义正确性不是 canary 的一部分。基于前五个答案停止或扩展会使确认产生偏差。
The cloud job produced all 200 expected rows:
云端任务产生了所有 200 个预期的行：
2 backends x 2 representations x 50 items = 200 generations
The clean paired analysis retained 49 items under the audit policy.
干净的配对分析在审计策略下保留了 49 个项。
net improvement: 6 items
accuracy delta: +12.2 percentage points
exact bootstrap 95% interval: [0.0, 26.5] points
treatment-only repairs: 9
control-only regressions: 3
both correct: 15
both wrong: 22
exact two-sided McNemar p: 0.145996
The intervention repaired three times as many items as it broke. It also preserved the external contract for every clean treatment output.
干预修复的项目是破坏的三倍。它还为每个干净的 treatment 输出保留了外部契约。
The point estimate clears the preregistered continuation threshold. It does not clear a conventional p < 0.05 threshold, and the interval touches zero. The honest conclusion is not "the compiler works universally." It is "the corrected evidence is strong enough to justify one bounded external-validity test."
点估计超过了预先注册的继续阈值。它没有超过传统的 p < 0.05 阈值，区间接触零。诚实的结论不是"编译器普遍有效"，而是"修正后的证据足以证明一次有界的外部有效性测试是合理的"。
Every item remains inspectable
Aggregate accuracy can hide cherry-picked wins or a concentrated replacement failure. The item map below shows both outcomes for every clean-analysis item in the frozen dataset order.
每个项目仍然可以检查
总体准确率可能隐藏精心挑选的胜利或集中的替换失败。下面的项目图显示了冻结数据集顺序中每个干净分析项目的两种结果。
The nine repaired items were:
九个修复的项目是：
gsm8k_test_98
gsm8k_test_416
gsm8k_test_712
gsm8k_test_739
gsm8k_test_765
gsm8k_test_1205
gsm8k_test_1232
gsm8k_test_1251
gsm8k_test_1272
The three regressions were:
三个退化的项目是：
gsm8k_test_506
gsm8k_test_601
gsm8k_test_996
The regressions remain in the primary denominator. They are not treated as outliers.
退化项目保留在主要分母中。它们不被视为异常值。
The two backends became an implementation agreement check
Outlines and XGrammar emitted byte-identical raw output on all 50 signed-string items and all 50 integer items.
Outlines 和 XGrammar 在所有 50 个带符号字符串项和所有 50 个整数项上发出字节相同的原始输出。
That is strong evidence that both corrected integrations realized the same canonical token path in this run. It also changes the statistical interpretation.
这是有力的证据，表明两个修正的集成在这个运行中实现了相同的规范 token 路径。这也改变了统计解释。
These are not two independent semantic replications. They are two implementations agreeing on one effective 49-item paired experiment. Drawing separate effect bars and calling the result replicated twice would overstate the evidence.
这不是两个独立的语义复制。它们是两个实现对一个有效的 49 项配对实验的一致意见。绘制单独的效果条并称结果被复制两次会夸大证据。
The observed semantic difference is associated with the model-facing representation, not backend identity, under these matched settings.
在这些匹配设置下，观察到的语义差异与面向模型的表示相关，而不是后端身份。
The mechanism is subtler than "integers forbid negatives"
Both negative signed-string outputs were repaired by the integer treatment:
两个负的带符号字符串输出都被整数处理修复了：
gsm8k_test_712: -14 -> 14
gsm8k_test_1205: -1 -> 4
But JSON integers allow negative values. The integer grammar did not prohibit the minus token.
但 JSON 整数允许负值。整数语法没有禁止减号 token。
The XGrammar boundary traces show that the integer grammar removes quote and structural-space continuations at the answer boundary while leaving minus legal. The model therefore traverses a different legal token path. That path can affect the first answer token and, because the answer follows reasoning, earlier generation as well.
XGrammar 边界追踪显示整数语法在答案边界处移除引号和结构空格延续，同时保留减号合法。因此模型遍历不同的合法 token 路径。该路径可以影响第一个答案 token，并且由于答案跟在推理之后，也会影响更早的生成。
This is consistent with a representation-boundary mechanism. It is not proof that every repair came from one local mask decision.
这与表示边界机制一致。这不是每个修复都来自一个局部 mask 决策的证明。
A correct benchmark answer can still have unfaithful reasoning
One repaired item is an important counterexample.
For gsm8k_test_712, the integer output's reasoning still computes -14, but its final answer is 14. The benchmark score is correct. The generated explanation is internally inconsistent.
一个修复的项目是一个重要的反例。
对于 gsm8k_test_712，整数输出的推理仍然计算 -14，但其最终答案是 14。基准分数是正确的。生成的解释内部不一致。
So the evidence supports improved final-answer fidelity. It does not establish improved reasoning faithfulness.
因此证据支持提高的最终答案保真度。它没有建立改进的推理忠实度。
This distinction matters for any production system that exposes explanations, executes intermediate values, or relies on chain-of-thought consistency.
这个区别对于任何公开解释、执行中间值或依赖思维链一致性的生产系统都很重要。
The artifact validator does not trust the remote summary
After the Kaggle run completed, a separate local validator checked:
200/200 expected rows;
50 unique items in every condition;
source, runner, manifest, and result hashes;
run signatures and frozen item order;
prompt equivalence within each representation;
byte equality across the two backends;
generation errors, cap hits, and blank outputs;
internal and external schema validity;
the three preregistered boundary traces.
在 Kaggle 运行完成后，一个独立的本地验证器检查了：
200/200 预期行；
每种条件下 50 个唯一项；
source、runner、manifest 和 result 哈希；
运行签名和冻结项顺序；
每个表示内的 prompt 等价性；
两个后端之间的字节相等性；
生成错误、容量命中和空白输出；
内部和外部模式有效性；
三个预先注册的边界追踪。
It reported zero failures and zero warnings. The run contained zero generation errors, zero cap hits, zero blank outputs, and zero schema failures.
它报告了零失败和零警告。运行包含零生成错误、零容量命中、零空白输出和零模式失败。
The summary was then regenerated locally. The paired bootstrap interval is computed exactly from the empirical paired distribution by finite convolution, not estimated from a random 10,000-resample loop.
摘要随后在本地重新生成。配对自举区间是通过有限卷积从经验配对分布精确计算的，而不是从随机 10,000 次重采样循环估计的。
What exists beyond the experiment
The repository now contains a conservative compiler prototype rather than only a one-off transducer:
a canonical, hashable contract IR;
explicit unsupported-construct records;
deterministic, serializable alignment plans;
backend capability and whitespace requirements;
integer-string, key-alias, field-order, scratch-field, and whitespace transforms;
exact reverse transduction;
final validation against the original schema;
typed fail-closed refusals;
property and adversarial tests.
仓库现在包含一个保守的编译器原型，而不仅仅是一次性换能器：
一个规范的、可哈希的 contract IR；
明确的不支持构造记录；
确定性、可序列化的对齐计划；
后端能力和空白要求；
整数-字符串、键别名、字段顺序、暂存字段和空白转换；
精确的逆转换；
根据原始模式的最终验证；
类型化的闭式拒绝；
属性和对抗性测试。
The implementation refuses references and schema features outside its initial supported subset. That is intentional. A compiler that silently discards contract semantics would be worse than no compiler.
该实现拒绝其初始支持子集之外的引用和模式特性。这是刻意的。一个静默丢弃契约语义的编译器比没有编译器更糟糕。
The current test suite passes 87 tests, including 1,501 integer property cases and real pinned-backend grammar probes. Those tests establish deterministic contract behavior and integration. They do not substitute for external semantic evidence.
当前测试套件通过 87 个测试，包括 1,501 个整数属性用例和真正的固定后端语法探测。这些测试建立了确定性契约行为和集成。它们不能替代外部语义证据。
What changed relative to the historical result
The historical mixed-runner paths reported:
历史混合运行器路径报告：
signed string: 61.2%
internal integer: 75.5%
delta: +14.3 points
The corrected paired paths report:
修正后的配对路径报告：
signed string: 36.7%
internal integer: 49.0%
delta: +12.2 points
The absolute rates changed substantially. The direction and approximate treatment effect survived. Historical and corrected rates must not be pooled because prompt handling and canonical grammar behavior differ between runner versions.
绝对比率发生了实质性变化。方向和近似处理效应存活了下来。历史和修正后的比率不能混合，因为 prompt 处理和规范语法行为在运行器版本之间不同。
This is why preserving old artifacts matters. A correction should create a new evidence layer, not rewrite what happened.
这就是保留旧 artifacts 重要的原因。修正应该创建一个新的证据层，而不是重写发生的事情。
The decision: Green, but scoped
The corrected result satisfies the registered continuation rule:
修正结果满足注册的继续规则：
The point recovery exceeds five percentage points.
Final external validity is 100%.
Paired wins exceed losses.
No systematic replacement failure was observed.
Boundary evidence remains consistent with the representation hypothesis.
点恢复超过五个百分点。
最终外部有效性为 100%。
配对胜利超过失败。
没有观察到系统性替换失败。
边界证据仍然与表示假设一致。
The limitations are equally concrete:
限制同样具体：
only 49 clean paired items;
interval touching zero and p = 0.146;
treatment accuracy only 49.0%;
one model family and one benchmark;
one correct answer with inconsistent reasoning;
no independent semantic replication across backends.
仅 49 个干净的配对项；
区间接触零且 p = 0.146；
处理准确率仅 49.0%；
一个模型家族和一个基准；
一个答案正确但推理不一致的情况；
没有跨后端的独立语义复制。
The next authorized gate is not broad schema expansion. It is either:
下一个授权关卡不是广泛的模式扩展。它是以下之一：
the same paired contract test on a second model family; or
在第二个模型家族上进行相同的配对契约测试；或者
an executable tool-call task where contract-valid correctness has direct practical value.
一个可执行的工具调用任务，其中契约有效的正确率具有直接的实际价值。
If neither preserves a useful positive effect, the project should narrow into a schema-risk linter and measurement system rather than continue claiming a general compiler direction.
如果两者都不能保留有用的正向效应，项目应该缩小为模式风险 linter 和测量系统，而不是继续声称通用编译器方向。
Reproduce the evidence
The corrected evidence lives under:
修正的证据位于：
experiments/corrected-replication/
The main entry points are:
主要入口点是：
python scripts/validate_corrected_replication.py \
--run-dir experiments/corrected-replication/results/qwen2.5-7b-corrected/results/corrected-replication \
--dataset data/gsm8k_50_seed0.jsonl \
--source-root deployment/kaggle/corrected-replication/source-snapshot \
--kernel-source deployment/kaggle/corrected-replication/run_kaggle.py \
--out /tmp/corrected-artifact-validation.json
python scripts/build_corrected_replication_figures.py
python -m pytest -q
Read the exact artifacts:
阅读确切的 artifacts：
Independent validation
I am treating the correction as part of the result, not as an embarrassing footnote. The point of an evidence pipeline is not to protect a claim. It is to make the claim survive contact with its own implementation.
我将修正作为结果的一部分，而不是一个令人尴尬的脚注。证据管道的要点不是保护一个声明，而是让声明在与其自身实现的接触中存活下来。
For further actions, you may consider blocking this person and/or reporting abuse