60 分钟建一套 AI 评审 golden set 测试集

Every team that adds an AI code reviewer gains a new reviewer, but almost nobody adds a test for that reviewer. A golden set of fifteen real review cases, scored against a small rubric, is enough to catch the regressions that quietly break review quality. This workshop builds that harness in sixty minutes using free model tokens and a free server. You leave with a gate you can rerun on every model change.

Recent DEV discussions have made the same point from a different direction: AI promoted every developer to a reviewer, and nobody tested the reviewer. The fix is not a bigger benchmark; it is a small reproducible evaluation that runs on demand. A public benchmark measures a model against generic data, but your review workload lives in your diffs, your style, and your bug history. A project-specific golden set therefore beats a leaderboard for this decision.

准备工作

Python 3.11 或更高版本，仅使用标准库

需要获取一个你想评估的 chat model 的 API key

一个用于定时重跑的免费服务器；MonkeyCode 的免费套餐同时包含 model tokens 和服务器，写稿时附赠 10M token 额度。额度会变动，启动前请在文档中核实最新数字。

MonkeyCode 是开源项目，其免费套餐覆盖本实验需要的两个资源：运行所需的 tokens 和用于定时重跑的服务器。披露：本文是 MonkeyCode 产品推广的一部分。以下 harness 不使用任何 vendor SDK；它调用的是 OpenAI 兼容的 chat 端点。你只需改三个环境变量就能切换 provider。

Exercise 1: 编写 golden set（10 分钟）

A golden set is a small file of real inputs plus the criteria that separate a good answer from a bad one. You do not need model-written expected outputs; you need the judgment rules your team already applies in review. Save this shape as golden_set.json, then grow it to fifteen cases from your last month of real comments.

{
  "task": "code_review_comment",
  "criteria": [
    "catches the real bug, not a style nit",
    "names the exact file and line",
    "explains why the bug matters",
    "proposes a concrete fix",
    "stays under 120 words"
  ],
  "examples": [
    {"id": "case-01", "input": "checkout.py: total = sum(item.price for item in cart)", "context": "The sum ignores item.qty, so multi-item orders are undercharged.", "expected": "Flag that total ignores item.qty before apply_promo runs."}
  ]
}

Include five genuinely tricky cases, because a reviewer that passes easy ones and fails hard ones is exactly the failure mode you want to catch early.

Exercise 2: 构建 reviewer 和 judge（15 分钟）

The harness has three parts. First, call the candidate model with a review prompt built from the case input. Second, grade the produced comment with a short rubric prompt that returns JSON only. Third, write every raw output to disk, because a run you cannot inspect is a run you cannot debug.

import json, os, sys, time, urllib.request

BASE_URL = os.environ.get('LLM_BASE_URL')
MODEL = os.environ.get('LLM_MODEL')
API_KEY = os.environ.get('LLM_API_KEY')
DATA = json.load(open('golden_set.json'))

def call_model(prompt, temperature=0.2):
    body = json.dumps({
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': temperature,
    }).encode()
    req = urllib.request.Request(
        BASE_URL + '/chat/completions', data=body,
        headers={'Authorization': 'Bearer ' + API_KEY,
                 'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)['choices'][0]['message']['content']

def judge(comment, example):
    rubric = '\n'.join(f'{i+1}. {c}' for i, c in enumerate(DATA['criteria']))
    prompt = f'''Score this review against the criteria.
1 point if fully met, 0.5 if partially met, 0 if not met.
Reply with JSON only, keys scores and total.

Criteria:
{rubric}

Review:
{comment}

Target:
{example['expected']}'''
    try:
        return float(json.loads(call_model(prompt, 0.0))['total'])
    except Exception:
        return 0.0

The judge is a second call to the same model in this lab, which is a known bias. You accept that for a short workshop, but you should switch the judge to a stronger model when the budget exists. The try/except converts a malformed judge reply into a zero, which keeps scoring strict instead of silently optimistic.

Exercise 3: 运行完整集合并查看 scorecard（15 分钟）

Run every case, collect per-case scores, and print the mean. The scorecard should show not only the total but also the weakest criteria. A review assistant that always misses the "explains why" criterion needs a prompt fix, not a model swap.

def build_prompt(example):
    src = example['input']
    ctx = example['context']
    return 'Review this diff in ' + src + '. Context: ' + ctx + '. Reply in under 120 words.'

results = []
for ex in DATA['examples']:
    comment = call_model(build_prompt(ex))
    results.append({'id': ex['id'], 'score': judge(comment, ex), 'comment': comment})

n = len(DATA['criteria'])
mean = sum(r['score'] for r in results) / len(results)
json.dump({'ts': time.time(), 'model': MODEL, 'mean': mean, 'results': results},
          open('run.json', 'w'), indent=2)
print(f'model={MODEL} mean={mean:.2f} max={n}')

Typical honest results land between 2.5 and 4.0 on a five-point scale. Do not be surprised by low scores on the tricky cases; the value of the lab is knowing which criteria fail before you ship the reviewer to your team.

Exercise 4: 把 mean 变成 decision gate（10 分钟）

The gate is three lines of logic: read the saved mean, compare it with a threshold, and exit non-zero on failure. In a repository, run it as a CI step that blocks a model upgrade when quality drops. On the free server, schedule the same script with cron if background jobs are allowed.

threshold = float(os.environ.get('SCORE_THRESHOLD', '4.0'))
print('PASS' if mean >= threshold else 'FAIL')
sys.exit(0 if mean >= threshold else 1)

A nightly run is the minimum cadence that makes sense, because model endpoints change behavior without changing version numbers. When the mean drops, the saved run.json tells you whether the model changed or the judge changed.

何时不适用此方案

Golden-set tests complement benchmarks; they do not replace them. Use a full benchmark to choose the model family, then use this lab to decide whether that model works on your actual diffs. The free server's limits around CPU and background jobs are documented on the provider site, and the lab itself needs only a few seconds of compute per run.

用另一个模型重新跑一遍

The whole exercise costs a few thousand tokens per run, which is why a free allowance matters. You can rerun the lab dozens of times per month without thinking about spend, and each rerun produces a comparable scorecard. The harness is provider-agnostic, so the same golden set can rank two models side by side. If you want to try it against the current free tier and server, the MonkeyCode docs are the fastest way to verify today's token limit and sign-up steps.