8.0
热点
AI SCORE
技术实践2026-09-11 23:13
部署 AI 推理网关前，先查队年龄龄而非面板利用率
dev.to · AI#AI推理#SRE#运维
Editor brief · 编辑速览
当 AI worker 空闲容量看似充裕时，盲目重试仍可能超时。正确做法是先比对 queue_age_ms 与 deadline_slack_ms，在队年龄超过剩余时限时应拒绝请求，而非依赖利用率面板的绿色指示。
凌晨 02:14，你收到了推理网关的告警。免费 Worker 的 CPU 只有 12%。队列年龄已经达到了 47 秒，而期限是 30 秒。
利用率面板看起来还有充足的余量。但队列年龄说明完成 SLO 已经无法达成。你仍然又重试了 3 次。
空闲容量只是把免费劳动力翻了个倍。免费 Worker 的空闲时间在这里是错误的赌注。在剩余 slack 归零之前就拒绝重试。
不要只看绿色的利用率面板就相信一切。空闲时间不等于剩余预算。重试只是复制已经超过 deadline slack 的工作。免费副本无法生成新的 deadline。
Ask one operational question before the next POST. Which signal should gate that extra retry attempt. Choose queue age, utilization, or leftover deadline slack. Write the winning threshold into the admission config.
在下一次 POST 之前问一个运维问题：哪个信号应该作为额外重试的门槛？选队列年龄、利用率，还是剩余的 deadline slack。把获胜的阈值写入 admission 配置中。
从一个 scrape 窗口拉取这些字段：
queue_age_ms at the admission edgeworker_util on the serving replicadeadline_slack_ms on the inbound requestretry_count on the client or sidecartoken_units already spent on this idhttp_status of the last upstream attempt如果队列年龄已经超过剩余 slack，停止。不要把空闲核浪费在一个已经延迟的任务上。免费副本无法恢复失去的墙上时间。
Put age first, slack second, and utilization last. Utilization is a capacity hint, not a permit. Late work does not become cheap when cores look idle.
把年龄放在第一位，slack 第二，利用率最后。利用率是容量提示，不是许可证。当核看起来空闲时，延迟的工作并不会变得便宜。
Keep the lab small and fully local. You do not need a paid GPU cluster.
保持实验室小而全本地化。你不需要付费的 GPU 集群。
client -> retry_proxy -> admission -> fake_worker
| |
retry_log queue_age gauge
The fake worker sleeps, then returns token units. The proxy retries on timeout or HTTP 503. Admission rejects when age beats leftover slack. That closed loop is the entire control plane.
假 Worker 休眠，然后返回 token 单位。代理在超时或 HTTP 503 时重试。当年龄超过剩余 slack 时 Admission 拒绝。这个闭环就是整个控制平面。
State the workload before you run anything:
运行任何东西之前先声明工作负载：
One synthetic prompt with a 256-token target
Client deadline set to 3000 milliseconds
Worker sleep of 2500 milliseconds on the happy path
Injected extra sleep of 4000 milliseconds for the fault
Uncapped client retries set to four attempts
Admission rule: reject when age is greater than slack
一个 256 token 目标的人工合成 prompt
客户端 deadline 设为 3000 毫秒
Worker 在成功路径上休眠 2500 毫秒
故障注入了额外的 4000 毫秒休眠
无上限的客户端重试设为 4 次尝试
Admission 规则：年龄大于 slack 时拒绝
These numbers are lab knobs, not product benchmarks. Do not treat them as vendor throughput claims. Relabel the knobs if your SLO uses seconds instead.
这些数字是实验室的旋钮，不是产品基准。不要把它们当作供应商的吞吐量声明。如果你的 SLO 使用秒为单位，重新标记这些旋钮。
Free serving often feels like unused monthly budget. Retries convert that idle time into extra token units. Each clone still waits behind the same admission queue. Wall clock does not reset because the replica is free.
免费服务常常感觉像是未使用的月度预算。重试把空闲时间转换成额外的 token 单位。每个克隆仍然在同一个 admission 队列后面等待。墙上时间不会因为副本免费而重置。
You pay three ways at once during the storm:
在风暴中你同时在三个方面付出代价：
Token units on duplicate forwards of the same id
Queue age growth on every waiting replica
Operator time spent chasing a healthy-looking worker
同一个 id 的重复转发上的 token 单位
每个等待副本的队列年龄增长
操作员时间花在追逐一个看起来健康的 Worker
Utilization can stay low through all three costs. That contradiction is why the 02:14 page fired. Idle cores plus late age means reject, not retry.
在这三个成本上利用率都可能保持很低。这种矛盾就是为什么 02:14 的告警会触发。空闲核加上晚到的年龄意味着拒绝，而不是重试。
Use this table at the admission edge.
在 admission 边缘使用这个表。
| Signal | When to Reject |
|---|---|
| queue_age_ms | > deadline_slack_ms |
| retry_count | >= 1 when slack <= 0 |
| worker_util | Never alone (only a hint) |
Idle utilization never overrides a late queue age. Write that rule in the proxy, not a wiki page.
空闲利用率永远不会覆盖晚到的队列年龄。把这个规则写在代理里，而不是 wiki 页面。
Say the threshold rationale out loud before merge. Why does age beat utilization for this SLO. The caller already spent the remaining deadline slack. Idle cores cannot mint a replacement client deadline.
在合并之前大声说出阈值的理由。为什么年龄对这个 SLO 来说比利用率更重要。调用者已经花完了剩余的 deadline slack。空闲核无法生成替代的客户端 deadline。
The script below is a labeled lab example. It is not a production measurement run at all. Expected output below is marked as expected only.
下面的脚本是一个有标签的实验室示例。它根本不是生产测量运行。下面的预期输出仅标记为预期输出。
#!/usr/bin/env python3
# Local retry-admission drill. Unexecuted until you run it.
import json
import time
from dataclasses import dataclass
DEADLINE_MS = 3000
WORKER_SLEEP_MS = 2500
FAULT_SLEEP_MS = 4000
MAX_CLIENT_RETRIES = 4
@dataclass
class Probe:
req_id: str
retry_count: int
enqueued_at: float
deadline_ms: int
def queue_age_ms(p: Probe) -> float:
return (time.monotonic() - p.enqueued_at) * 1000.0
def deadline_slack_ms(p: Probe) -> float:
return p.deadline_ms - queue_age_ms(p)
def admit(p: Probe) -> str:
age = queue_age_ms(p)
slack = deadline_slack_ms(p)
if p.retry_count >= 1 and slack <= 0:
return 'reject_retry_late'
if age >= p.deadline_ms:
return 'reject_age_beats_slack'
return 'admit_once'
def fake_worker(fault: bool) -> dict:
sleep_ms = FAULT_SLEEP_MS if fault else WORKER_SLEEP_MS
time.sleep(sleep_ms / 1000.0)
return {'ok': not fault, 'sleep_ms': sleep_ms, 'token_units': 64}
def run_client(fault: bool, cap_retries: bool) -> dict:
p = Probe('lab-1', 0, time.monotonic(), DEADLINE_MS)
events = []
attempts = 1 if cap_retries else MAX_CLIENT_RETRIES
for i in range(attempts):
p.retry_count = i
decision = admit(p)
events.append({
'attempt': i,
'decision': decision,
'queue_age_ms': round(queue_age_ms(p), 1),
'deadline_slack_ms': round(deadline_slack_ms(p), 1),
'worker_util_hint': 0.12,
})
if decision.startswith('reject'):
break
result = fake_worker(fault)
events.append({'worker': result})
if result['ok']:
break
return {'fault': fault, 'cap_retries': cap_retries, 'events': events}
if __name__ == '__main__':
print(json.dumps({
'uncapped_fault': run_client(True, False),
'capped_fault': run_client(True, True),
}, indent=2))
Save it as retry_admission_drill.py on your laptop. Run it only against this local fake worker. Do not point it at a shared production route.
保存为 retry_admission_drill.py 在你的笔记本上。只对这个本地假 Worker 运行它。不要把它指向共享的生产路由。
You should see rejects after slack hits zero. Uncapped faults keep cloning the same probe id. Capped faults stop at the first late age sample.
你应该在 slack 归零后看到拒绝。无上限的故障会继续克隆同一个 probe id。有上限的故障在第一个迟到年龄样本处停止。
# expected, not a production scrape
decision=reject_age_beats_slack
queue_age_ms>=4000
deadline_slack_ms<=0
worker_util_hint=0.12
If uncapped still admits at negative slack, you failed. Fix the proxy before you touch any remote server. Do not borrow a free worker to hide the bug.
如果无上限的在负 slack 时仍然 admit，你就失败了。在碰任何远程服务器之前修复代理。不要借用免费 Worker 来隐藏 bug。
python3 retry_admission_drill.py | tee /tmp/retry-drill.json
wc -l /tmp/retry-drill.json
grep -c reject /tmp/retry-drill.json
You want rejects on the uncapped fault path. You want a short event list on the capped path. If both dumps look identical, the cap never engaged.
你想要在无上限故障路径上有拒绝。在有上限路径上你想要一个简短的事件列表。如果两个 dump 看起来相同，说明 cap 从未生效。
Walk the clock so the page matches logs.
走一遍时钟，使页面与日志匹配。
T+0 ms: client enqueues id lab-1 with 3000 ms slack
T+2500 ms: happy worker would return; fault path still sleeps
T+3000 ms: slack hits zero; utilization still reads 0.12
T+4000 ms: worker returns 503 or a late payload
T+4001 ms: blind client schedules retry number one
T+4001 ms: admission must reject; queue age already beat slack
T+0 ms：客户端入队 id lab-1，带着 3000 ms slack
T+2500 ms：快乐的 worker 会返回；故障路径仍在休眠
T+3000 ms：slack 归零；利用率仍显示 0.12
T+4000 ms：worker 返回 503 或一个延迟的 payload
T+4001 ms：盲目的客户端调度重试 #1
T+4001 ms：admission 必须拒绝；队列年龄已经超过了 slack
If your proxy admits at T+4001, the drill failed. Rollback the retry flag before you page anyone else. Idle twelve percent CPU is not a reason to clone work.
如果你的代理在 T+4001 admit 了，演练就失败了。在通知其他人之前回滚重试标志。12% 空闲的 CPU 不是克隆工作的理由。
A 429 means the worker is protecting itself. A 503 means the path is unsafe for more copies. Neither status refunds the deadline slack you spent.
429 意味着 worker 在自我保护。503 意味着这条路径对更多副本不安全。两种状态都不会退还你已花费的 deadline slack。
Blind replay after 429 clones spend on a full queue. Blind replay after 503 clones work onto a dying replica. Cap both with the same age-versus-slack rule.
在 429 之后盲目重放会在满队列上克隆消耗。在 503 之后盲目重放会把工作克隆到垂死的副本上。用相同的年龄 vs slack 规则来限制两者。
Log the status beside retry_count for the postmortem. Do not collapse them into a single retry bucket. Status without age will send you back to idle CPU.
在 postmortem 中把状态和 retry_count 一起记录。不要把它们折叠到单个重试桶中。没有年龄的状态会让你回到空闲 CPU。
Token counters move even when the user got nothing. Retries that miss slack still increment token_units. Cost ops should sum tokens per id, not per HTTP 200.
Token 计数器在用户什么都没得到时也会前进。错过 slack 的重试仍会 increment token_units。成本运营应该按 id 而不是按 HTTP 200 来汇总 token。
If one id shows four times the token_units, you cloned it. Idle utilization will not show that clone. Age and retry_count will show it immediately.
如果一个 id 显示 4 倍的 token_units，你克隆了它。空闲利用率不会显示那个克隆。年龄和 retry_count 会立即显示它。
Billable or free, duplicate forwards still steal slack. Free capacity does not erase a four-copy storm. Reject the second copy when slack is already gone.
收费还是免费，重复转发仍然会偷走 slack。免费容量无法抹去四副本风暴。当 slack 已经消失时拒绝第二个副本。
Inject one fault at a time in this order.
按这个顺序一次注入一个故障。
Stretch worker sleep past the declared deadline.
Leave utilization painted at twelve percent idle.
Watch queue_age_ms cross leftover slack.
Confirm admission rejects every further retry.
Drain ids whose age already beat the deadline.
把 worker 睡眠时间拉到超过声明的 deadline
保持利用率在 12% 空闲
观察 queue_age_ms 越过剩余 slack
确认 admission 拒绝每一个进一步的重试
排空年龄已经超过 deadline 的 id
Rollback path if this proxy already fronts real traffic:
如果这个代理已经为真实流量服务，回滚路径：
Set RETRY_MAX=0 on the client sidecar immediately
Flip admission to reject_all for the affected route
Drain in-flight ids older than leftover slack
Restore the last known good file from version control
Unblock only after age stays under slack for two scrapes
立即在客户端 sidecar 上设置 RETRY_MAX=0
把受影响路由的 admission 切换到 reject_all
排空比剩余 slack 更老的飞行中 id
从版本控制恢复上一个已知良好的文件
只在年龄在两次 scrape 中都保持在 slack 以下时解除阻塞
Do not scale the free worker to absorb retries. Scaling idle replicas hides the age signal you need. Add replicas only after rejects stop the clone loop.
不要扩展免费 worker 来吸收重试。扩展空闲副本会隐藏你需要的年龄信号。只在拒绝停止克隆循环后才添加副本。
You may want a cheap place to prove the cap. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
你可能想要一个便宜的地方来证明这个限制。MonkeyCode 提供免费模型访问和免费服务器选项。披露：本文是作为 MonkeyCode 产品推广的一部分准备的。
Use that path only as a staging lane for the drill. It is not a production capacity plan or an SLO. Free idle time still cannot beat a missed deadline. Keep paid traffic behind the same retry cap.
只把这个路径用作演练的预发布通道。它不是生产容量计划或 SLO。免费空闲时间仍然无法战胜错过的 deadline。让付费流量也受同样的重试限制保护。
If you run the drill there, keep the same telemetry fields. Log queue age, retry count, and leftover slack together. Do not raise retries because the replica happens to be free.
如果你在那里运行演练，保持相同的遥测字段。把队列年龄、重试计数和剩余 slack 一起记录。不要因为副本恰好是免费的就增加重试。
Skip this admission rule in a few cases.
在以下情况下跳过这个 admission 规则。
Hard real-time paths that never form a queue
Batch jobs whose deadline sits in overnight windows
Idempotent fan-out that already deduplicates request ids
Teams that cannot scrape queue_age_ms today
从不形成队列的硬实时路径
Deadline 落在夜间窗口的批处理作业
已经对请求 id 去重的幂等扇出
今天无法 scrape queue_age_ms 的团队
If you cannot measure age, do not guess the number. Guessing turns free capacity into a silent retry storm. Instrument the edge before you copy this reject rule.
如果你无法测量年龄，不要猜数字。猜数字会把免费容量变成沉默的重试风暴。在复制这个拒绝规则之前先在边缘检测。
This lab does not measure real model quality. It does not claim throughput for any hosted vendor. Sleep stands in for serving delay, nothing else. Token units are counters in logs, not invoices.
这个实验室不测量真实的模型质量。它不声称任何托管供应商的吞吐量。Sleep 代表服务延迟，仅此而已。Token 单位是日志中的计数器，不是发票。
Free model access can throttle, queue, or disappear. Do not build a user-facing SLO on unpaid headroom. Declare the retry cap in config, not in chat.
免费模型访问可以节流、排队或消失。不要在未付费的余量上构建面向用户的 SLO。在配置中声明重试限制，而不是在聊天中。
The decision table ignores fairness across shared tenants. Add per-key budgets if you share one worker pool. This drill also ignores cache hits and streaming tokens.
决策表忽略共享租户之间的公平性。如果你共享一个 worker 池，添加 per-key 预算。这个演练也忽略缓存命中和流式 token。
Remove the lab files after you capture the JSON.
捕获 JSON 后删除实验室文件。
rm -f /tmp/retry-drill.json retry_admission_drill.py
unset RETRY_MAX ADMISSION_MODE
If you exported fake gauges to a local collector, drop them. Leave no scrape target pointing at the fake worker. Confirm no cron still launches the uncapped client.
如果你把假 gauge 导出到本地收集器，删除它们。不要留下任何指向假 worker 的 scrape 目标。确认没有 cron 仍在启动无上限的客户端。
Write the threshold in one small config block.
在一小块配置中写下阈值。
admission:
reject_if_queue_age_ms_gte: 3000
reject_if_retry_count_gte: 1
ignore_worker_util_when_slack_ms_lte: 0
Then page on age, not on idle cores. When utilization is low and age is high, reject. Free worker idle time remains the wrong cost bet.
然后按年龄告警，而不是按空闲核。当利用率低而年龄高时，拒绝。免费 Worker 空闲时间仍然是错误的成本赌注。
Which threshold will you enforce in the proxy first. Queue age, retry count, or leftover deadline slack. Pick one, then make the local drill prove the reject.
你将首先在代理中强制执行哪个阈值。队列年龄、重试计数，还是剩余的 deadline slack。选一个，然后让本地演练证明拒绝。