认证聊天机器人流式后端的测试策略

Short answer: Choose the backend API whose completed stream can be proved, replayed, and evaluated independently of any provider SDK.

对于应用内聊天机器人，"简单"不应该意味着 notebook 里最少几行代码。它应该意味着：一个服务器所有的认证边界、一个小型流式契约，以及足够的证据来区分一个真正完成的回答和一个只是在浏览器里看起来已经完成的响应。我在意这个区别，因为我的通常路径是从 notebook 到生产：先在 Python 里测试检索和提示词，然后再把它们迁移到一个有登录态的 Web 应用后面，同时不把模型客户端变成架构的一部分。

My evaluation constraint is blunt. A candidate passes only when the browser can reconnect, the server can associate the turn with the correct user, and an automated check can reconcile the visible answer with the committed conversation record. Raw token speed comes later. This rules out the deceptively easy design where browser code holds provider credentials and imports a provider SDK directly. It also gives me a portable alternative to any single SDK: my application owns the interface, while a thin adapter owns provider-specific calls.

What should an authenticated web app demand from a chatbot streaming backend API?

The backend has two trust relationships, and I don't let them blur together. The browser authenticates to my application. My application authenticates to whichever model service sits behind the adapter. A user session is not a model credential, and a model credential never belongs in browser code. That separation also gives the server a stable place to enforce conversation ownership before generation starts.

后端有两个信任关系，我不会让它们混在一起。浏览器向我的应用认证。我的应用向位于适配器背后的模型服务认证。用户会话不是模型凭证，模型凭证也绝不应该出现在浏览器代码里。这种分离也让服务器有了一个稳定的地方，在生成开始前强制执行会话所有权。

The request itself can stay boring: conversation identity, a client-generated turn identity, the user's text, and a version for the prompt or behavior under evaluation. The response is a sequence of typed events rather than an unstructured pile of token fragments. I need, at minimum, a start event that identifies the accepted turn, zero or more deltas for display, and a terminal event that states whether the turn was committed. The exact wire encoding matters less than the semantics — the browser must not infer durable success from a closed connection.

请求本身可以保持简单：会话标识、客户端生成的轮次标识、用户文本，以及正在评估的提示词或行为的版本号。响应是一系列类型化的事件，而不是一堆非结构化的 token 碎片。我至少需要一个起始事件来标识已接受的轮次、零个或多个用于展示的增量事件，以及一个终端事件来声明该轮次是否已被提交。精确的传输编码不如语义重要——浏览器不能从关闭的连接中推断出持久化的成功。

I learned that the unpleasant way. In one release, a call returned HTTP 200 and the full reply appeared on screen, but the side effect that saved the assistant turn never happened; 6 hours later, support found conversations that vanished after refresh. The transport had succeeded. The product action had not. Since then, "done" means the terminal event and the durable record agree.

我是从惨痛的教训中学到这一点的。在一个版本中，一次调用返回了 HTTP 200，完整的回复也显示在了屏幕上，但保存 assistant 轮次的后置效应从未发生；6 小时后，支持团队发现对话在刷新后消失了。传输成功了，产品动作却没有。从那以后，"完成"意味着终端事件和持久化记录是一致的。

That's the test. It sounds stricter than a quick demo because it is.

这就是测试。它听起来比快速演示更严格，因为确实如此。

I also want cancellation to be explicit. A closed tab may tell the server that nobody needs more display deltas, but it doesn't automatically answer whether the accepted turn should be abandoned, completed, or billed to a usage ledger. I'm not sure why so many prototypes leave that policy implicit; perhaps the happy path hides it. Your mileage may vary, but writing the cancellation rule down before choosing an API exposes more architectural risk than another latency shootout.

我还希望取消是显式的。关闭的标签页可能告诉服务器不再需要更多的展示增量，但它不会自动回答已接受的轮次是否应该被放弃、完成，或者被计入用量账本。我不确定为什么这么多原型把这个策略弄得隐式的；也许是因为愉快路径掩盖了它。各人情况不同，但在选择 API 前把取消规则写下来，比再来一轮延迟对比更能暴露架构风险。

Treat streaming as an application contract

把流式处理当作应用契约

I keep a narrow Python interface between chat orchestration and model transport. It lets an eval harness consume the same events as the web delivery layer, without pretending every provider exposes identical options. The adapter may translate a remote stream, but the rest of the application sees only behavior my team is prepared to support.

我在聊天编排和模型传输之间维护一个狭窄的 Python 接口。它让 eval 测试工具能够消费与 Web 交付层相同的事件，而不假装每个 provider 暴露的是相同的选项。适配器可能会转换远程流，但应用的其他部分只看到我的团队准备支持的行为。

from dataclasses import dataclass
from typing import AsyncIterator, Literal, Protocol

@dataclass(frozen=True)
class ChatEvent:
    kind: Literal["started", "delta", "committed", "rejected"]
    turn_id: str
    text: str = ""
    record_version: int | None = None

class ChatRuntime(Protocol):
    async def stream_turn(
        self,
        *,
        user_id: str,
        conversation_id: str,
        turn_id: str,
        message: str,
    ) -> AsyncIterator[ChatEvent]: ...

async def deliver_turn(
    runtime: ChatRuntime,
    session_user_id: str,
    conversation_id: str,
    turn_id: str,
    message: str,
) -> AsyncIterator[ChatEvent]:
    async for event in runtime.stream_turn(
        user_id=session_user_id,
        conversation_id=conversation_id,
        turn_id=turn_id,
        message=message,
    ):
        yield event
        if event.kind in {"committed", "rejected"}:
            return

This example is intentionally smaller than production code. Session validation and the ownership lookup happen before deliver_turn; persistence belongs inside the runtime transaction boundary or an adjacent application service. The important bit is that the terminal state is data. Tests can assert that every started turn reaches exactly one accepted terminal state, while the UI can wait for committed before representing the answer as durable.

这个示例有意比生产代码更精简。会话验证和所有权查找发生在 deliver_turn 之前；持久化属于运行时事务边界内部或相邻的应用服务。重要的是终端状态是数据。测试可以断言每个启动的轮次都达到一个可接受的终端状态，而 UI 可以等待 committed 事件后才将回答呈现为持久化的。

Don't flatten every remote event into delta. Usage, tool requests, citations, safety outcomes, and termination reasons may need separate application types later. I add one only when the product has defined its meaning and my eval set can exercise it. Otherwise an allegedly generic schema becomes a bag of optional fields copied from one provider.

不要把每个远程事件都扁平化成 delta。用量、工具调用、引用、安全结果和终止原因以后可能需要单独的应用类型。我只在产品已经定义了它的含义、而且我的 eval 集合能够执行它的时候才添加一个。否则，一个号称通用的 schema 就会变成从某个 provider 复制过来的一袋可选字段。

This is also where a "compatible" API can mislead. Matching a request shape doesn't guarantee matching retry, cancellation, ordering, or terminal-event behavior. I run the adapter through contract tests, then swap it without changing browser code. If that test requires importing the provider's classes throughout the app, the boundary isn't real yet.

这也是"兼容"的 API 可能误导的地方。匹配请求形状并不能保证重试、取消、排序或终端事件行为的匹配。我让适配器通过契约测试，然后用它替换而不改变浏览器代码。如果那个测试需要在整个应用中导入 provider 的类，那么边界还没有真正建立起来。

Compare failure semantics before feature lists

在特性列表之前比较失败语义

I score candidates with a short failure matrix before I compare model catalogs. It keeps the selection tied to the in-app workflow rather than to an impressive notebook. No row gets a pass from documentation alone; I exercise it against a disposable conversation and inspect both emitted events and stored state.

我在比较模型目录之前先用简短的失败矩阵对候选者评分。它让选择与应用内工作流程绑定，而不是与一个令人印象深刻的 notebook 绑定。没有哪一行仅凭文档就能通过；我用一次性会话来执行它，检查发出的事件和存储的状态。

The catch is that a tiny wrapper costs engineering time. It is not suitable when I'm building a disposable, unauthenticated experiment whose output will never be persisted. In that case, I stick with the provider's SDK because its native types and examples shorten the feedback loop. I also keep the native SDK when the product depends on provider-specific realtime media or event semantics that my abstraction would merely conceal. Portability isn't free, and a false abstraction is worse than an explicit dependency.

问题是小包装器需要工程时间。当我在构建一个一次性的、未认证的实验、其输出永远不会被持久化时，它就不适用了。在这种情况下，我坚持使用 provider 的 SDK，因为它的原生类型和示例缩短了反馈循环。当产品依赖于 provider 特定的实时媒体或事件语义，而我的抽象只会掩盖它们时，我也会保留原生 SDK。可移植性不是免费的，错误的抽象比显式的依赖更糟糕。

For the ordinary authenticated text chatbot, though, the table usually finds the real gaps. A reconnect can duplicate a turn. Two tabs can race on one conversation. A user can submit against a stale record version. Partial text can linger after cancellation. None of these is fixed by choosing a fashionable client library; they need application semantics and tests.

但对于普通的已认证文本聊天机器人，这个表通常会发现真正的差距。重新连接可能复制一个轮次。两个标签页可能在同一个会话上竞态。用户可能针对一个过时的记录版本提交。取消后部分文本可能残留。没有一个有流行客户端库的噱头能修复这些问题；它们需要应用语义和测试。

Small differences matter. I record the candidate's result for each scenario as an artifact, including event order and final stored version, so a later adapter upgrade runs against the same expectations. That makes API selection reproducible instead of a meeting where everyone remembers a different demo.

小差异很重要。我把候选者在每个场景下的结果记录为一个产物，包括事件顺序和最终存储的版本，这样后来的适配器升级就针对相同的期望运行。这让 API 选择可复现，而不是变成一场每个人记得不同演示的会议。

Measure the choice from notebook to production

从 notebook 到生产环境衡量选择

My last gate is an eval run that joins quality, reliability, latency, and usage without collapsing them into one magic score. I start with representative conversations: short factual turns, retrieval-backed questions, prompt-injection attempts, cancellations, reconnects, and concurrent submissions. Embeddings can support retrieval and similarity-based analysis, but the metric still has to reflect the product decision; proximity alone doesn't establish that an answer is correct. The embeddings guide in References is a useful primary starting point, while the prompt engineering guide provides broader patterns to test rather than accept on faith.

我的最后一个关卡是一次 eval 运行，它把质量、可靠性、延迟和用量join在一起，而不是把它们压缩成一个魔术分数。我从有代表性的对话开始：简短的事实性轮次、基于检索的问题、prompt 注入尝试、取消、重连和并发提交。Embeddings 可以支持检索和基于相似度的分析，但指标仍然必须反映产品决策；仅靠相似度不能确立一个回答是正确的。《References》中的 Embeddings guide 是一个有用的主要起点，而 Prompt Engineering Guide 提供了更广泛的模式来测试，而不是盲目接受。

For every fixture, I retain prompt version, adapter version, turn identity, time to first visible delta, time to committed terminal state, final record version, and whatever usage units the adapter can report faithfully. Then I score the answer separately. This keeps prompt-cost awareness grounded: I can see whether a longer retrieval context improved the eval cases enough to justify its added input, without making a volatile price claim or optimizing for short prompts that fail users.

对于每个 fixture，我保留提示词版本、适配器版本、轮次标识、首次可见 delta 的时间、到达 committed 终端状态的时间、最终记录版本，以及适配器能够忠实报告的任何用量单位。然后我分别对回答评分。这让 prompt 成本意识有据可依：我可以看到更长的检索上下文是否足够改善了 eval 案例来证明其额外输入的合理性，而不用做出一个不稳定的成本声明或为那些会让用户失望的短 prompt 做优化。

I don't promote a candidate merely because its median stream feels fast. Tail behavior and correctness decide whether the interaction survives real traffic. The deployment check replays duplicate turn identities, expires a session before generation, cancels after the first delta, and reconnects before the terminal event. Observability must link those attempts without logging raw private conversation text by default.

我不会仅仅因为一个候选者的中位流感觉很快就提升它。尾部行为和正确性决定交互是否能在真实流量中存活。部署检查会重放重复的轮次标识、在生成前让会话过期、在第一个 delta 后取消，以及在终端事件前重连。可观测性必须把这些尝试串联起来，而默认不记录原始的私人对话文本。

One more constraint: ownership has to remain obvious to the team. The application owns authorization, conversation state, eval definitions, and the public event contract. The adapter owns translation into a model service and translation back. If changing adapters requires edits in React components, persistence records, and eval fixtures, I haven't isolated it.

还有一个约束：所有权必须对团队保持清晰。应用拥有授权、会话状态、eval 定义和公共事件契约。适配器拥有到模型服务的翻译以及翻译回来。如果改变适配器需要在 React 组件、持久化记录和 eval fixtures 中编辑，那就说明我没有隔离好它。

Before copying this choice, measure the cases your users will actually create and decide which provider-specific capabilities you are willing to give up. A simple backend API is the one that leaves the fewest ambiguous states after those tests, not the one that wins a screenshot of the happy path.

在复制这个选择之前，衡量你的用户实际会创造的场景，决定你愿意放弃哪些 provider 特定的能力。一个简单的后端 API 是那些在这些测试后留下最少模糊状态的，而不是在愉快路径截图上赢的那个。

OpenAI, "Embeddings guide": https://platform.openai.com/docs/guides/embeddings

Prompt Engineering Guide: https://www.promptingguide.ai