深入解析结构化输出：让 AI 生成绝对可解析的 JSON

当你的客服回复助手从聊天记录中提取客户姓名、订单号和问题类别时，你需要这些数据每次都能被机器读取。Structured outputs 通过将 JSON Schema 转换为上下文无关语法，并在生成过程中用它来屏蔽无效 token，使这一点成为可能。模型在字面上无法产生违反你 schema 的输出。

反直觉的地方在于：最可靠的 structured-output 系统并不会在事后解析模型的输出。它们从一开始就阻止了不良输出的产生。当你的代码收到响应时，数据已经保证符合你要求的格式。不需要正则表达式、不需要用 try/catch 包裹 JSON.parse、不需要重试循环。约束存在于解码循环本身。

How does constrained decoding actually work under the hood?

An autoregressive language model generates text one token at a time by sampling from a probability distribution over its vocabulary, given the context so far. Normally, every token is fair game. Constrained decoding introduces a formal language, derived from your JSON Schema, that defines which sequences are legal. At each step, the system masks out every token that would lead to an illegal state and renormalizes the distribution over the remainder. The model never steps outside the allowed language.

The formal language comes from compiling your JSON Schema into a context-free grammar. OpenAI's implementation converts the schema into a CFG, pre-processes it into a cached data structure, and consults that structure on every token generation step.

A schema like:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "order_id": { "type": "integer" },
    "category": { "enum": ["billing", "shipping", "returns"] }
  },
  "required": ["name", "order_id", "category"],
  "additionalProperties": false
}

becomes a grammar where an extraction object must contain exactly those three keys with those types, and the category field must be one of three string literals. The grammar tracks position: inside an object, expecting a property name; inside a value, constrained to a specific type; inside an enum, only allowing listed options. At any prefix, the grammar state encodes exactly which tokens keep the partial sequence on a path that can still complete to a valid JSON document matching the schema.

This means the model cannot omit required fields, cannot add extra properties when additionalProperties is false, cannot use a string where an integer is expected, and cannot pick a category outside the enum. These become token-level impossibilities.

Why is JSON mode different from full structured outputs?

JSON mode, available in OpenAI's API by setting response_format to {"type": "json_object"}, constrains the model to generate syntactically valid JSON. The grammar is generic: any valid JSON document is allowed. You are guaranteed that JSON.parse will succeed, but the resulting object can have arbitrary keys, types, and nesting. The model might call the field orderId instead of order_id or return extra commentary alongside the JSON.

Structured outputs with json_schema and strict: true replace the generic JSON grammar with one derived specifically from your schema. The model cannot produce anything that fails schema validation at the structural level — required fields, types, enum membership are all enforced during generation. The internal evaluations for gpt-4o-2024-08-06 report 100% adherence to complex schemas under this setup.

For our customer-support assistant, this distinction matters. With JSON mode, the assistant might correctly extract name and order_id but put the category in a field called issue_type. Your downstream code breaks, and you are back to writing defensive parsers. With structured outputs, the category field will exist and will contain exactly one of billing, shipping, or returns because those are the only tokens the grammar allows at that position.

{
  "type": "comparison",
  "title": "JSON mode vs Structured Outputs",
  "caption": "JSON mode guarantees valid JSON syntax. Structured outputs enforce your specific schema.",
  "before": {
    "label": "JSON mode",
    "points": [
      "Produces any valid JSON",
      "Does not enforce required keys",
      "Can add extra properties",
      "Only guarantees syntactic correctness"
    ]
  },
  "after": {
    "label": "Structured Outputs",
    "points": [
      "Conforms to your JSON Schema",
      "Required keys always present",
      "No extra properties allowed",
      "Enum values strictly enforced"
    ]
  }
}

How do function calling and structured outputs relate?

Function calling is structured outputs in disguise. When you define a tool with a JSON Schema describing its parameters, the provider uses constrained decoding to guarantee that any function call arguments match that schema. The model chooses a function name and fills in arguments; the grammar ensures the arguments conform to the declared types and required fields.

For data extraction tasks, you can define a function whose only purpose is to accept the structured data you want. The function never executes. It exists purely to give the decoding engine a schema to enforce. This works well when your extraction maps cleanly onto a function's argument list. For our assistant, you might define log_customer_issue(name: string, order_id: int, category: enum) and treat the function call as your structured output.

graph TD
  A["User defines tool with JSON Schema"] --> B["Provider compiles schema to grammar"]
  B --> C["Constrained decoding enforces schema"]
  C --> D["Valid function arguments produced"]

Response-level structured outputs with json_schema are better when you need arbitrary nesting, want to reuse a schema across many calls without the function abstraction, or find the tool-calling paradigm awkward for pure data extraction. Both mechanisms rely on the same underlying constrained decoding stack. The choice is ergonomic, not technical.

What happens when constraints clash with what the model wants to say?

Constrained decoding forces the model into a narrower token space. When the schema is strict or the model is small, this can degrade output quality because the model's preferred tokens keep getting masked. The Draft-Conditioned Constrained Decoding paper formalizes this as a projection loss: you are discarding probability mass assigned to illegal sequences, and that "tax" grows when the schema excludes many high-probability paths.

The fix is to separate reasoning from formatting. Generate an unconstrained draft first, letting the model think freely, then feed that draft as context into a constrained decoding step that maps the content into the schema. This two-phase approach improved structured accuracy on GSM8K from 15.2% to 39.0% using a 1B parameter model.

For our assistant, this means: if the extraction requires reasoning (this customer mentioned three separate issues, which one is primary?), do that reasoning in an unconstrained first pass. Then constrain the formatting pass. OpenAI's reasoning tokens and Anthropic's extended thinking features achieve something similar by keeping internal reasoning outside the schema envelope and only constraining the final visible output.

What guarantees do I actually get, and where do I still need validation?

Structured outputs guarantee structural correctness: valid JSON, required keys present, types correct, enum values within the allowed set. What they do not guarantee is semantic correctness. The model can put a plausible but wrong name in the name field or assign billing to a shipping complaint. The grammar only constrains form, not meaning.

Numeric bounds (minimum, maximum) are also tricky. A CFG cannot enforce that an integer falls within a range at the token level because the constraint depends on the full value, and tokens are generated incrementally. Libraries like Guidance explicitly note that numeric bounds "cannot really be supported in the context of LLM generation." Providers implement a practical subset and leave the rest to post-hoc validation.

{
  "type": "stat",
  "title": "What Structured Outputs Guarantee",
  "caption": "Syntactic guarantees are exhaustive; numeric bounds need separate validation.",
  "stats": [
    {
      "value": "Yes",
      "label": "Valid JSON syntax"
    },
    {
      "value": "Yes",
      "label": "Required keys present"
    },
    {
      "value": "Yes",
      "label": "Correct types & enums"
    },
    {
      "value": "No",
      "label": "Numeric range enforcement"
    }
  ]
}

Production systems layer semantic validation on top of syntactic constraints. Guardrails AI wraps any LLM call with JSON Schema validation and Python-level validators. If a field must be a valid email or a URL must be reachable, you write a validator. When validation fails, Guardrails re-prompts the model with error feedback. This adds latency but catches the errors that grammars cannot express. Instructor takes a similar approach with Pydantic models and automatic retries.

For the customer-support assistant, you would use structured outputs to guarantee that order_id is an integer and category is one of the three enum values. Then you would add a validation layer that checks the order_id against your database and flags cases where the extracted category contradicts the order's actual status. Structured outputs eliminate the parsing headache. Validation catches the business logic violations.

Anatomy of a reliable structured output pipeline

Here is what a production-grade pipeline looks like for our support assistant, combining decoding-time constraints with post-hoc validation:

flowchart LR
    A[Chat transcript] --> B[Unconstrained reasoning pass]
    B --> C[Draft: identified primary issue]
    C --> D[Constrained decoding with JSON Schema]
    D --> E{Schema valid?}
    E -->|Yes| F[Semantic validators]
    E -->|No| D
    F --> G{Business logic OK?}
    G -->|Yes| H[Return structured object]
    G -->|No| I[Re-prompt with error context]
    I --> D

The unconstrained pass does the reasoning. The constrained pass enforces the schema. The validator checks business rules. If anything fails, the feedback loop re-prompts. This architecture treats the LLM as a component in a reliable data pipeline, not as a magic box you hope behaves.

OpenAI caches the compiled grammar per schema, so only the first request with a new schema pays the compilation cost. Subsequent requests reuse the cached artifact. The token-level masking adds modest per-step overhead, but the elimination of parsing failures and retries typically makes the overall system faster and cheaper than best-effort JSON mode with defensive code.

Frequently Asked Questions

Q: Does structured outputs cost more in tokens or latency?

The schema itself counts against input tokens, and the first request with a new schema incurs a one-time compilation cost for building the grammar artifact. Per-token generation overhead is small because the provider caches the pre-processed grammar and uses efficient data structures for token masking. The net cost is often lower because you eliminate retries from parsing failures.

Q: Can I use structured outputs with streaming?

Yes. Providers like OpenAI support streaming with structured outputs. The model sends tokens as they are generated, and those tokens are guaranteed to be prefixes of a valid schema-conforming JSON document. You may need to buffer partial output and parse only on completion, depending on your parser's tolerance for incomplete JSON.

Q: What if my schema is too complex for the grammar compiler?

Providers support a practical subset of JSON Schema. Nested objects, arrays, enums, required fields, and additionalProperties are well supported. Features like $ref, allOf, and anyOf have varying levels of support depending on the provider. Check the provider's documentation for the exact supported subset. For constraints that cannot be expressed in a CFG (numeric ranges, cross-field dependencies), use post-hoc validation.

Q: How do I handle optional fields in my schema?

Define them in your JSON Schema without listing them in the required array. The grammar will allow the model to include or omit optional fields. If the model omits an optional field, the resulting JSON simply will not have that key. If it includes the field, the value must match the declared type. This works identically to standard JSON Schema semantics.

Q: What is the fallback if I cannot use provider-native structured outputs?

Use a library like Guardrails or Instructor. These parse the model's output (handling common formatting issues like markdown code fences), validate against your schema, and re-prompt with error feedback on failure. You get schema adherence at the cost of potentially multiple LLM calls, but this works with any provider, including local models.

Your customer-support assistant uses structured outputs to extract order_id (integer, required) and refund_amount (number, required, with minimum: 0). On one call, the model returns {"order_id": 12345, "refund_amount": -50.00}. The JSON is structurally valid: both fields exist, types are correct. But refund_amount is negative, which violates the minimum: 0 constraint. Your downstream code processes the negative refund and accidentally charges the customer. What went wrong, and how do you fix it?

Answer: The grammar compiled from your JSON Schema enforces required keys and types but cannot enforce numeric bounds at the token level because a CFG has no mechanism to check that a completed number falls within a range while tokens are being generated incrementally. The model can output -50.00 as a valid number token sequence that satisfies the number type constraint, and the grammar will allow it. The minimum: 0 constraint is part of JSON Schema but lives outside what the CFG can express. The fix is to add a post-hoc validation layer: after receiving the structured output, validate refund_amount >= 0 in your application code or using a library like Guardrails with a field-level validator. If validation fails, re-prompt the model with an explicit error like "refund_amount must be non-negative." Structured outputs guarantee structural correctness, not semantic correctness. Always validate business constraints in code.

If getting this kind of breakdown every week on how real systems work under the hood sounds useful, subscribe to Internals Decoded at internalsdecoded.com. Next up in the series: what happens when you need the model to use external APIs, databases, or your own code during generation.

OpenAI Structured Outputs documentation

OpenAI guide on how structured outputs works

OpenAI JSON mode documentation

OpenAI function calling documentation

Anthropic extended thinking documentation

Draft-Conditioned Constrained Decoding paper

Guidance documentation on JSON Schema limitations

Guardrails AI documentation

Instructor documentation

OpenAI reasoning documentation

Originally published at Internals Decoded. AI internals, explained conversationally.