LLM调用生产API的工程教训：别做快乐演示

The first time I gave a language model access to a real API, it worked perfectly in the demo. It looked up an order, summarized the status, and everyone in the meeting nodded. Two weeks later, in production, the same setup tried to issue the same refund three times in a row because a timeout made it think the first attempt failed.

Nothing was "wrong" with the model. Everything was wrong with how I had wired it up.

Function calling (or tool calling, same thing) is the moment your LLM stops being a text generator and becomes an actor in your system. Most tutorials stop at the happy path: define a function, pass the schema, watch the model call it. That gets you a demo. This article is about everything after the demo.

Here is the mental model that changed how I build these systems: designing tools for an LLM is API design for a brilliant intern with no fear. The intern is smart, fast, tireless, and reads documentation more carefully than most senior engineers. The intern also has zero survival instinct. If a button exists, the intern will eventually press it, at 3am, with arguments you never imagined. Every practice below falls out of that one idea.

Never trust the model's arguments as typed data

When a model "calls a function", what actually happens is that it emits text that looks like JSON. Your SDK parses it and hands you an object. That object has the epistemic status of user input from a public form, because that is functionally what it is.

Models produce arguments that are almost right constantly: a string where you expected a number, an ISO date with the wrong timezone, an enum value that is a plausible synonym of a real one ("cancelled" when your API says "canceled"), a negative quantity, an ID copied from the wrong part of the conversation.

So the first rule is schema-first, validated at runtime. Define the schema once, derive both the tool definition and the validator from it:

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const refundArgs = z.object({
  orderId: z.string().regex(/^ord_[a-z0-9]{12}$/),
  amountCents: z.number().int().positive().max(50_000),
  reason: z.enum(["damaged", "not_delivered", "customer_request"]),
});

// The same source of truth feeds the model...
const refundTool = {
  name: "issue_refund",
  description: "Issue a partial or full refund for an order.",
  parameters: zodToJsonSchema(refundArgs),
};

// ...and guards the execution.
function executeRefund(rawArgs: unknown) {
  const parsed = refundArgs.safeParse(rawArgs);
  if (!parsed.success) {
    return { ok: false as const, reason: parsed.error.message };
  }
  // parsed.data is now actually typed, not just claimed to be
  return performRefund(parsed.data);
}

Notice the max(50_000). Business rules belong in the schema too. The model should be structurally incapable of requesting a $40,000 refund, not merely discouraged by the prompt. Prompts are suggestions; validators are laws.

Which tools should the model even see?

Here is a question I ask on every integration now: what is the minimum set of tools this specific context needs?

Most systems dump every tool into every conversation. That is like giving the intern a master keycard on day one. Instead, scope capabilities:

Split read from write. get_order and issue_refund are different risk classes. A conversation that is just answering questions should receive only read tools. The model literally cannot misuse a tool it was never given.

Scope by context. A logged-out visitor's session gets public lookup tools. An authenticated customer's session gets tools bound to their own account ID, injected server-side, never as a model-supplied argument. If the model can pass customerId, the model can pass the wrong customerId.

Scope by conversation state. You can add tools mid-conversation. The refund tool only enters the toolset after an order has been located and verified. This shrinks the blast radius of every earlier turn.

The counterintuitive part: fewer tools also make the model smarter. Tool selection is a decision the model can get wrong, and every irrelevant tool in the list is a chance to get it wrong. Small, sharply described toolsets improve accuracy and safety at the same time. It is one of the few free lunches in this field.

Return failures, never throw them

This one cost me real money, so I will be emphatic about it.

Most agent frameworks, and most hand-rolled loops, have retry logic somewhere. A thrown exception looks like a transient infrastructure failure, so something retries it: the framework, the queue, your own catch block. Now walk through what happens when a payment API times out. The charge may have succeeded. Your code throws. The retry fires. The customer is charged twice.

A thrown error is a promise to your infrastructure that retrying is safe. For non-idempotent operations, that promise is a lie.

The fix is to make failure a value, not an exception:

type ToolResult<T> =
  | { ok: true; data: T }
  | { ok: false; reason: string; retryable: boolean };

async function chargeCard(args: ChargeArgs): Promise<ToolResult<Charge>> {
  try {
    const charge = await payments.charge(args);
    return { ok: true, data: charge };
  } catch (err) {
    // Ambiguous outcome: do NOT signal "safe to retry"
    return {
      ok: false,
      reason: "Payment provider did not confirm the charge.",
      retryable: false,
    };
  }
}

The { ok: false, reason } object goes back to the model as the tool result. And here is the part I underestimated: the model is genuinely good at handling it. It reads the reason, explains the situation to the user in plain language, and offers a next step. The model apologizes; your infrastructure does not replay a charge. That is the correct division of labor.

Idempotency keys: assume every write will happen twice

Even with structured failures, writes get duplicated. Networks flap, users double-click, the model occasionally decides to call the same tool twice in one turn (yes, really). The defense is the same one payment APIs have used for a decade: an idempotency key on every write.

Derive the key from stable identifiers, not from randomness:

import { createHash } from "node:crypto";

function idempotencyKey(conversationId: string, toolCallId: string) {
  return createHash("sha256")
    .update(`${conversationId}:${toolCallId}`)
    .digest("hex");
}

Pass it through to any downstream API that supports one, and enforce it yourself (a unique constraint on the key column) for internal writes. Now a duplicate call is a harmless no-op that returns the original result, instead of a second refund.

Budgets, timeouts, and the runaway loop

The intern has no fear, and also no sense of time or money. An agent loop with tools will happily call a slow search API fourteen times in a row, each call informing the next, while your user watches a spinner and your bill climbs.

Give every tool call, and every conversation, a budget:

A hard timeout per tool call (a few seconds for reads, tighter than you think for writes; a slow write is an ambiguous write).

A cap on tool calls per turn and per conversation. When the cap hits, the tool loop returns a structured failure ("call budget exhausted") and the model tells the user it could not finish, which is honest and recoverable.

A cost ceiling where tools have real unit costs (external API fees, compute-heavy queries).

None of this is exotic. It is the same bulkheading you would put around any untrusted client, which is exactly what the model is.

Gate the consequential stuff behind a human

Some actions should never complete on model judgment alone: refunds above a threshold, account deletion, sending email to a customer list, anything legally significant. For these, the tool does not perform the action. It stages it.

The tool writes a pending action record and returns { ok: true, data: { status: "pending_confirmation", confirmUrl } }. A human (the end user, or an operator, depending on the action) clicks confirm, and only that click executes the write. The model's role ends at proposing.

This pattern is also a gift to your future self, because staged actions come with a natural place to show a diff: here is exactly what will happen if you confirm. Humans are much better at reviewing a concrete proposed action than at supervising an abstract conversation.

Log every call like you will be asked about it

Because you will be asked about it. When a customer says "your bot refunded the wrong order", the difference between a five-minute answer and a very bad week is an audit log.

Log, for every tool call: timestamp, conversation ID, tool name, the full validated arguments, the result (or structured failure), latency, and which model and prompt version produced the call. Redact secrets before logging, and treat argument logs with the same care as any user data, but do log the arguments. "The model called issue_refund" is useless in an incident; "the model called issue_refund with orderId: ord_x, amountCents: 1900 at 14:02 and received ok: false" is everything.

The log has a second life too: it is your best source of evaluation data. Real failed calls from production are worth fifty synthetic test cases.

If you remember nothing else

Validate every argument with a real schema. Business limits live in the validator, not the prompt.

Give each context the minimum toolset. Reads and writes are different risk classes.

Return { ok: false, reason } instead of throwing. Thrown errors invite retries; retried writes double-charge people.

Idempotency keys on every write, derived from stable IDs.

Timeouts, call caps, and cost budgets on the loop.

Human confirmation for consequential actions; the model proposes, a person disposes.

Audit-log every call with its arguments and outcome.

None of these steps is hard. What is hard is remembering that the demo working proves almost nothing, because the happy path was never the risk. The intern is brilliant. Build the guardrails like you believe the other half of the sentence.

I work on Fetchply, an AI support agent for ecommerce, where custom API tools run behind exactly these guardrails in production, and the structured-failure rule exists because thrown errors really did try to retry a non-idempotent request.