非开发者用Agent Coding构建生产系统的经验: Eval是关键

Yes — and I am the evidence, with a condition attached. I have never hand-written a line of production code. narenkatakam.com runs a RAG pipeline on Cloudflare Workers AI, a Vectorize index of 256 chunks, a D1 database, and an MCP server with four tools. It was built entirely through agentic coding with Claude Code. What made it production was not the agent. It was the eval suite.

Writing the code was never what made someone a builder. Verifying it was. Agentic coding hands you the writing for free and hands you nothing else. The proof was never the code. It was the eval you can re-run in front of someone.

What "production" actually means here

Precision first, because this phrase gets stretched until it means nothing.

This is a personal site, not a bank. No PII, no payments, no on-call rotation, no SLA anyone signed. What it does have: a live endpoint answering questions from strangers, a vector index that has to stay in sync with the content, an MCP server other agents connect to, and five months of machine traffic — 9,238 crawler visits from Googlebot, ClaudeBot, Bingbot, OAI-SearchBot, Applebot and PerplexityBot — reading it without a human in the loop.

That is small production. It is still production, in the only sense that matters for this question: it breaks in ways I do not find out about by looking at it, and something other than my opinion has to tell me it broke.

That last sentence is the entire argument. Everything below is the receipt.

Why the bottleneck was never typing

The standard objection to a non-developer building anything real is about syntax. It has always been the wrong objection.

I am a product person — systems thinker, designer, mechanical engineer turned consultant. I know what a thing should do and why. For twenty years the how required a team, and the reason was never that I could not learn a language. It was that writing code is the cheap part of building software and the expensive parts sit either side of it: knowing what to build, and knowing whether what got built is correct.

Agentic coding collapsed the middle. Claude Code writes the D1 migration, the Pages Function, the embedding pipeline. It writes them faster and more idiomatically than I would have. The how stopped being a gate somewhere around week three of thirty days with the tool.

What it did not collapse is the far side. An agent will hand you a migration that runs, returns exit code 0, and quietly does the wrong thing. It will hand you an API that responds 200 with a body nobody has read. Exit 0 is not correctness. Rendered is not right.

So the non-developer question inverts. It is not can you produce the code. It is can you tell whether the code is lying to you. That is a judgment skill, not a syntax skill, and it is the one I actually had.

Call it the verification gap: the distance between what an agent produces and what you can prove about it.

Every failure mode I have hit in eighteen months of building this way lives in that gap. Not one of them would have been caught by reading the source more carefully. All of them were caught by something that ran and returned a number.

The database that had been empty for five months. The site's full-text search index, search_index, had zero rows in production. Not the seven the local seed implied — zero. /api/search returned nothing, the MCP search_projects tool returned nothing, and the RAG fallback path returned nothing, for every query, since March. The local database was fine. Nobody had ever asked production the same question.

The upsert that succeeded and did nothing. Pushing 208 vectors into a freshly created Vectorize index returned success: true with a mutation ID. It was silently dropped — the index was still provisioning. vectorCount stayed at 0 for twenty-five minutes across six polls. An identical re-run indexed in forty seconds. A mutation acknowledgement is not indexing.

The IDs that collided. The function generating chunk IDs truncated from the left at 56 characters, so all four sections of one long essay produced the same ID and the upsert kept only the last. Three quarters of that essay was simply absent from retrieval. The build was green the entire time.

The citations pointing at a route that did not exist. The RAG endpoint's fallback path emitted /playbook/<slug> as its source URL. The live API had been handing out broken citations to anyone who asked.

Four bugs. Four different subsystems. Zero of them visible in a diff, a build log, or a page that looked fine.

What the evals caught that reading the code would not have

The instrument is boring and that is the point. Thirty-three questions, six categories — factual, philosophical, synthesis, adversarial, voice, edge cases — each with an expected answer and a scoring rubric. A full run takes about four minutes.

First run scored 54.5%. Eighteen of thirty-three, on a system I had been telling people was working. The full climb is written up in an eval journey from 54% to 100%, but three findings from it are the ones that answer this essay's question.

Roughly 30% of what I first scored as model failures were failures in my own scorer. One check looked for a project slug when the model had correctly used the full project title. Another looked for the word "won" in an answer about awards, so a correct refusal — "no awards won" — scored as a fabrication. I was testing the tester. Fix the ruler before you fix the thing being measured.

The last stuck question was not solved by a bigger model. Every model I tried — Llama 3.3 70B, Qwen3 30B, Mistral Small 3.1 24B — answered a project-year question with an inferred date range, because the narrative text mentioned a later year. Re-embedding did not fix it. Editing the source did not fix it. One sentence added to the system prompt did: state only the Year from metadata; don't infer date ranges from narrative text. The fix was a constraint, not compute.

And the suite is now a regression guard, which is the part that compounds. Run 8 went out after four simultaneously risky changes — an API token roll, a Vectorize index recreation, a remote database migration, and eighteen essays newly embedded. It scored 38 of 39, with categories 1 through 8 at 33/33: zero regression. Without the suite, the honest report on that deploy would have been "seems fine."

None of that required me to write a line of code. All of it required me to define what correct meant, in writing, before the machine had a chance to be plausible.

Write the eval before you write the feature, not after

The workflow that makes this durable is one ordering decision.

Before the agent builds anything I care about, I write down how I will know it worked — as a question with an expected answer, or a check that exits non-zero. Then the agent builds. Then the check runs against the deployed thing, not the local one.

The same discipline generalises past evals. This site has a build gate — a script with seven check groups that refuses to build if the site makes a claim the backend cannot support: if a documented MCP tool name has no matching identifier in the worker source, if the two copies of the identity schema disagree, if a page links somewhere that does not resolve. On its first run it reported seventeen failures. Four were real. Thirteen were bugs in the checker itself.

Fixing those thirteen mattered more than fixing the four. A checker that cries wolf gets disabled, and a disabled checker reintroduces exactly the problem it exists to prevent.

The broader version of this is what I have written about as the encoding crisis: agents cannot absorb tacit knowledge, so the standards have to be explicit, machine-readable and present at runtime. An eval suite is that idea applied to your own output instead of the agent's. So is a CLAUDE.md that carries your actual rules rather than your aspirations. So is KERN, the rules schema I published for encoding those corrections in a form an agent evaluates before it acts.

What a non-developer still cannot do

The honest limits, because a claim without them is marketing. The hands in that picture do not meet, and that is the accurate version of this working relationship — not a merger, a handover across a gap that never quite closes.

I cannot pass an unassisted live coding screen, and I don't pretend otherwise. Given a whiteboard and no tools, I would not produce the migration I shipped last week.

I am slower than a senior engineer at reading an unfamiliar stack trace, and I lean on the agent to narrate it. When the agent is wrong about the cause, the loop is longer than it would be for someone who has debugged that class of failure a hundred times.

I depend on the tools existing and staying good. That is a real exposure. It is also, increasingly, everyone's exposure.

And the failure mode specific to this way of working is overconfidence in a green result. Exit 0 is the most dangerous output in the toolchain, because it looks like an answer. Four of the bugs above shipped under a green build.

The trade is straightforward: I gave up the ability to write it myself and bought the obligation to prove it works. On balance, for the systems I build, that has been the better half of the trade.

Isn't this just vibe coding?

No, and the difference is one artefact.

Vibe coding is prompt, accept, ship, move on. It works, and it never compounds, because nothing carries between sessions and nothing scores the output. Every session starts at zero and every claim about quality is a feeling.

The thing that separates the two is not talent or prompt technique. It is whether there is a number you can re-run. Thirty-three questions and a scorer is not a sophisticated instrument. It is just an instrument, and having a crude one puts you in a different category from having none.

If you can't score the output, you're vibing. Vibing doesn't compound.

So: can a non-developer build production AI systems with agentic coding? Yes, on one condition, and the condition is the whole answer.

You do not have to be the person who writes the code. You do have to be the person who defines what correct means, writes it down before the machine gets a vote, and runs it against the deployed system rather than the one on your laptop. Agentic coding removed the syntax gate. It did not remove the verification gate, and it never will — that one was always the actual job.

Everything on this site that works, works because something other than my opinion says so. The proof was never the code. It was the eval you can re-run in front of someone.

Originally published at narenkatakam.com.