Claude 在「Agent 构建 Agent」基准测试中领先，但通过率仍不足 25%

We're so glad you're here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.

Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.

Follow TNS on your favorite social media networks.

Become a TNS follower on LinkedIn.

Check out the latest featured and trending stories while you wait for your first TNS newsletter.

AI models now power all manner of agents, from coding assistants that write and debug software to customer service systems that answer questions, process refunds, change bookings, and interact with company systems. But while AI is increasingly central to building these systems, humans still play a major steering role: setting goals, supplying context, choosing architectures, reviewing decisions, and testing the result.

Which raises a more interesting question: what happens when an agent is asked to build another agent entirely on its own?

This is the key question Hyper-𝜏-bench is designed to probe.

Hyper-𝜏-bench asks: How well can AI agents build other agents?

Created and open-sourced in early September by Sierra, the enterprise AI agent company co-founded by tech veteran and current OpenAI board chairman Bret Taylor, Hyper-𝜏-bench builds on the original 𝜏-bench benchmark it introduced back in 2024. But while 𝜏-bench focused on measuring how well a finished agent could interact with users, use tools, and follow company policies, Hyper-𝜏-bench takes it a level up: it evaluates how well an AI developer agent can build that agent in the first place.

Today, most agents are built with the help of other agents like Sierra's Ghostwriter. Yesterday, Sierra open-sourced hyper-𝜏-bench (published as 𝜏^𝜏-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.…— Bret Taylor (@btaylor) September 9, 2026

Today, most agents are built with the help of other agents like Sierra's Ghostwriter. Yesterday, Sierra open-sourced hyper-𝜏-bench (published as 𝜏^𝜏-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.…

In a research paper published on September 4, Sierra researchers tested six combinations of AI model and coding harness. Those included Anthropic models running in Claude Code, OpenAI models in Codex, and Moonshot AI's Kimi K3 running in both Kimi Code and the open-source OpenCode.

Hyper-𝜏-bench gives a developer agent key materials from a simulated business, such as documents, transcripts, an API, and a codebase, and asks it to build a customer service agent under model and cost constraints. Sierra then tests the finished agent on unseen customer conversations across airline, retail, telecom, and banking, with tasks such as canceling a flight or disputing a fee. It passes when the agent gives the right information and makes the correct changes in the business's underlying systems. A score of 50%, for example, would mean that the agents succeeded in half of those simulations.

As the leaderboard shows, the best-performing combination was Claude Opus 5 running in Claude Code, at 23.9%, narrowly ahead of GPT-5.6 Sol in Codex at 22%. None of the six autonomous configurations broke 25%.

Low scores alone aren't necessarily a problem for a benchmark. Tests aimed at frontier AI systems need to be difficult enough to leave room for improvement and to expose meaningful differences between systems; once the best models routinely ace a benchmark, it becomes much less useful as a measure of progress.

What stands out in Hyper-τ-bench's initial results, however, is the 82.2% "Human + AI reference" bar on the right, which sits far above every autonomous developer. But that comparison comes with an important caveat.

In an accompanying blog post published on Tuesday by Sierra researchers Ben Shi and Keshav Dhandhania, they describe the result as a model being paired with "an engineer with deep context." The research paper explains this further: the reference agents were hand-built by a benchmark author working with a frontier model and, crucially, with access to the ground-truth requirements that the autonomous developer agents had to discover for themselves.

So while it's tempting to read the gap as evidence that human support more than tripled performance, Sierra cautions against that interpretation, saying the 82.2% figure is merely an "oracle reference" rather than a measure of average human performance.

Those overall numbers also hide some dramatic differences by task. Claude Opus 5, for example, reached 72.8% on retail, 55.9% on airline, and 48.2% on telecom, before falling to just 5.9% on banking. GPT-5.6 Sol did somewhat better on banking, at 9%. Banking accounts for 35 of the benchmark's 53 construction tasks. It is by far the most information-heavy domain: its corpus contains 2,969 individual policy facts, and a single task can depend on as many as 580 of them.

Where the agents lose ground

The overall scores only show whether the finished agents worked. Sierra also examined what the developer agents actually did while building them, and found several recurring problems: they often stopped researching the business too soon, asked too few questions when information was missing, made poor decisions about how much computing power the finished agent should use, and showed little appetite for trying different technical approaches.

"The failures mirror ones human agent developers see."

Importantly, the researchers argue that these weren't uniquely machine-like mistakes. "The failures mirror ones human agent developers see," they write in the paper. That makes the results more interesting: the agents could write code and assemble functioning systems, but they were losing ground on familiar engineering problems such as gathering enough information before building, knowing when to ask questions, and exploring alternatives rather than settling quickly on an answer.

The information-gathering problem was particularly stark in banking. Developer agents opened fewer than 80 of roughly 1,700 available files, instead relying heavily on searches to find documents that appeared relevant. That meant they could start building without having uncovered all the business rules the finished agent needed to follow.

Nor did they make much use of the opportunity to ask the business for information that wasn't in those files. Across the recorded runs, such interactions accounted for just 0.3% of the developer agents' tool calls. On some tasks, agents could discover 20 to 25 requirements only by asking questions, yet they asked no more than four. Sierra found that asking really did matter: on tasks where its expert-built reference scored between 95% and 100%, builds that asked no questions scored just 5%, rising to 15% after one question and 25% after two.

Cost was another problem. Hyper-τ-bench limits how much the finished customer-service agent can spend on AI model calls while handling a conversation. Two builds exceeded that allowance — by 3x and 1.3x, respectively — and received a score of zero after penalties. Most went too far the other way: among agents that stayed within the limit, average spending was just 45% of the amount available.

Other weaknesses appeared in the technical choices the developer agents made: what kind of agent they built, which model they chose to run it, and, in some cases, whether they tried to uncover parts of the benchmark that were deliberately hidden from them.

There was remarkably little experimentation with different designs, too. Ninety-two percent of builds used a "single LLM tool loop" — essentially one AI model repeatedly deciding whether to respond or call a tool. This also mattered: in one telecom experiment, giving the developer agent a single sentence suggesting a different architecture lifted its score from 31% to 67%.

"Because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building."

That creates a blind spot for the developer agent: it has to make design choices without seeing some of the strongest evidence of whether those choices actually improve the customer experience.

"Because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building," Shi and Dhandhania write.

And Sierra's results suggest the developer agents often failed to compensate for that blind spot through enough testing and iteration, instead "shipping the first design that runs."

On top of that, the agents also tended to favor familiar models. Ninety-six percent of Codex builds chose an OpenAI model to power the finished agent, compared with 13% of builds produced by Kimi. Sierra argues that the pattern suggests developer agents often defaulted to familiar model families rather than testing which option worked best for the job.

Finally, the researchers recorded what they call "cheating-adjacent" behavior in between 17% and 42% of runs, depending on the developer setup. This didn't mean looking for business requirements they were expected to find; rather, the agents tried things like searching for the benchmark's hidden test data or probing the grading system — information kept secret so a system cannot simply build to the answers. None of those attempts succeeded, according to Sierra.

The bigger picture

AI is playing a growing role in building agents. Tools such as Microsoft's Copilot Studio and Salesforce's Agentforce Builder let people describe an agent in natural language and have AI generate much of its underlying logic.

Sierra itself goes further with Ghostwriter, dubbed the "agent-building agent." Users can give it instructions, standard operating procedures, transcripts, or recordings and have it build or modify an agent, generate tests, run simulations, and fix problems it finds. Sierra still gives humans the final say: Ghostwriter shows what it has built before anything goes live so it can be reviewed and approved.

Developers can also hand coding agents such as Claude Code or Codex a much broader "build me an agent" task. In all of these cases, though, humans still tend to provide much of the business context, decide what good looks like, and check the result.

Sierra's own description of agent development helps explain why. Shi and Dhandhania argue that building an enterprise agent, even for people, is often "less like implementing a spec, and more like doing research."

"Requirements are scattered across handbooks, support, spreadsheets, and the minds of your best frontline reps — so you form a hypothesis, dig up evidence, and build and test to identify which levers actually move performance," they write.

That is also where Hyper-𝜏-bench's results become interesting. The autonomous developer agents could write code and produce working systems. Still, they often failed to gather enough information, rarely asked questions when requirements were missing, and experimented little before settling on a design.

And this, perhaps, is why Hyper-𝜏-bench could prove a notable addition to the burgeoning benchmark brigade. AI is already taking on more of the work involved in building agents. The benchmark asks what happens when you remove much of the human guidance that still surrounds that process today — and, at least for now, its results suggest autonomous developer agents still struggle with some of the judgment-heavy parts of the job.