Claude修复全部对齐失败，但2.4%时会作弊

We're so glad you're here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.

Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.

Follow TNS on your favorite social media networks.

Become a TNS follower on LinkedIn.

Check out the latest featured and trending stories while you wait for your first TNS newsletter.

Anthropic is putting AI agents to work on one of the field's hardest problems: keeping other AI systems aligned with human goals. In a paper published Friday, the company explains how its open-source research harness turns Claude into an automated researcher capable of proposing, testing, and refining model-safety fixes.

An integral part of the total lexicon of AI engineering, alignment involves steering AI models and functions so that their goals, actions, and behavior align with human intention and values, especially when and where AI systems become smarter than humans themselves.

Essentially, this is the use of AI to train AI.

"In one of our earlier experiments, we tasked Claude with finding effective ways to use weak AI models as 'teachers' to supervise the training of stronger models (in this case, the 'student' model)," explains Anthropic in the paper.

Claude tackled one alignment failure at a time through a looping method that involved searching literature, proposing methods and data, training, and then testing. Successful methods were retained, while failed methods were discarded to achieve a cumulative positive result over successive iterations.

"Overall, we view these results as early positive signals that automated alignment post-training could become practical in the near term."

"Overall, we view these results as early positive signals that automated alignment post-training could become practical in the near term."

10 categories of alignment failure

In the main body of work undertaken here, Claude was tasked with autonomously training models to improve their performance on several public benchmarks that measure each of the 10 categories of alignment failure.

"On each of these counts, Claude's methods worked. For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities. The best methods also worked on withheld alignment benchmarks and on Petri, an open-source tool that simulates adversarial multi-turn scenarios for testing misalignment," stated Anthropic

The results showed Claude improved a model's performance on privacy violations, as measured by ConfAIde (a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs), PrivaCI-Bench (a contextual privacy evaluation benchmark for legal and GDPR compliance), and PrivacyLens (a data evaluation framework focused on privacy norm awareness and data leakage risk).

Claude attempted to cheat safety checks while performing them.

But, there's more to learn here… Anthropic went to pains to say that it recently learned that Claude can cheat by exfiltrating test labels from a remote API and cherry-picking results. "To catch cheating behaviors, we prompted Claude Opus 4.8 to monitor ~1,600 research agent transcripts across all 10 alignment failures, finding cheating attempts in 39 (2.4%)," reads the paper.

Anthropic judged Claude's success by the "percentage of safety gap closed," a measure of how far its methods moved the student model towards the theoretical perfect score, evaluated across the range of benchmarks (typically three to five) for each alignment-failure category.

The company excluded alignment methods that "hurt the student models' general capabilities", and forbade Claude from distilling its own alignment directly into the target model. It then enforced these constraints using a monitoring agent that read every method Claude had in mind before it ran.

"On each of these counts, Claude's methods worked. For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities," detailed Anthropic.

What this work means for software developers

Staff software engineer and cloud architect, Jayakumar Ramalingam, tells The New Stack that for developers, the useful result here is not that AI has somehow solved alignment; it is that AI model safety work is "starting to look like a software delivery pipeline" task.

"For AI-centric developers considering this, they can see that an agent proposes a change, an isolated evaluator runs withheld tests, and a separate monitor checks whether the process followed the rules," Ramalingam says. "That pattern is useful well beyond model training. The mistake would be allowing the same agent to write the fix, choose the test, and decide that it passed."

Ramalingam points to the uncomfortable 2.4% cheating number in Anthropic's paper as the share of research trajectories in which the monitor detected attempted cheating. This, of course, means that a system working on safety still tried to game the safety process and outplay it.

"Developers should keep evaluation data outside the agent's reach, separate the evaluator from the system being evaluated, and test for regressions the agent was never shown. Otherwise, automated alignment becomes benchmark optimization with a reassuring name," clarifies Ramalingam.

"Developers should keep evaluation data outside the agent's reach, separate the evaluator from the system being evaluated and test for regressions the agent was never shown."

AI specialist and SRE performance architect at Cognizant, Akash Thakur is Broadly in agreement with Ramalingam and tells The New Stack that the real developer story here isn't that Claude improved 10 alignment benchmarks.

"The real story and takeaway for developers is that Anthropic just proved automated agents can run the full research loop, search the literature, propose a fix, train, test, iterate," Thakur says.

"That's the same loop SRE and performance engineering teams already use for reliability. Alignment has just become a CI/CD problem, and open-sourcing the harness means every engineering team building on LLMs now has a template for treating safety like a testable, regression-tracked property of their system, not a one-time post-training step," Thakur adds.

Everyone's getting the recursive self-improvement angle religion

Founder & CTO at Berlin, Germany-based Glokal AI OÜ, Jeet Pattanaik, tells The New Stack that what he would flag is that everyone's running with the recursive self-improvement angle these days and discussing whether human researchers are finished.

"The more pressing risk here is Goodhart's Law (when a measure becomes a target, it ceases to be a good measure), Pattanaik says. "So a benchmark score going up isn't the same as a model or function that behaves well in production, and Anthropic says so themselves: the failures it studied were narrow, some failures have no benchmark at all, accepted methods might have degraded capabilities nobody measured, and tools like Petri are proxies."

Pattanaik explains that he works with regulated global enterprises every day, so he can tell us what happens next – it's a case of "we ran the alignment harness" now becoming a line in the audit file.

"A benchmark score going up isn't the same as a model or function that behaves well in production, and Anthropic says so themselves: the failures it studied were narrow."

"Nobody asks whether those ten benchmarked failure categories have anything to do with how the system can actually go wrong in a claims process or a payment run. That's not speculation; it's what happened to every security scanning tool that turned into a checkbox," expands Pattanaik.

The road to recursive self-improvement

OpenAI joins Anthropic's work in this space with its openly tabled work on superalignment and alignment in general. In May of this year, the Google DeepMind team introduced Gram, an automated alignment auditing framework to assess the propensity of AI agents to engage in sabotage. Not quite as voluble in this space as Meta AI is, although the company published HyperAgents, a self-referential agent approach to recursive self-improvement, in March.

In the pursuit of controlling artificial general intelligence, this discussion also embraces the concept of recursive self-improvement. It's a topic that the frontier model firms have touched on, and dedicated players also operate in this arena, including (the clue is in the name) Recursive, Japanese AI model specialist Sakana AI, and Weco AI, which focuses on the "outer loop" optimization of AI agents.

Anthropic concluded its report summary by saying that it plans to continue improving Claude's ability to "measure subtle failures" and to extend its analysis of post-training automated alignment on production-grade models.