多 Agent 系统调试:你的 Trace Tree 在说谎
深入剖析多 Agent 系统中从用户反馈(人名+时间)定位到具体调用链路的方法,讲解 trace 机制设计,对构建 Agent 系统的程序员有直接指导意义。
完整中文译文
Why
When you deploy an agent-based system to production, you discover something uncomfortable: your system works correctly 99% of the time, but when it fails, it fails in ways you don't expect. Not because of bugs in your business logic—but because of control-flow regressions where an agent starts routing to the wrong tool, or tool-safety regressions where a tool returns an unexpected schema that your agent doesn't handle.
Traditional debugging fails here. Logs give you the inputs and outputs, but the trace tree—the actual decision path your agent took—is opaque. You can't set a breakpoint in a live LLM call. And by the time you reproduce the issue locally, the LLM provider may have updated their model, making exact reproduction impossible.
This is the problem Chronicle solves.
Core Concepts
Decision Graphs
Multi-agent systems aren't linear pipelines. They're decision graphs—directed acyclic graphs where each node represents a decision point (an LLM call), and each edge represents a routing choice (tool calls, conditional branches). When a failure occurs, the question isn't "what went wrong?"—it's "which path through the graph led to the failure?"
Recording
Chronicle intercepts the execution flow at each decision point. For each LLM call, it records:
- The model name and parameters
- The full prompt (including system prompt, conversation history, and tool definitions)
- The raw tool calls and their results
from agent_chronicle import record
with record() as ctx:
response = agent.complete("Summarize the latest security advisories")
# Chronicle records: model, params, prompt, tool_calls, results
Cut-Point Replay
Once a failure is recorded, you can replay it deterministically by setting cut points—specific decision points where you want to inject a modified response instead of re-playing the original LLM call:
from agent_chronicle import CutPoint, replay
# Replay the recorded session, but use a fixed response at decision point 3
with replay(ctx.session_id) as session:
session.cut(
at_decision=3,
using=FixedResponse("Here is a safe summary...")
)
result = session.resume()
Cut points let you test "what if the LLM had returned X instead of Y?"—without needing live LLM calls.
Quick Start
Installation
pip install agent-chronicle
Record a Session
from agent_chronicle import AgentChronicle
chronicle = AgentChronicle(
project="my-agent",
api_key=os.environ["CHRONICLE_API_KEY"]
)
# Wrap your agent
agent = chronicle.wrap(my_agent)
# Run normally—Chronicle records in background
result = agent.run("Query the database for active users")
Create a Regression Test
# From a recorded session, generate a regression test
test = chronicle.generate_test(
session_id="session_abc123",
name="test_active_users_query"
)
# Save as a committed test file
test.save("tests/regressions/test_active_users_query.py")
Run Without Live LLM Calls
from agent_chronicle import replay
# Replay a recorded failure
with replay("session_abc123") as session:
# Apply a fix at decision point 2
session.cut(at_decision=2, using=MockResponse(...))
# Verify the fix works
result = session.resume()
assert result.success
Verification
Chronicle's regression tests are deterministic—they replay the exact same prompt/response pairs every time, without calling the LLM. This means:
- Fast feedback: No API latency or cost
- Reproducible: Same result every run
- Shareable: Commit the test and anyone can reproduce the failure
def test_database_tool_timeout_handling():
"""Regression test: agent should retry on tool timeout."""
with replay("session_timeout_001") as session:
# Simulate a timeout at decision point 4
session.cut(
at_decision=4,
using=ToolError("Connection timeout after 30s")
)
result = session.resume()
# Agent should have retried, not failed
assert result.tool_calls[-1].name == "retry"
assert result.success
Compare Runs
Compare two recorded sessions to understand what changed:
diff = chronicle.compare(
baseline="session_old_001",
candidate="session_new_001"
)
# See which decision points differ
for decision in diff.changed_decisions:
print(f"Decision {decision.id}:")
print(f" Old: {decision.old_response[:100]}...")
print(f" New: {decision.new_response[:100]}...")
Recording Architecture
Chronicle uses a layered recording architecture:
┌─────────────────────────────────────────┐
│ Application Layer │
│ (your agent code, tool definitions) │
├─────────────────────────────────────────┤
│ Chronicle Core │
│ ┌──────────┐ ┌──────────────────┐ │
│ │ Recorder │ │ Decision Graph │ │
│ │ │ │ Builder │ │
│ └──────────┘ └──────────────────┘ │
│ ┌──────────┐ ┌──────────────────┐ │
│ │ Cut-Point│ │ Session Store │ │
│ │ Engine │ │ (SQLite/Postgres)│ │
│ └──────────┘ └──────────────────┘ │
├─────────────────────────────────────────┤
│ LLM Provider │
│ (OpenAI, Anthropic, local models) │
└─────────────────────────────────────────┘
The Recorder intercepts all LLM calls and tool executions. The Decision Graph Builder reconstructs the execution path from these recordings. The Cut-Point Engine manages replay with injected responses.
Session Storage
Sessions are stored in a local SQLite database by default, or you can use PostgreSQL for production:
chronicle = AgentChronicle(
project="my-agent",
storage=StorageBackend.postgres(
connection_string=os.environ["DATABASE_URL"]
)
)
Each session captures:
- Decision graph (nodes and edges)
- Full prompt/response pairs
- Tool call inputs and outputs
- Timing information
- Custom metadata
Cut-Point Replay Deep Dive
Cut points are the key to turning recorded failures into actionable tests. A cut point can inject:
Fixed Responses
session.cut(at_decision=3, using=FixedResponse("Fixed output"))
Mock Tool Results
session.cut(
at_decision=2,
using=ToolResult("query_users", {"users": []})
)
Error Injection
session.cut(
at_decision=4,
using=ToolError("Simulated connection refused")
)
Partial Modification
# Only modify the tool_calls, keep other fields
session.cut(
at_decision=3,
using=PartialResponse(
tool_calls=[{"name": "safe_tool", "arguments": {}}]
)
)
Use Cases
Tool-Safety Regressions
When a tool's schema changes (e.g., a field is renamed or removed), Chronicle replays the recorded session and cuts at the decision point to test how your agent handles the new schema:
# Tool schema changed from {user} to {users}
session.cut(
at_decision=2,
using=ToolResult(
"query_users",
{"users": [{"id": 1, "name": "Alice"}]} # New schema
)
)
result = session.resume()
assert result.success # Agent handled the schema change
Routing Regressions
When routing logic changes and breaks existing paths:
# Simulate a new routing condition
session.cut(
at_decision=1,
using=RoutingDecision(next_node="secure_branch")
)
result = session.resume()
Context Window Overflows
Replay with truncated context to test graceful degradation:
session.cut(
at_decision=5,
using=ContextTruncation(max_tokens=2048)
)
FAQ
Q: Does Chronicle work with any LLM provider? A: Yes. Chronicle intercepts at the application layer, so it works with OpenAI, Anthropic, local models, and custom LLM backends.
Q: How is this different from just logging LLM calls? A: Traditional logging gives you inputs and outputs. Chronicle records the full decision graph, enabling cut-point replay and deterministic testing.
Q: Can I use Chronicle with existing test suites? A: Yes. Chronicle generates standard pytest-compatible test files that integrate with your existing CI/CD pipeline.
Q: Is there a size limit on recorded sessions? A: Sessions are stored in your own database (SQLite or PostgreSQL), so storage limits depend on your infrastructure.
Roadmap
- [ ] v0.3: Multi-agent session correlation (trace across agent boundaries)
- [ ] v0.4: Visual decision graph debugger
- [ ] v0.5: Cloud replay service for team collaboration
- [ ] v0.6: Integration with existing observability platforms (Datadog, New Relic)
Get Started
pip install agent-chronicle
Then follow the step-by-step onboarding guide to record your first session and create a committed regression test.
Chronicle is MIT-licensed and maintained by The Agent Plane. Contributions welcome.