我用56个场景评估健身房控制AI Agent开销

After my AI agent spent $2,800 in 60 seconds at 3 AM, I knew I needed pre-flight spend control. But how do you test something that's supposed to catch runaway AI spending before it happens?

The answer: an eval gym with 56 labeled scenarios across 9 categories.

When you're building spend-control logic, you quickly realize that simple rules have subtle edge cases:

A $50 limit with a $50.00 transaction — should it block or approve?

What about $49.99 vs $50.01?

Daily total resets at midnight — but what timezone?

Velocity limits: 5 calls in 5 minutes — is the 6th call blocked or just the 5th?

These edge cases compound across 7 rule types. Here's how the scenarios break down:

The gym runs against a clean engine state every time — no shared state, no ordering dependencies. Every scenario is independent.

The Hardest Edge Cases

Amount exactly at limit: If the limit is $50.00 and the transaction is $50.00, should it be approved? We chose approve — "exceeds" means strictly greater than. This is documented and tested.

Decimal precision: All monetary arithmetic uses Python's Decimal type. Never float. 0.1 + 0.2 is fine for graphics; it's catastrophic for money.

Priority ordering: Rules evaluate in priority order. If rule #1 BLOCKs but rule #2 would APPROVE, the first match wins. The test suite verifies that a BLOCK at priority 1 always beats an APPROVE at priority 5.

The Gym is a Universal Benchmark

The eval gym is useful even if you don't use AgentShield. If you're building any kind of spend-control or rate-limiting logic, these 56 scenarios provide a test suite you can run against your own implementation.

Check the live results: 56/56 across 9 categories. The spec is at agentshield.fly.dev/eval-gym-spec.

I'm working on adding more scenario categories:

Multi-agent shared budget scenarios

Timezone-aware daily reset edge cases

Cascade cost with variable failure rates

The gym is MIT licensed. Use it. Fork it. Add your own scenarios. If you find a bug in the test suite, open an issue — I'll fix it.

Previously: I built a firewall for AI agent spending — here's the architecture

For further actions, you may consider blocking this person and/or reporting abuse