8.0
热点
AI SCORE
编程提效2026-08-25 17:24
150行Python代码：给免费LLM配额装上实时仪表盘
dev.to · AI#LLM#用量监控#Python
Editor brief · 编辑速览
作者在两周内发现免费LLM端点无法追踪用量，自己写了150行logger，一小时内发现三个配额泄露（重试、评估调用等），避免了月内配额提前耗尽。
免费 LLM 配额的 Token 计量器：我那 150 行日志代码一小时内发现的三个漏洞
A free LLM quota is a number without a story, and that is exactly why it will surprise you. I had been running a nightly batch summarization job against MonkeyCode's free model endpoint for two weeks when I realized I could not answer the simplest question in engineering: how many tokens am I burning per day? The provider gives you a monthly allowance and a hope, not a meter, so I built the meter myself in about 150 lines of Python, and within an hour it had found three leaks that would have exhausted my quota before the month ended. This is the story of that meter, the leaks it caught, and the pattern you can copy into any LLM project.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Every free-tier LLM provider gives you a number, and that number is almost always useless in practice. The number says "10 million tokens per month," but it does not tell you how many you used yesterday, how many your retries are burning, or how many your evaluation calls are eating in the background. The provider has a meter somewhere, but it is not exposed to you in real time, and the first time you learn about it is when your requests start failing with a 429 that means "out of quota" instead of "slow down."
My project was a batch summarization pipeline that processed about 200 documents per night, deployed on MonkeyCode's free server option alongside the code that called the model. When I moved it to the free tier, I assumed the math was simple: 200 documents times a few thousand tokens each equals well under the monthly allowance. Was that assumption correct? The meter said no, and the meter was right.
The first piece of the meter is a wrapper that intercepts every LLM call, records the request and response metadata, and writes it to a local SQLite database. The wrapper does not change the behavior of the call; it just observes it, and that observation is what turns a vague quota into a precise accounting system.
# token_meter.py — wrap any OpenAI-compatible client call
import sqlite3
from datetime import datetime, timezone
from functools import wraps
DB_PATH = "token_usage.db"
def init_db():
conn = sqlite3.connect(DB_PATH)
conn.execute("""
CREATE TABLE IF NOT EXISTS usage (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
model TEXT NOT NULL,
prompt_tokens INTEGER NOT NULL,
completion_tokens INTEGER NOT NULL,
total_tokens INTEGER NOT NULL,
status_code INTEGER NOT NULL,
retry_count INTEGER NOT NULL,
endpoint TEXT NOT NULL
)
""")
conn.commit()
return conn
def record_usage(model, prompt_tokens, completion_tokens, status_code, retry_count, endpoint):
conn = init_db()
conn.execute(
"INSERT INTO usage (timestamp, model, prompt_tokens, completion_tokens, total_tokens, status_code, retry_count, endpoint) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
(datetime.now(timezone.utc).isoformat(), model, prompt_tokens, completion_tokens,
prompt_tokens + completion_tokens, status_code, retry_count, endpoint)
)
conn.commit()
conn.close()
def metered(api_call):
"""Decorator that records token usage for any API call function."""
@wraps(api_call)
def wrapper(*args, **kwargs):
retry_count = 0
try:
result = api_call(*args, **kwargs)
usage = result.get("usage", {})
record_usage(
model=kwargs.get("model", "unknown"),
prompt_tokens=usage.get("prompt_tokens", 0),
completion_tokens=usage.get("completion_tokens", 0),
status_code=200,
retry_count=retry_count,
endpoint=kwargs.get("endpoint", "unknown"),
)
return result
except Exception as e:
record_usage(
model=kwargs.get("model", "unknown"),
prompt_tokens=0,
completion_tokens=0,
status_code=getattr(e, "status_code", 0),
retry_count=retry_count,
endpoint=kwargs.get("endpoint", "unknown"),
)
raise
return wrapper
The key design decision is to record every call, including failed ones, because a failed call that triggers a retry will burn tokens twice: once for the original prompt and once for the retry. If you only record successful calls, you are missing exactly the calls that are costing you the most, and the meter will lie to you in the most flattering way possible.
A database of usage records is not a meter; it is a pile of numbers. The meter appears when you query it with questions like "how many tokens per day?" and "which endpoint is the biggest consumer?" and "how much do retries cost?" These three queries are the core of the meter, and they fit on a single screen.
-- Daily token consumption
SELECT date(timestamp) as day, SUM(total_tokens) as tokens
FROM usage
GROUP BY day
ORDER BY day;
-- Retry cost: tokens burned by failed calls
SELECT status_code, COUNT(*) as calls, SUM(total_tokens) as tokens
FROM usage
WHERE status_code != 200
GROUP BY status_code;
-- Top consumers by endpoint
SELECT endpoint, SUM(total_tokens) as tokens, COUNT(*) as calls
FROM usage
GROUP BY endpoint
ORDER BY tokens DESC;
When I ran these queries for the first time, the results were embarrassing in the best possible way. The daily trend was there, the retry cost was there, and the allocation by endpoint was there, and all three told a story I had been too lazy to ask for.
The first query showed a pattern I had not noticed: every 429 response was followed by three or four retries, and each retry re-sent the full prompt, including the system prompt and all few-shot examples. A single rate-limited call that should have cost 2,000 tokens was costing 8,000 or 10,000 tokens by the time the retry succeeded.
The fix was not to eliminate retries; it was to make them cheaper. I moved the system prompt and few-shot examples into a cached prefix, so retries only re-sent the variable part of the prompt. This cut the retry cost by roughly 60%, and it also reduced the latency of every retry because there was less to process. Why had I never measured this before? Because the paid endpoint had made me lazy.
The second query revealed something worse: my evaluation harness was calling the model with the full test set every night, and those calls were not going through the metered wrapper. They were raw client calls in a separate script, and they were consuming about 25% of my monthly allowance without appearing in any log. The meter could not see them, which meant I could not see them, which meant they were free in the only sense that matters: invisible.
The fix was to route every model call through the same wrapper, including evaluation, testing, and one-off experiments. If a call does not go through the meter, it does not count, and if it does not count, it will eventually surprise you. The rule is simple: no meter, no call.
The third query showed that my average prompt size had grown by 40% over two weeks. The cause was a shared system prompt that several team members had appended context to, one line at a time, without ever reviewing the whole thing. Each addition was tiny, and the cumulative effect was invisible until the meter showed the trend.
The fix was a prompt review ritual: every Friday, print the system prompt and ask whether each line is still earning its token cost. The meter made the cost visible, and the review made the prompt smaller. This is not a technical fix; it is a hygiene fix, and it is the kind of thing that only happens when you have a number to argue with.
The last piece of the meter is a predictor that answers the question "at this rate, when will I run out?" The predictor is deliberately simple: take the average daily consumption over the last 7 days, multiply by the days remaining in the month, and compare to the remaining allowance.
# predict.py — estimate quota exhaustion
import sqlite3
from datetime import date, timedelta
def daily_average(days=7):
conn = sqlite3.connect("token_usage.db")
cutoff = (date.today() - timedelta(days=days)).isoformat()
row = conn.execute(
"SELECT AVG(daily) FROM (SELECT date(timestamp) as day, SUM(total_tokens) as daily FROM usage WHERE date(timestamp) >= ? GROUP BY day)",
(cutoff,)
).fetchone()
conn.close()
return row[0] or 0
MONTHLY_QUOTA = 10_000_000 # at time of writing
today = date.today()
days_remaining = 30 - today.day + 1
avg = daily_average(7)
projected = avg * days_remaining
print(f"Daily average (7d): {avg:,.0f} tokens")
print(f"Days remaining: {days_remaining}")
print(f"Projected usage: {projected:,.0f} tokens")
print(f"Quota: {MONTHLY_QUOTA:,.0f} tokens")
print(f"Status: {'OK' if projected < MONTHLY_QUOTA else 'OVER BUDGET'}")
The predictor is not sophisticated, and it does not need to be. Its job is to give you a number that is better than a guess, and a number that is better than a guess is enough to change behavior. Would I have noticed the trend without the predictor? Maybe, but the predictor made it impossible to ignore.
The meter changed my behavior in three concrete ways. First, I stopped treating the free tier as infinite and started treating it as a budget. Second, I found the evaluation leak and routed those calls through the meter, which made them visible and therefore controllable. Third, I started reviewing the system prompt weekly, and the prompt got smaller instead of bigger.
The most surprising result was the retry cost. I had assumed retries were free because they were rare, and the meter proved that rare was not the same as cheap. A 1% retry rate was consuming 8% of my token budget, and that ratio is the kind of thing you cannot see without a meter. The meter is a pattern, not a product, and the pattern is the part that lasts.
If your LLM usage is a few calls per day and you are nowhere near the quota, a token meter is overkill. If you are on a paid tier with a predictable bill, the meter is still useful, but the urgency is lower. And if you cannot modify the code that makes LLM calls, the wrapper approach will not work; you will need a proxy or a network-level logger instead.
The free tier from MonkeyCode is a good place to practice this discipline, because the quota is real and the meter is not provided. The pattern applies to every LLM project, free or paid, and the sooner you build it, the fewer surprises you will have. Build the meter this weekend, run it for a week, and let the numbers tell you what you have been missing.