8.0
热点
AI SCORE
编程提效2026-08-08 08:56
本地推理何时比调用 API 更划算
dev.to · AI#本地部署#成本优化#LLM
Editor brief · 编辑速览
比较本地推理与 API 成本需将硬件折旧、实际利用率（单卡大部分时间空闲）和吞吐量统一折算成每百万 token 成本；结论显示在日均调用量超过某阈值后本地推理更便宜，阈值取决于硬件配置而非硬件本身。
The comparison is usually made badly, because one side is quoted per million tokens and the other as a purchase. Convert the purchase into the same unit and the argument resolves itself — and the deciding term turns out not to be the hardware at all.
An API price is currency per million tokens. To compare, express your own machine the same way. That requires three quantities: what the box costs per hour to own and run, how many tokens it can produce per hour, and what fraction of the hours it is actually producing.
The third one is where nearly every published comparison goes wrong. A GPU that is idle still costs its amortisation, and a machine bought for one developer is idle the overwhelming majority of the time.
One preliminary, because it invalidates the whole exercise if you skip it: the two sides must be serving comparable capability. Comparing a small open model on your own hardware against a frontier API is comparing the price of two different things, and it will always flatter the local side. Either compare against the hosted price of the same open model, or accept that you are making a quality trade as well as a cost one and say so explicitly. The arithmetic below assumes you have already fixed the model.
hourly_cost = amortised_capital_per_hour
+ (power_kw * electricity_per_kwh)
+ operational_per_hour
amortised_capital_per_hour = purchase_price / (life_years * 8760)
tokens_per_hour = sustained_tokens_per_sec * 3600 * utilisation
cost_per_million = hourly_cost / (tokens_per_hour / 1e6)
Every input is something you either know or can measure. sustained_tokens_per_sec comes from running llama-bench or a serving benchmark on the actual hardware with the actual model — not from a published figure. utilisation is generated tokens divided by tokens the box could have generated, over a real week. power_kw is the wall draw under load, which is meaningfully higher than the GPU's own figure once the rest of the system is counted.
Note what is not in the formula: nothing about how good the model is. This computes the price of a token, and a token from a smaller model is not the same good as a token from a larger one. Compare like with like or the exercise is decorative.
Take a single-GPU box serving one mid-size open model, and suppose your measurement gives 30 tokens per second sustained for a single stream. Call the all-in hourly cost H, in whatever currency you computed it in.
one developer, one stream, busy 40% of an 8-hour day
utilisation = 0.40 * (8/24) = 0.133
tokens_per_hour = 30 * 3600 * 0.133 = 14,400
cost_per_million = H / 0.0144 = 69 * H
the same box, batching server, 10 concurrent users,
aggregate 250 tok/s, busy 30% of the whole day
tokens_per_hour = 250 * 3600 * 0.30 = 270,000
cost_per_million = H / 0.270 = 3.7 * H
the same box, saturated batch job, aggregate 400 tok/s, 95% busy
tokens_per_hour = 400 * 3600 * 0.95 = 1,368,000
cost_per_million = H / 1.368 = 0.73 * H
The hardware did not change. The cost per million moved by a factor of ninety-four. Look up a current per-million price for a comparable open model, divide by your H, and you have the threshold: if the quotient is above 69 in the first scenario, self-hosting is cheaper; below it, it is not. For most single-user setups the quotient is not close, and the machine is being justified by something other than cost.
Two structural facts drive that table, and they are worth stating plainly because they invert the usual intuition.
An API bills only for tokens; a box bills for time. Idle API capacity costs nothing. Idle GPU costs full amortisation. All of the economics of self-hosting is the fight to keep the card busy.
Batching is nearly free throughput. Generation is memory-bandwidth bound, so the weights streamed for one token can serve many sequences at once. Going from one stream to ten concurrent streams multiplies aggregate tokens per second by far more than it multiplies cost — which is why a batching server rather than a single-stream runtime is the difference between the second row and the first.
The practical consequence: if you are self-hosting for cost reasons, the first engineering task is not choosing hardware. It is making sure the hardware you have is shared, queued and batched.
Your time. Setup, upgrades, driver breakage, model updates, an on-call path when it dies at 2am. At any professional hourly rate this is frequently the largest term in the model and it is almost never included.
Redundancy. One box is a single point of failure. Costing a production service honestly means costing two, or costing the fallback path.
Peak versus average. Hardware must be sized for peak concurrency; the cost model above divides by average. The larger the ratio, the worse self-hosting looks.
Residual value and obsolescence. The amortisation period is a guess about how long the card stays adequate, and capability per unit of memory has been moving quickly.
On the API side: caching and batch tiers. Discounted asynchronous processing and cached prompt prefixes can move the per-million figure substantially for the workloads that qualify. Compare against the price you would actually pay.
Self-hosting wins on sustained high utilisation — continuous batch processing, an embedding pipeline over a large corpus, a classifier in a hot path — and on any workload where the data may not leave your control, where the cost model is beside the point. It also wins when the hardware exists already and its amortisation is sunk into something else.
An API wins on spiky, low-average traffic, on anything needing frontier capability, and on every early-stage system where you do not yet know your volume — which is nearly all of them.
The sequence that avoids the expensive mistake is to start hosted, instrument the traffic, and re-run this calculation once you have a month of real token counts and a measured utilisation figure. By then the answer is arithmetic rather than argument, and the case for buying hardware — if there is one — will be obvious enough to make without a spreadsheet fight. Buying first and measuring afterwards inverts that, and the machine tends to acquire justifications after the fact.
If the arithmetic lands near the line, there is a middle option worth pricing before you buy anything: the same open weights served by somebody else, where per-million prices for one open model differ noticeably between hosts. That keeps the model choice and drops the utilisation problem. It does not help at all if the reason you were self-hosting was that the data cannot leave — in which case the utilisation fight is simply the job.