llama.cpp 服务端并发Slot机制与503处理

dev.to · AI#llama.cpp#本地部署#LLM推理

Editor brief · 编辑速览

深入解析llama-server在高并发下的Slot排队机制，详解如何通过slots端点的fail_on_no_slot参数主动返回503而非拒绝请求。

文章思维导图

Knowledge map

拖拽缩放

Full translation

完整中文译文

The server is running, the model is loaded, and under a burst of traffic some requests come back refused instead of slow. The phrase people search for is "slot unavailable", and it is not a bug — it is llama-server telling you that its fixed pool of concurrent sequences is full.

Where the string comes from

The exact text depends on which endpoint you hit and which build you are on, which is why searching for it is confusing. The one that is documented and stable is the slots endpoint. Query it with the fail-on-no-slot flag and it answers with HTTP 503 rather than 200:

curl -s -o /dev/null -w '%{http_code}\n' \
  'http://127.0.0.1:8080/slots?fail_on_no_slot=1'
# 503

curl -s 'http://127.0.0.1:8080/health'
# {"status": "no slot available", "slots_idle": 0, "slots_processing": 32}

The llama.cpp server README documents the 503 behaviour for the slots endpoint. Older builds returned the same body from the health endpoint, and older builds still would refuse a completion outright rather than queue it — that refusal is what issue 3821 in the llama.cpp tracker is about, and it is where the phrasing most people quote originally came from.

The server has since moved from examples/server to tools/server, the default for --parallel changed from a fixed 1 to an automatic value, and continuous batching became the default rather than an opt-in -cb flag. Check llama-server --help on your own binary before trusting any flag list, including this one.

What a slot actually is

A slot is one sequence the server can hold in flight: its own KV cache region, its own position counter, its own sampling state. The number of slots is set by --parallel N (short form -np N), and it is fixed at load time because the KV cache for all of them is allocated up front. Nothing about a slot is created on demand. That is the whole reason a full pool produces a refusal rather than a slower response — there is no memory left to make another one.

The part that surprises people is what -c means once --parallel is above one. In the classic non-unified layout, the context size you pass is the total across slots, and each slot gets n_ctx / n_parallel. Run --ctx-size 32768 --parallel 4 and every request is capped at 8,192 tokens, not 32,768. A reader who raises --parallel to clear this error and then starts seeing prompts truncated has found that division, not a second bug.

Newer builds add --kv-unified (-kvu), a single shared KV buffer across sequences rather than one region per slot. It changes the allocation shape but not the underlying constraint: total KV memory is still bounded, and the bound is still what you are hitting.

Deriving your real concurrency ceiling

You can compute how many slots a machine affords, and the arithmetic is worth doing once because it is the only thing that turns this error into a decision rather than a guess. The KV cache stores a key and a value vector per layer per token. With grouped-query attention the count that matters is the number of key/value heads, not attention heads:

bytes_per_token = 2            # K and V
                * n_layers
                * n_kv_heads
                * head_dim
                * bytes_per_element

Take Llama 3.1 8B, whose config Meta publishes on its Hugging Face model card (the weights are gated; the config file is not): 32 layers, 8 key/value heads, head dimension 128. At fp16 — two bytes per element, which is llama.cpp's default cache type — that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token, or 128 KiB. Every assumption there is stated; nothing is measured.

So a slot with an 8,192-token window costs about 1.07 GB of KV, and four of them cost about 4.3 GB — before weights, before the compute buffer, before the CUDA context. On a 24 GB card holding an 8B model at Q4 (roughly 4.7 GB of weights), the KV budget is what is left, and dividing it by the per-slot figure gives you the ceiling. Do the same sum for a 70B and the per-token cost falls out differently: more layers, same eight KV heads, so the KV grows more slowly than the weights do.

If that arithmetic is unfamiliar, the general treatment is in the KV cache page; this page is the operational layer under it.

The four fixes, in order of cost

Raise the slot count and the context together. If you want four concurrent requests each with a 8,192-token window, pass --parallel 4 --ctx-size 32768. Raising --parallel alone silently shrinks every window.

Quantize the cache. --cache-type-k q8_0 --cache-type-v q8_0 halves the per-token figure derived above, which doubles the slots the same memory affords. It is a quality trade and it is not free at low bit widths; q8_0 is the conservative setting.

Shorten the windows. Most production prompts do not need the model's full trained context. The KV cost is linear in tokens, so halving the window is exactly a doubling of concurrency.

Stop refusing and start queueing. See below. This is usually the right answer and it is the one nobody reaches for, because the error reads like a capacity problem rather than an admission-control one.

Why a queue in front is usually the answer

Slots are not a throughput knob past a certain point. Every additional in-flight sequence shares the same memory bandwidth, so beyond the point where the batch saturates the device, adding slots lowers per-request tokens per second without raising aggregate throughput much — and it costs KV memory that is now unavailable for longer contexts. There is a genuine sweet spot, it depends on the model and the card, and it is lower than most people assume.

The failure you are looking at is therefore better handled as admission control: hold requests in a bounded queue with a timeout, and return 429 with a Retry-After when the queue itself is full. That gives you one place to reason about latency instead of two, and it stops a traffic burst turning into a wall of refusals that clients will immediately retry and make worse. There is also a reported case in the llama.cpp tracker of a slot getting stuck under concurrent load, so a health check that watches the idle-slot count is worth having regardless.

Once a queue exists, the next question is what to do with the overflow, and "refuse it" is rarely the best available answer. Sending the overflow to a hosted model means one call site that speaks to two backends with different streaming shapes and different token accounting; a gateway such as Multigrid exists to make that one call site rather than two, with the local server as the primary route and a hosted model as the fallback.