The KV cache is not your bottleneck

July 18, 2026 · 8 min

Every serving thread I read last year opened the same way: the KV cache is eating your memory, so quantize it, page it, evict it. All true. But when I finally sat down with a profiler in front of a 70B deployment that was missing its latency budget by 40%, the cache was not where the time went. Roughly a third of a request's wall clock was spent in places nobody writes blog posts about.

This is a walk through what the profile actually said, in the order I found it.

The setup

Single node, eight accelerators, tensor parallel across all eight, continuous batching with a target of 64 concurrent sequences. Prompts skew long — median around 6k tokens, a long tail past 30k — and generations are short, usually under 300 tokens. That shape matters, and I'll come back to it.

The symptom was p99 time-to-first-token drifting past two seconds under load, while p50 sat comfortably at 380ms. A tail problem, not a throughput problem. Those get misdiagnosed constantly, because the first instinct is to look at the thing that is largest rather than the thing that is most variable.

What the profile said

Three findings, in descending order of how much they embarrassed me:

Tokenization ran on the request thread. For a 30k-token prompt that is a real number of milliseconds, and it happened before the request was ever admitted to a batch.
The scheduler recomputed the full block table on every admission decision, not just for the admitted sequence. At 64 concurrent sequences this went quadratic in a way that was invisible at 8.
Prefill and decode shared a queue. One long prefill would sit in front of thirty decode steps that each needed four milliseconds, and every one of those thirty sequences took the hit.

The cache quantization work I had queued up would have addressed none of these. It would have bought memory headroom, which I did not need, at the cost of a small accuracy regression, which I did not want.

The largest line in a profile is rarely the one worth fixing. The one with the widest distribution usually is.

The fix that mattered

Splitting prefill and decode into separate scheduling lanes, with decode given strict priority, moved p99 TTFT from 2.1s to 640ms. No model changes, no kernel changes, about ninety lines of scheduler diff.

# Decode always wins. Prefill fills whatever budget is left.
CHUNK_SIZE = 2048

def next_batch(decode_queue, prefill_queue):
    if decode_queue and decode_budget_remaining():
        return build_decode_batch(decode_queue)
    return build_prefill_batch(prefill_queue, token_budget=CHUNK_SIZE)

The chunking matters as much as the split. Admitting a 30k prompt as one unit reintroduces exactly the head-of-line blocking you just removed; capping each prefill step at a fixed token budget keeps the decode lane's worst-case wait bounded by one chunk instead of one prompt.

What I'd check first next time

Plot the distribution, not the mean. A tail problem and a throughput problem have different fixes and identical dashboards.
Profile at production concurrency. The quadratic scheduler cost simply does not exist at the concurrency most benchmarks run at.
Move everything off the request thread before touching the model. Tokenization, validation, template rendering — all of it is free latency to reclaim.

None of this makes the KV cache unimportant. It made my memory ceiling, and eventually I did quantize it. But it bought throughput, months later, for a different reason — and by then I knew which number I was moving.