perf(server): decode runs ~11% slower after a prompt-cache hit than after a cold prefill

[inureyes]

Context

While validating the #632 lookahead decode pipeline on GB10 (CUDA 13.0, branch perf/632-scheduler-async-eval-lookahead, 2026-07-10), repeated bench_serving_concurrency.py --concurrency 1 --prompt-tokens 512 --max-tokens 512 runs against one server instance (qwen2.5-0.5b-bf16) are bimodal on decode throughput depending only on whether the synthetic prompt hit the server prompt cache.

Evidence

Cache-miss runs (TTFT ~285-296 ms, full prefill): decode 182.4-185.0 tok/s. Cache-hit runs (TTFT 13-44 ms, prompt-cache: cached=448/461): decode 161.9-164.6 tok/s, i.e. ~11% slower decode for the identical request. The /metrics lookahead counter increments 509 per 512-token run in BOTH modes, so the #632 pipeline engages fully either way; the penalty is not pipeline disengagement. Reproduced 5 times (3+2 samples).

Suspects (unverified)

The restored/donated KV cache's internal layout after a prompt-cache hit (block rounding, fragmentation, or a dense-concat restore that grows differently during decode), per-step cache-append cost differing between a freshly prefilled cache and a restored one, or attention kernel behavior at slightly different internal cache capacities.

Suggested next step

Profile one hit run and one miss run with nsys (--cuda-graph-trace=node) and diff the per-step kernel composition and any reallocation/copy activity in the decode phase; check whether the restored cache path skips a capacity reservation the prefill path performs.

[inureyes]

Not reproduced on Apple Silicon (metal)

I tried to reproduce the ~11% cache-hit decode penalty on metal before touching any code. It does not reproduce here on either decode backend: cache-hit decode stays within 0.5% of cache-miss decode with fully overlapping sample ranges. Details below so the finding is auditable, plus where the restore and prefill paths genuinely differ in the code.

Environment

• Apple M1 Ultra, 128 GB, macOS 26.5.2 (25F84), metal,accelerate release build.
• main at e63dae0b1. PR perf(server): pipeline scheduler decode with lookahead async_eval #729 (1ad7aeedd, pipeline scheduler decode with lookahead async_eval) is an ancestor, so the perf(server): port lookahead async_eval pipelining into BatchScheduler decode #632 pipeline under test is present.
• Model qwen2.5-0.5b-bf16 (same as the report), 471-token synthetic prompt, max_tokens=512, temperature=0 (greedy), streaming /v1/chat/completions.

Methodology

To isolate cache provenance from context length, I alternated MISS/HIT samples with identical geometry rather than restarting the server:

• MISS: a fresh 12-hex-char nonce is prepended to a fixed base prompt. The nonce sits at the start of the user content, so the whole prefix from the nonce onward differs from any previously donated entry, forcing a full prefill (cached=0/471).
• HIT: the identical nonce-prompt is re-sent immediately, so the server adopts the just-donated prefix (cached=448..464/471, matching the report's cached=448/461).

Because the prompt length is identical and greedy decoding makes the continuation identical, both requests decode the same 471 to ~983 token window doing the same work. The only variable is fresh-prefill vs restored cache. Decode throughput is computed client-side exactly as bench_serving_concurrency.py does: (completion_tokens - 1) / (t_last_token - t_first_token). Seven clean sample pairs per run after one discarded warmup pair, 2 s cooldown between requests, machine otherwise idle. The /metrics lookahead counter is sampled before and after every request.

Results

Default backend (--parallel 4, paged decode pool, max_batch_size=4, matching the GB10 default). Two independent runs, decode tok/s:

Run	mode	median	min	max	n
1	MISS	241.9	240.7	242.7	7
1	HIT	241.5	239.9	242.0	7
2	MISS	242.8	240.2	243.6	7
2	HIT	242.6	240.4	243.8	7

HIT vs MISS median delta: +0.2% (run 1) and +0.1% (run 2), i.e. hit is marginally slower than the noise floor, and the sample ranges overlap completely. TTFT tracked hit vs miss as expected (paged run 2: MISS median 93 ms, HIT median 36 ms).

Dense backend (--parallel 1, max_batch_size=1), to also exercise the "dense-concat restore" suspicion. One run, decode tok/s:

mode	median	min	max	n
MISS	254.1	252.7	255.8	7
HIT	253.1	252.7	254.4	7

HIT vs MISS median delta: +0.4%, ranges overlap. TTFT MISS median 103 ms vs HIT median 43 ms.

Lookahead and cache-hit confirmation

mlxcel_batch_decode_lookahead_steps_total incremented by exactly 509 per 512-token request in every mode and run, MISS and HIT alike, reproducing the report's observation that the #632 pipeline engages either way. The prompt-cache: request completed log confirmed cached=0/471 on every miss and cached=448..464/471 on every hit.

Where restore and prefill actually differ (and why metal shows no delta)

• Backend selection: effective_decode_storage_backend (src/server/batch/scheduler.rs:156) sets paged_available = max_batch_size > 1 && supports_batching && supports_paged_decode_backend. With the default --parallel 4, the effective backend for this model is Paged, so the single-concurrency default runs on the paged pool, not the dense cache.
• Paged hit restore floors the match to the 32-token pool block boundary: adoptable = (matched_len.min(paged.seq_len()) / block_size) * block_size (src/server/batch/scheduler.rs:1624; the take-path repeats it at :1727). A fresh prefill fills the same pool in 32-token blocks. So on the default backend a restored cache and a fresh cache are both block-granular at the same block size, and there is no dense-concat / step-rounding geometry difference at all.
• Dense hit restore truncates the detached buffer to matched_len (src/server/batch/scheduler.rs:1700), which is not 256-aligned. But the dense cache grows in a 256-token step (step: 256 at src/lib/mlxcel-core/src/cache.rs:487,:535; grow at update_fp16, n_steps = (step + new_seq_len - 1) / step; buf_size = n_steps * step, cache.rs:866). When matched_len < prompt_len (the partial-hit case here, 448/471), the divergent tail is re-prefilled immediately after adoption, and that re-prefill re-rounds the buffer back up to the same ceil(471/256)*256 = 512 capacity a fresh prefill lands on. So the restored dense cache converges to the same geometry as a fresh one before decode begins. Only an exact full-prefix match (matched_len == seq_len, truncate skipped) keeps the donor's padding, and that path is bit-exact by construction. This is consistent with the +0.4% dense measurement.

Lookahead priming appends and trims one speculative KV position through the same update/trim machinery (record_lookahead_step at src/server/batch/scheduler.rs:5514), orthogonal to the restore path, which is why the counter is identical in both modes.

Conclusion and next step

The penalty did not reproduce on metal on either backend, so the nsys profiling step on GB10 remains the right next action. On the paged default path, block-floored adoption keeps the restored cache block-aligned exactly like a fresh prefill, so the geometry-mismatch suspicions do not hold on Apple Silicon's MLX gather + SDPA attention.

One CUDA-specific hypothesis for the nsys diff to check: after a partial paged adopt the sequence's block table is [block-floored donor blocks shared copy-on-write] + [fresh tail block] + [decode blocks], whose block ids can be non-contiguous, whereas a fresh prefill tends to draw a contiguous run from the pool free list. If the CUDA paged-attention gather kernel is sensitive to block-table contiguity (coalescing / cache locality per gather) while MLX gather + SDPA on metal is not, that would explain an ~11% decode delta visible only on GB10. The suggested nsys --cuda-graph-trace=node diff of per-step gather/copy activity and block-table layout between one hit run and one miss run should confirm or refute this.

No code change is proposed from the metal side; this is a not-reproduced report to keep the investigation honest.