feat(server): non-consuming paged adoption so concurrent same-prefix requests share one stored prefix

[inureyes]

Measurement context

Found during a cross-version server memory-footprint measurement (v0.1.4 vs main, 2026-06-12) using scripts/bench_memory_footprint.py (commit 18c88c1). Method: one fresh mlxcel serve process per (config, scenario), fixed HTTP workloads at temperature 0, sampling the process phys-footprint via /usr/bin/footprint at 2.5 s (RSS does not see MLX Metal buffers at all). Model: llama-3.2-1b-4bit on M1 Ultra, --max-batch-size 8. Scenarios: idle, shared_prefix_burst (8 concurrent requests sharing a ~3.7k-token system prompt), seqshare (same prefix, 8 sequential requests), multiturn (one conversation, 8 turns), churn (32 distinct prompts). Run-to-run noise: idle/multiturn within 0.5 percent, burst/seqshare/churn peaks within 10 to 25 percent.

Problem

CacheEntry::take_detached is one-shot: the first adopter consumes the entry and the remaining concurrent siblings cold-prefill until the adopter completes and re-donates. Measured with APC+dense on the 8-way burst: 3664 cached tokens total, which is roughly one adoption out of eight, while the sequential variant of the same workload (seqshare) reaches 25,648 cached tokens.

Opportunity

Paged blocks are refcounted with copy-on-write, so adoption can clone the block table and bump refcounts without consuming the entry. N concurrent requests then share one stored prefix.

Fix sketch

For paged entries, replace the one-shot take with a clone-and-pin operation: the entry stays in the store, each adopter gets its own pins, and pins are released through the existing release paths. The dense path stays one-shot.

Acceptance criteria

• 8 concurrent same-prefix requests: at least 7 adopt (cached>0), and the pool stores the shared prefix once (blocks_live close to a single prefix's blocks).
• Byte-identical outputs vs cold prefill.
• Burst footprint at or below the v0.1.4 baseline on the bench harness.

Depends on the partial-adoption fix (the divergent-suffix case only matters on paged once partial adoption exists).

Related

Filed together from the same measurement: #224, #225, #226, #228.

[inureyes]

Field observation from the #226 measurement that raises this issue's priority.

With --apc-enabled, the one-shot take_detached consume is not just a missed-sharing limitation: combined with #225's partial adoption it actively destroys cache. A request that shares only a short prefix with a stored entry (the ~30-token chat-template preamble every request shares is enough) takes the entry, trims it to the matched pool block, and adopts one block. The remaining thousands of tokens of pinned KV are released. Observed live on llama-3.2-1b: prompt-cache adopt: paged partial adoption trimmed detached set from=3848 to=32, leaving the pool with a single sequence's blocks while the ledger still carried the drained shell.

Consequences while this issue is open: every divergent-prefix request cannibalizes the newest overlapping entry under APC, so cross-conversation traffic keeps cache hit depth near one block. Default APC-off configs are unaffected (whole-containment matches only).

Design note for the fix: with clone-and-pin, a partial match should NOT trim the stored set at all. Clone the first floor(matched_len / block_size) block ids per layer, bump their refcounts, and build the adopted state from the clone; the stored entry stays intact for deeper future matches. The #225 trim path then remains only as the fallback for whatever still consumes sets wholesale.