feat: enable Automatic Prefix Caching by default#233
Merged
Conversation
With non-consuming clone-and-pin adoption (#227) a partial match shares prefix blocks without copying or destroying the stored entry, and the real-bytes ledger (#226) makes the capacity cap genuinely bound retention, so the reasons to keep APC opt-in are gone. Without APC a stored prefix is reusable only when fully contained in the new request, which made the default server config unable to reuse KV across requests that share a system prompt but diverge afterwards. Both serve binaries flip --apc-enabled to default true (disable with --apc-enabled=false; the APC_ENABLED env fallback still applies when the flag is absent). Hybrid SSM models keep the automatic runtime disable. Docs updated: turbo-kv-cache.md describes the default paged backend, clone-and-pin adoption, APC partial matching, real-bytes retention accounting, and adds the measured end-to-end footprint table (v0.1.4 vs current default, llama-3.2-1b, M1 Ultra); CONTINUOUS_BATCHING.md aligned. Measured on the new defaults (scripts/bench_memory_footprint.py): 8-way shared-prefix burst peak 1627 MiB vs v0.1.4's 1653 with all 8 requests adopting (29184 cached tokens) and the request phase 3.4x faster; sequential share 1413 vs 1650; multiturn 1301 vs 2104 with 10976 cached tokens; churn retains donated entries (real memory, governed by --prompt-cache-capacity-bytes). cli_help_consistency 10/10, prefix-share parity 7/7 real-model, server lib 3150, cold clippy and fmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #228, the capstone of the memory-footprint issue set (#224, #225, #226, #227).
With non-consuming clone-and-pin adoption (#227) a partial match shares prefix blocks without copying or destroying the stored entry, and the real-bytes ledger (#226) makes
--prompt-cache-capacity-bytesgenuinely bound retention, so the reasons to keep APC opt-in are gone. Without APC, a stored prefix is reusable only when fully contained in the new request, which left the default server config unable to reuse KV across requests that share a system prompt but diverge afterwards (the central finding of the cross-version footprint measurement that produced this issue set).mlxcel serve,mlx-server) flip--apc-enabledto default true. Disable with--apc-enabled=false; theAPC_ENABLEDenv fallback still applies when the flag is absent (the fallback keys on whether clap actually received the flag, not on the default value). Hybrid SSM models keep the automatic runtime disable.turbo-kv-cache.mdnow describes the default paged backend, clone-and-pin adoption, APC partial matching, real-bytes retention accounting, and a measured end-to-end footprint table;CONTINUOUS_BATCHING.mdaligned.Measured (new defaults, no flags, llama-3.2-1b, M1 Ultra, phys-footprint peak during the scenario phase)
v0.1.4 never engaged the prompt cache from the HTTP path, so it is the no-cache floor; the new default matches or beats it on every sharing scenario while also skipping the shared prefill work. The churn growth is donated-entry retention, now real-bytes-accounted and bounded by the capacity cap (validated in #231: eviction triggers).
Validation
cli_help_consistency10/10 (both binaries agree on the new default).mlxcel serveshows cached>0 on burst, seqshare, and multiturn (29184 / 25536 / 10976); shared-prefix scenario footprint at or below the v0.1.4 baseline; scenario walls all at or below v0.1.4 (burst 15.7 s vs 23.0, seqshare 14.0 vs 19.0, multiturn 13.0 vs 14.8); docs carry the measured table.Closes #228