docs: unified paged KV cache + disaggregated serving (B3c)#194
Merged
Conversation
Document the epic #116 capstone: extend turbo-kv-cache.md with a unified paged KV cache section (cross-request prefix reuse, copy-on-write, block budget, and measured payoff) and add CONTINUOUS_BATCHING.md covering the batching scheduler, paged decode, and the disaggregated prefill/decode/router serving roles. Benchmark numbers (M1 Ultra, qwen3-0.6b-4bit): a shared 1024-token prefix across 8 requests stores once instead of eight times, saving about 784 MiB; an adopting request skips the shared prefill entirely and decodes byte-identically to a cold run; the live native block-table decode kernel runs 1.9x faster at a 512-token prompt and 10.9x faster at 4096 than the gather reference (batch 4). Paged adopt and donate are documented as supported; no "disabled" limitation note exists to remove. docs/model_tests*.md and docs/en/prompt_cache.md named in the issue do not exist in the current flat docs layout, so the unified-cache content lives in turbo-kv-cache.md. Closes #126
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
B3c, the final sub-step of #126 (epic #116 capstone): benchmarks and documentation for the unified paged KV cache and the disaggregated serving roles. The correctness and parity tasks for #126 landed across the earlier sub-issues and the B-track PRs (#186, #187, #189, #190, #191, #192, #193); this PR adds the measured payoff and brings the docs in line, closing the epic.
What this does
docs/turbo-kv-cache.md: a "Unified paged KV cache" section covering cross-request prefix reuse, copy-on-write block sharing, the--kv-cache-budgetadmission/eviction, and theGET /v1/cache/statssurface, plus a "Measured payoff" subsection with concrete numbers. States that paged adopt and donate are supported.docs/CONTINUOUS_BATCHING.md(new): the continuous-batching scheduler flags, how it uses paged decode, and the disaggregatedprefill/decode/routerroles (topology A) with the--node-role/--serving-bind/--prefill-peers/--decode-peersflags and a three-process localhost example.docs/README.md: index entry for the new doc.Benchmark numbers (M1 Ultra,
models/qwen3-0.6b-4bit)tests/paged_prefix_share_parity.rs).Notes
docs/model_tests*.mdanddocs/en/prompt_cache.mdnamed in the issue do not exist in the current flat docs layout; the unified-cache content lives inturbo-kv-cache.md.Closes #126