Skip to content

docs: unified paged KV cache + disaggregated serving (B3c)#194

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-126-b3c-bench-docs
Jun 9, 2026
Merged

docs: unified paged KV cache + disaggregated serving (B3c)#194
inureyes merged 1 commit into
mainfrom
feature/issue-126-b3c-bench-docs

Conversation

@inureyes

@inureyes inureyes commented Jun 9, 2026

Copy link
Copy Markdown
Member

Summary

B3c, the final sub-step of #126 (epic #116 capstone): benchmarks and documentation for the unified paged KV cache and the disaggregated serving roles. The correctness and parity tasks for #126 landed across the earlier sub-issues and the B-track PRs (#186, #187, #189, #190, #191, #192, #193); this PR adds the measured payoff and brings the docs in line, closing the epic.

What this does

  • docs/turbo-kv-cache.md: a "Unified paged KV cache" section covering cross-request prefix reuse, copy-on-write block sharing, the --kv-cache-budget admission/eviction, and the GET /v1/cache/stats surface, plus a "Measured payoff" subsection with concrete numbers. States that paged adopt and donate are supported.
  • docs/CONTINUOUS_BATCHING.md (new): the continuous-batching scheduler flags, how it uses paged decode, and the disaggregated prefill/decode/router roles (topology A) with the --node-role / --serving-bind / --prefill-peers / --decode-peers flags and a three-process localhost example.
  • docs/README.md: index entry for the new doc.

Benchmark numbers (M1 Ultra, models/qwen3-0.6b-4bit)

  • Memory: a shared 1024-token prefix is ~112 MiB of KV; across 8 concurrent requests the pool stores it once instead of eight times, saving ~784 MiB.
  • Prefill avoided: an adopting request skips the shared prefill and decodes byte-identically to a cold run (tests/paged_prefix_share_parity.rs).
  • Decode throughput (batch 4): the live native block-table kernel runs at 276 tok/s (512-token prompt) and 84 tok/s (4096), versus 146 and 7.7 tok/s for the gather reference (1.9x and 10.9x). The gather reference degrades sharply with context, which is why the live path uses the native kernel.

Notes

  • docs/model_tests*.md and docs/en/prompt_cache.md named in the issue do not exist in the current flat docs layout; the unified-cache content lives in turbo-kv-cache.md.
  • The issue's "remove the paged adopt/donate disabled note" task: no such note exists in the repo (it was never written), so the positive "supported" statement is added instead.
  • Docs-only change; no code touched.

Closes #126

Document the epic #116 capstone: extend turbo-kv-cache.md with a unified paged
KV cache section (cross-request prefix reuse, copy-on-write, block budget, and
measured payoff) and add CONTINUOUS_BATCHING.md covering the batching scheduler,
paged decode, and the disaggregated prefill/decode/router serving roles.

Benchmark numbers (M1 Ultra, qwen3-0.6b-4bit): a shared 1024-token prefix across
8 requests stores once instead of eight times, saving about 784 MiB; an adopting
request skips the shared prefill entirely and decodes byte-identically to a cold
run; the live native block-table decode kernel runs 1.9x faster at a 512-token
prompt and 10.9x faster at 4096 than the gather reference (batch 4).

Paged adopt and donate are documented as supported; no "disabled" limitation
note exists to remove. docs/model_tests*.md and docs/en/prompt_cache.md named in
the issue do not exist in the current flat docs layout, so the unified-cache
content lives in turbo-kv-cache.md.

Closes #126
@inureyes
inureyes merged commit c2386c6 into main Jun 9, 2026
5 checks passed
@inureyes
inureyes deleted the feature/issue-126-b3c-bench-docs branch June 9, 2026 15:47
@inureyes inureyes self-assigned this Jun 10, 2026
@inureyes inureyes added status:done Completed type:docs Documentation improvements or additions priority:medium Medium priority labels Jun 10, 2026
@inureyes inureyes added this to the 0.2 milestone Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:medium Medium priority status:done Completed type:docs Documentation improvements or additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

End-to-end validation, parity, benchmarks, and docs

1 participant