perf(cuda/kv): validate and enable quantized KV cache modes on CUDA

[inureyes]

Part of #623.

Context

Quantized KV cache modes exist and are wired through both CLI and server: KVCacheMode = Fp16 (default), Int8, Turbo4Asym (fp16 K + 4-bit V), Turbo4, Turbo3Asym, Turbo4Delegated (src/lib/mlxcel-core/src/cache.rs:153-186; group size default 64 in cache/batch_quant.rs; server config kv_cache_mode/batch_kv_quant at src/server/config.rs:556-557, applied in scheduler.rs:2007; env gates MLXCEL_TURBO4_* in docs/environment-variables.md:171-176).

These were developed and benchmarked on Metal (scripts/bench_m5_kv_int8.sh, M5-era docs). Their CUDA status is unverified: the Turbo paths involve custom SDPA/dequant kernels that may be Metal-only (same situation as paged attention, #634), and no CUDA benchmark exists.

Why it matters on GB10: decode is bandwidth-bound, and at long context the KV read stream rivals or exceeds the weight stream. An 8B model at 32k context reads ~4 GB of fp16 KV per token step (GQA-dependent); Int8/4-bit KV halves/quarters that, directly raising long-context decode tok/s. This is one of the few levers that improves single-stream decode beyond the weight-bandwidth roofline.

Scope

1. Audit which KV quant modes function on CUDA today.
2. Fix or port what is broken (Int8 first, Turbo second).
3. Benchmark and document CUDA guidance.

Implementation plan

1. Audit: grep the Turbo implementation (src/lib/mlxcel-core/src/cache/turbo/, cache.rs quant branches at :3702-3718 for rotating, :2499-2624 update/fetch) for metal_kernel or Metal-only assumptions. Run each mode on GB10 CLI (--kv-cache-mode int8, turbo variants; check the actual flag names in src/commands/generate.rs and server config) on llama-3.1-8b-4bit and record: works / wrong output / crash ("No Metal back-end" abort expected wherever a metal_kernel hides).
2. Int8 path: expected to be plain MLX ops (quantize/dequantize/qmm); if functional, benchmark 8k/32k-context decode vs Fp16 (use perf(bench): long-prompt prefill benchmark coverage and serving TTFT/decode-rate telemetry #624 long-prompt harness with long generations) and measure quality drift (greedy divergence step count + a small perplexity spot check).
3. Turbo paths: where a Metal kernel blocks them, either port via fast::cuda_kernel (pattern: src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp) or explicitly gate them off on CUDA with a clear error instead of an abort. Coordinate with perf(cuda/attn): port the native paged-attention decode kernel to CUDA #634 (a quantized paged-attention CUDA kernel would subsume some of this; do not duplicate).
4. Server integration check: batch_kv_quant applied at scheduler.rs:2007 must interact correctly with prompt-cache donation/adoption (quantized detached caches) and --max-kv-size trimming (the audit note at scheduler.rs:4310-4313 says Turbo modes skip trim; verify Int8 trim on CUDA).
5. Document a CUDA recommendation in docs (e.g. "Int8 KV for >8k contexts on CUDA: +X% decode at 32k, quality delta Y").

Acceptance criteria

• Status matrix (mode x works/output-quality/perf on CUDA) committed to a dated docs/benchmark_results/ note.
• Int8 KV: functional on CUDA CLI and server, measured long-context table (2k/8k/32k) vs Fp16 committed; no crash paths remain for unsupported modes (clean error, not abort).
• At 32k context on llama-3.1-8b-4bit, Int8 KV decode tok/s exceeds Fp16 KV by a measured margin (expected 15%+ from halved KV traffic; record actual).
• Prompt-cache adopt/donate and max-kv-size trim work under Int8 on CUDA (integration test or scripted E2E).
• cargo test --features cuda passes.

Validation

cargo build --release --features cuda
./target/release/mlxcel-bench-decode --model ./models/llama-3.1-8b-4bit --prompt-tokens 32768 --max-tokens 256                    # fp16 KV
./target/release/mlxcel-bench-decode --model ./models/llama-3.1-8b-4bit --prompt-tokens 32768 --max-tokens 256 --kv-cache-mode int8

References

• Modes: src/lib/mlxcel-core/src/cache.rs:153-186; turbo dir src/lib/mlxcel-core/src/cache/turbo/; batch quant cache/batch_quant.rs.
• Server plumbing: src/server/config.rs:556-557, src/server/batch/scheduler.rs:2007,4310-4313.
• Metal-era benches: scripts/bench_m5_kv_int8.sh, scripts/bench_kv_cache.sh.

[inureyes]

Chain-run note (2026-07-09): this issue was queued in a batch implementation run on an Apple M1 Ultra session, and was skipped as hardware-blocked rather than partially implemented.

Every acceptance criterion requires executing on CUDA hardware: the mode-by-mode status matrix, the Int8 2k/8k/32k decode table and the 32k margin measurement on llama-3.1-8b-4bit, the prompt-cache adopt/donate and max-kv-size trim checks under Int8, and cargo test --features cuda (the CUDA feature does not even build on macOS). No GB10/CUDA node was reachable from this session (office hosts and the previously used remote node both unreachable), so an attempt here would have produced either an unvalidated port or a status matrix based on guesswork, which the issue's own audit-first plan explicitly rules out.

What a future CUDA-equipped session can pick up directly from the plan: (1) the static audit grep targets (src/lib/mlxcel-core/src/cache/turbo/, cache.rs quant branches) and the expected "No Metal back-end" abort sites, (2) the Int8-first ordering since it is plain MLX ops and most likely functional, (3) the coordination note with #634 so a quantized paged-attention CUDA kernel is not duplicated, and (4) the ready-made validation commands in the Validation section.

The batch run completed #398, #331, #330, #633, and #628 instead; nothing from this issue's scope was landed, so the issue remains fully open.

[inureyes]

PR #717 (status:review). Summary of findings on GB10 (llama-3.1-8b-4bit):

• Root cause fixed: the single-stream Int8 quantizer clamped every per-token scale up to a floor of 1.0, collapsing quantization into round-to-nearest-integer and producing degenerate greedy output on CUDA. Fixed to guard only the exact-zero (all-zero token) case. Int8 greedy output is now byte-identical to Fp16 on the probe prompt.
• Audit: all six modes run on CUDA with no Metal-only abort (Turbo kernels are macOS-gated and fall through to plain-MLX graph paths), so nothing needed porting or hard-gating.
• 32K margin (record-actual): Int8 decode is slower, not faster: 0.74x at 2K, 0.53x at 8K, 0.31x at 32K. The path dequantizes the full window to Fp16 before standard SDPA, so it saves KV memory (32K peak 10.37 -> 8.18 GB) but not decode bandwidth. Int8 KV is a memory lever on CUDA, not a speed lever, until a fused INT8-KV kernel (perf(cuda/attn): port the native paged-attention decode kernel to CUDA #634) exists.
• Server: Int8 chunked prefill/decode and prompt-cache donate/adopt verified E2E (second prefix-sharing request got cached_tokens=256, coherent). The KV-cache-layer Int8 front-trim is proven correct by new unit tests.
• Known limitation (pre-existing, mode-independent): --max-kv-size on the dense batched-decode backend mis-decodes prompts longer than the cap. --kv-cache-mode fp16 --decode-storage-backend dense --max-kv-size N fails identically, and Int8 always uses the dense backend so it always exposes it. A startup warning now flags the Int8 + --max-kv-size combination; a full fix of the dense-decode trim orchestration is a follow-up.