feat(cuda): continuous batching on the CUDA backend (per-sequence GPU KV caches + batched decode + packed prefill)

[pekkah]

Context

Issue #183 / PR #188 closed the scheduling gaps in ContinuousBatchingEngine — but the engine remains CPU-only: InferenceEngineLoader enables it only when the forward pass is the CPU ForwardPass (batchingSupported is false on every GPU/hybrid path). CudaForwardPass has no CreateCache / PrefillWithCache / BatchForwardMulti / PrefillPackedMulti equivalents — multi-user serving on the GPU falls back to the serialized single-user InferenceEngine.

This is the real remaining "GPU eating air" gap behind #183's theme: a 4070 Ti decoding one Qwen3-8B stream at ~75 t/s is using a fraction of its matvec bandwidth; batched decode amortizes weight reads N× exactly as the CPU BatchForwardMulti does.

Scope (large — on the order of #124/#136)

1. Per-sequence GPU KV caches — today _gpuKCache[layer] is a single [maxSeqLen, kvDim] allocation owned by the forward pass. Needs a per-sequence cache object (paged or slab) + VRAM-budgeted allocation, which dovetails with the q8_0/bf16 KV follow-ups: auto-narrow default + Tc/half2-flash q8 thunks (#179) #185 KV-budget machinery and PR GPU saturation gaps: chunked+packed prefill admission, KV budget, PCIe probe (#183) #188's admission backpressure (KvBytesPerToken analogue for the GPU dtypes fp32/bf16/q8_0).
2. Batched decode kernels — N tokens, N cache pointers per layer: batched matvec is the easy half (cuBLAS GEMM at N>1); attention needs per-sequence (cache, position) indirection (block-table or pointer-array kernel signature).
3. Packed multi-prompt prefill — the chunked batched-prefill trunk (Gemma 4 CUDA perf: collapse per-token kernel launches + enable batched prefill #136, PrefillBatchChunk=4096) already exists for one sequence; a cu_seqlens-style packed variant mirrors ForwardPass.PrefillPackedMulti from PR GPU saturation gaps: chunked+packed prefill admission, KV budget, PCIe probe (#183) #188.
4. Engine generalization — ContinuousBatchingEngine takes the concrete ForwardPass; needs an interface (e.g. IBatchedForwardPass) so the CUDA implementation can slot in, plus the loader gate.

Suggested order: 1 → 2 (decode batching alone is most of the throughput win) → 4 → 3.

Constraints

• CUDA graphs (Gemma 4 CUDA perf: collapse per-token kernel launches + enable batched prefill #136) assume a single cache layout — batched decode either bypasses graphs initially or captures per batch-size.
• Gemma 4 per-layer head_dim / k_eq_v stays out of scope initially (same exclusion as the CPU engine, see PR GPU saturation gaps: chunked+packed prefill admission, KV budget, PCIe probe (#183) #188 loader gate).
• SnapKV/TurboQuant composition out of scope.

[pekkah]

PR #192 lands the core of this issue for the dense path:

• ✅ Scope 1 — per-sequence GPU KV caches (CudaSequenceKvCache)
• ✅ Scope 2 — batched decode (BatchForwardMulti: batched GEMM-N trunk + per-sequence attention); 75 → 99 t/s (N=4) / 106 t/s (N=8) on Qwen3-8B Q4_K_M @ 4070 Ti
• ✅ Scope 4 — engine generalization (IBatchedForwardPass impl + SupportsContinuousBatching loader gate)
• ◻️ Scope 3 — packed multi-prompt prefill is not done (PrefillPackedMulti is a per-sequence loop today) → tracked in feat(cuda): packed multi-prompt prefill for continuous batching — PrefillPackedMulti is sequential (#190 follow-up) #193

Follow-ups split out:

• feat(cuda): packed multi-prompt prefill for continuous batching — PrefillPackedMulti is sequential (#190 follow-up) #193 — packed multi-prompt prefill (cu_seqlens) [scope item 3]
• perf(cuda): weight-stationary batched-decode matmul — amortize weight HBM reads across the batch (#190 follow-up) #194 — weight-stationary batched-decode matmul (GEMM-N only gets L2 reuse, not weight-HBM amortization — the lever past ~1.4×)
• feat(cuda): Gemma 4 continuous batching — per-layer head_dim / SWA rings / shared-KV / PLE (#190 follow-up) #195 — Gemma 4 batching (per-layer head_dim / SWA rings / shared-KV / PLE; deferred here)
• feat(cuda): SnapKV-aware continuous batching — long-context batched serving silently falls back to single-user (#190 follow-up) #196 — SnapKV-aware batching (auto-SnapKV at ctx ≳ 2048 silently disables batching; out of scope here)

This issue can close once #193 lands (or stay open as the umbrella for #193–#196).