perf(cuda): decode CUDA graphs for non-Gemma dense Forward (#156 Item B)

[pekkah]

Goal

Generalize the Gemma 4 decode CUDA-graph capture/replay (#136) to the non-Gemma per-token CudaForwardPass.Forward, so dense models (Qwen3-8B Q4_K etc.) get the same launch-overhead reduction. Split out of the #156 umbrella as Work Item B.

Decode is HBM-bandwidth-bound (see #142 roofline notes), so the win is not kernel throughput — it's killing the ~1k host launches/token. Same overhead exists on Qwen3-8B's decode loop.

Background

ForwardGemma4 (CudaForwardPass.cs:~1082) is structured as: token-varying prefix (embed/PLE) → capturable device region (RunGemma4DeviceRegion, the layer loop + final norm/output) → logits download. The graph helpers it uses are already generic (nothing Gemma-specific in CudaBackend):
TryBeginGraphCapture / TryEndGraphCaptureAndInstantiate / LaunchGraphForPosition.

The non-Gemma Forward (:~890) inlines embed + layers + output + download in one method with no capture.

Tasks

• Refactor Forward into: token-varying prefix (embed) → RunDeviceRegion(position) (layer loop + final norm + output projection) → logits download + sync.
• Keep the TQ ring-advance (:~1064) and the SnapKV Q-capture (:~964) outside the captured region (they mutate host/ring state that breaks static topology).
• Wrap RunDeviceRegion with the generic capture helpers; mirror TryRunGemma4DeviceRegionViaGraph (:~1304), reusing the _tqEnabled || _kvEvictedCount > 0 bail.
• Pre-grow any decode scratch (e.g. the dp4a Q8_1 input buffer) before first capture — capture forbids cudaMalloc. Note: the Q4_K matvec Q8_1 scratch currently grows on demand; ensure it's sized before capture.
• Verify bit-identical: graph replay == direct launch on Qwen3-8B (decode N tokens both ways, assert exact logits). A Gemma4CudaGraphParityTests analogue for dense already-exists as a template.
• Measure decode t/s on Qwen3-8B Q4_K (SHARPI_CUDA_GRAPH=0 to bisect); update README + close out in perf(cuda): generalize Gemma 4 batched prefill + flash attn + decode graphs to dense Q4_K models #156.

Risk

Lower than #156 Item A — graph replay is numerically identical to direct launch (no new math). Main cost is the Forward refactor; keep TQ/SnapKV state mutation out of the captured region.

Refs: #156 (umbrella), branch perf/cuda-dense-q4k-batched-prefill-156 (Item A, prerequisite context).

[pekkah]

Done on branch perf/cuda-dense-q4k-batched-prefill-156 (commit e1de163).

Refactor (B1). CudaForwardPass.Forward split into: token-varying embed prefix → RunDeviceRegion(position) (layer loop + final norm + output projection) → TQ ring-advance + logits download/sync. The TQ ring-advance and the per-layer SnapKV Q-capture stay outside the captured region.

Capture wrapper (B2). New TryRunDeviceRegionViaGraph mirrors TryRunGemma4DeviceRegionViaGraph using the generic helpers (TryBeginGraphCapture / TryEndGraphCaptureAndInstantiate / LaunchGraphForPosition). The position-varying ops (RoPE/KvAppend/Attention) already self-register their graph nodes, so nothing model-specific was needed. Bail conditions:

• _tqEnabled — host-synced rotate/compress + ring advance,
• _kvEvictedCount > 0 — compacted cache breaks seqLen == position+1,
• _snapKvCaptureSlot >= 0 — prefill Q-capture writes a host-computed slot offset (dormant in pure decode, so the capture block never enters the graph),
• _isMoE — MoeFfn does a router Download+Synchronize mid-layer (illegal during capture).

Q4_K/Q8_0 dp4a Q8_1 input scratch is pre-grown to max(embDim, intermDim) before the first capture (EnsureQ81Scratch), since DispatchMatVecQ4K grows it on demand via cudaMalloc which capture forbids.

Parity (B3). New Qwen3CudaGraphParityTests (Item-B analogue of Gemma4CudaGraphParityTests): graph replay is bit-identical to direct launch over 8 decode steps on Qwen3-8B Q4_K, plus a SnapKV-configured-but-no-evict guard. Both pass and assert GraphReady so a silent fallback fails the test. Existing Gemma4 graph tests (3), Qwen3 batched-prefill tests (2), and CUDA SnapKV/MoE/TurboQuant tests (23) all still green.

Measurement. Qwen3-8B Q4_K -g -1 decode, same-session A/B (RTX 4070 Ti):

• graphs off (SHARPI_CUDA_GRAPH=0): 65.0 / 65.6 t/s
• graphs on (default): 70.0 / 70.1 t/s → +7%

--no-thinking shares the path (70.0 t/s). --tq is unchanged (65.7 t/s; graphs bail on _tqEnabled as designed). README Qwen3-8B CUDA rows updated.

Closes Item B of #156.

[pekkah]

Done in PR #161 (e1de163). Forward refactored into embed prefix → RunDeviceRegion(position) (layer loop + final norm/output) → download; TQ ring-advance and SnapKV Q-capture kept outside the captured region; decode Q8_1 scratch pre-grown before capture; reuses the generic TryBeginGraphCapture/TryEndGraphCaptureAndInstantiate/LaunchGraphForPosition helpers and the _tqEnabled || _kvEvictedCount > 0 bail.

Bit-identical (graph replay == direct launch) proven by Qwen3CudaGraphParityTests; decode 65 → 70 t/s on Qwen3-8B Q4_K, SHARPI_CUDA_GRAPH=0 bisects. Closing via PR #161.