GPU saturation gaps: chunked+packed prefill admission, KV budget, PCIe probe (#183)

[pekkah]

Closes #183.

Gap 1 — chunked / interleaved prefill admission

Admitted prompts prefill in prefillChunkTokens slices (default 256; SHARPI_PREFILL_CHUNK / PrefillChunkTokens; 0 = legacy blocking) with one BatchForwardMulti decode step between chunks. Cancellation honoured between chunks; SnapKV-enabled engines fall back to blocking (eviction needs the whole prompt at startPos==0).

A/B (SharpInference.Bench -- --cb, SmolLM2 CPU, 3 active decoders + 1040-token prompt injected):

	worst decode stall	aggregate
blocking (pre-#183)	25.1 s	8.1 tok/s
chunked+packed (256)	7.9 s	7.1 tok/s

Smaller chunks are a trap on CPU — every MatMulBatched call re-pays weight dequant, so prefill collapses 42→20→13 t/s at chunk 256/64/32. 256 is the default compromise, operator-tunable.

Gap 2 — packed multi-sequence prefill

ForwardPass.PrefillPackedMulti(chunks, startPos, caches, wantLogits): in-flight prompts' chunks run as ONE packed varlen pass — batched GEMMs over the combined token count, per-token RoPE/append/attention against each sequence's own PagedKvCache, no padding, no cross-sequence attention. Logits only for prompt-completing chunks. Same guards as BatchForwardMulti (no MoE/TQ/gemma4).

A/B (--cb --packed, 4×256-token prompts): whole-prompt serial 35.8 tok/s · chunked serial 20.4 · chunked packed 33.7 (+65% vs serial chunks, −6% vs blocking).

Gap 3 — KV token-budget admission backpressure

Each request reserves min(prompt + MaxNewTokens, MaxSeqLen) KV tokens at admission (ForwardPass.KvBytesPerToken); the head request waits when the budget would be exceeded instead of risking OOM. A lone oversized request is always admitted (no deadlock). SHARPI_KV_BUDGET_MB / KvBudgetMb: 0 = auto (half RAM), negative = unlimited.

HardwareProfile: measured PCIe bandwidth

CudaBackend.MeasurePcieBandwidthGBps() (pinned 64 MiB probe, min(H2D, D2H), heuristic fallback) replaces the VRAM-bucket guess in Detect(CudaBackend). 4070 Ti: ~26 GB/s measured vs 20 guessed.

CUDA 13.3 evaluation (no default change)

All pinned entry points exist in 13.3. Qwen3-8B -g -1: prefill 2354→2390 t/s (+1.5%), decode 58.0→57.9 (noise) — decode is HBM-bound in our NVRTC kernels. CudaLibraryResolver (assembly's single DllImportResolver, NVRTC probing folded in) keeps CUDA 12 default, SHARPI_CUDA13=1 opts into the 13 pair, and 13-only installs now load instead of hard-failing.

Test-suite fix found along the way

Gemma4_12B_CudaForward_ProducesCoherentDecode has been failing on master at default settings since #186: #185's auto-narrow makes the 12B tests run with bf16 KV on a 12 GB card, and their synthetic OOD prompt's tiny logit margin lets bf16 rounding tip greedy into a one-token attractor (forced bf16 failed identically at #184 — nothing regressed; CLI bf16 output is coherent). The 12B tests now pin SHARPI_KV_DTYPE=fp32 (they guard trunk math, not dtype noise).

Tests

5 new in ContinuousBatchingTests (chunked parity, packed parity incl. startPos>0 continuation, engine chunked-vs-unchunked greedy equality, tiny-budget completion). Full suite green: Core 134, ForwardPass 481→ all passing after the 12B pin, Pipeline 37, TurboQuant 41, Server 103.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request addresses concurrent load scheduling issues (issue #183) by introducing chunked/interleaved prefill, packed multi-sequence prefill, and KV token-budget admission backpressure to the continuous batching engine. It also adds dynamic CUDA 12/13 library resolution and a real PCIe bandwidth measurement probe. Feedback on the changes highlights three critical improvements: disabling continuous batching for Gemma 4 models to prevent a guaranteed crash due to unsupported per-layer head dimensions in packed prefill, wrapping the main batcher loop in a try-finally block to ensure resources are properly drained and native memory leaks are avoided on unexpected exceptions, and optimizing the CUDA library resolver to load DLLs lazily rather than eagerly loading both CUDA 12 and 13 pairs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Continuous batching is currently enabled for Gemma 4 models on CPU, but ForwardPass.PrefillPackedMulti explicitly throws a NotSupportedException when _layerHeadDim is not null (which is the case for Gemma 4 models due to their per-layer head dimensions). This will cause a guaranteed crash and silent hang of the background batcher thread on the very first request.

To prevent this, we should disable continuous batching for Gemma 4 models and let them fall back to the single-user InferenceEngine (which is fully supported) by checking hp.LayerHeadDim == null in the admission condition on line 94:

if (opts.MaxBatchSize > 1 && batchingSupported && hp.LayerHeadDim == null && fwd is ForwardPass cpuFwd)

[gemini-code-assist]

The main while (!_disposed) loop in BatcherLoop is not wrapped in a try-finally block. If any unexpected exception is thrown inside the loop (e.g., from BatchForwardMulti or RunPrefillStep due to CUDA out-of-memory, transient GPU errors, or bugs in custom kernels), the background thread will die silently.

Because the "Drain" phase is located outside the loop and is not protected by a finally block, it will be bypassed entirely. This leads to:

1. Permanent silent hangs: Active and prefilling requests' output channels are never completed, causing all active clients to hang indefinitely.
2. Native memory leaks: The PagedKvCache of all active and prefilling requests are never disposed, leaking native memory.

We should wrap the entire loop in a try-finally block to guarantee that the Drain phase is always executed on failure:

try
{
    while (!_disposed)
    {
        // ... loop body ...
    }
}
finally
{
    // Drain: complete everything still in flight ...
}

[gemini-code-assist]

The current implementation of DecideRuntimeMajor eagerly calls CanLoadPair for both CUDA 12 and CUDA 13. Since NativeLibrary.TryLoad actually loads the libraries into the process memory and increments their reference counts, this causes the unused version's heavy DLLs (like cublas, which is hundreds of megabytes) to be loaded and remain in memory forever.

This wastes significant system memory and can potentially cause runtime conflicts between different CUDA versions. We should optimize the detection flow to only load the preferred version first, and only fall back to the other version if the preferred one is not available.

        bool prefer13 = Environment.GetEnvironmentVariable("SHARPI_CUDA13") == "1";
        if (prefer13 && CanLoadPair("cudart64_13", "cublas64_13"))
        {
            major = 13;
        }
        else if (CanLoadPair("cudart64_12", "cublas64_12"))
        {
            major = 12;
        }
        else if (CanLoadPair("cudart64_13", "cublas64_13"))
        {
            major = 13;
        }
        else
        {
            major = 12;
        }

[pekkah]

Addressed all three review findings in 2bf8f98:

1. Gemma 4 + continuous batching — gated in BuildForwardPass (batchOk now requires hp.LayerHeadDim is null), falling back to the single-user InferenceEngine. Note the failure mode was a clean per-request NotSupportedException (prefill exceptions are caught and completed onto the request channel), not a batcher hang — but every request failing made the engine useless for gemma4, so the gate is the right call.
2. Batcher loop resilience — BatcherLoop is now try/catch/finally: an exception the per-request handlers didn't isolate fails every in-flight request with that exception via the finally drain (no hung callers, no leaked caches).
3. Eager dual-version DLL loading — DecideRuntimeMajor probes the preferred pair first, only falls back when absent, and frees probe handles, so the unused CUDA major is never left mapped.

Follow-ups filed from this work: #189 (dequant-once MatMulBatched cache → smaller prefill chunks), #190 (continuous batching on the CUDA backend), #191 (narrowed-KV decode-coherence coverage, refs #166).