Gemma 4 GPU decode is ~1.8-2.9x slower than llama.cpp (kernel efficiency + low-ctx clock cliff)

[pekkah]

✅ RESOLVED (2026-06-14)

Decode reached ~70.5 t/s (E4B Q8, RTX 4070 Ti) — greedy 71.5 vs llama.cpp 76.6 = 1.07×, within the "≤1.3× and flat" acceptance — via the top-k-first penalty-aware sampler (#154, +44%), dp4a/Q8_1 decode matvec (#143), and default-on CUDA graphs (#139). Problem 1 (the "low-ctx clock cliff", 27 t/s at depth 0) was diagnosed as a --verbose-prompt full-vocab-LINQ-sort bench artifact, not a GPU stall — real matvecs run at ~90% HBM. Steady-state decode is bandwidth-bound at the ~504 GB/s ÷ 8 GB-per-token ceiling, so there is no structural lever left. Closing.

---

Finding

Gemma 4 GPU decode is ~1.8–2.9× slower than llama.cpp, and unlike llama.cpp it degrades
at low context.

Benchmark (RTX 4070 Ti, gemma-4-E4B-it-Q8_0, all layers on GPU):

Decode	llama.cpp b9529	SharpInference	gap
near-zero ctx (tg128 @ d0)	78.5 t/s	~27 t/s	~2.9×
~1K ctx (tg128 @ d1024)	77.7 t/s	~42–45 t/s	~1.8×

llama.cpp holds ~78 t/s flat from depth 0 → 1024. Ours is faster at 1K (42) than at
near-zero (27) — backwards.

Two distinct problems

1. Low-context under-utilization / clock cliff. Our near-zero-ctx decode (27) is slower
   than our 1K decode (42) because the GPU never reaches boost clock — the per-token work is a
   long chain of tiny, serialized kernels with idle gaps, so power draw stays low. llama.cpp
   keeps the GPU busy enough to boost even at depth 0 (78 flat). This is a symptom of an
   inefficient/launch-gappy decode, not an inherent GPU limit. (The merged CUDA-graph path
   targets exactly these launch gaps but the per-token SetParams overhead makes it −9.7% at
   short ctx; the device-side-position follow-up CUDA Graph decode: device-side position to remove short-context regression (follow-up to #136) #140 is the only graph variant that might
   actually help here — but it's secondary to the kernel work below.)

2. Steady-state decode is ~1.8× behind even when boosted (1K: 42–45 vs 78). Likely the
   attention and matvec kernels: ours uses a per-head scalar attention kernel + shared-memory
   scores; llama.cpp uses fused flash-attention and tuned dequant-dot matvecs. This is the
   structural gap.

Direction

• Profile a single decode token vs llama.cpp (kernel-level) to attribute the 1.8× — attention
  vs QKV/FFN matvec vs launch overhead.
• Likely wins: a fused flash-attention-style kernel (one launch/​layer instead of score+softmax+
  AV), and tighter Q8_0 dequant-dot matvecs. These also keep the GPU busier → fix problem (1)'s
  clock cliff as a side effect.

Acceptance

• Decode within ~1.3× of llama.cpp and flat across context (no low-ctx cliff).

Scope note

Per "don't optimize paths that won't reach competitive speed": this issue is the structural
decode work. The CUDA-graph micro-opt (#140) is deprioritized behind it — it can't close a
1.8× gap on its own.

[pekkah]

Progress — PR #143 landed the dp4a decode matvec.

Decode went 49 → 59 t/s on the RTX 4070 Ti (Gemma 4 E4B Q8_0): dp4a-based Q8_0 decode matvec + CUDA graphs default-on.

Finding: decode is matvec-bound, not attention-bound at low ctx, and is BW-bound rather than launch-bound after the #136/#137 graph work — so further graph micro-opt (#140) is deprioritized. Gap to llama.cpp (~78 t/s) remains; next levers are the decode kernels themselves (e.g. dropping the Q8_0 qs-misalignment funnelshift, MMQ-style for decode). Keeping open.

[pekkah]

Diagnosis update + #154 (sampler) landed

Re-measured on RTX 4070 Ti with gemma-4-E4B-it-Q8_0.gguf (llama.cpp tg128 = 76.6 t/s on the same box). Two of this issue's premises turned out to be wrong, and most of the apparent gap was a benchmark artifact + the CPU sampler, not the CUDA kernels.

The "1.8–2.9×" was inflated. scripts/bench-textgen.ps1 hard-coded --verbose-prompt, whose per-token DecodeLoop debug log does a full-vocab (262144) LINQ OrderByDescending every token (~1.5–2.5 ms). Removing it (and fixing the sampler, below) brings the real numbers to:

• Recommended args (--temp 1.0 --top-k 64 --top-p 0.95): 48.7 → 70.3 t/s after perf(sampler): top-k-first fast path — Gemma 4 E4B decode 48.7→70.3 t/s (#142) #154.
• Greedy: 71.5 t/s vs llama.cpp 76.6 → the real gap is ~1.07× (7%), not 1.8–2.9×.

"Low-ctx clock cliff" — disproven. nvidia-smi during sustained decode shows the GPU holds 2865 MHz (92% of the 3120 boost), 97–99% util, 180/285 W, 54 °C. No clock/power cliff.

"Kernel efficiency" — the big matvecs are already near-optimal. New CudaDecodeMatvecRooflineProbe (cold/L2-defeated) shows the FFN matvecs (49% of decode) run at 88–91% of the ~504 GB/s HBM peak. Decode is HBM-bandwidth-bound; the matvec kernels have little headroom. (qkv/o-proj are smaller at 72–80%, ~20% of decode.) SHARPI_DECODE_REGIONS=1 profiler: the 42-layer device region is 95% of forward time; PLE prepass and logits download are negligible.

What #154 fixed (the actual recommended-args bottleneck): Sampler.Sample sorted a 262144-element array with a delegate comparator + 2 MB alloc every token in ApplyTopP (the CLI default --rep-penalty 1.1 routed through it). Replaced with a top-k-first fast path (one vocab pass → softmax/top-p over ≤64 candidates, penalty folded in by over-selection). Recommended-args decode now meets the ≥70 t/s target.

Remaining ~7% greedy gap (71.5 → 76.6). Since decode is BW-bound with the matvecs at ~90%, the levers are not the big kernels:

• Per-token small-kernel tails (the smaller matvecs at 72–80% + the many norm/rope/scale/add launches) — kernel fusion, overlaps with Gemma 4 CUDA perf: collapse per-token kernel launches + enable batched prefill #136/CUDA Graph decode: device-side position to remove short-context regression (follow-up to #136) #140.
• Fewer weight bytes per token — self-speculative decode (EAGLE-style, Research: EAGLE-3 self-speculative decoding for non-MTP models #52); E4B has no MTP head, so this is a larger effort.

Suggest retitling this issue to drop the "clock cliff" framing and rescoping it to the ~7% greedy gap via fusion/speculative, now that the recommended-args target is met.