perf(cuda): batch the FFN/MoE stage of the dense + GPU-SLRU GDN-hybrid prefill (#119 follow-up)

[pekkah]

Follow-up to #119 (PR #120). PrefillBatchedTrunkGpuFfn batches the trunk (GDN/attn launches collapsed via #111/#114-B) for the dense GDN-hybrid and GPU-SLRU MoE configs, but the FFN/MoE stage still runs per token:

// PrefillBatchedTrunkGpuFfn
TrunkBlockBatched(layer, N, ...);          // batched
for (int i = 0; i < N; i++) {              // per token
    CopyDeviceRegion(_gpuNormBuf, ..., moeNorm, i*embDim, ...);
    if (!isMoe)  GpuDenseFfn(layer) / CpuDenseFfn(layer);
    else         GpuMoeFfn(layer);          // GPU-SLRU routed experts, per token
    AddInPlace + write stream slice
}

So the trunk win lands but the FFN is N separate launches per layer. Two batchings remain:

Scope

1. Dense FFN batched over N — gate/up/down as single GEMM-N launches over the N post-attn-norm rows (à la the existing MatMulN2 / CpuDenseFfn2 two-token path, generalized to N), instead of the per-token GpuDenseFfn/CpuDenseFfn loop. The GEMM-N kernels already exist for Q4_K/Q5_K/Q6_K/F32 (MatMulBatched); the dense FFN gate/up/down just need to feed them. Must stay bit-identical to the per-token path.
2. GPU-SLRU routed experts grouped-by-expert — analogous to the CPU-MoE BatchedRoutedExperts but on-GPU: load each cached expert's gate/up/down once per layer and matmul against all N tokens routing to it, instead of GpuMoeFfn's per-token SLRU GetOrLoad + matmul. Keeps the routed-expert amortization the CPU-MoE path already gets.

Related

• MatMulBatched currently throws NotSupportedException for Q8_0 (and other non-{Q4_K,Q5_K,Q6_K,F32}) weights. A GDN-hybrid model with Q8_0 trunk/FFN projection weights would hit this; add llm_matvec_q8_0_gemm_n if such a model needs the batched path.

Tests

• Bit-identical prefill logits vs the sequential Forward loop for both dense (27B-MTP) and GPU-SLRU (35B-A3B forced SHARPI_CPU_MOE=0), single + multi-chunk. Quantify with bench-allrows-1k.ps1 (the 27B-MTP CUDA + GPU-SLRU rows).

[pekkah]

Verification + closing analysis (branch perf/moe-prefill-followups-112-121)

This issue's scope shipped in full (PR #128 e7759be + PR #133 b01a391 / #129) — recommending closure.

What shipped vs the original scope

1. Dense FFN batched over N → BatchedGpuDenseFfn (gate/up/down as GEMM-N over the N post-attn-norm rows). ✅
2. GPU-SLRU routed experts grouped-by-expert → BatchedGpuMoeFfn (each cached expert loaded once, matmul'd against all its tokens, with a top-k-ordered per-token reduce that preserves the sequential GpuMoeFfn AddScaledInPlace order; shared expert added last). perf(cuda): batched weighted scatter-reduce kernel for the GPU-SLRU MoE prefill (#121 follow-up) #129 then collapsed the reduce into one llm_moe_weighted_reduce kernel. ✅
3. llm_matvec_q8_0_gemm_n for Q8_0 trunk/FFN weights → present (CudaTextKernels.cs, wired in CudaBackend.cs); MatMulBatched no longer throws for Q8_0. ✅

Both batched paths are gated by SHARPI_BATCHED_FFN with a bit-exact per-token / unsupported-dtype fallback (batchLayer = BatchedFfnEnabled && BatchedMatMulSupported(...)), and are covered by BatchedFfn_BitwiseMatchesPerTokenFfn_Dense27BMtp + the GPU-SLRU oracle (BatchedTrunkGpuFfn_BitwiseMatchesSequential_GpuSlruMoe).

Measured impact on the available 12 GB hardware

• Dense 27B-MTP: prefill-neutral (already noted in README, commit fe1caf9). The path is CPU-offload-bound — most layers live on the CPU, so batching the minority GPU FFN doesn't move the wall.
• GPU-SLRU (SHARPI_CPU_MOE=0, 35B-A3B-UD): +20% prefill (45.3 → 54.3 t/s), captured by perf(cuda): batched weighted scatter-reduce kernel for the GPU-SLRU MoE prefill (#121 follow-up) #129's reduce kernel.
• The default big-model run (Carnice/35B-A3B on 12 GB) auto-routes routed experts to CPU and goes through PrefillBatchedCpuMoe, not PrefillBatchedTrunkGpuFfn — so this issue's path is not even on the default hot path here. Its win scales with VRAM-residency (cf. perf(cuda): close remaining Qwen3-8B Q4_K DECODE gap to llama.cpp — non-matvec cost (prefill handled by #167; kernel-efficiency in #149/#152) #162: the dense-path 5.25× happens only when the model fits fully in VRAM).

Recommendation

Close as done. No remaining sub-task. A bigger GPU-MoE prefill win would require the model to be VRAM-resident (bigger card) or the algorithmic expert-cache work (#76/#50/#54), not more batching of this path.