perf(cpu,engine): dequant-once multi-input MoE dot kernels ("one GEMM per expert") — routed MoE is dequant-bound (#110 follow-up)

[pekkah]

Summary

Follow-up to #110. The batched routed-MoE in #110 groups prompt tokens by selected expert and reads each expert's weight rows once per chunk, dotting them against every token routing to that expert. This amortized the DRAM weight read to ~1 ms/tok (all 256 experts are hit once per 512-token chunk). But each weight row is still re-dequantized once per token-dot via DispatchDot/DispatchDotQ8K, so the routed MoE is now dequant/compute-bound at ~9.2 ms/tok (≈35% of prefill wall).

Root cause

BatchedRoutedExperts loops, per (used-expert, row), over the M tokens routing to that expert and calls DispatchDotQ8K(weightRow, tokenInputQ8KS, …) M times. The weight row stays hot in cache (read amortized), but the Q3_K nibble/scale unpack is repeated M times. For Carnice's Q3_K routed experts that unpack is a large fraction of each dot.

Proposed work

Add dequant-once, multi-input dot kernels so an expert weight row is unpacked once and dotted against all M of its tokens — the "one GEMM per expert" the original issue called for. The int-domain arithmetic must stay bit-identical to the single-input kernels (unpack the weight row to its int8 + per-sub-block scales once, then do M int8 dots — same accumulation, same scale-multiply order).

• SimdKernels.DotQ3K_Q8KS_NIn(byte* row, byte** q8ksInputs, float* outputs, int n, int cols)
• SimdKernels.DotQ8_0_Q8KS_NIn(...) (lower priority — Q8_0 has little to amortize)
• Optionally Q4_K/Q5_K/Q6_K N-input variants for other models.
• DotQ6K_Q8K_2In (SimdKernels.cs) is the existing 2-input precedent to generalize.

Then rewrite BatchedRoutedExperts Phase A (gate/up) and Phase C (down) to call the N-input kernels per weight row instead of looping DispatchDot.

Byte-parity (mandatory)

The A/B oracle BatchedPrefill_BitwiseMatchesSequential_Carnice (#110) compares batched-vs-sequential prefill bitwise — it will catch any reordering. Keep the per-token top-k reduce order (downPartial reduce) unchanged. See the Q8_KS validation note (#107) and the K/V MatVecDual parity regression note for the failure mode.

Expected impact

Routed MoE is ~35% of prefill wall; amortizing the Q3_K unpack across ~16 tokens/expert (512-chunk) could roughly halve the dequant cost. Best paired with #111 (trunk batching) since the trunk dominates once the MoE shrinks.

Scope (files)

• src/SharpInference.Cpu/SimdKernels.cs (or SimdKernels.*.cs) — N-input dot kernels
• src/SharpInference.Engine/CudaHybridGdnForwardPass.cs — BatchedRoutedExperts Phase A/C
• tests/SharpInference.Tests.ForwardPass/ — unit tests for the N-input kernels vs the single-input reference

[pekkah]

Verification + closing analysis (branch perf/moe-prefill-followups-112-121)

Revisited on an RTX 4070 Ti + Ryzen 9 7900X (Zen 4). The substance of this issue shipped and the remaining headroom is at the bench-noise floor — recommending closure.

What shipped

• DotQ3K_Q8KS_2In / DispatchDot2In / DispatchDotQ8K2In (perf(cpu,engine): dequant-once multi-input MoE dot kernels ("one GEMM per expert") — routed MoE is dequant-bound (#110 follow-up) #112, PR perf(engine,cuda,cpu): GEMM-batch the GDN-hybrid prefill trunk + dequant-once MoE dots (#111, #112) #115).
• DotQ3K_Q8KS_4In + DotQ4K_4In register-tiled quads + DispatchDot4In / DispatchDotQ8K4In (perf(engine,cpu,cuda): remaining GDN-hybrid prefill headroom after #111/#112 (N-input MoE dots + per-position recurrence/SDPA batching) #114-A, PR perf(cpu,engine): register-tiled N-input routed-MoE dots (#114-A) #116).
• BatchedRoutedExperts Phase A/C already tier quad → pair → single, decoding each Q3_K/Q4_K weight row once per tier and dotting against all its tokens.

Profiler evidence (Carnice 35B-A3B, SHARPI_CPU_MOE=1, warm, N=1212, SHARPI_PREFILL_PROFILE=1)

[batched-prefill] N=1212 total=9598ms trunk=2370ms router=902ms routedMoE=6270ms combine=37ms
[moe-subphase] bucket=8ms normQ=32ms phaseA(gate+up)=3779ms silu/reduce=141ms gateQ=59ms phaseC(down)=2249ms

([moe-subphase] is a new diagnostic added on this branch.)

routedMoE is 66% of prefill, and 96% of routedMoE is the int8 dots themselves (phaseA+phaseC = 6028ms). The Q8_KS quantization / bucketing / silu / reduce overhead is only ~3% — so amortizing it further buys nothing.

Why there is no bigger lever here (all bit-parity-preserving options)

1. The dots are compute-bound, not memory-bound. Used-expert weights for a 1212-token chunk total ~13.7 GB read once at ~60 GB/s RAM BW ≈ 0.23 s, vs 6.3 s of compute. So it's the maddubs chains, not the weight stream.
2. Decode is already well-amortized. In DotQ3K_Q8KS_4In the weight unpack (q3u/scales) is shared across the quad; per-input is ~8 irreducible vector ops/sub-block. 4-input already captures ~64% of the available decode amortization. Pushing to 8/16-input yields only ~5-8% prefill (op-count model), with AVX2 register-spill risk — below the ±10% page-cache variance of this path.
3. No VNNI. Confirmed .NET 10 still exposes no Avx512Vnni (vpdpbusd), so no faster int8 dot from managed code. AVX-512BW (512-bit maddubs) is available but Zen 4 double-pumps it, and prior AVX-512 attempts on this path landed at the noise floor.
4. GPU offload is slower on 12 GB, not faster: perf(cuda): batched weighted scatter-reduce kernel for the GPU-SLRU MoE prefill (#121 follow-up) #129 measured GPU-SLRU prefill at 54 t/s vs CPU-MoE's ~126 t/s — PCIe-streaming + tiny-expert (expertDim=512) GPU under-utilization lose to the CPU dots.

Recommendation

Close as done (4-input shipped; further N-input is diminishing). The genuinely higher-leverage MoE work is algorithmic — 2-bit quant (#53), mixed-precision expert cache (#76), predictive prefetch (#50) — which change the fundamental compute/memory rather than shaving the int8 dot. Reopen only if a model with a much higher tokens/expert ratio makes 8/16-input clearly worth the register pressure.