perf(cuda,engine): CPU-MoE mode + expert-aware TierPlanner for CudaHybridForwardPass — auto placement is structurally wrong for pure-attention MoE on 12 GB

[pekkah]

Context

Target: single-user decode/prefill on 12 GB VRAM / 64 GB RAM. The GDN-hybrid class (CudaHybridGdnForwardPass) already has the right low-VRAM MoE configuration: all trunk layers on GPU + routed experts on CPU, with a capacity-driven auto-router (CudaHybridGdnForwardPass.cs:886-909: SLRU capacity ratio < 0.5 → CPU MoE, SHARPI_CPU_MOE override). The #100 verification showed why this matters: on a 4070 Ti 12 GB, CPU MoE is 11.8 t/s vs 6.1 t/s GPU-SLRU on the structurally identical qwen3.6-A3B.

The pure-attention MoE hybrid (CudaHybridForwardPass — Qwen3-Coder-30B-A3B, OLMoE, qwen2moe-class models) has neither piece:

Problem 1 — TierPlanner books all routed experts per layer, but the runtime never uploads them eagerly

TierPlanner.MeasureLayerBytes (TierPlanner.cs:165-180) includes ffn_gate_exps / ffn_up_exps / ffn_down_exps — i.e. every expert — in each MoE layer's byte cost. But CudaHybridForwardPass uploads routed experts lazily via the SLRU slot manager, not per layer (CudaHybridForwardPass.cs:2805-2806: "Routed-expert weights are uploaded lazily by CudaExpertSlotManager on cache miss"; the shared expert and router stay resident).

Consequence of -g -1 (InferenceEngineLoader.cs:209-242, same pattern in RunCommand.cs): the greedy packer charges ~N×expert bytes per layer, stops packing far too early, and trunk layers whose actual eager footprint is tiny (attn q/k/v/o + norms + router + shexp) land on CPU. Each CPU layer then pays CPU attention plus the per-token pinned-buffer boundary crossings in Forward (CudaHybridForwardPass.cs:890-936), while the "saved" VRAM just becomes SLRU headroom for fewer GPU layers.

Problem 2 — no "GPU trunk + CPU routed experts" mode at all

CudaHybridForwardPass only supports a wholesale layer split: GPU layers get GPU attention + SLRU experts, CPU layers get everything on CPU. The config that wins on 12 GB for the GDN class (full GPU trunk, CpuMoeFfn-style routed experts on CPU with the Q8_KS SIMD path) cannot be expressed. SHARPI_CPU_MOE (#93) is only read by CudaHybridGdnForwardPass.

Problem 3 (minor) — KV budget always priced at fp32

TierPlanner.cs:99,124 size KV at sizeof(float) per element even when SHARPI_KV_DTYPE=bf16/q8_0 halves/quarters the real cost, so the auto context/layer trade-off is mispriced for narrowed-KV runs.

Proposed work

1. CPU-MoE mode for CudaHybridForwardPass: mirror the GDN class — per layer, GPU computes attention/norms (and shared expert), downloads the normed hidden, CPU runs the routed experts (reuse the existing CPU MoE core / SimdKernels Q8_KS path), uploads the result. Same SHARPI_CPU_MOE=0|1 override + capacity auto-router (predicted SLRU slots vs total experts, 50% threshold) so behavior matches CudaHybridGdnForwardPass.cs:886-909.
2. TierPlanner MoE-aware accounting: for MoE models, measure layers as trunk-only bytes (attn + norms + router + shexp; exclude ffn_*_exps) and report an explicit expert-cache budget (leftover VRAM after trunk + KV) in LayerPlacement, so the loader can decide trunk split and MoE placement (SLRU vs CPU-MoE) from real numbers instead of phantom expert bytes.
3. KV-dtype-aware pricing in Plan (pass the resolved KV dtype; fp32 default unchanged).

Acceptance

• Qwen3-Coder-30B-A3B (or OLMoE) with -g -1 on a 12 GB profile places all trunk layers on GPU; placement summary shows the expert budget.
• A/B decode + prefill vs today's auto placement on the same model/hardware; CPU-MoE mode beats the current wholesale split (expect a multiple, mirroring the GDN-class 11.8-vs-6.1 t/s data point).
• No placement change for dense models or explicit -g N; existing batched-prefill parity tests stay green.

References

• perf(cuda): llm_matvec_q3_k NVRTC + UploadRaw Q3_K/Q8_0 + TierPlanner byte accounting #100 (capacity threshold + CPU-vs-SLRU bench data), Evaluate NVIDIA Model-Optimizer techniques for low-VRAM (12GB/64GB) perf #104 (TierPlanner places by size only — this is the MoE half), Server: surface SHARPI_CPU_MOE as a SharpInferenceServerOptions field (mirror #80) #93 (SHARPI_CPU_MOE surface), perf(cuda): batched-trunk prefill for the pure-attention layer-split hybrid (CudaHybridForwardPass / Qwen3-Coder-30B) #123 (batched prefill on this class).

[pekkah]

Verified complete on master (commits 50b5db7 CPU-MoE execution mode + 9e7e332 MoE-aware TierPlanner / KV-dtype pricing). Acceptance re-checked on a 4070 Ti 12 GB with Qwen3-Coder-30B-A3B-Instruct-Q4_K_M:

(a) Placement — PASS. -g -1 plans GPU: 48 layers, CPU: 0 layers (all trunk on GPU), and the summary prints the expert budget:
expert-cache budget: 3957 MB (routed experts: 16740 MB) → auto-selects CPU MoE (capacity ≈ 989/6144 = 16% < 50%). (Budget is 3957 MB here because ctx defaulted to 32768 → 6144 MB KV, now priced at real dtype per #220/#215; with a capped ctx the budget is larger. Either way the trunk is fully GPU-resident.)

(b) A/B decode + prefill — PASS. Same model/hardware, 48/0 placement, warm (each cell run twice, first discarded to warm the routed-expert mmap pages), 100 decode tokens:

mode	prefill t/s	decode t/s
CPU-MoE (SHARPI_CPU_MOE=1)	22.7	25.5
GPU-SLRU (SHARPI_CPU_MOE=0)	16.1	14.4
CPU-MoE / SLRU	1.41×	1.77×

CPU-MoE wins on both, mirroring the GDN-class 11.8-vs-6.1 t/s (1.93×) data point. The 25.5 t/s decode matches the per-token baseline recorded in #225.

(c) No regression elsewhere — PASS. CudaHybridBatchedPrefillTests 9/9 green with CPU-MoE auto-selected (so the CPU-MoE execution path is proven bit-identical to the sequential reference); the >4096 wave oracle is green separately and is orthogonal to MoE execution. Dense models and explicit -g N are unchanged: all MoE accounting in TierPlanner is gated on hp.IsMoE, and explicit -g N bypasses the auto-packer entirely.

The batched-CPU-MoE follow-up (#225) is closed separately as a measured negative — the shipped per-token CpuMoeRouted is the bandwidth/cache optimum for these Q4_K targets.