Add Gemma 4 12B (dense gemma4_unified) support — CUDA-Hybrid/CUDA first, Q4_K_M

[pekkah]

Track support for the dense Gemma 4 12B model (gemma4_unified, released 2026-05-23). Full research + phased plan: docs/gemma4-12b-implementation-plan.md.

Context

The merged gemma4 path (E4B, #82) is metadata/tensor-driven and already implements the hard parts the 12B needs: per-layer head-dim variance, dual-RoPE, SWA, shared-KV, post-attn/post-ffn norms, GeLU-tanh FFN, final-logit softcap, and proportional RoPE (rope_freqs). The dense 12B is essentially E4B minus PLE / layer_output_scale, plus a 1024 sliding window, plus a possible arch-string rename. So this is mostly validation of the never-exercised dense (no-PLE) path + decode parity, not new kernels.

Target

• Primary model: google/gemma-4-12B-it-qat-q4_0-gguf (~7.0 GB) — Google's official quantization-aware-trained (QAT) 4-bit weights. Best quality-per-byte at 4-bit; fits full-GPU offload on 12 GB. → https://hf.co/google/gemma-4-12B-it-qat-q4_0-gguf
• Fallback model: unsloth/gemma-4-12b-it-GGUF @ Q4_K_M (~7.3 GB) — community K-quant of the non-QAT weights, for cross-checking. → https://hf.co/unsloth/gemma-4-12b-it-GGUF
• Parity oracle: ggml-org/gemma-4-12B-it-GGUF (llama.cpp reference quant) for Phase 3 decode parity.
• Hardware: 12 GB VRAM (RTX 3060/4070), full-GPU offload + KV at ~8K-32K context; 64 GB system RAM for CUDA-Hybrid spill if going higher than Q6.
• Backends (in order): CUDA-Hybrid + CUDA → CPU → Vulkan.
• Out of scope (iter 1): encoder-free multimodal (vision/audio) — separate mmproj, follow-up epic.

Gap analysis

• G0 (NEW) — QAT weights are stored as q4_0, not Q4_K_M. Confirm the q4_0 dequant path is wired across CPU/CUDA/CUDA-Hybrid for the gemma4 tensors (token-embd, attn/ffn weights). If only K-quants were exercised for gemma4 so far, validate/enable q4_0. (The unsloth Q4_K_M fallback covers the K-quant path.)
• G1 — Verify general.architecture: gemma4 vs gemma4_unified. If the latter, alias it in the NEOX allowlist, isGemma4, _isGemma4Like, and metadata-key prefix logic (ModelGraph.cs:289,352).
• G2 — Dense (no-PLE) path has never been exercised; all gemma4 tests are E4B/PLE. Validate HasPerLayerTokenEmbd == false end-to-end across CPU/CUDA/CUDA-Hybrid.
• G3 — layer_output_scale absence: confirm clean skip on the HasLayerOutputScale == false branch.
• G4 — Sliding window = 1024 (vs E4B's 512); confirm nothing hard-codes 512.
• G5 — Proportional RoPE (rope_freqs) on global layers; validate at long positions, not just position 0.
• G6 — KV/context budget: no PLE table to subtract; size per-layer KV by min(maxSeqLen, window); 256K context not realistic on 12 GB.
• G7 — Tokenizer / chat template / EOG smoke for the 12B GGUF.

Phases

• Phase 0 — Acquire + dump real 12B GGUF headers → tests/fixtures/gemma4_12b_header.md for both the QAT q4_0 primary and the unsloth Q4_K_M fallback (note quant-type differences) (needs HF egress)
• Phase 1 — Arch-string + metadata wiring (G1) + Gemma4_12B_Dense_PopulatesAllFields test
• Phase 2 — Dense (no-PLE) CUDA + CUDA-Hybrid decode (G0/G2/G3/G4/G5)
• Phase 3 — CUDA / CUDA-Hybrid decode parity vs llama.cpp (acceptance gate) → tests/fixtures/gemma4_12b_greedy.json
• Phase 4 — Server + CLI smoke (G7)
• Phase 5 — CPU backend parity
• Phase 6 — Vulkan backend parity
• Phase 7 (future epic) — Encoder-free multimodal (vision + audio)

Open questions (resolve from Phase 0 dump)

1. general.architecture string — gemma4 or gemma4_unified?
2. block_count, embedding_length, feed_forward_length, head counts.
3. Confirm per_layer_* and layer_output_scale absent (dense).
4. rope_freqs.weight present? length (maxHeadDim/2)?
5. sliding_window == 1024? pattern length == block_count, final entry global?
6. shared_kv_layers count.
7. Does the text GGUF embed any multimodal projection tensors, or is mmproj fully separate?
8. QAT GGUF tensor quant types — all q4_0, or are norms/embeddings kept at higher precision (f16/q8_0)?

Plan doc PR: develops on claude/gemma-4-12b-support-4Reex.

[pekkah]

Phase 3 — Q4_0 fast paths land; llama.cpp decode gap essentially closed

Branch claude/gemma-4-12b-gh-issues-Nduw0 (commits 78050a4, 5dac1e0, c53d48e). RTX 4070 Ti, full offload -g -1, gemma-4-12b-it-qat-q4_0.gguf. llama-bench -ngl 99 on this box: pp512 4290 / tg128 56.96 t/s.

	llama.cpp	before	after	gap
Prefill	4290 t/s	~23 t/s	1692 t/s	180× → 2.5×
Decode	57.0 t/s	~25 t/s	53.8 t/s	2.8× → 1.06×

Both fast paths are argmax-stable (greedy output identical in A/B); SHARPI_BATCHED_PREFILL=0 / SHARPI_Q40_DP4A=0 revert.

Task 1 — Q4_0 batched prefill GEMM (23 → 1692 t/s, ~73×)

Root cause was structural, not kernel: BatchableDType excluded Q4_0, so IsBatchedPrefillSupported() was false and the whole prompt ran the per-token decode loop. Fix mirrors the #162 Q6_K/Q5_K prefill path:

• New llm_dequant_q4_0_to_f16 + Q4_0 wired into MatMulBatchedGemm (colAlign 32) and GpuMatMulBatchedCore (dequant→fp16→cuBLAS GEMM, weight read once per batch).
• The batched trunk had never seen the 12B's global layout — wired it in: per-layer KV heads (8 GQA / 1 MQA, was hardcoded _numKvHeads), attention_k_eq_v V=raw-K reuse, gated pure V-norm via new llm_head_norm_pure_batched/HeadNormPureBatched. Guarded the null _wv[i] of the 8 global layers in IsBatchedPrefillSupported.

Task 2 — Q4_0 dp4a decode (25 → 53.8 t/s, 2.1×)

llm_matvec_q4_0 was the basic per-element fp32 kernel. New llm_matvec_q4_0_dp4a (mirror #142): Q8_1-quantize activation, int8·int8 __dp4a. Q4_0 is symmetric, so the asymmetric trick dp4a's the raw nibbles (0..15) and subtracts 8·Σq once per block via the stored Q8_1 sum (no per-nibble centering, no min-bias term). SHARPI_Q40_DP4A default-on. SoA (#156) was not pursued — Q4_0 has no per-block scale-unpack chain (one fp16 d), so that lever doesn't apply.

Tests

CudaMatVecQ40Tests: fp32 oracle pinned to Q40Dp4aEnabled=false; new MatVecQ4_0_Dp4a_TracksCpuReference (2%-row-RMS). 34 Gemma4 + 2 CudaMatVecQ40 + 22 adjacent batched-matmul tests green; README row added.

Phase checkboxes 0/1/2 done (prior commits); Phase 2 now also has the prefill/decode fast paths. Remaining follow-ups tracked in a new issue.