perf(cuda): attribute the gemma-4-26b-a4b and glm4-flash decode drops from the 2026-07-12 GB10 sweep (re-measure after reboot first)

[inureyes]

Summary

The 2026-07-12 GB10 full sweep (mlxcel 0.4.0-rc.1, MLX pin 57c66cac / 0.32.1) recorded moderate decode drops versus the 2026-06-17 (0.3.1) sweep for gemma-4-26b-a4b-it-4bit, gemma-4-26b-a4b-it-qat-4bit, and glm4-flash-4bit. PR #751 hypothesized these were the same L=1 depthwise-conv dispatch cost that hit lfm2; issue #752 / PR #753 (06a8f38) refuted that on the record: neither architecture calls conv1d at all (gemma4 MoE and glm4_moe_lite). The drops are therefore unattributed.

Because the same sweep carried an unattributed environmental anomaly (the SSM/hybrid cluster read 2-3x HIGHER than every earlier record) and the host had ~5.5 days of uptime with the known post-benchmark-day driver signature, sweep deltas from that day are not trustworthy attribution evidence on their own. The first discriminating step is a clean re-measurement after a host reboot.

Measurements (2026-06-17 / 0.3.1 -> 2026-07-12 / 0.4.0-rc.1)

Model	Prefill tok/s	Decode tok/s
gemma-4-26b-a4b-it-4bit	130.81 -> 114.81 (-12.2%)	58.59 -> 50.19 (-14.3%)
gemma-4-26b-a4b-it-qat-4bit	136.06 -> 114.43 (-15.9%)	50.33 -> 45.29 (-10.0%)
glm4-flash-4bit	106.65 -> 99.04 (-7.1%)	53.33 -> 45.72 (-14.3%)

Related SSM data point, same status: mamba2-130m (bf16) decode 181.05 -> 162.23 (-10.4%). The #752 nsys audit showed its conv runs on the fast engine (slow engine 0 launches, conv only 1.6% of decode), so its drop is equally unattributed.

Source CSVs: benchmarks/cuda_gb10_2026-06-17.csv, benchmarks/cuda_gb10_2026-07-12.csv.

Already ruled out

• Conv dispatch (issue perf(cuda): audit L=1 depthwise conv1d dispatch across SSM/hybrid decode paths and apply the #751 elementwise step where confirmed #752 / PR refactor(models): share the short-conv decode helper and audit L=1 conv dispatch across SSM/hybrid decode paths #753): neither gemma4 MoE nor glm4_moe_lite calls conv1d; the nsys audit found only the fast conv1d_c1_k1_nhwc engine across the conv families measured.
• Pure single-run noise is unlikely for the gemma pair: both independent 26b-a4b variants moved together (-10 to -14%), suggesting a systematic cause; the environmental confounder remains, hence the reboot gate.

Environmental context

• Same-sweep anomaly: the SSM/hybrid cluster (granite-4.0-h, falcon-h1, nemotron-h/nas/omni, plamo-2) read 2-3x higher than both the 2026-06-17 sweep and the 2026-07-09 subset with no relevant code change in between; still unattributed.
• Host state: ~5.5 days uptime at sweep time; the known GB10 post-benchmark-day driver signature was active (nondeterministic CUDA graph-capture aborts in broad test suites on 2026-07-12, recorded on PR fix: bind gemma-4 NVFP4 vision tower so image input works #750).

Plan (ordered; step 1 gates everything else)

1. Re-measure after a fresh host reboot. Run single-model benchmarks for the three subjects plus mamba2-130m and two stable controls, ideally in the same post-reboot session as the planned SSM-cluster re-verification so one reboot settles both anomalies:

   ./scripts/bench_decode.sh models/gemma-4-26b-a4b-it-4bit --cooldown 30
   ./scripts/bench_decode.sh models/gemma-4-26b-a4b-it-qat-4bit --cooldown 30
   ./scripts/bench_decode.sh models/glm4-flash-4bit --cooldown 30
   ./scripts/bench_decode.sh models/mamba2-130m --cooldown 30
   ./scripts/bench_decode.sh models/qwen3-30b-a3b-4bit --cooldown 30
   ./scripts/bench_decode.sh models/lfm2-8b-a1b-4bit --cooldown 30

   (qwen3-30b-a3b-4bit is the MoE control that was stable on 2026-07-12 at +1%; lfm2-8b-a1b-4bit is the hybrid control at +3%.) If the subjects return to the 2026-06-17 envelope while controls stay put, the verdict is environmental: update the doc rows and notes with the post-reboot values and close with the evidence.

2. Only if the drops persist post-reboot: profile a warm decode of gemma-4-26b-a4b-it-4bit against the healthy MoE control with the perf(cuda): audit L=1 depthwise conv1d dispatch across SSM/hybrid decode paths and apply the #751 elementwise step where confirmed #752 method (nsys kernel histogram, MLX_USE_CUDA_GRAPHS=0) and diff the kernel mix; then review MoE-path changes in the 0.3.1 -> 0.4.0-rc.1 window for the suspects (the perf(cuda/quant): amortize small-M quantized matmul with a multirow qmv path #740 multirow qmv path on expert matmuls, perf(cuda): single-dtype decode graph to eliminate per-token AsType #732 single-dtype decode-graph eligibility for these architectures, fused-MoE dispatch changes).

3. Root-cause and fix, or document as an accepted change, and refresh the affected rows in docs/benchmark_results/model_tests_gb10.md with committed singles CSVs.

Acceptance Criteria

• Post-reboot single-model measurements recorded for the three subjects, mamba2-130m, and both controls (singles CSVs committed; numbers in the issue or PR)
• Verdict recorded: environmental (numbers recover post-reboot) or real regression (drops persist)
• If environmental: doc rows and the unattributed-drop notes updated to the post-reboot values and this issue closed with the evidence
• If real: root cause identified with profiling evidence, then fixed or documented as accepted, with doc rows refreshed
• The unattributed-drop mentions in docs/benchmark_results/model_tests_gb10.md reference this issue number

References

• Issue perf(cuda): lfm2-350m-8bit decode regressed ~10x on GB10 (409 -> ~40 tok/s) between 0.3.1 and 0.4.0-rc.1 #748 / PR fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 (3fa2789), issue perf(cuda): audit L=1 depthwise conv1d dispatch across SSM/hybrid decode paths and apply the #751 elementwise step where confirmed #752 / PR refactor(models): share the short-conv decode helper and audit L=1 conv dispatch across SSM/hybrid decode paths #753 (06a8f38), PR fix: bind gemma-4 NVFP4 vision tower so image input works #750 (3fa7cc1)
• benchmarks/cuda_gb10_2026-06-17.csv, benchmarks/cuda_gb10_2026-07-12.csv
• GB10 driver-state history: PR fix: bind gemma-4 NVFP4 vision tower so image input works #750 comment (2026-07-12) and the 2026-07-08 record

[inureyes]

Cross-hardware check from the M1 Ultra 0.4.0-rc.1 refresh (PR #754, benchmarks/metal_m1ultra_2026-07-12.csv): the drops do NOT reproduce on Metal.

model	M1 decode 0.2.1 (06-15)	M1 decode 0.3.3 (07-06)	M1 decode rc.1 (07-12)	GB10 decode 0.3.1 -> rc.1
gemma-4-26b-a4b-it-4bit	79.92	81.36	79.55	58.59 -> 50.19 (-14.3%)
gemma-4-26b-a4b-it-qat-4bit	77.99	79.46	77.56	50.33 -> 45.29 (-10.0%)
glm4-flash-4bit	45.53	48.96	48.84	53.33 -> 45.72 (-14.3%)
mamba2-130m	(see 07-06/07-12)	240.82	259.90 (+7.9%)	181.05 -> 162.23 (-10.4%)

The M1 flat window (0.2.1 through rc.1, within about 2%) fully covers the GB10 comparison window (0.3.1 -> rc.1), so a platform-independent code regression is ruled out. Remaining candidates are narrowed to two: GB10 host/driver state on the 2026-07-12 bench day, or a CUDA-only code path change in the window (Metal cannot see those). The plan's gating step is unchanged: the post-reboot re-measurement separates these two. If the drops persist after reboot, step 2 profiling should focus exclusively on CUDA-specific suspects (#740 multirow qmv on expert matmuls, #732 single-dtype decode graph eligibility, fused-MoE dispatch). M5 Max cannot contribute a data point yet: its latest sweep is 0.2.1 (2026-06-15), with no rc.1 re-benchmark.