VLM embedding-prefill throughput regressed 1.5-2.3x on M1 Ultra since v0.1.0

[inureyes]

Summary

The VLM prefill path (language-model prefill over pre-computed image embeddings) regressed 1.5-2.3x on Apple M1 Ultra between v0.1.0 (benchmark sweep 2026-05-19) and v0.2.1 (sweep 2026-06-15). Text prefill is unaffected, so the regression is isolated to the VLM embedding-prefill path. The current binary reproduces the slow v0.2.1 numbers, confirming this is current behavior and not a one-off sweep artifact.

Evidence (M1 Ultra, prompt "What is in this image?", test image, warmup=20, prefill tok/s)

Model	mlxcel v0.1.0	mlxcel current	mlx-vlm 0.4.4 (0.1.0 ref)	mlx-vlm 0.4.4 (today)
llava-interleave-qwen-0.5b-bf16	8589	3746	4186	3937
gemma-4-26b-a4b-it-4bit	476	270	259	261
molmo2-4b	1012	687	730	706

Prompt-token counts are identical across sweeps (754 / 277 / 438), so this is not a prompt or tokenization change.

This is not environmental

Re-running the same mlx-vlm version (0.4.4) on the same host today reproduces its own v0.1.0 prefill numbers (4186->3937, 259->261, 730->706, all within run variance). The host is stable; only mlxcel's VLM prefill dropped. Current mlxcel VLM prefill is therefore roughly at parity with mlx-vlm (~0.95-1.03x), versus the ~1.33x median advantage reported for the v0.1.0 campaign (which was inflated by anomalously fast mlxcel measurements, e.g. llava at 8589 tok/s = 2.05x).

Reproduction

cargo build --release --features metal,accelerate
./target/release/mlxcel-bench-decode \
  -m models/llava-interleave-qwen-0.5b-bf16 \
  -p "What is in this image?" \
  --image tests/fixtures/test_image.png \
  --warmup-tokens 20 --max-tokens 100
# Observe Prefill ~3700 tok/s; the v0.1.0 sweep recorded ~8589 tok/s for the same model.

Scope / where to look

The measured prefill is generate_with_stats_and_embeddings in src/bin/bench_decode.rs (the vision encoder runs separately in prepare_and_compute_vlm_embeddings and is excluded from the prefill stat). So the regression is in the language-model prefill over input embeddings, not in the vision encoder and not in text prefill (text prefill held at ~1.79x).

Suspect window: changes landed between v0.1.0 (2026-05-19) and v0.2.1 (2026-06-15). Candidate causes to bisect:

• Unified paged KV cache going LIVE (epic [Epic] Unified KV cache: radix-tree-indexed paged attention with shared physical blocks #116, PR docs: unified paged KV cache + disaggregated serving (B3c) #194)
• Automatic prefix cache default-on (feat: enable Automatic Prefix Caching by default #233)
• Chunked-prefill work (Disaggregated serving handoff: support chunked-prefill prompts #197 / fix: support chunked-prefill prompts in the disaggregated serving handoff #213)
• MLX C++ pin bump to a6ec712 (chore: bump vendored MLX to upstream main 2026-06-11 and retire the steel GEMM overlay #223)

Acceptance criteria

• Bisect to the introducing change and root-cause why the embeddings-prefill path slowed on M1 Ultra.
• Restore VLM embedding-prefill throughput, or document why the slowdown is an intended/unavoidable cost of the new cache/prefill path.
• Re-measure on M1 Ultra against same-version mlx-vlm to confirm.

[inureyes]

Investigation update: this is a general long-sequence prefill regression, not VLM-specific

Re-measuring on M1 Ultra shows the slowdown is prefill length-dependent, not VLM-specific. VLM prompts just happen to always be long (image-expanded, ~750 tokens), so they surface it.

mlxcel vs mlx-lm prefill at increasing length (qwen2.5-7b-4bit, M1 Ultra, current binary)

prompt tokens	mlxcel tok/s	mlx-lm tok/s	ratio
128	648	689	0.94x
1600	795	884	0.90x
3520	778	863	0.90x
7040	745	820	0.91x
14080	668	746	0.90x

At every real prefill length (128+ tokens) mlxcel is ~0.90x mlx-lm. The headline ~2.7x text-prefill number is only at 8-token prompts, i.e. a fixed-launch-overhead regime, not GEMM-bound prefill.

In the v0.1.0 campaign, mlxcel's long prefill was 1.4-2.0x faster than the Python reference (the VLM rows: llava 8589 tok/s = 2.05x, gemma-4-26b-a4b 476 = 1.84x, molmo2 1012 = 1.39x). It is now ~parity. mlx-lm / mlx-vlm are stable on this host across the same window, so mlxcel's long-sequence prefill regressed ~2x in absolute terms.

Root cause: the #223 MLX bump

mlxcel's prefill/attention code is essentially unchanged since v0.1.0. The only prefill-relevant change in the v0.1.0..HEAD window is #223 (0ebd984d0): MLX pin 84961223 -> a6ec712, which deleted the vendored steel GEMM mma.h overlay (1146 lines) and adopted upstream a6ec712's mma.h. The steel GEMM is the core attention/matmul kernel, and the regression is exactly in GEMM-bound long-sequence prefill. The overlay was originally a safe-load correctness patch, so the slowdown comes from the upstream mma.h change between 84961223 and a6ec712, not the overlay retirement itself.

Next

• Confirming by building at the pre-chore: bump vendored MLX to upstream main 2026-06-11 and retire the steel GEMM overlay #223 state (0ebd984d0^, MLX 84961223) and re-measuring long prefill. If it returns to ~2x, the MLX bump is confirmed.
• Likely fix direction: report upstream to ml-explore/mlx (steel GEMM long-seq prefill regression 84961223->a6ec712), and/or re-vendor the faster kernel as a perf overlay. Pinning MLX back is not an option (the bump carries diffusion and other fixes).

Title should be generalized from "VLM embedding-prefill" to "long-sequence prefill".

[inureyes]

Resolution: not a real regression — the 2026-05-19 sweep numbers were anomalous

I rebuilt the exact binaries and measured VLM prefill head-to-head on M1 Ultra. The 0.1.0-era sweep's VLM-prefill figures do not reproduce with the very binary that recorded them.

llava-interleave-qwen-0.5b-bf16, VLM prefill (754 prompt tokens, same test image)

build	llava prefill tok/s
recorded in the 2026-05-19 M1U sweep (v0.0.28)	8589
v0.0.28 rebuilt (the exact campaign binary)	3714 – 3785
v0.1.0	3612 – 3724
pre-#223 (e099821af, MLX 84961223)	3793
current (a6ec712)	3732 – 3901
mlx-vlm 0.4.4 (fresh, same host)	~3937

Same pattern for the others (recorded -> rebuilt v0.0.28): gemma-4-26b-a4b 476 -> 278, molmo2 1012 -> 705. Prompt-token counts are identical (754 / 277 / 438), so it is not a prompt difference, and the MLX pin is the same (84961223) for v0.0.28 / v0.1.0 / pre-#223.

Conclusion

mlxcel VLM prefill has been stable at roughly mlx-vlm parity (~0.95–1.0x) across v0.0.28 -> v0.1.0 -> pre-#223 -> current. There is no regression. The 2026-05-19 M1 Ultra VLM-prefill column (8589 / 476 / 1012, which produced the "1.33x M1 Ultra VLM prefill" headline) was a transient measurement artifact specific to that sweep run; it is not a property of any binary and does not reproduce.

The earlier "general long-sequence prefill regression" reading was also wrong: it came from extrapolating the bad VLM numbers. Real long-text prefill (qwen2.5-7b) is ~0.90x mlx-lm and identical pre/post #223.

Recommend closing this as not a bug. Follow-ups:

• README: the M1 Ultra VLM-prefill figure should be ~parity (the fresh same-host run gives ~1.0x), not 1.33x.
• Benchmark harness: the 05-19 VLM-prefill anomaly suggests a flaky VLM prefill measurement path worth a guard (e.g., sanity-bound prefill tok/s, or repeat-and-median), but that is a separate harness-hardening item, not an inference bug.

[inureyes]

Closing as not-a-bug per the investigation above: the 2026-05-19 VLM-prefill figures were a non-reproducible measurement anomaly, and mlxcel VLM prefill is stable at ~mlx-vlm parity across v0.0.28 -> current. README will be corrected to reflect ~parity. The flaky-VLM-prefill-measurement harness guard is noted as a separate potential follow-up.