fix(vision): capture the functional slice_update return in DeepStack and MRoPE-section loops#666
Merged
Merged
Conversation
…and MRoPE-section loops mlxcel_core::slice_update wraps the functional mlx::core::slice_update, which returns a new array and never mutates its src; four loops called it as a bare statement and silently lost every write (issue #650, the same class as the Granite 4 Vision blindness fixed by inject_at_positions). In qwen3_vl.rs and qwen3_vl_moe.rs the DeepStack injection loop dropped the whole per-layer visual refinement (the models ran on the main embeddings alone); the fix captures the return and, mirroring the Granite fix, collapses the contiguous single-image run to one slice_update with a per-row fallback for disjoint runs. In qwen2_vl.rs and paddleocr_vl.rs the apply_mrope section assembly dropped the H and W section writes, so every head dim kept the temporal frequencies and image tokens carried no intra-image spatial position information; the fix captures the return. Four co-located regression tests pin the semantics (deepstack_process must add visual embeds at masked positions; apply_mrope must place per-axis frequencies into their sections). Real-checkpoint validation: qwen2-vl-2b-4bit, qwen3-vl-4b-instruct-4bit, and qwen3-vl-30b-a3b-4bit all localize a left-red/right-blue split image correctly through the CLI; paddleocr_vl shares the identical fixed function with qwen2_vl and is covered by its regression test (no local checkpoint).
inureyes
added a commit
that referenced
this pull request
Jul 6, 2026
…s) (#673) Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlxcel_core::slice_updateis functional (wrapsmlx::core::slice_update, which returns a new array and never mutatessrc). Four loops called it as a bare statement and silently lost every write — the same class as the Granite 4 Vision blindness fixed byinject_at_positionsin PR #649.src/models/qwen3_vl.rs(DeepStack loop)inputs_embeds)src/models/qwen3_vl_moe.rs(DeepStack loop)src/models/qwen2_vl.rs(apply_mrope)src/models/paddleocr_vl.rs(apply_mrope)Fix
Capture the return. The DeepStack loops mirror the Granite fix: a contiguous single-image run collapses to one
slice_update, disjoint runs fall back to per-row with the return captured. Theapply_mropesites capture the return of each section write.Validation
deepstack_processmust add visual embeds at masked positions (qwen3_vl, qwen3_vl_moe);apply_mropemust place per-axis frequencies into their sections (qwen2_vl, paddleocr_vl). All four fail on the pre-fix code.qwen2-vl-2b-4bit: "The color on the left side of the image is red, and the color on the right side is blue."qwen3-vl-4b-instruct-4bit: "far left is red ... far right is blue."qwen3-vl-30b-a3b-4bit(MoE): "the color on the left side is red, and the color on the right side is blue."paddleocr_vlshares the identical fixed function withqwen2_vland is covered by its regression test (no local checkpoint on the validation machine).cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Closes #650