fix(qwen3.5-moe): sanitize toward the gate/up/down expert names SparseMoeBlock actually loads#671
Merged
Merged
Conversation
…eMoeBlock actually loads PR #588 renamed the shared qwen3_next SwitchGLU/SparseMoeBlock expert projections from w1/w2/w3 to gate_proj/up_proj/down_proj, but qwen3_5::sanitize_weights kept normalizing in the OLD direction: step 6 stacked per-expert experts.{e}.{w1,w2,w3} into switch_mlp.{w1,w2,w3}, and step 8 renamed pre-stacked switch_mlp.{gate,up,down}_proj INTO switch_mlp.{w1,w3,w2}. Every qwen3.5-MoE / qwen3.6-MoE checkpoint has therefore failed to load since #588 with "Missing weight: model.layers.0.mlp.switch_mlp.gate_proj" (issue #670; found by the 2026-07-06 benchmark sweep, where qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit regressed from 70/67 tok/s on 2026-06-12 to FAIL). The #587 per-expert gate/up/down stacking pass also targeted the stale w1/w3/w2 slots; its tests asserted sanitize output keys only and never loaded the result, so they passed against the wrong convention. The fix aligns all three passes with the block naming: per-expert w1/w2/w3 stacks into gate_proj/down_proj/up_proj, per-expert gate/up/down stacks into the same-name slots, and step 8 now renames legacy stacked w1/w3/w2 forward into gate/up/down while pre-stacked gate/up/down checkpoints pass through untouched. The missing gate is added as a regression test: sanitize output must load end to end through the shared SwitchGLU for all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). Real-checkpoint validation: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit load and answer coherently again (57.8 and 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template), and dense qwen3.5-4b-4bit is unaffected.
inureyes
added
type:bug
inureyes
added a commit
that referenced
this pull request
Jul 6, 2026
…s) (#673) Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
inureyes
pushed a commit
that referenced
this pull request
Jul 7, 2026
Keep PR #653 as a focused test-only follow-up after the loader/sanitizer implementation landed through #671. - add a sanitizer unit test that verifies per-expert gate_proj/up_proj/down_proj tensors preserve expert order and projection values when stacked into switch_mlp.*_proj - leave the qwen3_5 sanitizer implementation unchanged from current main Validation: - cargo fmt --all -- --check - cargo test qwen3_5::sanitize_tests -- --nocapture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every qwen3.5-MoE / qwen3.6-MoE checkpoint has failed to load since PR #588 with
Missing weight: model.layers.0.mlp.switch_mlp.gate_proj. Found by the 2026-07-06 full benchmark sweep:qwen3.5-35b-a3b-4bitandqwen3.6-35b-a3b-4bitbenched at 70 / 67 tok/s on 2026-06-12 and wereFAIL:benchon 2026-07-06.Root cause
PR #588 renamed the shared
qwen3_nextSwitchGLU/SparseMoeBlockexpert projections fromw1/w2/w3togate_proj/up_proj/down_proj, butqwen3_5::sanitize_weightskept normalizing in the OLD direction: step 6 stacked per-expertexperts.{e}.{w1,w2,w3}intoswitch_mlp.{w1,w2,w3}, and step 8 renamed pre-stackedswitch_mlp.{gate,up,down}_projINTOswitch_mlp.{w1,w3,w2}. The block then requestsswitch_mlp.gate_projand finds nothing. PR #587's per-expert gate/up/down stacking pass also targeted the stalew1/w3/w2slots; its tests asserted sanitize OUTPUT keys only and never loaded the result, so they passed against the wrong convention — the same "synthetic passes, real checkpoint fails" class as #588 root-caused.Fix
All three passes now align with the block naming: per-expert
w1/w2/w3stacks intogate_proj/down_proj/up_proj(w1→gate, w2→down, w3→up), per-expertgate/up/downstacks into the same-name slots, and step 8 renames legacy stackedw1/w3/w2FORWARD intogate/up/downwhile pre-stacked gate/up/down checkpoints pass through untouched.Validation
SwitchGLUfor all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). It fails on the pre-fix sanitize by construction (the old direction producesw1keys the block never requests).qwen3.5-35b-a3b-4bitandqwen3.6-35b-a3b-4bitload and generate again (57.8 / 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template); denseqwen3.5-4b-4bitunaffected.cargo fmt,cargo clippy --all-targets -- -D warnings, and the qwen3_5 unit suite are clean. The two models will be re-added to the 2026-07-06 benchmark CSV after merge.Closes #670