fix(qwen3.5-moe): stack per-expert gate_proj/up_proj/down_proj weight names#587
Merged
Merged
Conversation
… names
In sanitize_weights step 6, also match experts.{e}.{gate_proj,up_proj,down_proj}.weight and stack them into switch_mlp.{w1,w3,w2} (gate->w1, up->w3, down->w2), mirroring the fused gate_up_proj path's naming convention.
The existing path only handled experts.{e}.{w1,w2,w3}.weight, so checkpoints shipping per-expert tensors under the gate_proj/up_proj/down_proj names failed with 'Missing weight: model.layers.N.mlp.switch_mlp.w1'.
The gate_proj pass skips any slot already filled by the w1/w2/w3 pass, preventing accidental clobber when both conventions appear in the same checkpoint. Quantized scales and biases are forwarded with the same slot rename.
Three unit tests cover: plain float stacking, stacking with quantized scales/biases, and the priority guard against clobbering w1/w2/w3 slots.
4 tasks
inureyes
added a commit
that referenced
this pull request
Jul 6, 2026
…eMoeBlock actually loads (#671) PR #588 renamed the shared qwen3_next SwitchGLU/SparseMoeBlock expert projections from w1/w2/w3 to gate_proj/up_proj/down_proj, but qwen3_5::sanitize_weights kept normalizing in the OLD direction: step 6 stacked per-expert experts.{e}.{w1,w2,w3} into switch_mlp.{w1,w2,w3}, and step 8 renamed pre-stacked switch_mlp.{gate,up,down}_proj INTO switch_mlp.{w1,w3,w2}. Every qwen3.5-MoE / qwen3.6-MoE checkpoint has therefore failed to load since #588 with "Missing weight: model.layers.0.mlp.switch_mlp.gate_proj" (issue #670; found by the 2026-07-06 benchmark sweep, where qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit regressed from 70/67 tok/s on 2026-06-12 to FAIL). The #587 per-expert gate/up/down stacking pass also targeted the stale w1/w3/w2 slots; its tests asserted sanitize output keys only and never loaded the result, so they passed against the wrong convention. The fix aligns all three passes with the block naming: per-expert w1/w2/w3 stacks into gate_proj/down_proj/up_proj, per-expert gate/up/down stacks into the same-name slots, and step 8 now renames legacy stacked w1/w3/w2 forward into gate/up/down while pre-stacked gate/up/down checkpoints pass through untouched. The missing gate is added as a regression test: sanitize output must load end to end through the shared SwitchGLU for all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). Real-checkpoint validation: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit load and answer coherently again (57.8 and 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template), and dense qwen3.5-4b-4bit is unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a weight loading failure for Qwen3.5-MoE checkpoints that store per-expert tensors under
gate_proj/up_proj/down_projnames instead ofw1/w2/w3. The previous code only handled thew1/w2/w3naming, leaving those checkpoints withMissing weight: model.layers.N.mlp.switch_mlp.w1.What changed
src/models/qwen3_5.rs(sanitize_weights step 6): added a second inner pass that stacksexperts.{e}.gate_proj/up_proj/down_projintoswitch_mlp.w1/w3/w2(gate->w1, up->w3, down->w2), matching the fusedgate_up_projpath naming. The pass skips any slot already filled by thew1/w2/w3pass to prevent clobbering. Quantized.scalesand.biasesreceive the same rename.sanitize_testsmodule: plain float stacking, stacking with quantized scales/biases, and the priority guard.Test plan
cargo test --lib models::qwen3_5::sanitize_tests-- 3 tests passcargo check --lib --tests-- cleancargo clippy --lib --tests -- -D warnings-- no warningsCloses #519