feat(models): port GLM-4V MoE (ViT + GLM-4 MoE text with half-split MRoPE)#598
Merged
Conversation
…RoPE) Add GLM-4V MoE (model_type "glm4v_moe", GLM-4.5V-class) as a vision-language model, reusing the GLM-4V ViT vision encoder and the GLM-4 MoE text machinery, wired end-to-end through the shared Qwen-VL runtime. Text backbone (src/models/glm4v_moe.rs): a GLM-4 MoE backbone driven by sectioned half-split MRoPE. Unlike GLM-4V (sectioned even/odd), GLM-4V MoE uses the sectioned half-split style (GPT-NeoX rotation), applies no q/k norm, and uses two RMSNorm layers per decoder block. The MoE and dense MLP machinery (grouped noaux_tc routing, shared experts, first_k_dense_replace dense layers) is reused from src/models/glm4_moe.rs. Vision (src/vision/glm4v_moe.rs): reuses the GLM-4V ViT (src/vision/encoders/glm4v.rs) and its config, plus the shared Qwen-VL image processor, prompt-token plumbing, and LLaVA merge, with 3D MRoPE positions from compute_rope_index. The GLM-4V and GLM-4V MoE vision towers differ only in a mathematically equivalent windowed-attention formulation (per-segment vs block-diagonal mask), so the encoder is shared. Generalized src/models/glm4v.rs: Glm4vMRoPE gains a pairing parameter (even/odd vs half-split) and is promoted to pub(crate) so both GLM-4V and GLM-4V MoE share one sectioned-MRoPE implementation. Wiring: ModelType::Glm4vMoe variant and detection arm for "glm4v_moe", model_metadata registration, the VLM load route (load_glm4v_moe reusing the GLM-4V weight remap and the GLM-4 MoE args), LoadedModel variant and delegation, vlm_runtime dispatch, a QwenVlRuntime impl, tensor-parallel arch fallback (glm4_moe), and a docs/supported-models.md row. Tests: a half-split MRoPE-selector unit test plus a glm4v_moe_parity integration test (config parse and detection run in CI; real-model forward passes are gated on checkpoint presence). Closes #530
inureyes
added
type:enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds GLM-4V MoE (
model_type: "glm4v_moe", GLM-4.5V-class) as a vision-language model, reusing the GLM-4V ViT vision encoder from #529 and the GLM-4 MoE text machinery, wired end-to-end through the shared Qwen-VL runtime. Ported faithfully fromreferences/mlx-vlm/mlx_vlm/models/glm4v_moe/.What changed
src/models/glm4v_moe.rs(new): GLM-4 MoE text backbone with sectioned half-split MRoPE (GPT-NeoX rotation), no q/k norm, two RMSNorm layers per decoder block. The MoE/dense MLP machinery (groupednoaux_tcrouting, shared experts,first_k_dense_replacedense layers) is reused fromsrc/models/glm4_moe.rs.src/vision/glm4v_moe.rs(new): VLM wrapper reusing the GLM-4V ViT (src/vision/encoders/glm4v.rs) and its config, the shared Qwen-VL processor/prompt plumbing, LLaVA merge, andcompute_rope_index. ImplementsLanguageModel+ sharedQwenVlRuntime. The GLM-4V and GLM-4V MoE vision towers differ only in a mathematically equivalent windowed-attention formulation, so the encoder is shared.src/models/glm4v.rs: generalizedGlm4vMRoPEwith apairingparameter (even/odd vs half-split), promoted topub(crate)so both GLM-4V and GLM-4V MoE share one sectioned-MRoPE implementation.ModelType::Glm4vMoe+detection.rsarm for"glm4v_moe",model_metadataregistration, VLM load route (load_glm4v_moereusing the GLM-4V weight remap + GLM-4 MoE args),LoadedModelvariant + delegation,vlm_runtimedispatch, tensor-parallel arch fallback (glm4_moe), and adocs/supported-models.mdrow.tests/glm4v_moe_parity.rs(config parse + detection in CI; real-model forward gated on checkpoint presence).Test plan
cargo fmtcargo check --lib(clean)cargo test --lib models::glm4v(4 passed, incl. half-split selector)cargo test --test glm4v_moe_parity(4 passed; real-model arms skip without a checkpoint)cargo test --test glm4v_parity(5 passed; feat(models): port GLM-4V (ViT + GLM-4 text) #529 regression check)cargo clippy --lib --tests -- -D warnings(clean)Real-checkpoint forward-pass validation is deferred to the orchestrator.
Closes #530