feat(models): port DeepSeek-V4 backbone (MLA + group-limited MoE)#592
Closed
inureyes wants to merge 1 commit into
Closed
feat(models): port DeepSeek-V4 backbone (MLA + group-limited MoE)#592inureyes wants to merge 1 commit into
inureyes wants to merge 1 commit into
Conversation
Register a `deepseek_v4` text model that reuses the DeepSeek-V3 backbone: MLA attention, group-limited sigmoid-routed MoE, the unclamped shared-expert DenseMLP, and V3's multi-token-prediction layer handling. `DeepSeekV4Model` is a thin wrapper over `DeepSeekV3Model`; `DeepSeekV4Config` is parsed independently and mapped onto the V3 config via `to_dsv3_config`, mirroring the existing `glm_moe_dsa` wrapper. Config defaults track the mlx-vlm `deepseek_v4` reference. Scope is the backbone only. The two V4-specific features, HiSA hierarchical sparse attention and the shared-expert SwiGLU `swiglu_limit` clamp, are tracked as separate follow-up issues; their config fields are parsed for forward-compatibility but not applied by this port. Wiring: `ModelType::DeepSeekV4` plus the `deepseek_v4` detection arm, the `LoadedModel` variant and delegation, the config-backed metadata entry, the tensor-parallel fallback arch string, `ALL_MODEL_TYPES`, `metadata()`, and a `docs/supported-models.md` row. Tests: `deepseek_v4_tests` pin the config defaults and the V3 mapping; `tests/deepseek_v4_parity.rs` asserts the config maps to reference-derived V3 values and that the V4 backbone produces bit-identical logits to DeepSeek-V3 on shared synthetic weights.
Member
Author
|
Closing without merging, per maintainer decision. This PR ported DeepSeek-V4 as a thin wrapper over DeepSeek-V3. Investigation against the in-tree reference (references/mlx-vlm/mlx_vlm/models/deepseek_v4/) and the real DeepSeek-V4-Flash checkpoint showed the real deepseek_v4 architecture is genuinely novel (HyperConnections, pooled-KV Compressor, HiSA Indexer, hash-routed MoE with sqrtsoftplus, MultiLinear output projection), not a V3 variant, so a V3 wrapper cannot load or run a real V4 checkpoint. Issue #523 is re-scoped to a faithful port of the real architecture. This PR's diff is preserved here for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds DeepSeek-V4 support by porting the backbone only.
DeepSeekV4Modelreuses the DeepSeek-V3 stack (MLA attention, group-limited sigmoid-routed MoE, unclamped shared-expertDenseMLP, and V3's multi-token-prediction layer handling) as a thin wrapper:DeepSeekV4Configis parsed independently and mapped ontoDeepSeekV3Configviato_dsv3_config, mirroring the existingglm_moe_dsawrapper over DeepSeek V3.2. Config defaults track the mlx-vlmdeepseek_v4reference.A
deepseek_v4checkpoint previously errored with "Unsupported model type"; it now detects, loads, and runs on the V3-equivalent backbone.Scope
Backbone only, exactly as the issue directs. The two V4-specific features are deferred to separate follow-up issues and are intentionally not implemented here:
swiglu_limitclampTheir config fields (
head_dim,swiglu_limit,num_nextn_predict_layers) are parsed for forward-compatibility but not applied by the backbone.What changed
src/models/deepseek_v4.rs(new):DeepSeekV4Config+DeepSeekV4Config::to_dsv3_config+DeepSeekV4Model(wrapsDeepSeekV3Model).src/models/deepseek_v4_tests.rs(new): config default and V3-mapping unit tests.tests/deepseek_v4_parity.rs(new): reference-config parity + a synthetic forward-equivalence proof that the V4 backbone is bit-identical to DeepSeek-V3.ModelType::DeepSeekV4and thedeepseek_v4detection.rsarm;LoadedModel::DeepSeekV4+ delegation;model_metadata.rsconfig-backed entry;tensor_parallel/inference.rsfallback arch string;ALL_MODEL_TYPES,metadata(), and theall_variants!registry list inmod.rs.docs/supported-models.md: DeepSeek-V4 backbone row.Test plan
cargo fmtcargo check --lib --testscargo clippy --lib --tests -- -D warningscargo test --lib deepseek_v4(5 passed)cargo test --test deepseek_v4_parity(2 passed, incl. forward-equivalence vs DeepSeek-V3)cargo test --lib -- models::detection metadata_tests model_metadata(registry completeness, 19 passed)deepseek_v4export yet)Closes #523