Skip to content

feat(models): port DeepSeek-V4 backbone (MLA + group-limited MoE)#592

Closed
inureyes wants to merge 1 commit into
mainfrom
feature/issue-523-deepseek-v4-backbone
Closed

feat(models): port DeepSeek-V4 backbone (MLA + group-limited MoE)#592
inureyes wants to merge 1 commit into
mainfrom
feature/issue-523-deepseek-v4-backbone

Conversation

@inureyes

@inureyes inureyes commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

Adds DeepSeek-V4 support by porting the backbone only. DeepSeekV4Model reuses the DeepSeek-V3 stack (MLA attention, group-limited sigmoid-routed MoE, unclamped shared-expert DenseMLP, and V3's multi-token-prediction layer handling) as a thin wrapper: DeepSeekV4Config is parsed independently and mapped onto DeepSeekV3Config via to_dsv3_config, mirroring the existing glm_moe_dsa wrapper over DeepSeek V3.2. Config defaults track the mlx-vlm deepseek_v4 reference.

A deepseek_v4 checkpoint previously errored with "Unsupported model type"; it now detects, loads, and runs on the V3-equivalent backbone.

Scope

Backbone only, exactly as the issue directs. The two V4-specific features are deferred to separate follow-up issues and are intentionally not implemented here:

  • HiSA (hierarchical sparse attention)
  • the shared-expert SwiGLU swiglu_limit clamp

Their config fields (head_dim, swiglu_limit, num_nextn_predict_layers) are parsed for forward-compatibility but not applied by the backbone.

What changed

  • src/models/deepseek_v4.rs (new): DeepSeekV4Config + DeepSeekV4Config::to_dsv3_config + DeepSeekV4Model (wraps DeepSeekV3Model).
  • src/models/deepseek_v4_tests.rs (new): config default and V3-mapping unit tests.
  • tests/deepseek_v4_parity.rs (new): reference-config parity + a synthetic forward-equivalence proof that the V4 backbone is bit-identical to DeepSeek-V3.
  • Registration: ModelType::DeepSeekV4 and the deepseek_v4 detection.rs arm; LoadedModel::DeepSeekV4 + delegation; model_metadata.rs config-backed entry; tensor_parallel/inference.rs fallback arch string; ALL_MODEL_TYPES, metadata(), and the all_variants! registry list in mod.rs.
  • docs/supported-models.md: DeepSeek-V4 backbone row.

Test plan

  • cargo fmt
  • cargo check --lib --tests
  • cargo clippy --lib --tests -- -D warnings
  • cargo test --lib deepseek_v4 (5 passed)
  • cargo test --test deepseek_v4_parity (2 passed, incl. forward-equivalence vs DeepSeek-V3)
  • cargo test --lib -- models::detection metadata_tests model_metadata (registry completeness, 19 passed)
  • Real-checkpoint validation deferred (no public deepseek_v4 export yet)

Closes #523

Register a `deepseek_v4` text model that reuses the DeepSeek-V3 backbone: MLA attention, group-limited sigmoid-routed MoE, the unclamped shared-expert DenseMLP, and V3's multi-token-prediction layer handling. `DeepSeekV4Model` is a thin wrapper over `DeepSeekV3Model`; `DeepSeekV4Config` is parsed independently and mapped onto the V3 config via `to_dsv3_config`, mirroring the existing `glm_moe_dsa` wrapper. Config defaults track the mlx-vlm `deepseek_v4` reference.

Scope is the backbone only. The two V4-specific features, HiSA hierarchical sparse attention and the shared-expert SwiGLU `swiglu_limit` clamp, are tracked as separate follow-up issues; their config fields are parsed for forward-compatibility but not applied by this port.

Wiring: `ModelType::DeepSeekV4` plus the `deepseek_v4` detection arm, the `LoadedModel` variant and delegation, the config-backed metadata entry, the tensor-parallel fallback arch string, `ALL_MODEL_TYPES`, `metadata()`, and a `docs/supported-models.md` row.

Tests: `deepseek_v4_tests` pin the config defaults and the V3 mapping; `tests/deepseek_v4_parity.rs` asserts the config maps to reference-derived V3 values and that the V4 backbone produces bit-identical logits to DeepSeek-V3 on shared synthetic weights.
@inureyes inureyes added type:enhancement New features, capabilities, or significant additions priority:medium Medium priority area:models Model architectures, weights, loading, metadata status:review Under review status:done Completed and removed status:review Under review labels Jul 1, 2026
@inureyes

inureyes commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

Closing without merging, per maintainer decision. This PR ported DeepSeek-V4 as a thin wrapper over DeepSeek-V3. Investigation against the in-tree reference (references/mlx-vlm/mlx_vlm/models/deepseek_v4/) and the real DeepSeek-V4-Flash checkpoint showed the real deepseek_v4 architecture is genuinely novel (HyperConnections, pooled-KV Compressor, HiSA Indexer, hash-routed MoE with sqrtsoftplus, MultiLinear output projection), not a V3 variant, so a V3 wrapper cannot load or run a real V4 checkpoint. Issue #523 is re-scoped to a faithful port of the real architecture. This PR's diff is preserved here for reference.

@inureyes inureyes closed this Jul 1, 2026
@inureyes
inureyes deleted the feature/issue-523-deepseek-v4-backbone branch July 2, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:medium Medium priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): port DeepSeek-V4 (MLA + group-limited MoE backbone)

1 participant