Skip to content

feat(models): port DeepSeek-V4 (MLA + group-limited MoE backbone) #523

Description

@inureyes

Summary

Add DeepSeek-V4 support. mlxcel has DeepSeek V2 / V3 / V3.2 but not V4.

Scope correction (2026-07)

An initial port attempt (PR #592) assumed "V4 = a V3 backbone + HiSA + a shared-expert swiglu_limit clamp" and implemented DeepSeekV4Model as a thin wrapper over DeepSeekV3Model. That premise is incorrect. Verified against the in-tree reference references/mlx-vlm/mlx_vlm/models/deepseek_v4/ and the real DeepSeek-V4-Flash checkpoint, DeepSeek-V4 is a genuinely novel architecture, not a V3 variant. A V3 wrapper does not load or run a real V4 checkpoint, so PR #592 was closed without merging and this issue is re-scoped to a faithful port of the real architecture.

Real V4 architecture (port target)

Port references/mlx-vlm/mlx_vlm/models/deepseek_v4/ faithfully. The distinguishing components are:

  • HyperConnections in place of plain residual connections.
  • A pooled-KV Compressor (not V3-style MLA).
  • HiSA hierarchical sparse attention via an Indexer.
  • Hash-routed MoE with sqrtsoftplus gating (not V3 group-limited softmax routing).
  • A MultiLinear output projection.

Scope

  • Add ModelType::DeepSeekV4 + a detection.rs arm for deepseek_v4 backed by a real V4 backbone (not a V3 alias).
  • Implement the components above as net-new modules, reusing V3 pieces only where genuinely shared.
  • Register across the arch/metadata tables and docs/supported-models.md; add unit + reference-parity tests.
  • Validate on the real DeepSeek-V4-Flash checkpoint (available locally at /home/inureyes/models/deepseek-v4-flash-4bit).

Effort

High: net-new, multi-component architecture. Consider splitting HyperConnections, the pooled-KV Compressor, the HiSA Indexer, and hash-routed MoE into dedicated sub-issues.

Notes

The rejected V3-wrapper attempt is preserved in the closed PR #592 for reference only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:readyReady to be worked ontype:enhancementNew features, capabilities, or significant additions

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions