tracking: non-generative vision models present in the VLM reference but out of mlxcel's LLM/VLM-generation scope

[inureyes]

Summary

A model-coverage audit surfaced the model families below as not supported by mlxcel. They are task-specific vision models (object detection, instance segmentation, 3D body estimation, grounding, diffusion image generation) rather than the chat/completions LLMs and VLMs that mlxcel's text-generation runtime targets. Loading any of them today fails in src/models/detection.rs::get_model_type: a config.json whose model_type has no match arm errors with Unsupported model type: <type>, and checkpoints without a top-level model_type (the diffusion image models) fail earlier with model_type not found.

This issue records the scope decision for each family so that coverage questions have a recorded answer. It is a tracking document, not a work item: closing it requires no code. One family on the original list, RT-DETRv2, has since been implemented and is noted below as resolved.

mlxcel is not strictly text-in/text-out today. Three non-chat paths already exist and serve as precedent for any future re-scoping:

• mlxcel detect (src/commands/detect.rs, src/vision/detection/): object detection, bounding-box output.
• Whisper (src/models/whisper/): encoder-decoder speech-to-text via /v1/audio/transcriptions.
• Kokoro (src/models/kokoro/): text-to-speech, audio output via /v1/audio/speech.

Families

rt_detr_v2 (RT-DETRv2): resolved, already supported

Real-time detection transformer: ResNet-vd CNN backbone, hybrid encoder (AIFI plus FPN/PAN), and a 6-layer deformable-attention decoder with iterative box refinement. Outputs class logits and boxes for a fixed label set resolved through id2label. Task: object detection, including document layout detection (Docling layout checkpoints).

Status: supported. Implemented in src/vision/detection/rt_detr_v2/ and exposed through the mlxcel detect subcommand (src/commands/detect.rs), which prints l, t, r, b, label, confidence rows or JSON. It sits outside the generate loop by design; detection models produce boxes, not a token stream.

sam3 (SAM 3)

Open-vocabulary detection, instance segmentation, and video tracking (~860M parameters). A ViT-L backbone with windowed plus global attention and 2D RoPE feeds an FPN neck, a DETR encoder/decoder stack (200 queries, box refinement), and a mask decoder; a CLIP text encoder embeds the text prompt for open-vocabulary scoring. A separate memory-based tracker propagates masklets across video frames. Checkpoints ship a hierarchy of component types under a top-level model_type: "sam3_video" with a "sam3" detector sub-config.

Task: detection plus segmentation plus tracking. Output: boxes, per-instance masks, confidence scores. It generates no text; the text encoder only embeds prompts. Supporting it would require a segmentation-mask output path (mask tensors or encoded mask images), a DETR/mask-decoder model family, and for video, streaming frame input with tracker memory state, none of which the generation runtime or the detect box path provides.

sam3_1 (SAM 3.1)

Extension of SAM 3 (~873M parameters) adding multi-object "Object Multiplex" tracking: a tri-branch FPN neck (detection, interactive, propagation), a multiplex mask decoder handling up to 16 objects per forward pass, and decoupled memory attention. Detection stack and outputs match SAM 3. Ships as model_type: "sam3.1_video" with a "sam3.1" detector sub-config.

Task: detection plus segmentation plus multi-object video tracking. Output: boxes, masks, tracked masklets. Same gap as SAM 3: mlxcel has no mask output path and no video/tracker-state loop; the two families would be one port effort since they share most components.

sam3d_body (SAM 3D Body)

Single-image 3D human body mesh estimation (~720M parameters). A DINOv3-H+ ViT backbone encodes a cropped person image; a prompt encoder (keypoint and hand-box embeddings) plus a 6-layer cross-attention decoder produce pose, camera, and keypoint tokens; regression heads drive a parametric body model (forward kinematics, blend shapes, linear blend skinning). Ships as model_type: "sam3d_body" with a "dinov3" vision sub-config and no text component at all.

Task: 3D human pose/mesh regression. Output: mesh vertices, joint transforms, 3D keypoints, camera parameters. There is no text anywhere in the model, and the outputs are geometry buffers; supporting it would mean a new mesh/keypoint output path and a parametric body-model runtime with no overlap with mlxcel's existing code.

rfdetr (RF-DETR)

Real-time detection transformer built on a DINOv2 ViT backbone with windowed attention, a multi-scale projector, two-stage query selection, and a deformable-attention decoder (300 queries, group-DETR), with an optional instance-segmentation head. Fixed COCO-style label set, no text components. Ships as model_type: "rf-detr".

Task: object detection, optional instance segmentation. Output: boxes, scores, labels, optional masks. Its box path could in principle ride the existing mlxcel detect subcommand next to RT-DETRv2 (src/vision/detection/), so it is the cheapest family here to re-scope; it still needs a DINOv2 backbone, the deformable decoder, and (for masks) a mask output path.

locateanything (LocateAnything)

A 3B generative grounding VLM: a MoonViT vision tower with an MLP connector into a Qwen2.5-style decoder-only text backbone. Its standard decode path is ordinary autoregressive text generation whose output interleaves coordinate control tokens (box/coord/ref markers) encoding located regions; it also ships a specialized parallel-box-decoding path with multi-token-prediction heads (n_future_tokens: 6) for fast box emission. Ships as model_type: "locateanything" with a "qwen2" text sub-config and "moonvit" vision sub-config.

Task: visual grounding, referring-expression localization, open-vocabulary localization. Output: text containing box coordinates. Architecturally it is close to mlxcel's VLM runtime (MoonViT already exists for Kimi-VL in src/vision/encoders/kimi_vl.rs, Qwen2 in src/models/qwen2.rs), which makes it the strongest VLM-shaped re-scoping candidate. It stays here because its value is the grounding contract: coordinate-token post-processing into boxes and the parallel-box-decoding path, neither of which the chat-completions API represents. A minimal port (plain autoregressive decode, raw coordinate tokens in the text) would work but deliver little without that contract.

florence2 (Florence-2)

A unified multi-task vision seq2seq model. A DaViT vision backbone produces image tokens that are concatenated with the text prompt and fed to a BART-style encoder-decoder transformer (6 encoder plus 6 decoder layers, learned absolute positions). One text-generation interface covers captioning, OCR, detection, dense region captioning, grounding, and segmentation: for spatial tasks, the decoder emits location tokens that post-process into boxes or regions. Ships as model_type: "florence2" with a "davit" vision sub-config.

Task: prompt-keyed multi-task vision (caption/OCR/detect/ground). Output: text tokens, some of which encode locations. It does generate text autoregressively, but as an encoder-decoder: mlxcel's generation engine is decoder-only, and the only encoder-decoder precedent is Whisper's dedicated ASR path. Supporting it would require either a seq2seq generation loop in the engine or a Whisper-style dedicated pipeline, plus task-prompt handling and location-token post-processing.

flux2 (FLUX.2 Klein)

Latent flow-matching text-to-image generation and image editing (4B and 9B variants). A dual-stream plus single-stream MMDiT transformer denoises latents over an iterative scheduler, conditioned on prompt embeddings from a bundled LLM-style text encoder, with a VAE for latent encode/decode. Checkpoints are multi-component directories (transformer/, vae/, text encoder) with no single top-level model_type; the family identifier is flux2. In mlxcel such a directory fails at load with model_type not found.

Task: image generation and editing. Output: images. Supporting it would require an entirely new inference stack: a diffusion sampling loop, a VAE, multi-component checkpoint loading, and an image output path (an images-generations style endpoint), none of which exist in mlxcel.

ideogram4 (Ideogram 4)

Diffusion text-to-image generation. A DiT transformer (34 layers, 4608 hidden, multimodal RoPE) denoises VAE latents, conditioned on features from a large external LLM text encoder; sampler presets trade steps for quality. Checkpoints are identified by a model_index.json pipeline class rather than a config.json model_type; the family identifier is ideogram4. The public checkpoint is FP8 and license-gated.

Task: image generation. Output: images. Same gap as FLUX.2: diffusion loop, VAE, multi-component loading, image output path. If mlxcel ever grows an image-generation stack, FLUX.2 and Ideogram 4 would share most of that infrastructure.

nemotron_labs_diffusion (Nemotron Labs Diffusion 8B)

Despite the name, this is a text-only diffusion language model, not an image model: an 8B dense decoder-only transformer (GQA, SwiGLU, YaRN-scaled RoPE, 262k context) with an untied diffusion output head and a mask token. One checkpoint supports three decode modes: standard autoregressive, masked block-diffusion denoising (block size 32), and linear self-speculative decoding (diffusion draft, autoregressive verify, bundled LoRA adapter). Ships as model_type: "nemotron_labs_diffusion".

Task: text generation. Output: text. This is the clearest re-scoping candidate in this issue: mlxcel already runs a block-diffusion text model (DiffusionGemma, src/models/diffusion_gemma/, accepted as diffusion_gemma in src/models/detection.rs), so the runtime shape exists. It is recorded here only because it arrived in the same audit batch; it should graduate to a dedicated port issue when prioritized rather than be treated as out of scope.

Re-scoping criteria

A family moves from this issue to a dedicated port issue when all of the following hold:

1. A concrete user or product need names the family (a checkpoint someone wants to serve).
2. The output path exists or is being built: box outputs can ride mlxcel detect today; masks, meshes, and images need new output paths that should be scoped as their own infrastructure work first.
3. A specific public checkpoint is named for validation, with its config.json (or component layout) confirmed.
4. Weight layout is accounted for. Every family here carries convolution stacks (CNN backbones, ViT patch embeddings, VAEs), and checkpoints serialize 4-D convolution weights in (out_channels, in_channels, kernel_h, kernel_w) order, while mlxcel's MLX runtime runs channels-last convolutions expecting (out_channels, kernel_h, kernel_w, in_channels). A port must permute these at load; src/vision/detection/rt_detr_v2/sanitize.rs is the in-tree precedent (it also guards against double-transposing checkpoints already stored in MLX layout).

Ranked by current proximity to mlxcel's runtime:

• nemotron_labs_diffusion: text-only, DiffusionGemma precedent covers the decode modes; needs only a port issue.
• locateanything: standard VLM architecture with in-tree components; needs a decision on the grounding output contract.
• rfdetr: fits the existing detect box path; needs a DINOv2 backbone and deformable decoder.
• florence2: needs a seq2seq generation loop or a Whisper-style dedicated pipeline.
• sam3 / sam3_1: need a mask output path and (for video) tracker state streaming.
• flux2 / ideogram4: need a full diffusion image stack.
• sam3d_body: needs a geometry output path and a parametric body model; furthest from any existing code.

When re-scoping, follow the port-issue format used by #533 through #547 (Summary with the model_type string, What it is, In-tree reuse, Scope, Effort) and link the new issue here.

Non-goals

• Implementing any of these families as part of this issue; this is a decision record only.
• Building a general image-generation, segmentation-mask, or 3D-geometry output pipeline; each would be its own scoped infrastructure issue.
• Extending the OpenAI-compatible API surface with image/mask/mesh response types.
• Tracking generative VLM port candidates; those live in tracking: VLM-reference model families needing architecture investigation before a port #548 and in the individual port issues feat(models): port DeepSeek-VL2 (MoE text + SigLIP/SAM vision) #533 through feat(models): port GLM-OCR (ViT + GLM-4 text OCR) #547.