feat(minimax-h3): port the MiniMax-H3 omni-modal video+audio DiT (portable path complete; speed GPU-gated) - #26
Open
localai-bot wants to merge 65 commits into
Open
feat(minimax-h3): port the MiniMax-H3 omni-modal video+audio DiT (portable path complete; speed GPU-gated)#26localai-bot wants to merge 65 commits into
localai-bot wants to merge 65 commits into
Conversation
MiniMax-H3 (`MiniMaxAI/MiniMax-H3`) is the project's FIRST diffusion
architecture: a 33.1B CFG-distilled joint video+audio transformer served by
vLLM-Omni over `/v1/videos`. One request runs a 50-step flow-matching denoise
loop, forwarding the DiT ONCE PER STEP over the whole packed sequence, then
decodes latents to 24 FPS frames plus 32 kHz stereo through two VAEs. It is NOT
autoregressive: no KV cache, no sampler, no logits. It is therefore deliberately
NOT registered in the causal-LM registry, and the born-on-the-runner seam does
not apply to it by construction.
Ported from vllm-project/vllm-omni, vllm_omni/diffusion/models/minimax_h3/:
minimax_h3_transformer.py -> minimax_h3.{h,cpp} (arch config, 3D MM-RoPE,
time embedder, AdaLN proj, DiT block, token
refiner, final layer, packed forward, weight
contract, grouped-qkv reorder)
denoise_loop.py -> minimax_h3.cpp (CFG-distilled driver)
packed_sequence.py -> minimax_h3_packing.cpp (fl2va + ref2va layouts)
packed_tokens.py -> minimax_h3_packing.cpp (latent <-> token packing)
scheduling_..._euler_...py -> minimax_h3_packing.cpp (euler eta0 scheduler)
No new kernel was added. The packed varlen NON-CAUSAL attention routes through
the shared `vt::DFlashBlockAttention(causal=false)` -- its per-document
bidirectional contract is exactly upstream's cu_seqlens varlen FA call -- and
every projection through `vt::MatmulBT`.
HARDWARE VERDICT (recorded, not worked around): the checkpoint is ~354 GB and
upstream validates on 4x NVIDIA B300 at ~133 GB peak per rank. One GB10 has 119
GiB UNIFIED memory, so CPU offload cannot help. End-to-end H3 is impossible on
this project's hardware; no e2e result and no speed number is claimed.
What IS gated, and exactly: upstream's H3 modules are pure Python, so they are
executed at REDUCED DIMENSIONS as the oracle. gen-minimax-h3-goldens.py imports
packed_sequence/packed_tokens/scheduling by file path (bypassing the package
__init__, so neither vllm nor aenum is needed) and restates the DiT at TP=1;
both sides rebuild weights and inputs from an identical FNV-1a + splitmix64
stream, so not one weight byte is checked in.
test_minimax_h3: 10/10 cases, 2539 assertions, clean CPU build (0 warnings)
- fl2va + ref2va packed layouts EXACT, fp64 position grid BIT-EXACT
- patchify / unpatchify / audio pack EXACT + round-trip identity
- euler-ancestral eta0 scheduler EXACT
- DiT forward max abs diff 1.6e-7 (video) / 1.5e-7 (audio)
- denoise-loop invariants (pinned rows reset per step, targets advance)
The fp64 grid is gated bit-exact because it feeds RoPE, which required matching
upstream's arithmetic ORDER: numpy linspace(endpoint=False) evaluates
i*step + start, and upstream keeps a numpy-PAIRWISE and a Python-SEQUENTIAL span
sum deliberately separate (packed_sequence.py:101-113).
Not yet built, recorded honestly in the spec: device-resident/bf16 forward and
the fusion folds (W2b, where speed work begins -- upstream reports the DiT at
88% of request latency), the H3-Encoder on our existing Qwen3-VL tower (W3), the
two VAEs -- which are checkpoint REMOTE CODE under trust_remote_code and must be
REIMPLEMENTED in C++, not adapted (W4/W5) -- pipeline/tasks (W6), /v1/videos plus
MP4 muxing, which needs a new dependency decision (W7), and USP multi-GPU (W8).
OPEN: there is no vllm-omni parity PIN; the upstream-sync protocol covers only
the vLLM repo, and H3 is both beyond the pin and outside that repository.
Record: .agents/specs/minimax-h3.md, model-matrix row
MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit (PARTIAL) + checklist/rollup,
roadmap ROAD-V1-H3, docs/STATUS.md, docs/BENCHMARKS.md (PENDING, hardware
blocked), parity-ledger and state entries.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Three additions, and one correction to the previous commit's hardware verdict.
CORRECTION FIRST. The previous commit concluded MiniMax-H3 end-to-end was
"impossible on this project's hardware". That was wrong. It reasoned from ONE
artifact -- the BF16 release (~354 GB, validated on 4x B300). Quantized H3
checkpoints exist and they fit one GB10 (119 GiB unified):
realrebelai/MiniMax-H3_GGUFs DiT Q3_K_M 15.6 GB + Qwen3-VL encoder Q4_K_M
14.6 GB + VAEs ~11 GB => ~41 GB working set
lilcheaty/MiniMax-H3-NVFP4 NVFP4 DiT + AWQ encoder + both VAEs
So e2e AND a speed comparison are REACHABLE; they are gated on the remaining
bricks (encoder, VAEs, pipeline), not on hardware. NVFP4 is the likely speed
path: sm_121 has native FP4 tensor cores and our NVFP4 stack is the most tuned
one we own. Still no e2e or speed NUMBER is claimed here.
1. BF16 PRODUCTION STREAM (minimax_h3.cpp). The DiT forward now runs upstream's
production dtype policy as well as the f32 parity path: the block stream is
bf16 while the fp32 islands (both patch projections, the time embedder, both
output heads -- minimax_h3_transformer.py:85-101) stay fp32, with the explicit
casts of _modulate_scale_shift / _modulate_gate reproduced at their sites.
Gated against a bf16 upstream golden: max abs diff 2.4e-3 (bf16 scale; same
cast points, different GEMM accumulation order).
2. REQUEST PLANNING (minimax_h3_planner.cpp) <- time_request.py:5-61 and
pipeline_minimax_h3.py:121-122, 207-222, 374-434. Frame snapping to 17n+5,
video/audio latent shapes, the rectified-flow time-shift sigma schedule,
canvas resolution, reference-image rescale, and t2va/fl2va/ref2va dispatch.
EXACT vs upstream. Note Python's round() is half-to-EVEN and both
_align_multiple and _audio_latent_t depend on it.
3. GGUF ARM (minimax_h3_gguf.cpp). The ComfyUI-format H3 GGUF keeps the
checkpoint's own parameter names, so THE NAME MAP IS THE IDENTITY: all 535
tensors of MiniMax-H3-FL2VA-Q3_K_M.gguf match the contract
EnumerateMiniMaxH3DitTensors derived from upstream SOURCE -- the weight
contract is now validated against a real checkpoint. Two shape rules:
- GGUF `ne` is reversed relative to torch;
- `comfy.gguf.orig_shape.<name>` overrides it where ComfyUI reshaped a
tensor for quant-block alignment (the 50 AdaLN projections are logical
[96768, 2688], and 2688 is not a multiple of the 256-element Q3_K block).
Geometry is derived from SHAPES ALONE, because a ComfyUI GGUF ships no
transformer config -- that is what lets a GGUF load without the original repo.
The gate needs no download: a GGUF header is self-delimiting and at the front
of the file, so a 4 MiB HTTP range request yields the whole manifest, which
scripts/gen-minimax-h3-gguf-manifest.py freezes into a fixture (names, dims,
types, orig_shape -- no weight bytes).
test_minimax_h3: 13/13 cases, 3907 assertions, clean CPU build (0 warnings).
Record updated with the corrected verdict throughout: .agents/specs/minimax-h3.md
(section 0 rewritten, W9/W10 bricks added), the model-matrix row + checklist,
roadmap ROAD-V1-H3, docs/STATUS.md, docs/BENCHMARKS.md (the "not reproducible"
disposition is WITHDRAWN), plus ledger and state entries.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: quantized arms land, and the hardware verdict is correctedPushed
So e2e and a speed comparison are reachable, gated on the remaining bricks (encoder → VAEs → pipeline), not on hardware. NVFP4 is the likely speed path: sm_121 has native FP4 tensor cores and our NVFP4 stack is the most tuned one we own. What this push adds1. BF16 production stream. The DiT forward now runs upstream's production dtype policy alongside the f32 parity path — bf16 block stream with the fp32 islands (both patch projections, time embedder, both output heads) preserved. Gated against a bf16 upstream golden at max abs diff 2.4e-3. 2. Request planning ( 3. GGUF arm. The best result here: the name map is the identity. All 535 tensors of
Geometry is derived from shapes alone, since a ComfyUI GGUF ships no transformer config — that's what lets a GGUF load without the original repo. The gate needed no download: a GGUF header is self-delimiting and at the front of the file, so a 4 MiB HTTP range request yields the whole manifest, which Status
Next: download a quantized checkpoint and close the e2e loop (encoder on our existing Qwen3-VL tower → the two VAEs → pipeline), then the NVFP4 arm for speed. |
The biggest unknown in this lane was that H3's two VAEs are checkpoint REMOTE
CODE: they ship inside the HF repo (FL2VA/{audio,video}_vae/*.py) and are loaded
through get_class_from_dynamic_module under trust_remote_code. vLLM-Omni only
ADAPTS them (vae.py:41-53), so a no-Python engine has to reimplement them.
The remote code is now in hand (~130 KB of Python, NOT vendored here -- it ships
under the MiniMax H3 Community License with the checkpoint), and the AUDIO side
is done.
WHAT IT IS. A DAC-lineage BigVGAN vocoder, per the checkpoint's config.yaml +
metadata.json: dec_in_proj (32 -> 2048, k=1) then conv_pre -> 7 upsample stages
(ConvTranspose1d; rates 5,5,2,2,2,2,2 / kernels 9,9,4,4,4,4,4), each followed by
3 AMPBlock1 residual stacks (kernels 3,7,11, dilations 1,3,5) whose outputs are
AVERAGED -> anti-aliased SnakeBeta -> conv_post (1 ch, k=7, no bias) -> clamp to
[-1, 1] (H3 sets use_tanh_at_final=false). 32 kHz, 2 channels.
Two details that are easy to get wrong, both gated:
* Every conv is WEIGHT-NORMALIZED, so the checkpoint stores (g, v) pairs as
parametrizations.weight.original0/original1 and the loader materializes
g * v / norm(v) with the norm over every dim except dim 0. ConvTranspose1d
weight is [in, out, k], so its weight-norm dim 0 is the INPUT channel.
* The anti-aliased activation is up 2x -> SnakeBeta -> down 2x through a
KAISER-SINC filter COMPUTED at load time and never loaded -- needing a
Bessel I0, torch's periodic=false kaiser window, and REPLICATE padding. The
filter is gated separately so a filter bug cannot masquerade as a decoder bug.
GATE. scripts/gen-minimax-h3-audio-vae-goldens.py imports the CHECKPOINT'S OWN
modules and runs them at reduced dimensions as the oracle, with weights from the
shared H3Rand stream so no weight byte is checked in:
waveform max abs diff 4.2e-9 (f32 round-off)
kaiser filter max abs diff 3.0e-8
The first golden had 18 of 32 samples pinned at the final clamp, which would have
HIDDEN errors; the generator's weight scale is tuned so the output is fully
unsaturated, and the test asserts non-saturation explicitly.
test_minimax_h3: 14/14 cases, 3983 assertions, clean CPU build (0 warnings).
NOT done: the encode-side determinism-context semantics, and the VIDEO VAE (W4),
which is the largest remaining brick -- klvae.py alone is ~48 KB, plus a CNN/ViT
hybrid, tiling, and a parallel path.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: the audio VAE is reimplemented (
|
| max abs diff | |
|---|---|
| waveform | 4.2e-9 (f32 round-off) |
| kaiser-sinc filter | 3.0e-8 |
One methodology note worth flagging: the first golden had 18 of 32 samples pinned at the final clamp, which would have hidden errors. The weight scale is tuned so the output is fully unsaturated, and the test now asserts non-saturation explicitly.
test_minimax_h3: 14/14 cases, 3983 assertions, clean build, all checkers green.
Remaining
Video VAE (W4) is the largest brick left — klvae.py alone is ~48 KB, plus a CNN/ViT hybrid, tiling, and a parallel path. Then the encoder (W3, mostly reuse of our Qwen3-VL tower) and the pipeline (W6). After those, an e2e run on a quantized checkpoint is reachable — and only then is a speed number meaningful.
…m real manifests
Generalizes the GGUF header trick: a safetensors header is ALSO front-loaded (an
8-byte length plus JSON) and only tens of KB even for a 10 GB file, so one HTTP
range request captures the entire tensor manifest. scripts/gen-minimax-h3-
safetensors-manifest.py turns that into a C++ fixture, and two real MiniMax-H3
checkpoints are now gated without downloading a byte of payload.
1. THE NVFP4 CHECKPOINT IS OUR LAYOUT, EXACTLY.
lilcheaty/MiniMax-H3-NVFP4 (1051 tensors) is the textbook compressed-tensors
triple:
weight U8 FP4 packed 2-per-byte; [21504, 2688] for a logical
[21504, 5376] fused qkv
weight_scale F8_E4M3 one per group of 16 along K; [21504, 336]
weight_scale_2 F32 one global scalar
258 quantized projections, each carrying all three. The fp32/bf16 ISLANDS
(both patch projections, the time embedder, both output heads, the norms and
rope.inv_freq) are left unquantized, so the DiT's dtype policy survives. And
the names are IDENTICAL to the contract we derived from upstream source.
So W10 is loader WIRING onto the NVFP4 stack this project already tuned, not a
new quantization scheme. That matters because W10 is the speed path: sm_121
has native FP4 tensor cores.
2. THE VIDEO VAE DECODER IS A ViT, NOT A CNN.
W4 was scoped as "port a 48 KB klvae.py" and treated as the scary brick. The
real 560-tensor manifest says otherwise: the ENCODER is the 3D CNN (116
tensors, rank-5 Conv3d down blocks), but the DECODER -- the only half
generation needs -- is a 36-block TRANSFORMER (440 tensors: attn.to_qkv /
attn.to_out, ff.w1 / ff.w2, two norms and two learned residual scales per
block, plus x_embedder, mask_token, register_tokens, norm_out, proj_out). fp32
throughout. We have every primitive for that, so W4 is materially smaller than
previously recorded.
test_minimax_h3: 16/16 cases, 5908 assertions, clean CPU build (0 warnings).
Still NO end-to-end result and NO speed number. What changed is that the two
remaining unknowns on the critical path are now measured rather than guessed.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: NVFP4 arm grounded, video VAE re-scoped (
|
| tensor | dtype | shape (fused qkv, block 0) |
|---|---|---|
weight |
U8 | [21504, 2688] — FP4 packed 2-per-byte, logical [21504, 5376] |
weight_scale |
F8_E4M3 | [21504, 336] — one per group of 16 along K |
weight_scale_2 |
F32 | scalar global |
258 quantized projections, each carrying all three. The fp32/bf16 islands (patch projections, time embedder, output heads, norms, rope) are left unquantized, so the DiT's dtype policy survives. Names identical to the contract derived from source.
So W10 is loader wiring onto the NVFP4 stack we already tuned for Laguna — not a new quant scheme. That matters because this is the speed path: sm_121 has native FP4 tensor cores.
2. The video VAE decoder is a ViT, not a CNN
I had scoped W4 as "port a 48 KB klvae.py" and called it the scary brick. The real 560-tensor manifest says otherwise:
- Encoder = the 3D CNN (116 tensors, rank-5 Conv3d down blocks)
- Decoder = a 36-block transformer (440 tensors:
attn.to_qkv/to_out,ff.w1/w2, two norms + two learned residual scales per block, plusx_embedder,mask_token,register_tokens,norm_out,proj_out), fp32 throughout
Generation only needs the decoder. We have every primitive for a ViT, so W4 is materially smaller than previously recorded.
test_minimax_h3: 16/16 cases, 5908 assertions, clean build, all checkers green.
Honest status
Still no e2e result and no speed number. What changed is that the two biggest unknowns left on the critical path are now measured instead of guessed — and both came back smaller than feared. Remaining: W4 video-VAE decoder (ViT port), W3 encoder (Qwen3-VL reuse), W6 pipeline, then W10 loader wiring. Only after those does a speed measurement mean anything.
…rtial)
The video VAE is checkpoint REMOTE CODE like the audio one, so it must be
reimplemented rather than adapted. Its real 560-tensor manifest showed the
decoder -- the only half generation needs -- is a 36-block transformer, and this
ports that repeated unit.
Per block, all fp32 (base_module.py:200-281):
h += scale1 * Attention(RMSNorm(h))
h += scale2 * GatedSiLU_FeedForward(RMSNorm(h))
with scale1/scale2 LEARNED PER-CHANNEL vectors (not scalars) and per-head RMS
q/k normalization carrying NO affine weight.
THE TRAP THIS CATCHES. This ViT's to_qkv output is PER-HEAD INTERLEAVED: upstream
does qkv.view(B, S, -1, 3*dim_head) then chunk(3, dim=-1), so the layout is
[head0_q | head0_k | head0_v | head1_q | ...] -- NOT the [q_all | k_all | v_all]
that the H3 DiT's fused qkv uses. Both layouts are the same SIZE, so reading it
the DiT way raises no error and no shape mismatch; it just produces a
plausible-but-wrong image. Only a numeric gate against the real module finds it.
GATE. scripts/gen-minimax-h3-video-vae-goldens.py executes the CHECKPOINT'S OWN
base_module.TransformerBlock at reduced dimensions as the oracle, with weights
from the shared H3Rand stream (no weight bytes checked in):
video VAE decoder block max abs diff 6.0e-8
The bundle imports a handful of diffusers symbols (a logger, two mixin bases, two
no-op decorators); the generator STUBS them rather than taking the whole diffusers
dependency just to run an oracle.
test_minimax_h3: 17/17 cases, 5911 assertions, clean CPU build (0 warnings).
W4 REMAINS PARTIAL. Still to do: the 36-block stack surround (x_embedder,
mask_token / register_tokens, 3D RoPE, norm_out / proj_out, unpatchify, tiling)
and the 3D-CNN encoder, which is only needed for image/video CONDITIONING rather
than for generation output. Still no e2e run and no speed number.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: video-VAE decoder block ported (
|
Both VAE DECODERS are now done. This adds the stack surround around the already gated TransformerBlock, so the whole generation-critical half of the video VAE reproduces the checkpoint's own ViT3DDecoder: _pack_tensors_3d (channels-last flatten) x_embedder suffix = register tokens + a ZERO cls token 3D RoPE 36-block stack norm_out <- LAYER norm (the blocks use RMS) proj_out _unpack_tensors_3d -> [C, T*pt, H*ps, W*ps] Run at the REAL hyperparameters, read from the checkpoint's source/config.json::vit_decoder_kwargs: 36 layers, 32 heads x 64, rms_norm affine, qk rms_norm WITHOUT affine, gated SiLU, rope_theta 100.0, rope_dim_ratio 0.75. 3D ROPE DETAIL. RotaryEmbeddingND builds angles from LENGTH-NORMALIZED token ids ((i + 0.5)/n mapped into [-1, 1)), scales them by 2*pi (use_angle=True), and concatenates the three per-axis frequency blocks before TILING the result twice to fill rot_dim. The suffix tokens carry id 0, so their cos/sin are 1/0 -- an identity rotation -- which falls out of initializing the tables that way. GATE (vs the checkpoint's own module at reduced dimensions): video VAE full ViT3D decoder max abs diff 8.9e-8 video VAE decoder block max abs diff 6.0e-8 audio VAE decoder max abs diff 4.2e-9 test_minimax_h3: 18/18 cases, 5918 assertions, clean CPU build (0 warnings). REMAINING on the VAE side: video tiling (vae_tile_size 256, overlap 64) and the 3D-CNN ENCODER -- and the encoder is only needed for image/video CONDITIONING (fl2va/ref2va), not for producing output frames, so a t2va path does not need it. Still no e2e run and no speed number. W3 (H3-Encoder on our Qwen3-VL tower) and W6 (pipeline) are what stand between here and an end-to-end t2va run. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: video-VAE ViT3D decoder complete (
|
| max abs diff | |
|---|---|
| video VAE full ViT3D decoder | 8.9e-8 |
| video VAE decoder block | 6.0e-8 |
| audio VAE decoder | 4.2e-9 |
test_minimax_h3: 18/18 cases, 5918 assertions, clean build, all checkers green.
What's left
On the VAE side: video tiling (vae_tile_size 256 / overlap 64) and the 3D-CNN encoder — and the encoder is only needed for image/video conditioning (fl2va/ref2va), not for producing output frames, so a t2va path doesn't need it.
On the critical path to e2e: W3 (H3-Encoder on our existing Qwen3-VL tower) and W6 (pipeline). Then W10 NVFP4 loader wiring.
Still no e2e run and no speed number — and neither is reachable from this machine (no GPU). That's a run on the DGX once W3+W6 close.
The H3-Encoder produces the [seq, 5120] prompt_embeds the DiT consumes. Its
architecture is a Qwen3-VL, which this project already ports, so the value of
this change is pinning down and GATING the three H3-specific deltas:
1. LAYER TRUNCATION. num_layers = min(config.num_hidden_layers, 50). The gate's
config deliberately declares MORE layers than are selected so truncation is
actually exercised, plus an assertion that it never EXTENDS a shallower model.
2. UNNORMALIZED OUTPUT -- the load-bearing one. H3 consumes the hidden state
straight out of layer 49 with NO final RMSNorm, unlike a stock Qwen3-VL text
model. Applying one raises no error and changes no shape; it just silently
shifts every conditioning vector.
3. DEEPSTACK. Visual features are added at the visual token positions after each
of the first len(deepstack_visual_embeds) layers. The test asserts DeepStack
actually CHANGES the result, so a no-op injection cannot pass.
The layer itself is the familiar pre-norm block: RMSNorm -> fused-QKV attention
with per-head q/k RMSNorm and interleaved M-RoPE -> causal GQA -> o_proj ->
residual; RMSNorm -> gated-SiLU MLP -> residual.
GATE. scripts/gen-minimax-h3-encoder-goldens.py runs the UPSTREAM
MiniMaxH3Qwen3VLTextModel at reduced dimensions. encoder.py imports exactly one
vllm symbol (vllm.logger), so a one-line stub lets the oracle run without vllm or
any of its dependencies -- the same pattern as the diffusers stub for the video
VAE.
encoder text tower, plain path max abs diff 1.2e-7
encoder text tower, DeepStack path max abs diff 1.2e-7
test_minimax_h3: 19/19 cases, 5930 assertions, clean CPU build (0 warnings).
REMAINS for W3: the encoder's VISION tower (a reuse of our qwen3_vl_vision.cpp
rather than a new port) and the MM processor.
Still no e2e run and no speed number. W6 (pipeline assembly) is now the main
thing standing between here and an end-to-end t2va run.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Update: encoder text tower ported (
|
| Component | Gate |
|---|---|
| packed layout (fl2va + ref2va) | exact, fp64 grid bit-exact |
| latent packing | exact + round-trip |
| flow-matching scheduler | exact |
| DiT forward (f32) | 1.6e-7 |
| DiT forward (bf16 production stream) | 2.4e-3 |
| request planning | exact |
| GGUF manifest (535 real tensors) | exact |
| NVFP4 manifest (1051 real tensors) | exact, layout = ours |
| audio VAE decoder | 4.2e-9 |
| video VAE ViT3D decoder | 8.9e-8 |
| encoder text tower | 1.2e-7 |
Remaining: encoder vision tower + MM processor, W6 pipeline assembly, W7 serving/MP4, W10 NVFP4 loader wiring, W2b device-resident forward, video tiling.
Still no e2e run and no speed number — and neither is reachable from this machine (no GPU, 90 GB free vs a ~41 GB checkpoint). W6 is now the main thing between here and a t2va run on the DGX.
…es (W6)
MiniMaxH3GenerateT2va wires the separately-gated stages into one path:
prompt_embeds -> packed layout -> sigma schedules (video shift 12, audio 3)
-> denoise loop (one DiT forward per step, euler-eta0 update)
-> unpatchify / audio unpack -> per-channel denormalize
-> video ViT3D decoder + audio BigVGAN decoder
-> frames [3, T*pt, H*ps, W*ps] + stereo waveform at 32 kHz
A structural end-to-end test runs the whole thing at reduced dimensions with
random weights. That is explicitly NOT a quality result -- it is proof the stages
COMPOSE: shapes are right, every value is finite, the waveform lands inside
[-1, 1], and the denoise loop demonstrably moves the latents rather than passing
noise straight through.
ASSEMBLING IT CAUGHT A REAL GAP, which is why doing this before the checkpoint
arrives was worth it: the audio decode was missing the checkpoint's dec_in_proj
(Conv1d k=1, vae_latent_channels -> num_mels) ahead of BigVGAN. The DiT emits a
32-wide audio latent while BigVGAN expects 2048 mels; without that projection the
two never meet. It is now applied when the weight is present, leaving the
standalone BigVGAN gate untouched.
DESIGN NOTE: noise is an INPUT, not generated internally. Upstream seeds a torch
CPU generator (pipeline_minimax_h3.py:813-843); reproducing torch's RNG
bit-exactly decides WHICH sample you get, not whether the pipeline is correct, so
it is recorded as an open item rather than guessed at.
test_minimax_h3: 20/20 cases, 6370 assertions, clean CPU build (0 warnings).
REMAINING before a real generation: the encoder's VISION tower (reuse of
qwen3_vl_vision.cpp) and MM processor, fl2va/ref2va conditioning, the quantized
loader wiring (GGUF dequant / NVFP4), and a GPU with the checkpoint on it. The
device-resident forward (W2b) and NVFP4 wiring (W10) are where speed work begins.
Still no run on a real checkpoint and no speed number.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Milestone: the whole t2va path composes end to end (
|
| Component | Gate |
|---|---|
| packed layout (fl2va + ref2va) | exact, fp64 grid bit-exact |
| latent packing / scheduler / request planning | exact |
| DiT forward (f32 / bf16 production stream) | 1.6e-7 / 2.4e-3 |
| GGUF manifest (535 real tensors) | exact, identity name map |
| NVFP4 manifest (1051 real tensors) | exact, layout = our stack |
| audio VAE decoder | 4.2e-9 |
| video VAE ViT3D decoder | 8.9e-8 |
| encoder text tower | 1.2e-7 |
| whole t2va path | composes; correct shapes, finite, in range |
test_minimax_h3: 20/20 cases, 6370 assertions, clean build, all checkers + mutation suites green. No weight bytes checked in anywhere.
Remaining
Encoder vision tower (reuse of qwen3_vl_vision.cpp) + MM processor · fl2va/ref2va conditioning · quantized loader wiring (GGUF dequant / NVFP4) · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident forward (W2b) · video tiling.
No run on a real checkpoint and no speed number. Those need a GPU with the ~41 GB checkpoint on it — a DGX operation, not something reachable from this box.
LoadMiniMaxH3DitFromGguf turns a ComfyUI-format GGUF into a runnable DiT:
* resolve the manifest -- identity name map, `ne` reversal, and the
`comfy.gguf.orig_shape` reshape rule (all three already gated against the
real 535-tensor MiniMax-H3-FL2VA-Q3_K_M.gguf manifest);
* derive the geometry from SHAPES ALONE, because a ComfyUI GGUF ships no
transformer config;
* dequantize every tensor to f32 through the SHARED DequantGgufRowToF32, so
the Q2_K / Q3_K / Q4_K families the H3 GGUFs use are handled by the same code
path every other GGUF model in this tree uses -- no new quant code;
* bind the forward's non-owning views, with a missing tensor throwing BY NAME
rather than yielding a null view the forward would silently read as zeros.
Gated by a synthetic-file LOAD-AND-RUN test rather than a shape check: the
geometry comes back exactly, a loaded weight carries the LOGICAL (torch) shape
rather than the reversed ne, and a REAL DiT forward executes off the loaded
weights with finite, correctly-sized outputs.
test_minimax_h3: 21/21 cases, 6668 assertions, clean CPU build (0 warnings).
Full scripts suite: 128 passed.
NOT covered: a load of the ACTUAL 15.6 GB file (needs the download) and NVFP4
loader wiring (W10 -- the layout is already proven identical to ours).
ENVIRONMENT NOTE: dgx.casa is currently unreachable ("no route to host") and this
workstation has no GPU, so no run on a real checkpoint -- and therefore no speed
number -- is possible from here regardless of how much of the port is done.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
GGUF arm complete — load into a runnable DiT (
|
| Component | Gate |
|---|---|
| packed layout (fl2va + ref2va) | exact, fp64 grid bit-exact |
| latent packing / scheduler / request planning | exact |
| DiT forward (f32 / bf16 production stream) | 1.6e-7 / 2.4e-3 |
| audio VAE decoder | 4.2e-9 |
| video VAE ViT3D decoder | 8.9e-8 |
| encoder text tower | 1.2e-7 |
| GGUF manifest (535 real tensors) | exact, identity map |
| NVFP4 manifest (1051 real tensors) | exact, layout = our stack |
| GGUF load → runnable DiT | geometry from shapes; forward runs |
| whole t2va path | composes; correct, finite, in range |
No weight bytes checked in anywhere — every gate runs against upstream code, the checkpoint's own remote code, or real checkpoint headers fetched by range request.
What remains
Encoder vision tower (reuse of qwen3_vl_vision.cpp) + MM processor · fl2va/ref2va conditioning · NVFP4 loader wiring (W10 — layout already proven identical to ours) · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident forward (W2b) · video tiling.
Environment
dgx.casa is currently unreachable ("no route to host") and this workstation has no GPU. So a run on a real checkpoint — and therefore any speed number — is not possible from here regardless of how much of the port is finished. That's the gating constraint on the remaining goal, not the code.
LoadMiniMaxH3DitFromNvfp4 completes the loader half of the NVFP4 arm. The
compressed-tensors triple
<name>.weight U8 FP4 packed 2-per-byte, [out, in/2]
<name>.weight_scale F8_E4M3 one per group of 16 along K, [out, in/16]
<name>.weight_scale_2 F32 one global scalar
goes through this project's EXISTING DequantNvfp4ToBf16 -- no new quant code,
because the manifest gate had already proven the real checkpoint's layout IS
ours. The fp32/bf16 islands are read as-is, quant sidecars are excluded from the
model tensor set, and the geometry is recovered from the dequantized shapes.
The view binding is now SHARED between the GGUF and NVFP4 arms
(BindMiniMaxH3DitViews), since both land on the same weight contract.
Gated by a synthetic-file LOAD-AND-RUN test: a packed [out, in/2] weight comes
back as the logical [out, in], weight_scale/weight_scale_2 never appear as model
tensors, and a REAL DiT forward executes off the loaded weights.
A MISTAKE WORTH RECORDING. Extracting the shared binder, I applied a blind
`out.` -> `out->` rewrite that corrupted six tensor NAME STRINGS --
"time_embedder.proj_out.bias" became "...proj_out->bias". Two lessons: never
mechanically rewrite across code and string literals in one pass; and the
loader's throw-BY-NAME-on-missing-tensor design caught it instantly, where a
null-view default would have silently bound zeros and produced a
plausible-but-wrong forward. That design choice paid for itself the first time it
was exercised.
test_minimax_h3: 22/22 cases, 6971 assertions, clean CPU build (0 warnings).
NOT done: the DEVICE path that keeps FP4 packed and routes projections through
the cutlass FP4 GEMM -- that is where the speed actually is -- and a run on the
real file. dgx.casa is unreachable and this workstation has no GPU, so no speed
number is obtainable here.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…one (W3) The repeated unit of the H3-Encoder's Qwen3-VL vision tower now matches upstream at 6.0e-8, so both encoder towers' cores are ported (text tower was 1.2e-7). THE TWO ViTs IN THIS MODEL DISAGREE ON QKV LAYOUT. The video VAE decoder's ViT is PER-HEAD INTERLEAVED ([head][q,k,v]); this vision tower is [q_all, k_all, v_all]. Same tensor size, no error either way -- just a wrong result. Having now ported both, it is worth stating plainly: never assume a qkv layout carries across ViTs inside one checkpoint; read the reshape/permute in the source each time. Other deltas from the TEXT tower, all exercised: * LayerNorm WITH BIAS rather than RMSNorm (eps 1e-6); * rotary applied in fp32, cos/sin shared across heads; * the TANH-approximate GELU (gelu_pytorch_tanh), not exact erf; * NON-CAUSAL attention segmented by cu_seqlens. The test PROVES the segmentation rather than assuming it: perturbing a token in the second packed image leaves every output of the first BIT-IDENTICAL, while the second's outputs do change. A segmentation bug would otherwise slip through a plain tolerance check. test_minimax_h3: 23/23 cases, 7040 assertions, clean CPU build (0 warnings). Full scripts suite: 128 passed. REMAINING for the encoder: the vision surround (Conv3d patch embed, learned pos-embed interpolation, the 2D rotary table, patch + DeepStack mergers) and the MM processor. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Encoder vision block ported — both tower cores done (
|
| ViT | qkv layout |
|---|---|
| video VAE decoder | per-head interleaved — [head][q,k,v] |
| encoder vision tower | [q_all, k_all, v_all] |
Same tensor size, no error either way — just a wrong result. Never assume a qkv layout carries across ViTs inside one checkpoint; read the reshape/permute in the source each time.
Other deltas from the text tower, all exercised: LayerNorm with bias (not RMSNorm), fp32 rotary, tanh-approximate GELU (not exact erf), and cu_seqlens-segmented non-causal attention.
The test proves the segmentation rather than assuming it: perturbing a token in the second packed image leaves every output of the first bit-identical, while the second's do change. A segmentation bug would slip past a plain tolerance check.
test_minimax_h3: 23/23 cases, 7040 assertions; scripts suite 128 passed; clean build, all checkers green.
Branch: 11 commits
| Component | Gate |
|---|---|
| packed layout (fl2va + ref2va) | exact, fp64 grid bit-exact |
| latent packing / scheduler / request planning | exact |
| DiT forward (f32 / bf16) | 1.6e-7 / 2.4e-3 |
| audio VAE decoder | 4.2e-9 |
| video VAE ViT3D decoder | 8.9e-8 |
| encoder text tower | 1.2e-7 |
| encoder vision block | 6.0e-8 |
| GGUF + NVFP4 manifests (real checkpoints) | exact, identity map |
| GGUF load → runnable DiT | geometry from shapes; forward runs |
| NVFP4 load → runnable DiT | triple dequantized; forward runs |
| whole t2va path | composes; correct, finite, in range |
Remaining
Vision surround (Conv3d patch embed, pos-embed interpolation, 2D rotary table, patch + DeepStack mergers) · MM processor · fl2va/ref2va conditioning · video tiling · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident FP4 forward (where speed actually comes from).
No speed number. dgx.casa is unreachable and this workstation has no GPU.
The full Qwen3-VL vision tower lands, so both encoder towers are ported end to
end: Conv3d patch embed -> bilinear-interpolated position embedding -> 2D rotary
-> block stack over per-frame cu_seqlens -> DeepStack mergers + final patch
merger.
Gated over a RAGGED two-image batch (different h/w), which is what actually
exercises the position-embedding interpolation and the per-frame cu_seqlens.
Merged output and DeepStack features are both within 1e-4 of upstream.
DETAILS THAT WERE LOAD-BEARING, now pinned by the gate:
* the patch embed's Conv3d has kernel == stride, so it is a plain linear over
the flattened patch -- no sliding window;
* torch.linspace(0, n-1, 1) returns the START, not the end (single-row grids);
* .int() TRUNCATES when picking the bilinear corners, it does not round;
* the two merger flavours differ -- the FINAL merger norms the PRE-shuffle
width while the DEEPSTACK mergers norm the POST-shuffle width, and both use
exact-erf GELU rather than the block MLP's tanh approximation. Three
different GELU/norm conventions inside one encoder.
test_minimax_h3: 24/24 cases, 7050 assertions, clean CPU build (0 warnings).
ENCODER STATUS: complete apart from the MM processor (image/video preprocessing
into patches), which is input plumbing rather than model math.
Remaining in the lane: MM processor, fl2va/ref2va conditioning, video tiling,
/v1/videos + MP4 muxing (blocked on a dependency decision), and the
device-resident forward -- which is where speed work begins and needs a GPU to be
worth doing.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
Author
Encoder complete — full vision tower (
|
| Component | Gate |
|---|---|
| packed layout (fl2va + ref2va) | exact, fp64 grid bit-exact |
| latent packing / scheduler / request planning | exact |
| DiT forward (f32 / bf16) | 1.6e-7 / 2.4e-3 |
| audio VAE decoder | 4.2e-9 |
| video VAE ViT3D decoder | 8.9e-8 |
| encoder text tower | 1.2e-7 |
| encoder vision tower (full) | 6.0e-8 / ≤1e-4 |
| GGUF + NVFP4 manifests (real checkpoints) | exact, identity map |
| GGUF → runnable DiT · NVFP4 → runnable DiT | geometry from shapes; forward runs |
| whole t2va path | composes; correct, finite, in range |
No weight bytes checked in anywhere — every gate runs against upstream code, the checkpoint's own remote code, or real checkpoint headers fetched by range request.
Remaining
MM processor (input plumbing) · fl2va/ref2va conditioning · video tiling · /v1/videos + MP4 muxing (blocked on a dependency decision — the tree has no muxer or A/V encoder) · device-resident FP4 forward.
No speed number. The speed work is the device-resident forward; the reference forward is deliberately CPU and unoptimized. dgx.casa is unreachable and this workstation has no GPU, so there is nothing to measure on.
fl2va and ref2va pin their keyframe and reference-audio rows to a NOISED anchor
rather than the clean latent: out = noise_aug*clean + (1-noise_aug)*noise.
The mix is one line. The reason this needed a gate is the ROW ACCOUNTING around
it -- three parts, each easy to get subtly wrong and none of which would raise an
error:
* each visual condition draws noise of length
target_latent_t + imgvid_cond_num_frames and slices the PREFIX matching its
own latent_t -- a shorter condition does NOT get a shorter draw;
* every condition RESTARTS the same seed, so concatenating all conditions and
drawing once would be numerically different for multi-reference requests;
* rows advance by that condition's own patchified row count.
Gated EXACT (<= 1e-6) against upstream with the noise SUPPLIED, so the comparison
isolates the accounting from torch's RNG. That keeps the RNG a single tracked
open item rather than smearing it across every conditioning path -- the t2va
pipeline takes noise as an input for the same reason.
noise_aug == 1.0 is asserted as the documented identity, and shape/row-count
disagreements throw rather than silently mis-slicing.
TOOLING: condition_noise.py uses a RELATIVE import, so the golden generator's
by-path loader now registers a synthetic package whose __path__ is the upstream
directory -- still bypassing the real vllm_omni __init__ (which would drag in
vllm and aenum) while letting relative imports resolve.
test_minimax_h3: 25/25 cases, 8787 assertions, clean CPU build (0 warnings).
Full scripts suite: 128 passed.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Ports the PURE-MATH half of reference_video.py:
* the canvas pipeline -- aspect clamp to [1:4, 4:1], 768 short edge, max-pixel
rescale, nearest multiple of 32 (Python round-half-to-even);
* the frame schedule -- 24 FPS resampled to the 2 FPS Qwen video rate with
duplicate indices dropped, then timestamps averaged per temporal patch with
the tail padded by REPEATING the last.
Both gated EXACT against upstream.
A TEST-AUTHORING LESSON. I added an invariant of my own -- "the snapped canvas
respects the max-pixel budget" -- and it failed on 3840x1080 (1920x544 =
1,044,480 > 1,032,192). The port was right; MY invariant was wrong: upstream
applies the budget BEFORE snapping to 32 and never re-checks. I corrected the
test rather than the implementation. When a self-invented invariant fails, check
whether the REFERENCE actually holds it before touching the port.
DEPENDENCY BOUNDARY, now explicit. The rest of reference_video.py -- probe,
transcode, frame extraction, audio decode -- shells out to ffmpeg/soundfile.
That is the SAME blocker as /v1/videos MP4 muxing: one dependency decision
unlocks reference-video INPUT decode and generated-video OUTPUT encode together.
It is the single item in this lane that needs a project decision rather than more
porting.
test_minimax_h3: 26/26 cases, 8908 assertions, clean CPU build (0 warnings).
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Ports klvae.py's split_tiles and blend, so large canvases can be decoded in overlapping tiles. With this the video VAE side is complete for GENERATION -- only the 3D-CNN encoder remains, and that is needed for image/video CONDITIONING, not for producing output frames. The plan is NOT a simple stride, which is the whole reason it needed a gate: it takes the SMALLEST tile count whose MINIMUM overlaps still cover the axis, then distributes the leftover slack in whole vae_ratio units ROUND-ROBIN across the seams. Get that distribution wrong and every tile after the first shifts -- surfacing as seam artifacts in the output, not as an error anywhere. Shipped config: tile_size 256, tile_overlap_min 64, vae_ratio 16 (= prod(space_down) [2,2,2,2,1,1] -- the "f16" in f16t4; vae_ratio_t = 4 is the "t4"). Gated EXACT over six cases (tiled, exactly-one-tile, smaller-than-tile, and a non-default tile/overlap pair), plus structural invariants: tiles cover the axis, every seam meets the minimum overlap, overlaps stay congruent mod vae_ratio, and the cross-fade starts fully on the previous tile and ends fully on the next. test_minimax_h3: 27/27 cases, 9036 assertions, clean CPU build (0 warnings). Full scripts suite: 128 passed. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…ride) Closes a gap flagged when the denoise loop was ported: its contract says "token_tags must already carry any fl2va vision-span overrides", and nothing in the port produced them. Now it does. THE LOAD-BEARING DETAIL. A vision block is vision_start + pad*count + vision_end, and the WHOLE block -- markers included -- is tagged VIDEO(0). Tagging only the pads leaves two markers as TEXT(1) and shifts every AdaLN modulation index after them: no error, no shape change, just wrong modulation for the rest of the sequence. The test proves each VIDEO run in the output equals a whole emitted vision span, so an off-by-two cannot pass. Tokenization deliberately stays with the CALLER (it owns the tokenizer); this owns the span -> tag mapping, which is the part that must agree with the packed layout. The gate drives upstream with a stub tokenizer, since only span LENGTHS affect tags. test_minimax_h3: 28/28 cases, 9129 assertions, clean CPU build (0 warnings). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
… GroupNorm3D, ResnetBlock3D)
Ports the repeated unit of the video VAE's ENCODER stack. The encoder serves
image/video CONDITIONING (fl2va keyframes, ref2va references); a t2va generation
path does not need it.
Two details pinned by the gate:
* the convolution is CAUSAL in time -- padding[0]*2 frames on the LEFT, none on
the right, with CONSTANT (zero) temporal padding and `reflect` SPATIAL
padding. A symmetric temporal pad would let a frame see the future.
* GroupNorm's statistics span TIME as well as space (32 groups, eps 1e-6), so a
per-frame normalization would silently differ.
CAUSALITY IS PROVEN, NOT ASSUMED. On the bare convolution, changing the last
frame leaves earlier frames BIT-IDENTICAL while the last frame's own output
moves. At block level GroupNorm legitimately mixes across time, so exact equality
would be the wrong assertion there; the weaker claim (the perturbed frame moves
strictly more than the first) is asserted instead, with the strict proof done at
the layer where it actually holds.
test_minimax_h3: 29/29 cases, 9137 assertions, clean CPU build (0 warnings).
REMAINS on the encoder: the Downsample3D + EncoderFCN3D assembly.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The strided convolution between VAE encoder levels, plus stride support in the shared causal Conv3d it builds on. THE SUBTLETY: when the spatial stride is 2, the input is padded by ONE pixel on the RIGHT of W and the BOTTOM of H -- F.pad(x, (0,1,0,1,0,0)) -- BEFORE a stride-2 conv with padding (1, 0, 0). That asymmetric pre-pad is what keeps the sampling lattice aligned; padding symmetrically instead shifts everything by half a pixel, which is a silent wrong latent rather than an error anywhere. Gated EXACT against the checkpoint's own module, with output extents asserted (H/W halve; T halves under the causal pad). test_minimax_h3: 30/30 cases, 9143 assertions, clean CPU build (0 warnings). REMAINS on the VAE encoder: only the EncoderFCN3D level-loop assembly. The encoder is conditioning-only -- a t2va generation path does not use it. HOUSEKEEPING: a clean rebuild hit ENOSPC partway through; ~15 throwaway build trees at 4.7 GB each had accumulated across this session. Removed, 66 GB free, re-verified from scratch. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The encoder assembly lands, so BOTH halves of the video VAE are now ported and gated: the ViT3D decoder (8.9e-8) and this 3D-CNN encoder. conv_in -> per level [ResnetBlock3D x N, then a Downsample3D or a 1x1x1 channel match] -> GroupNorm -> SiLU -> conv_out The channel plan is the fiddly part and is now pinned: block_mid[i] = ch * ch_mult[i] block_in[0] = block_mid[0]; block_in[i>0] = block_mid[i-1] A level gets a Downsample3D when space_down[i]*time_down[i] > 1; otherwise a 1x1x1 conv ONLY if its channel count changes, and nothing at all when it does not. THE FAILURE WAS MINE, IN THE TEST. The first run mismatched at 2.25 with correct shapes. Cause: the generator's scale rule tested `".norm" in name`, which silently misses `norm_out.weight` (no leading dot), so generator and test seeded that group-norm gain differently. Rule corrected to `"norm" in name`; the port needed no change. Same lesson as the reference-video invariant earlier -- when a gate fails, establish WHICH side is wrong before touching the implementation. Two for two this session, the fixture was at fault. test_minimax_h3: 31/31 cases, 9150 assertions, clean CPU build (0 warnings). Full scripts suite: 128 passed. The encoder serves image/video CONDITIONING; a t2va generation path does not call it. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
`examples/minimax-h3-gen` opens the DiT (GGUF, dequant or keep-quant), both VAE
safetensors and both shipped config.json files, plans the request shape, generates,
writes PPM frames + WAV, and muxes to MP4. It lives in examples/ because that is
where the ffmpeg invocation is allowed.
Verified over REAL file formats on both GGUF paths: geometry recovered from the
GGUF, both VAEs through the loaders written this session, text_len derived from the
prompt-embeddings file, and the shape plan correct — 768x1344 canvas / VAE ratio 16
= 48x84 latent, latent_t 62, 50 steps.
New config parsers, gated against the REAL config.json files embedded verbatim
(~2 KB each, so embedding beats a hand-copied summary that could drift):
* `latent_dim` (2048) is BigVGAN's MEL count while `latent_channels` (32) is the
VAE latent width dec_in_proj maps FROM — both keys live in the same file, and
reading the wrong one is wrong by 64x.
* rope_apply_dim = int(dim_head * rope_dim_ratio) = int(64 * 0.75) = 48.
* Both configs carry per-channel latents_mean/latents_std (32 audio, 24 video),
which the pipeline denormalizes with; previously the caller had to supply them.
The parsed geometry is cross-checked against the MANIFEST rather than itself:
x_embedder's [2048, 24] must equal (block.dim, in_channels).
Deliberate scope: prompt EMBEDDINGS are an input file, not computed in the driver.
The encoder tower needs a tokenizer plus a 32B forward — its own driver — and
separating them keeps "do the checkpoints compose?" answerable on its own.
Noise comes from a local splitmix64 stream, seeded for reproducibility, and the code
says plainly that this does NOT reproduce torch's RNG: matching it decides WHICH
sample you get, not whether the pipeline is right.
HONEST LIMIT, recorded in BENCHMARKS and STATUS: no full generation on a REAL
checkpoint has run. Everything is gated on synthetic files plus real manifests and
configs. That download is the next step.
Gate: 47/47 (13763 assertions).
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
… reference Found while preparing the real-checkpoint run: MiniMaxH3DenoiseLoop called MiniMaxH3DitForward — the CPU REFERENCE — not MiniMaxH3DitForwardDevice. Every device-forward milestone this session was therefore reachable only through the DiT unit test, never through the loop that actually drives generation. That made a real run infeasible rather than merely slow: 50 layers x hidden 5376 x ~250k latent positions x 50 steps is not a CPU workload. On a non-CPU device the loop now stages the DiT weights ONCE and runs every step device-resident. Staging once is the point — a 50-step loop that re-uploaded ~16 GB per iteration would be dominated by transfer, which is precisely what StageMiniMaxH3DitWeights exists for. Gated by re-running the WHOLE t2va path on CUDA and comparing frames AND waveform against the CPU pipeline at 2e-3: the same tolerance class the DiT forward is held to, carried through both VAE decoders. `minimax-h3-gen` gains `--device cpu|cuda`. This is the third instance of the same pattern this session — post_quant_conv was implemented but uncalled, the keep-quant test ran on the CPU backend, and now this. A component can be gated and still be unreached by the path that matters. Ask what CALLS the thing, not just whether the thing is correct. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
`--video-dit` (plus the VAEs and their configs) makes examples/server load the H3 checkpoints at startup and register a real VideoRunner: generate → PPM frames + WAV → ffmpeg → MP4, returning the path. Omit it and the routes are never registered, so the server is byte-identical to before — 36/36 api-server cases unchanged. The runner is a callback living in examples/ because that is where process spawning is allowed; the library still builds only artifacts and argv. HONEST GAP, stated at STARTUP rather than buried in a doc: turning a PROMPT into conditioning needs the H3-Encoder (a 32B tower plus a tokenizer), which is not wired. Until it is, every request is conditioned on the same --video-prompt-embeds file, so the prompt text does NOT steer the output — and with no such file the runner REJECTS requests with that explanation rather than silently generating something unconditioned. Everything else is live from the request: shape via MiniMaxH3ResolveShape (task, duration, frames, height, width), steps, both flow shifts, and the seed driving the noise stream. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
A 50-step loop over a real checkpoint spends its minutes in ONE of {weight staging,
per-step forward}, and guessing which is exactly the trap this avoids. Traces which
path was taken (device vs CPU reference), staging seconds, and per-step forward
seconds with the sequence length.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
Downloaded the real stack (DiT Q3_K_M 15.58 GB, video VAE 5.21 GB, audio VAE 605 MB, both configs) and ran it. WORKS: the DiT loads with its geometry recovered from GGUF shapes alone (layers=50, hidden=5376, heads=56 — the shipped H3), both VAEs load through this session's loaders, both configs parse, and the DiT forward runs on the GPU. KEEP-QUANT IS REQUIRED, NOT AN OPTIMIZATION: the dequant path is OOM-killed, because Q3_K -> f32 is ~145 GB against 122 GB of unified memory. With --keep-quant the DiT stays at 15.6 GB. The arm built earlier today turns out to be the enabling condition for a real run, not a speed lever. MEASURED via the VT_H3_PROGRESS trace: staging 32.8 s once; DiT forward 199.96 s per step at seq_len 576. The 50-step default is therefore ~2.8 h of DiT alone. BLOCKER: the run timed out in the VIDEO VAE DECODE, which is still CPU-ONLY (a 36-layer dim-2048 ViT3D on the host). A missing device path, not a tuning problem. Method note recorded: nvidia-smi GPU utilization is unreliable on this Tegra-class board (it also reports Memory-Usage "Not Supported"). Twenty minutes went into treating "GPU 0%" as evidence the device path was not taken — it was. One stderr phase trace settled it in a single run. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
Downloaded the real stack (DiT Q3_K_M 15.58 GB, video VAE 5.21 GB, audio VAE 605 MB, both configs) and ran it. WORKS: the DiT loads with its geometry recovered from GGUF shapes alone (layers=50, hidden=5376, heads=56 — the shipped H3), both VAEs load through this session's loaders, both configs parse, and the DiT forward runs on the GPU. KEEP-QUANT IS REQUIRED, NOT AN OPTIMIZATION: the dequant path is OOM-killed, because Q3_K -> f32 is ~145 GB against 122 GB of unified memory. With --keep-quant the DiT stays at 15.6 GB. The arm built earlier today turns out to be the enabling condition for a real run, not a speed lever. MEASURED via the VT_H3_PROGRESS trace: staging 32.8 s once; DiT forward 199.96 s per step at seq_len 576. The 50-step default is therefore ~2.8 h of DiT alone. BLOCKER: the run timed out in the VIDEO VAE DECODE, which is still CPU-ONLY (a 36-layer dim-2048 ViT3D on the host). A missing device path, not a tuning problem. Method note recorded: nvidia-smi GPU utilization is unreliable on this Tegra-class board (it also reports Memory-Usage "Not Supported"). Twenty minutes went into treating "GPU 0%" as evidence the device path was not taken — it was. One stderr phase trace settled it in a single run. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
A playable MP4 out of the real 15.58 GB MiniMax-H3-FL2VA-Q3_K_M checkpoint on Thor's GPU. ffprobe reports h264 / yuv420p 128x128 + AAC stereo 32 kHz; EXIT=0 in 6m22s. The whole chain is proven on real weights rather than fixtures: GGUF keep-quant load -> device-resident denoise loop -> post_quant_conv -> ViT3D video decode + BigVGAN audio decode -> PPM frames + WAV -> ffmpeg mux. Measured at latent 2x8x8 / 2 steps: staging 22.3 s once, DiT forward 21.1 s/step at seq_len 64. The earlier 256x256 attempt measured 200 s/step at seq_len 576 and timed out in the CPU-only VAE decode — hence the smaller dims here. The milestone is that the chain COMPLETES on real weights, not that these dimensions are useful output. WHAT THIS DOES NOT SHOW: the video is not prompt-conditioned. The H3-Encoder is not wired, so conditioning is a fixed embeddings file and the prompt steers nothing. Two performance gaps, now measured rather than suspected: the DiT forward at 200 s/step (a 50-step default is ~2.8 h), and the video VAE decode being CPU-only, which is the wall at realistic resolutions. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…n Thor
The second half of the goal. `examples/server`, started with both the Qwen3-0.6B LLM
and the H3 checkpoints, logs `/v1/videos on (dit layers=50, device=cuda, keep-quant)`
and then over HTTP:
POST /v1/videos -> {"id":"vid_1","status":"queued"}
GET /v1/videos/vid_1 -> {"status":"succeeded","output_path":"..."}
ffprobe(output_path) -> h264 / yuv420p 128x128 + AAC stereo 32 kHz
GET /v1/videos/<bad id> -> 404 NotFoundError
So the async route, the job store's queued->running->succeeded lifecycle, the runner
callback, the ffmpeg mux and the 404 path are all exercised against the REAL
checkpoint rather than fixtures. Server-side trace: staging 58.0 s, DiT forward
22.4 s/step.
Still true, and stated at startup rather than buried in a doc: the PROMPT does not
steer the output — conditioning is the fixed --video-prompt-embeds file, because the
H3-Encoder is not wired. That is the next piece for a truthful API.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
Wiring the encoder is what turns the generated video from unconditioned output into something a prompt steers. Step one is loading it at all: the tower is 32B, and the safetensors loader materializes f32 (~128 GB), which does not fit the 122 GB box we test on. LoadMiniMaxH3EncoderFromGguf keeps the projections in their ggml blocks — the real file is qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf, 902 tensors, all Q4_K — holding the tower at ~14.6 GB. As with the DiT, the block-quant GEMM carries no arch gate, so it runs natively on hardware that cannot do FP4. THE INTERESTING PART: the two fusions the forward needs (q/k/v -> qkv_proj and gate/up -> gate_up_proj) are done on the QUANTIZED BYTES. That is sound because ggml rows are INDEPENDENT block sequences, and every K here is a multiple of the 256-element block, so concatenating whole rows yields a valid block-quant tensor whose rows are [q_all|k_all|v_all] — no dequantize/requantize round trip, and no precision lost to one. Gated BYTE-FOR-BYTE against `q ++ k ++ v` and `gate ++ up`. Also gated: geometry recovered from the fused shapes alone, truncation to min(num_hidden_layers, 50), and the H3 delta that `norm.weight` is NOT bound because H3 reads the UNNORMALIZED truncated output. The GGUF prefixes differ from the safetensors ones and are now pinned. REMAINS for conditioning: a DEVICE encoder forward consuming these quantized weights (the existing forward is a host f32 reference), plus tokenization and the embedding gather. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
Loads the 32B encoder tower keep-quant and reports the geometry it recovered plus the resident byte count, so the loader can be validated against the REAL qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf rather than only a synthetic fixture. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
The shipped Q4_K_M encoder stores v_proj as Q6_K while q_proj/k_proj are Q4_K — the usual K_M recipe of keeping V at higher precision. That group therefore CANNOT be byte-concatenated, and the loader's uniformity assertion caught it on the real file. Fusion is now conditional: uniform groups fuse (gate/up still do), mixed groups keep their members under their own names in their own encodings. Dequantizing to force a fusion was the alternative and would have thrown away exactly the precision the recipe exists to keep — so a mixed checkpoint costs extra GEMM launches, not precision. Geometry recovery handles both shapes. Gated with a second synthetic checkpoint whose v_proj uses a different encoding. Only Q8_0 has a CPU quantizer, so that tensor is written as correctly-SIZED arbitrary bytes; the test exercises the loader's ROUTING decision and nothing dequantizes, so no real payload is needed — stated at the site rather than left to look like an oversight. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
`--encoder` on the real 14.58 GB qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf: encoder layers=50 hidden=5120 heads=64 kv_heads=8 head_dim=128 ffn=25600 encoder resident (keep-quant) = 13.2421 GiB Geometry recovered from shapes alone, and it is Qwen3-32B exactly (64 heads x 128, 8:1 GQA, ffn 25600). Layers are 50 because H3 truncates — the file ships 64. It loads ALONGSIDE the keep-quant DiT, so the full stack fits the box. The real file also corrected a design assumption, and the loader's uniformity assertion is what caught it: the K_M recipe stores v_proj as Q6_K while q/k are Q4_K, so the attention group cannot be byte-concatenated. A loader that silently took the first member's dtype would have produced a tensor whose v rows were garbage, with nothing downstream to say so. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…g path
MiniMaxH3EncoderTextForwardDevice runs the 32B conditioning tower from its ggml
blocks. Gated at max abs diff 3.76e-4 against the gated host f32 reference on a scale
of ~1.0 — 0.04% relative, which is Q8_0 quantization error and nothing else: the test
feeds BOTH paths the same dequantized numbers, so the only difference is where
quantization enters, not the model.
No dequantization on the device path — the projections go through vt::MatmulBT, which
dispatches kMatmulBTQuant on a block-typed weight. That is what lets a 32B tower sit
in 13.2 GiB alongside the DiT on a 122 GB box.
Two things that looked like they needed bespoke kernels and did not:
* M-RoPE — upstream builds emb = cat(freqs, freqs), so cos/sin REPEAT across the
halves, which is exactly vt::RopeFromCache's layout. Only the ANGLES are unusual
(three axes interleaved [THW THW ...]), and those are host-side once per prompt.
* Causal GQA attention — vt::DFlashBlockAttention(causal=true) already broadcasts
kv heads across their query group.
So the tower is shared ops plus the loader.
The mixed-encoding path is handled here too: where the checkpoint kept q/k/v separate
(the shipped Q4_K_M does), it issues three GEMMs instead of one. Launches, not
precision.
All three H3 deltas preserved: layer truncation, the UNNORMALIZED output, and
DeepStack left to the caller.
Gate: 49/49 (14606 assertions).
REMAINS for a prompt-steered video: tokenization plus the embedding gather, then
wiring prompt -> embeddings into the driver and the server.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
`minimax-h3-gen --encoder <gguf> --prompt "..."` now goes prompt -> ids -> embeddings -> encoder -> conditioning, with no --video-prompt-embeds file in the path. TOKENIZER: the encoder GGUF carries its own vocab, so tok::Tokenizer::FromGguf needs no side-car tokenizer.json — one fewer file to keep in sync with the weights. EMBEDDING GATHER: the table is [151936, 5120] (~1.5 GB even quantized) and a prompt touches a few dozen rows, so MiniMaxH3EncoderEmbedTokens decodes ONLY the requested rows. Valid for the same reason the byte-level fusion was — ggml rows are INDEPENDENT block sequences, so a row decodes from its own bytes alone. Gated BIT-IDENTICAL against slicing a full-table dequant (with a repeated id and both vocab ends in the list), and out-of-range ids must THROW rather than read past the table. That needed each kept tensor's ggml TYPE ID recorded alongside its vt::DType, since the single-row dequant entry point is keyed by the ggml id. Gate: 50/50 (15254 assertions). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…ights-only I assumed the encoder GGUF carried its own vocab. It does not: the ComfyUI-style export is WEIGHTS ONLY and has no `tokenizer.ggml.*` metadata, unlike a llama.cpp one, so Tokenizer::FromGguf throws on it. The real file confirmed this in one run. `--tokenizer <tokenizer.json>` now supplies the vocab (the checkpoint ships FL2VA/text_encoder/tokenizer.json), falling back to FromGguf for exports that do embed one. The comment says which kind of GGUF is which rather than leaving the next reader to rediscover it. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
--prompt "a cat playing a piano"
-> tokenizer 5 tokens
-> block-quant embedding gather [5, 5120]
-> 32B keep-quant encoder ON DEVICE -> conditioning [5, 5120]
-> DiT (keep-quant, device denoise loop) 21.6 s/step at seq_len 64
-> ViT3D + BigVGAN decode -> PPM/WAV -> ffmpeg
-> h264/yuv420p 128x128 + AAC stereo 32 kHz, EXIT=0
No --video-prompt-embeds anywhere in the path. Both towers are resident at once —
DiT 15.6 GB plus encoder 13.2 GiB — which is only possible because both are
keep-quant.
Encoder staging is 162.3 s against the DiT's 32.8 s, the next obvious cost if prompts
are encoded per request rather than once.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
`--video-encoder` (plus `--video-tokenizer`) makes examples/server tokenize each request's prompt, gather its rows from the block-quant embedding table, and run the H3-Encoder — so `/v1/videos` is genuinely prompt-conditioned rather than replaying a fixed embeddings file. The tower is staged to the device ONCE at startup: staging costs ~162 s, which would otherwise dominate every single request. The startup warning is now conditional and truthful in all three states: conditions on the prompt (encoder present), ignores the prompt (only --video-prompt-embeds), or will reject requests (neither). The runner's error message names the flag that fixes it rather than describing the gap abstractly. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
Two changes aimed at the measurement that dominates everything else: the keep-quant DiT GEMM achieves ~103 GFLOP/s, and per-step cost scales LINEARLY with sequence length (seq x9.0 -> time x9.5), so it is GEMM-bound. Projected to full quality (62,496 video rows) that is ~5.7 h/step, i.e. ~12 days for 50 steps. 1. PRE-STAGED WEIGHTS. MiniMaxH3GenerateT2va/DenoiseLoop take an optional already-staged weight set. Staging costs tens of seconds, so a driver or server stages ONCE per process rather than per generation. Null keeps the old behaviour. 2. StageMiniMaxH3DitWeightsDequantBf16 dequantizes block-quant weights to bf16 on the way up instead of keeping their blocks, so the projections go through the tuned cuBLASLt MatmulBT. It trades memory for throughput — ~33 GB bf16 against 15.6 GB kept-quant — which is affordable exactly where 145 GB of f32 was not. Dequantizing straight to bf16 (not via f32) keeps the peak down. `minimax-h3-gen --dequant-bf16` selects it, and the driver prints staging seconds, so the two paths can be compared head to head on real weights rather than argued about. Gate: 50/50 unchanged. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
The measurement that forced this: our in-quant DiT GEMM achieves ~103 GFLOP/s, and the implied weight-streaming rate is CONSTANT across sequence lengths (28.5 GB/s at seq 64, 27.1 GB/s at seq 576). A constant rate means the whole ~9.4 GB weight set is re-read per SEQUENCE ROW — a batch-1 GEMV, not a tiled GEMM. At ~16k rows that is a ~16,000x traffic amplification, so it was never a FLOP problem. Upstream agrees: ComfyUI-GGUF dequantizes to bf16 and calls F.linear rather than computing in-quant (its ops class docstring is literally "Dequantize weights on the fly before doing the compute"). LoadMiniMaxH3DitFromGgufBf16 dequantizes STRAIGHT to bf16 — never materializing f32. That matters for fit, not just speed: keeping blocks leaves the AdaLN projections INELIGIBLE (K=2688 is not a whole number of 256-element Q3_K blocks), so they dequantize to ~52 GB of f32, and 15.6 + 52 + 33 GB staged is ~107 GB of 122 — which is what killed the first bf16 run. Straight to bf16 the whole DiT is ~33 GB. Staging now uploads bf16 weights verbatim, and --dequant-bf16 selects the path. Gate: 50/50 unchanged. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
PEAK memory, not total, is what took the test box off the network: loading to host bf16 (~33 GB) and then staging to device (~33 GB more) holds BOTH at once, and on a UNIFIED-memory box those come from one 122 GiB pool. Alongside the VAEs and page cache the machine thrashed hard enough to stop answering ping — it never rebooted, `uptime` showed it had been up the whole time at 122/122 used. StreamMiniMaxH3DitToDeviceBf16 dequantizes and uploads ONE TENSOR AT A TIME, letting each host buffer die before the next, so the peak is the device copy plus one tensor. It also opts the mapping into page release and drops each tensor's file pages after reading them once — on a unified box the page cache competes with the model for the same pool. The forward's views are bound over the DEVICE tensors by checkpoint name, so a missing tensor still throws BY NAME rather than reading as zeros. Gate: 50/50 unchanged. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…nd VT_H3_DROP_PAGES The streamed load was SIGKILLed early at only ~21 GB peak — not a memory problem, so the page-release call is no longer trusted by default. Progress is now traced every 50 tensors so the failure can be localized instead of inferred. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…ager
The streamed stager bound every tensor to device memory, including rope.inv_freq —
but that one is read on the HOST: BuildRopeCosSin computes the cos/sin cache from it
before any kernel runs. The first forward segfaulted on the device pointer.
It is now dequantized to host f32 and kept alive by the staged struct, which also
documents the constraint on the field so the next stager cannot repeat it. The
non-streaming stager already did this correctly ("consumed on the host"); the
streaming one did not inherit the reasoning.
Good failure mode at least: a device pointer read on the host segfaults loudly
rather than yielding plausible garbage.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
The streaming stager converted every tensor to bf16, but upstream keeps both patch projections, both time-embedder projections and both output heads in f32 (MINIMAX_H3_FP32_PARAM_NAMES). Their ACTIVATIONS are f32 too, so the first island GEMM hit vt::MatmulBT's dtype check: "unsupported dtype combo (f32,bf16)->f32". Failing loudly there is the good outcome — a stager that silently bf16'd an island would have degraded the patch projections and output heads with nothing to show for it. The split is now explicit and commented at the predicate, so the reason survives. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
Measured on Thor: at seq 3264 the DiT step is 509.78 s and attention accounts for ~429 s of it — roughly 36 GFLOP/s. Scaling is quadratic (seq x5.67 -> time x34.9), so attention, not the GEMM, now dominates at realistic sequence lengths. FA2 is DISABLED on sm_110, so this portable kernel is the only path. The reason it is slow is structural, not tuning: the existing kernel uses one BLOCK per query row and walks keys ONE AT A TIME, doing a shared-memory tree reduction plus ~11 __syncthreads() PER KEY. At seq 3264 that is ~36k barriers per query row, and Q is re-read from global memory for every key. The fast path gives one WARP a query row: Q and the output accumulator live in REGISTERS (d/32 each per lane), the dot product is a warp shuffle reduction, and the online softmax runs inside the warp. No __syncthreads at all, and Q is read once. The arithmetic is deliberately IDENTICAL — same sequential key order, same online-softmax recurrence, same f32 accumulation — so this is a scheduling change, not a numerics change. It engages when head_dim is a whole number of warp widths (<= 4), which covers every head this port uses; anything else keeps the general kernel. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
My first insertion matched a dim3/shmem/switch pattern that occurs TWICE and landed in the plain Attention launcher, which has no d_cu or DFlashBlockAttentionArgs — nvcc caught it immediately. Asserting an anchor EXISTS is not enough when the file has near-duplicate launchers; the anchor must be asserted UNIQUE. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…d without the VAEs The scaling run took Thor off the network: on a unified-memory box the bf16 DiT (host + device) plus both VAEs plus f32 activations is the OOM-reboot recipe, and none of the VAE memory is doing anything when the question is 'how fast is a DiT step'. Splits MiniMaxH3DenoiseT2va out of MiniMaxH3GenerateT2va rather than duplicating the packed-layout and sigma-schedule logic in the example -- GenerateT2va is now implemented in terms of it, so there is exactly one copy. Reports an AVERAGE over the requested steps: step 1 pays one-off RoPE-cache and allocator costs, so a single-step run overstates steady state. Sums the output so the loop cannot be elided and an all-NaN forward cannot pass as a fast one. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
The portable ViT3D decoder is scalar triple loops with double accumulation -- the
right shape for a golden, the wrong one for 36 layers over thousands of tokens. It
is what made a 256x256 decode time out, and it was the last CPU-only stage in the
video path.
NO NEW KERNELS. Two EXACT load-time weight rearrangements are what make the whole
decoder expressible in the existing shared ops:
* to_qkv is stored PER-HEAD INTERLEAVED ([head][q|k|v]) in this checkpoint, where
vt::QkvSplit -- and every other model here -- wants [q_all|k_all|v_all].
Permuting the weight ROWS once at stage time puts the GEMM output directly in
QkvSplit's layout, instead of adding an interleaved-split kernel that would buy
nothing.
* each branch ends h += scale * (x @ W^T + b), with a learned PER-OUTPUT-CHANNEL
scale, so scaling row d of W and element d of b by scale[d] is algebraically
the same thing and leaves a plain vt::Add.
Gated against the CHECKPOINT'S OWN remote code -- the same golden the portable
decoder answers to, not the portable decoder's output. Two implementations agreeing
on a wrong answer is a failure mode this tree has already been bitten by, and both
folds are silent-failure shaped: a wrong permutation still yields plausible finite
frames. max|diff| 1.19e-07, in the same class as the portable path's own 8.9e-8.
CPU and CUDA cases both registered, so CPU CI covers the graph.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
The device decoder existed but nothing called it, so a real run still went through the scalar reference. On a device the pipeline now stages and runs it; on CPU the portable path is unchanged, and stays what the device path is gated against. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
…o VAE
Both measured on the Thor sm_110 board:
* DiT step loop 509.78 -> 18.68 s/step at seq ~3.2k (~27x), and the scaling is
now near-linear in sequence rather than quadratic.
* video VAE decoder device-resident at 1.19e-07 vs the checkpoint's own remote
code, retiring the last CPU-only stage in the video path.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
…windows" A real prompted 512x512 run came out globally correct (recognisable subject, right scene layout) but covered in a grid of small squares. That is not quantization noise, and it was not argued away — it was A/B'd: MiniMaxH3SplitTiles plans 512 px as 3 tiles per axis, but plans 256 px as exactly ONE tile. So 256x256 is the one size where tiled and untiled are the SAME computation. 256x256 came out clean; 512x512 did not. Root cause: MiniMaxH3SplitTiles/MiniMaxH3BlendTiles were implemented and gated, and nothing ever called them. Tiling here is NOT a memory strategy. The ViT3D's RoPE coordinates are LENGTH-NORMALIZED — 2*((i+0.5)/n)-1 over whatever grid it is handed — so the grid EXTENT is part of the input. Handing the decoder a 32x32 latent when it was trained on 16x16 tiles gives every token a position the model has never seen, and the patches stop cohering with their neighbours. Every reduced-dimension gate in the suite is smaller than one tile, where tiled and untiled coincide. That is exactly why the whole suite missed this, and why the new gate covers BOTH ends: a single-tile canvas stays bit-identical to the untiled decode, and on a 2x2-tile canvas each tile's un-blended interior equals a standalone decode of that tile's own latent slice exactly. Both failure modes otherwise produce plausible finite frames. The blend extent converts through latent units rather than reusing the plan's canvas pixels directly: vae_ratio and patch_size are both 16 on the real checkpoint but differ in the reduced-dimension configs the gates run at. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports MiniMax-H3 — a 33.1B CFG-distilled joint video+audio diffusion transformer — into vllm.cpp. It is the project's first DIFFUSION architecture: no KV cache, no sampler, no logits, so it is not a causal-LM registry model. Upstream is
vllm-project/vllm-omni, not the vLLM repo, and it sits beyond the parity pin (555967922).Every portable piece is done. What remains needs a GPU, not more porting.
How this is gated (no weight bytes checked in)
Two techniques carry the whole PR:
trust_remote_codeand therefore had to be reimplemented in C++) and execute them at reduced dimensions. Both sides rebuild weights from a shared FNV-1a + splitmix64 PRNG, so a golden is reproducible from source alone.Correctness gates
rf_v_to_x0/v1/videoscontract, job store, route dispatchThe fp64 grid is gated bit-exact because it feeds RoPE. Matching it required reproducing upstream's arithmetic order, not just its formulas:
numpy.linspace(endpoint=False)evaluatesi*step + start, and upstream keeps a numpy-pairwise and a Python-sequential span sum deliberately separate because they diverge in the last ulp from n=16 (packed_sequence.py:101-113).Attention reuses the shared
vt::DFlashBlockAttention(causal=false)for packed varlen — no new kernel.The ffmpeg boundary
/v1/videosreturns an MP4, and upstream shells out to ffmpeg. This library has no subprocess precedent, so the split was put to the project owner and ratified: "re: ffmpeg invocation, correct - let's keep in the examples only".src/vllm/builds the artifacts (PPM frames, WAV) and the argv — and spawns nothing.examples/minimax_h3_mux/performs the invocation./v1/videostherefore takes a caller-suppliedVideoRunnercallback.Serving
POST /v1/videos(async),POST /v1/videos/sync,GET /v1/videos/{id}, registered onApiServervia the server's existing additive/opt-in pattern: they appear only whenset_video_runnerhas been called, so a server built without video support is byte-identical to before — no new constructor parameter, no existing caller touched.Async uses a joinable worker drained in
~ApiServer; a detached thread would outlivethisand write into a destroyed job store. A throwing runner fails the job, not the process.Hardware verdict (corrected mid-PR)
An early reading called this hardware-blocked. That was wrong and the developer corrected it: the bf16 release (~354 GB) does not fit one box, but the quantized arms do — GGUF (DiT 15.6 GB + encoder 14.6 GB + VAEs ~11 GB ≈ 41 GB) and NVFP4 both load. Both quantized loaders are implemented and gated against the real manifests.
The device-resident forward (W2b) — landed and GPU-verified
MiniMaxH3DitForwardDeviceruns the whole DiT graph with every activation in device memory, so the block stack — and above it the 50-step denoise loop — never round-trips through the host. Verified on a real GPU, not just compiled: the CUDA case is proven to have run (220 assertions execute; 9673 total on GPU vs 9453 on CPU, where it skips).Three H3 kernels were needed, and only three, because the port reuses the tuned shared ops (
MatmulBT,RmsNorm,QkvSplit,SiluAndMul,Add,IndexSelect/IndexCopy,DFlashBlockAttention). H3's RoPE looks exotic but is plain NeoX rotate_half — only the angles are unusual (three axes off the fp64 position grid), so a per-row cos/sin cache feedsvt::RopeFromCachewith no bespoke kernel. That left two indexed AdaLN modulates and an ungated SiLU, in akMiniMaxH3glue table mirroring thekLagunaprecedent — but registered on both kCPU and kCUDA (Laguna's is CUDA-only), so the whole device path is gated in CPU CI too.Not bit-identical to the CPU reference, deliberately:
vt::RmsNormreduces in f32 where the reference accumulates in double, and f32 is what upstream torch does. Held to the same goldens instead.Not yet built — honestly recorded
vt::FusedChainglue folds on the device forward (the bf16 fold also clears its merged-GEMM allowlist entry, added with a specific reason rather than silently).Open: there is no vllm-omni parity pin — the upstream-sync protocol covers only the vLLM repo, and H3 is both beyond the pin and outside that repository.
Record
.agents/specs/minimax-h3.md, model-matrix rowMODEL-DIFFUSION-minimax-h3-mini-max-h3-dit+ checklist/rollup, roadmapROAD-V1-H3,docs/STATUS.md,docs/BENCHMARKS.md, parity-ledger and state entries.Checkers green on committed HEAD:
check-agent-record,check-model-checklist,check-doc-checkpoint,check-readme-structure,check-fusion-consistency,check-runner-routing-consistency,check-device-leakage,check-env-doc, plus the checker mutation suites.🤖 Generated with Claude Code