lite-05: negotiate per-frame compression via SUBSCRIBE_OK (Rust + JS)#1531
Merged
Conversation
Add opt-in frame compression to the moq-lite-05 (WIP) draft. - Track gains a `compress: bool` hint marking a track worth compressing (e.g. the JSON catalog, which is now marked compressible). - SUBSCRIBE_OK carries a negotiated Compression codec, gated to lite-05+. The publisher picks Deflate when the resolved track is marked compress and the peer speaks lite-05; older drafts always negotiate None and stream frames verbatim, so the wire stays backwards compatible. - Subscribers block on SUBSCRIBE_OK before decoding any frame, since a group's QUIC stream can race ahead of it on its own stream. The codec is fanned out to group streams via a per-subscribe watch channel. - Frames are compressed independently (raw DEFLATE via flate2) so the codec never carries state across the lossy, out-of-order group boundary. Decompression caps the inflated size at MAX_FRAME_SIZE to reject zip bombs. Co-Authored-By: Claude <noreply@anthropic.com>
cargo doc runs with -D warnings in CI; linking a public doc comment to the pub(crate) MAX_FRAME_SIZE tripped rustdoc::private_intra_doc_links.
The new Track.compress field broke struct literals in moq-audio, moq-ffi, and libmoq that the first pass missed. Default them via ..Default::default().
…sor API broadcast.rs's linger_resubscribe_keeps_flowing_moq_lite_03 still awaited OriginConsumer::announced() as a future, but #1434 made it return an AnnounceConsumer cursor. The three sibling tests were updated; this one was missed, leaving the workspace test build broken on dev. Mirror the sibling pattern: build the cursor up front and poll it with .next().
…lementation-QIFwj
Mirror the Rust per-frame compression on the browser side. - New `Compression` codec (None/Deflate) backed by the browser-native CompressionStream/DecompressionStream with "deflate-raw", which matches the Rust `flate2` raw-DEFLATE bytes on the wire. Decompression caps the inflated size at 16 MiB to reject zip bombs. - `Track` gains a `compress` flag. - `SubscribeOk` carries the negotiated codec on draft-05+ only; older drafts stay byte-compatible (always None). - Subscriber blocks on SUBSCRIBE_OK before decoding any frame: the codec is fanned out to group streams via a per-subscribe signal, since a group's stream can arrive before SUBSCRIBE_OK on its own stream. - Publisher negotiates the codec from the track's compress flag and compresses each frame independently. Tests cover the codec round-trip (incl. max-size and garbage rejection) and the SUBSCRIBE_OK wire format across versions.
kixelated
changed the title
…lementation-QIFwj # Conflicts: # rs/moq-net/src/stats.rs
kixelated
force-pushed
the
claude/compression-implementation-QIFwj
branch
from
9ad465e to
db3b66e
Compare
kixelated
commented
May 29, 2026
| } | ||
| Ok(()) = self.track_priority.changed() => { | ||
| priority.set_track(*self.track_priority.borrow_and_update()); | ||
| match self.compression { |
Collaborator
Author
There was a problem hiding this comment.
Any way to split this into separate functions? It's very nested.
Address review feedback: the compression branch left serve_group deeply nested. Extract the repeated "await X while servicing priority updates" pattern into next_frame/read_chunk/read_all/write_chunk helpers, and move the codec branch into serve_frame. Behavior is unchanged.
…lementation-QIFwj
Fixes the `cargo sort --check` CI failure: flate2 was appended below the alphabetically sorted [workspace.dependencies] block. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kixelated
added a commit
that referenced
this pull request
May 29, 2026
#1531 added `compress: bool` to `moq_net::Track` but two struct literals in libmoq's lib still listed only `name` and `priority`, so the lib test build (which is part of `cargo check --all-targets`) errored with E0063 once #1531 reached this branch via the dev merge. Swap both to `Track::new(name)` since neither needed a non-default priority. Same construction the rest of the workspace uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kixelated
added a commit
that referenced
this pull request
May 30, 2026
Picked up #1528 (moq-rtc) and #1531 (compress field on Track) by merging origin/dev. The new moq-rtc binary still passes publisher.consume() to with_publish, which used to take an OriginConsumer but now takes an OriginProducer. Drop the .consume(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opt-in frame compression for the
moq-lite-05(WIP) draft, implemented on both the Rust (moq-net) and browser (@moq/net) sides. The publisher marks a track as worth compressing, the codec is negotiated inSUBSCRIBE_OK, and the subscriber blocks on that message before decoding any frame.Trackgains acompressflag (Rustcompress: bool/Track::with_compress(..); JStrack.compress) — a hint that the track's frames are worth compressing. The hang JSON catalog (Catalog::default_track) is marked compressible, since it's text and re-sent on every change.SUBSCRIBE_OKcarries a negotiatedCompressioncodec, gated to lite-05+. The publisher picksDeflatewhen the track is markedcompressand the peer speaks lite-05; older drafts (lite-04 and below) always negotiateNoneand stream frames verbatim, so the wire stays backwards compatible. Negotiation lives inSUBSCRIBE_OK(notSUBSCRIBE) specifically so the subscriber has to block on it to learn how to decode.SUBSCRIBE_OKbefore decoding frames. A group's QUIC stream can be accepted beforeSUBSCRIBE_OKlands on the subscribe stream, so the negotiated codec is fanned out to each group stream (Rust: a per-subscribewatchchannel; JS: a per-subscribe signal) and the group read path waits on it before touching any frame payload.MAX_FRAME_SIZE) to reject zip bombs.flate2(pure-Rustminiz_oxidebackend, no C deps — keepsmoq-fficross-compilation and Nix builds painless); the browser uses the nativeCompressionStream/DecompressionStreamwith"deflate-raw", which produces the same bytes. TheCompressionenum is the extension point if we want zstd later.Wire format (lite-05 only)
SUBSCRIBE_OKappends a singleCompression (i)varint after the existing fields (0= none,1= deflate). On a compressed track, eachFRAME's size prefix becomes the compressed length; the subscriber inflates it back, so the publicFrame/Group/TrackAPI is unchanged on both sides.Scope / known limitations (follow-ups)
SUBSCRIBE_OK, decompresses on ingest, and caches the plaintext. Its downstream producer track isn't markedcompress, so it re-serves uncompressed. Compression therefore applies on a hop where the publisher's track carries the flag (origin → direct subscriber). Propagating the flag through relays is a follow-up.@moq/netcreates the served track lazily when a subscribe arrives, so an app must settrack.compressbefore the track is served for the publisher to advertise it. The browser's primary role here is the subscriber (decoding a Rust-published compressed catalog), which is fully wired.Lite05Wip/DRAFT_05_WIP); nothing negotiates it by default.Cross-package sync
rs/moq-netwire/API →js/net@moq/netimplements the sameCompressioncodec,SUBSCRIBE_OKfield, blocking, and per-frame (de)compression.rs/moq-netwire/API →doc/conceptmoq-dev/drafts;doc/conceptis a stable-feature overview and shouldn't document an unadvertised draft yet.rs/hangcatalog →js/hangcompressis a transport-layer track hint, not part of the serialized catalog JSON (skip_serializing_if).Test plan
Compressioncodec round-trips (none + deflate), rejects garbage, caps inflated size, wire-code round-trip;SUBSCRIBE_OKwire round-trip (compression on lite-05, absent on lite-04).cargo test -p moq-net --lib(327) and-p hangpass;cargo check --all-targets --workspace,clippy -D warnings,cargo doc -D warningsclean (minus ffmpeg/gst crates needing system toolchains locally).SUBSCRIBE_OKwire format across versions;bun test(148 net tests) andbun run --filter='*' check+ Biome clean.moq-netdoesn't have yet; flagging as a follow-up.(Written by Claude)