feat(flate): extract group-scoped DEFLATE into moq-flate / @moq/flate#1905
Conversation
Pull the per-group DEFLATE codec out of moq-json into a standalone, dependency-light crate (and matching @moq/flate npm package) so any framed-stream library can reuse it without the JSON snapshot/delta layer. The two implementations stay wire-interoperable (raw DEFLATE + RFC 7692 sync-flush marker stripping); this is a pure refactor of the codec, no wire-format change, so existing tracks and the browser peer are unaffected. Also collapse the consumer backlog for late joiners: a single poll now applies every frame already buffered in the group but yields only the latest reconstructed value. Frames are still decoded in order (the DEFLATE window and merge patches are sequential); only the per-frame deserialize and yield are skipped, so a late joiner catches up to the head in one step instead of replaying every superseded state. Live consumers keeping up still see each update. Changes: - new rs/moq-flate: Encoder/Decoder, configurable level + max-frame-size, own Error (Decompress/TooLarge). moq-json depends on it; its Error wraps moq_flate::Error transparently (replaces the Decompress/TooLarge variants). - new js/flate (@moq/flate): pako-based Encoder/Decoder with level / maxFrameSize options. @moq/json depends on it; pako/fflate move over. - moq-json Consumer::poll_next drains buffered frames, deserializing once. - rs/moq-json/examples/telemetry.rs: measures real wire savings (~91% on a realistic telemetry stream) and verifies every tick round-trips. - bump moq-flate/@moq/flate to 0.1.0 and moq-json/@moq/json to 0.1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 4 minutes and 31 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (19)
✨ Finishing Touches✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The backlog-collapse poll_next conflated two break reasons: a group that is exhausted (poll_read_frame -> Ready(None)) vs. one that is still open with nothing buffered yet (Pending). On Pending it broke the drain loop and fell through to the track-finished check, ending the stream early. But a group appended before the track finished can still deliver frames, so the consumer must keep waiting on it. Track the two cases separately and return Pending when the current group is still open. Regression caught by moq-mux's hang catalog consumer test (waits_for_pending_catalog_group_payload_after_track_finish); add the equivalent test directly to moq-json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Summary
Extracts the per-group DEFLATE codec added in #1897 out of
moq-jsoninto a standalone, dependency-light cratemoq-flate(and a matching@moq/flatenpm package), so any framed-stream library can reuse it without the JSON snapshot/delta layer. The Rust and JS codecs stay wire-interoperable (raw DEFLATE + the RFC 7692 sync-flush marker trick).This is a pure refactor of the codec plus a small consumer optimization. No wire-format change, so existing tracks (and the hang catalog) and the browser peer are byte-identical on the wire.
What changed
New
moq-flatecrate (rs/moq-flate)Encoder/Decodermoved over verbatim, madepub. Configurable DEFLATE level (Encoder::with_level) and decompressed-size cap (Decoder::with_max_frame_size), withDEFAULT_LEVEL/DEFAULT_MAX_FRAME_SIZEconsts.Error(Decompress,TooLarge,#[non_exhaustive]).New
@moq/flatepackage (js/flate)Encoder/Decoderwith{ level }/{ maxFrameSize }options.pako(+fflatefor the corruption test) move here from@moq/json.moq-json/@moq/jsonErrornow wrapsmoq_flate::Errortransparently (Flate(#[from] …)), replacing theDecompress/TooLargevariants.@moq/json's public exports are unchanged (it never re-exported the codec).Late-joiner backlog collapse (
moq-jsonconsumer)Consumer::poll_nextnow applies every frame already buffered in the group in one poll, but yields only the latest reconstructed value and deserializes once. Frames are still decoded in order (the DEFLATE window and merge patches are sequential); only the per-frame deserialize + yield are skipped. A late joiner catches up to the head in one step instead of replaying every superseded state; live consumers keeping up still see each update. Mirrored in the JS consumer behavior via existing tests.Example
rs/moq-json/examples/telemetry.rs(cargo run -p moq-json --example telemetry): measures real wire savings on a realistic telemetry stream (~91% smaller with snapshot+delta+deflate) and asserts every tick round-trips (live + late joiner).Public API changes
moq-flatecrate (Encoder,Decoder,Error,Result,DEFAULT_LEVEL,DEFAULT_MAX_FRAME_SIZE);@moq/flatepackage (Encoder,Decoder, options + default consts).moq-json, Rust):Error::DecompressandError::TooLarge(u64)removed in favor ofError::Flate(moq_flate::Error). The enum is already#[non_exhaustive], so externalmatches keep compiling via their wildcard arm.@moq/json's exported surface.Versioning
Bumps
moq-flate/@moq/flateto 0.1.0 andmoq-json/@moq/jsonto 0.1.0. These are hand-edited because release-plz never promotes a0.0.xcrate to0.1.0on its own.Branch targeting
Targets main: the codec being refactored only landed on main (#1897) and is not on
dev, so this can't be based ondev. Themoq-jsonError change is technically breaking, butmoq-jsonis0.0.xwith no external consumers and the0.1.0bump marks the change intentional. Happy to redirect if you'd rather hold it fordev.Cross-package sync
rs/moq-flate↔js/flatekept in lockstep (wire-interoperable, mirrored tests).rs/moq-json↔js/jsonupdated together.Test plan
cargo test -p moq-flate(7) and-p moq-json(31) passbun test js/flate(9) andjs/json(45) passcargo fmt --check+ clippy clean via nixcargo run -p moq-json --example telemetryreports savings and verifies round-trips(Written by Claude)