Skip to content

feat(flate): extract group-scoped DEFLATE into moq-flate / @moq/flate#1905

Merged
kixelated merged 2 commits into
mainfrom
claude/sad-chebyshev-8488b5
Jun 24, 2026
Merged

feat(flate): extract group-scoped DEFLATE into moq-flate / @moq/flate#1905
kixelated merged 2 commits into
mainfrom
claude/sad-chebyshev-8488b5

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Summary

Extracts the per-group DEFLATE codec added in #1897 out of moq-json into a standalone, dependency-light crate moq-flate (and a matching @moq/flate npm package), so any framed-stream library can reuse it without the JSON snapshot/delta layer. The Rust and JS codecs stay wire-interoperable (raw DEFLATE + the RFC 7692 sync-flush marker trick).

This is a pure refactor of the codec plus a small consumer optimization. No wire-format change, so existing tracks (and the hang catalog) and the browser peer are byte-identical on the wire.

What changed

New moq-flate crate (rs/moq-flate)

  • Encoder / Decoder moved over verbatim, made pub. Configurable DEFLATE level (Encoder::with_level) and decompressed-size cap (Decoder::with_max_frame_size), with DEFAULT_LEVEL / DEFAULT_MAX_FRAME_SIZE consts.
  • Own Error (Decompress, TooLarge, #[non_exhaustive]).

New @moq/flate package (js/flate)

  • pako-based Encoder / Decoder with { level } / { maxFrameSize } options. pako (+ fflate for the corruption test) move here from @moq/json.

moq-json / @moq/json

  • Depend on the flate packages; their compression modules are deleted.
  • Rust Error now wraps moq_flate::Error transparently (Flate(#[from] …)), replacing the Decompress / TooLarge variants.
  • @moq/json's public exports are unchanged (it never re-exported the codec).

Late-joiner backlog collapse (moq-json consumer)

  • Consumer::poll_next now applies every frame already buffered in the group in one poll, but yields only the latest reconstructed value and deserializes once. Frames are still decoded in order (the DEFLATE window and merge patches are sequential); only the per-frame deserialize + yield are skipped. A late joiner catches up to the head in one step instead of replaying every superseded state; live consumers keeping up still see each update. Mirrored in the JS consumer behavior via existing tests.

Example

  • rs/moq-json/examples/telemetry.rs (cargo run -p moq-json --example telemetry): measures real wire savings on a realistic telemetry stream (~91% smaller with snapshot+delta+deflate) and asserts every tick round-trips (live + late joiner).

Public API changes

  • New: moq-flate crate (Encoder, Decoder, Error, Result, DEFAULT_LEVEL, DEFAULT_MAX_FRAME_SIZE); @moq/flate package (Encoder, Decoder, options + default consts).
  • Breaking (moq-json, Rust): Error::Decompress and Error::TooLarge(u64) removed in favor of Error::Flate(moq_flate::Error). The enum is already #[non_exhaustive], so external matches keep compiling via their wildcard arm.
  • No change to @moq/json's exported surface.

Versioning

Bumps moq-flate / @moq/flate to 0.1.0 and moq-json / @moq/json to 0.1.0. These are hand-edited because release-plz never promotes a 0.0.x crate to 0.1.0 on its own.

Branch targeting

Targets main: the codec being refactored only landed on main (#1897) and is not on dev, so this can't be based on dev. The moq-json Error change is technically breaking, but moq-json is 0.0.x with no external consumers and the 0.1.0 bump marks the change intentional. Happy to redirect if you'd rather hold it for dev.

Cross-package sync

rs/moq-flatejs/flate kept in lockstep (wire-interoperable, mirrored tests). rs/moq-jsonjs/json updated together.

Test plan

  • cargo test -p moq-flate (7) and -p moq-json (31) pass
  • bun test js/flate (9) and js/json (45) pass
  • tsc + Biome clean; cargo fmt --check + clippy clean via nix
  • cargo run -p moq-json --example telemetry reports savings and verifies round-trips

(Written by Claude)

Pull the per-group DEFLATE codec out of moq-json into a standalone,
dependency-light crate (and matching @moq/flate npm package) so any
framed-stream library can reuse it without the JSON snapshot/delta layer.
The two implementations stay wire-interoperable (raw DEFLATE + RFC 7692
sync-flush marker stripping); this is a pure refactor of the codec, no
wire-format change, so existing tracks and the browser peer are unaffected.

Also collapse the consumer backlog for late joiners: a single poll now
applies every frame already buffered in the group but yields only the
latest reconstructed value. Frames are still decoded in order (the DEFLATE
window and merge patches are sequential); only the per-frame deserialize
and yield are skipped, so a late joiner catches up to the head in one step
instead of replaying every superseded state. Live consumers keeping up
still see each update.

Changes:
- new rs/moq-flate: Encoder/Decoder, configurable level + max-frame-size,
  own Error (Decompress/TooLarge). moq-json depends on it; its Error wraps
  moq_flate::Error transparently (replaces the Decompress/TooLarge variants).
- new js/flate (@moq/flate): pako-based Encoder/Decoder with level /
  maxFrameSize options. @moq/json depends on it; pako/fflate move over.
- moq-json Consumer::poll_next drains buffered frames, deserializing once.
- rs/moq-json/examples/telemetry.rs: measures real wire savings (~91% on a
  realistic telemetry stream) and verifies every tick round-trips.
- bump moq-flate/@moq/flate to 0.1.0 and moq-json/@moq/json to 0.1.0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@kixelated, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 4 minutes and 31 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 54c8651d-14a3-45d8-a577-f7b1daa16fc5

📥 Commits

Reviewing files that changed from the base of the PR and between a7c937e and e7f0874.

⛔ Files ignored due to path filters (2)
  • Cargo.lock is excluded by !**/*.lock
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (19)
  • Cargo.toml
  • js/CLAUDE.md
  • js/flate/README.md
  • js/flate/package.json
  • js/flate/src/index.test.ts
  • js/flate/src/index.ts
  • js/flate/tsconfig.json
  • js/json/package.json
  • js/json/src/compression.test.ts
  • js/json/src/consumer.ts
  • js/json/src/producer.ts
  • package.json
  • rs/CLAUDE.md
  • rs/moq-flate/Cargo.toml
  • rs/moq-flate/src/lib.rs
  • rs/moq-json/Cargo.toml
  • rs/moq-json/examples/telemetry.rs
  • rs/moq-json/src/compression.rs
  • rs/moq-json/src/lib.rs
✨ Finishing Touches
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch claude/sad-chebyshev-8488b5

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kixelated kixelated enabled auto-merge (squash) June 24, 2026 19:18
The backlog-collapse poll_next conflated two break reasons: a group that is
exhausted (poll_read_frame -> Ready(None)) vs. one that is still open with
nothing buffered yet (Pending). On Pending it broke the drain loop and fell
through to the track-finished check, ending the stream early. But a group
appended before the track finished can still deliver frames, so the consumer
must keep waiting on it.

Track the two cases separately and return Pending when the current group is
still open. Regression caught by moq-mux's hang catalog consumer test
(waits_for_pending_catalog_group_payload_after_track_finish); add the
equivalent test directly to moq-json.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kixelated kixelated merged commit fbf706d into main Jun 24, 2026
2 checks passed
@kixelated kixelated deleted the claude/sad-chebyshev-8488b5 branch June 24, 2026 19:50
@moq-bot moq-bot Bot mentioned this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant