fix(moq-relay,moq-native): stop the cert-reload busy loop, then dedupe FileWatcher

[kixelated]

Summary

Fixes a self-sustaining TLS cert-reload loop that took a production relay fully offline, then removes the duplication that made the fix land in two places.

Two commits:

1. fix: only reload certs on real change events — the bug.
2. refactor: share moq-native's FileWatcher — cleanup so the next fix lands once.

The bug

FileWatcher watches the parent directory of the cert/key files (correct: rotations swap the inode via atomic rename or symlink, so a path-level watch goes deaf after the first rotation, and the K8s ..data swap never names the cert file). But notify's inotify backend reports IN_OPEN/IN_ACCESS, and the callback treated every event as a change. The reload path opens and reads those same files, so each reload fired events that re-triggered it:

• ~400 reloads/sec, pinning a core at ~86% CPU.
• ServeCerts takes the RwLock write lock on every reload; inbound handshakes take the read lock. The write-lock churn + CPU burn starved all TLS handshakes, so QUIC/WebTransport and the wss:// TCP fallback failed.
• Kicked off the first time anything else opened a file in the watched dir (e.g. an hourly cert read). No file ever actually changed (mtimes stable), confirming a pure read-feedback loop.

The fix

Filter the notify callback through an explicit match allowlist of events that can mean new cert bytes:

• Reload on: Create, Modify (data/rename), and a finished write (IN_CLOSE_WRITE → Access(Close(Write))).
• Ignore: read-only access (open/read/close-without-write) and bare Remove (nothing to load until the replacement's create).
• A watcher error (e.g. inotify queue overflow) still reloads, since it may have dropped a real change.

Behavior change to note: a bare Remove no longer triggers a reload. Real rotations (cert-manager, mv-into-place, K8s ..data) are unaffected — they arrive as create/modify/rename.

Added a is_reload_trigger predicate with unit tests that don't depend on real inotify timing.

The dedupe

moq-relay and moq-native each carried a near-identical FileWatcher, so the fix above had to land twice. Consolidated onto moq-native's copy (the superset, with empty-parent handling):

• Exposed as moq_native::watch::FileWatcher behind a new watch feature (implied by the quinn/noq backends, so default builds are unchanged). Minimal surface: new + changed public, is_reload_trigger stays private. Re-exported notify (since new returns notify::Result), consistent with the crate's existing pub use rustls; pub use moq_net;.
• moq-relay enables moq-native/watch unconditionally (web + cluster reload paths need it regardless of QUIC backend), drops its own notify dep and watch.rs, and points web.rs/cluster.rs at the shared type.

Public API changes

• moq-native (additive, non-breaking): new pub mod watch with pub struct FileWatcher (new, changed), new watch feature, and pub use notify (both feature-gated). A major notify bump becomes a breaking change for moq-native, documented at the re-export. Not in the dev-gated list and purely additive, so targeting main; reviewer may want a minor version bump for the new surface.
• moq-relay: removed the private mod watch and the notify dependency. No public API change.

Branch targeting

main: a bug fix plus an additive, non-breaking library change. No wire/protocol or catalog/container changes.

Cross-package sync

rs/moq-net wire/API untouched, so no js/net or doc/concept updates needed. Relay config/behavior unchanged (hot-reload still works; it just no longer self-triggers), so no doc/bin/relay/ change.

Test plan

All via nix develop (CI's pinned toolchain):

• cargo test -p moq-native — watch tests pass (read/remove ignored, write/create/modify trigger)
• cargo clippy -p moq-native -p moq-relay --all-targets — clean
• Feature-matrix: moq-native compiles both with and without watch
• cargo-shear — clean (confirms the notify removal)
• taplo format --check — moq-native/moq-relay Cargo.toml canonical

Deploy note

The live incident still needs a moq-relay release + fleet redeploy; immediate per-node mitigation is stop moq-health (removes the trigger) then restart moq-relay.

(Written by Claude)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2bf96e64-e3f9-4561-898c-8c24fe739672

📥 Commits

Reviewing files that changed from the base of the PR and between 160daf0 and db7cfcd.

📒 Files selected for processing (1)

• rs/moq-native/Cargo.toml

🚧 Files skipped from review as they are similar to previous changes (1)

• rs/moq-native/Cargo.toml

---

Walkthrough

The FileWatcher implementation previously duplicated in moq-relay is extracted into moq-native as a public API. A new standalone watch feature is added to moq-native/Cargo.toml; the existing quinn and noq features now depend on it instead of directly pulling dep:notify. In lib.rs, the watch module becomes pub mod watch gated on the watch feature, and the notify crate is re-exported. In watch.rs, FileWatcher, new, and changed become fully public, a new private is_reload_trigger helper filters notify event kinds to avoid reload loops, and unit tests cover the filtering logic. In moq-relay, the duplicate watch.rs is deleted, the internal mod watch declaration is removed, and all call sites in cluster.rs and web.rs are updated to use moq_native::watch::FileWatcher.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately captures the two main changes: fixing the cert-reload busy loop and eliminating FileWatcher duplication between moq-relay and moq-native.
Description check	✅ Passed	The description provides detailed context on the bug, fix, and dedupe refactoring, directly aligned with the changeset modifications across moq-native and moq-relay.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/reverent-volhard-faf82b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rs/moq-native/src/lib.rs`:
- Around line 46-49: The re-export of the notify crate is currently documented
with regular comments (`//`) instead of rustdoc comments (`///`), which means
the explanation for the re-export will not appear in generated documentation.
Convert the two comment lines explaining why notify is re-exported (the lines
before `pub use notify;` in the watch feature block) to use `///` syntax instead
of `//` so that the documentation is included in the generated rustdoc output.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: aa2af580-7f4f-47ba-9773-033f646bbabf

📥 Commits

Reviewing files that changed from the base of the PR and between 2788d79 and 068a226.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (8)

• rs/moq-native/Cargo.toml
• rs/moq-native/src/lib.rs
• rs/moq-native/src/watch.rs
• rs/moq-relay/Cargo.toml
• rs/moq-relay/src/cluster.rs
• rs/moq-relay/src/lib.rs
• rs/moq-relay/src/watch.rs
• rs/moq-relay/src/web.rs

💤 Files with no reviewable changes (2)

• rs/moq-relay/src/watch.rs
• rs/moq-relay/src/lib.rs