Skip to content

CI never runs the buzz-db replica-routing Postgres fixtures — enforce them before enabling BUZZ_REPLICA_READ_MAX_AGE_MS #3622

Description

@tlongwell-block

Summary

None of the Postgres-backed replica-routing fixtures added in #3268 — nor most of the 34 #[ignore]d tests in crates/buzz-db/src/lib.rs — are ever executed by CI. They were verified exhaustively by hand at dd26caa9f (see the PR body), but a future regression on the routed-read path can merge green.

Found by Dawn during her #3268 sign-off amendment; independently confirmed by Max and Eva at the same SHA.

Evidence (at dd26caa9f)

  • Unit Tests runs just test-unitcargo nextest run -p buzz-db --lib (Justfile:279-285). Ignored tests are excluded by design, and the Justfile comment says so.
  • The only --run-ignored ignored-only nextest invocations in .github/workflows/ are ci.yml:690 and :702, both filtered to (package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/)).
  • Therefore, of the 34 #[ignore]d buzz-db tests, CI runs only the relay_invite::tests subset. The one-budget acquire regression (routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold), the seven-seam community-isolation fixture, and all fence/floor-guard/fallback fixtures have zero automated execution.

Why this matters

  • The one-budget fixture exists precisely because a "fix is present" claim survived review without a measurement (feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268 thread, Dawn's Pass 11 postmortem). An #[ignore]d test no job selects is a comment, not a gate.
  • Max's rollout condition: do not set BUZZ_REPLICA_READ_MAX_AGE_MS on bb-block until this is fixed. Merge itself is safe (new seams are dark by default), but enabling the new seams in production without CI enforcement of their regressions is not.

Constraints on the fix

  1. Ordering dependency on test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619: the moment any filter widening selects test_usage_metrics_lock_has_single_owner_and_releases_on_drop, the shared-DB advisory-lock collision fires in CI. test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 (scratch-DB isolation for that test) must land first, or the selection must explicitly exclude it.
  2. A naive regex is insufficient. Dawn's first-draft filter test(/tests::(routed|replica|reader|fence|floor_guard)/) misses at least: query_events_routed_*, count_events_routed_*, lazy_reader_pool_*, head_fetch_routes_by_configured_budget, channel_window_routes_*, thread_replies_cursor_pages_*, read_session_degrades_*. A partially-covering filter recreates this bug one level down — green CI, unenforced invariants.
  3. Several fixtures assume scratch DBs and serial execution; the selection should run serially against the existing backend-integration Postgres archive (that job already has the infra).

Suggested shape (per Max)

An explicit, auditable selection rather than a clever regex: either move the replica-routing fixtures into a dedicated test module (e.g. tests::replica_routing::) and filter on that module, or maintain an explicit nextest filter list in ci.yml with a comment stating it must cover every #[ignore]d routing fixture. Initially cover the exact routing matrix verified by hand in #3268. Keep test_usage_metrics_lock... excluded until #3619 lands.

Acceptance

Refs: #3268, #3619.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions