Summary
tests::test_usage_metrics_lock_has_single_owner_and_releases_on_drop (crates/buzz-db/src/lib.rs:5782) runs against the shared TEST_DATABASE_URL database and hardcodes the production advisory lock key (lib.rs:5792: 0x4255_5A5A_4D45_5452, identical to USAGE_METRICS_LOCK_KEY at crates/buzz-relay/src/main.rs:84).
Postgres advisory locks are per-database, so any live buzz-relay pointed at the same database deterministically holds that key (it re-arms on the usage_metrics_interval_secs ~300s tick, main.rs ~:1328), and the test reds at the "first database handle becomes leader" assertion. Whether it passes is a race against the relay's tick — several of us burned real diagnosis time on this during #3268 verification (thread cf5deba4 in #buzz-read-only-replica-usage), including one near-miss where the "remedy" almost terminated a 3-day-old live dev relay.
Proven remedy (verified 2026-07-29 by Dawn, reproduced by Eva and Max)
The test passes on a private scratch database with the relay still running and still holding its lock — advisory locks don't collide across databases:
TEST_DATABASE_URL=postgres://…/scratch_<uuid> cargo test -p buzz-db --lib -- --ignored \
--exact tests::test_usage_metrics_lock_has_single_owner_and_releases_on_drop
→ ok. 1 passed
Fix options
- Preferred: have the test create/drop its own scratch DB via the existing
create_scratch_db helper (lib.rs:6362), like the routing tests already do.
- Alternative: derive the lock key from the database name/test run instead of hardcoding the production constant (keeps shared-DB runs safe but loses the "same key as prod" documentation value — a comment + constant import would preserve it).
Notes for whoever picks this up
- Never
pg_terminate_backend an advisory holder without resolving client_port → owning process via lsof first; an idle holder is likely a live dev relay, not a leaked test backend.
- Related but separate pre-existing flake:
api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo (Redis-dependent, documented history in git log) — not caused by this lock.
Pre-existing test-isolation bug; predates PR #3268 and is not changed by it.
Summary
tests::test_usage_metrics_lock_has_single_owner_and_releases_on_drop(crates/buzz-db/src/lib.rs:5782) runs against the sharedTEST_DATABASE_URLdatabase and hardcodes the production advisory lock key (lib.rs:5792:0x4255_5A5A_4D45_5452, identical toUSAGE_METRICS_LOCK_KEYatcrates/buzz-relay/src/main.rs:84).Postgres advisory locks are per-database, so any live
buzz-relaypointed at the same database deterministically holds that key (it re-arms on theusage_metrics_interval_secs~300s tick,main.rs~:1328), and the test reds at the "first database handle becomes leader" assertion. Whether it passes is a race against the relay's tick — several of us burned real diagnosis time on this during #3268 verification (thread cf5deba4 in #buzz-read-only-replica-usage), including one near-miss where the "remedy" almost terminated a 3-day-old live dev relay.Proven remedy (verified 2026-07-29 by Dawn, reproduced by Eva and Max)
The test passes on a private scratch database with the relay still running and still holding its lock — advisory locks don't collide across databases:
Fix options
create_scratch_dbhelper (lib.rs:6362), like the routing tests already do.Notes for whoever picks this up
pg_terminate_backendan advisory holder without resolvingclient_port→ owning process vialsoffirst; an idle holder is likely a live dev relay, not a leaked test backend.api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo(Redis-dependent, documented history in git log) — not caused by this lock.Pre-existing test-isolation bug; predates PR #3268 and is not changed by it.