Skip to content

test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619

Description

@tlongwell-block

Summary

tests::test_usage_metrics_lock_has_single_owner_and_releases_on_drop (crates/buzz-db/src/lib.rs:5782) runs against the shared TEST_DATABASE_URL database and hardcodes the production advisory lock key (lib.rs:5792: 0x4255_5A5A_4D45_5452, identical to USAGE_METRICS_LOCK_KEY at crates/buzz-relay/src/main.rs:84).

Postgres advisory locks are per-database, so any live buzz-relay pointed at the same database deterministically holds that key (it re-arms on the usage_metrics_interval_secs ~300s tick, main.rs ~:1328), and the test reds at the "first database handle becomes leader" assertion. Whether it passes is a race against the relay's tick — several of us burned real diagnosis time on this during #3268 verification (thread cf5deba4 in #buzz-read-only-replica-usage), including one near-miss where the "remedy" almost terminated a 3-day-old live dev relay.

Proven remedy (verified 2026-07-29 by Dawn, reproduced by Eva and Max)

The test passes on a private scratch database with the relay still running and still holding its lock — advisory locks don't collide across databases:

TEST_DATABASE_URL=postgres://…/scratch_<uuid> cargo test -p buzz-db --lib -- --ignored \
  --exact tests::test_usage_metrics_lock_has_single_owner_and_releases_on_drop
→ ok. 1 passed

Fix options

  1. Preferred: have the test create/drop its own scratch DB via the existing create_scratch_db helper (lib.rs:6362), like the routing tests already do.
  2. Alternative: derive the lock key from the database name/test run instead of hardcoding the production constant (keeps shared-DB runs safe but loses the "same key as prod" documentation value — a comment + constant import would preserve it).

Notes for whoever picks this up

  • Never pg_terminate_backend an advisory holder without resolving client_port → owning process via lsof first; an idle holder is likely a live dev relay, not a leaked test backend.
  • Related but separate pre-existing flake: api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo (Redis-dependent, documented history in git log) — not caused by this lock.

Pre-existing test-isolation bug; predates PR #3268 and is not changed by it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions