NIP-50 search pool uses READ_DATABASE_URL raw, bypassing all replica fencing — live in bb-block and bb-public today

[tlongwell-block]

Summary

NIP-50 search creates its own eager, default-timeout SQLx pool directly from READ_DATABASE_URL (crates/buzz-relay/src/main.rs:389-402 at dd26caa9f; same code at origin/main ab55fee81). This pool bypasses all of the replica fencing machinery in buzz-db: no heartbeat/freshness proof, no 150ms bounded acquire, no writer fallback, and BUZZ_REPLICA_READ_MAX_AGE_MS cannot gate it (the budget gates hydration only — the initial FTS query that produces the IDs runs unfenced).

This is not a #3268 defect and it is live in production today:

• Introduced by Route lag-tolerant reads to an optional Postgres read replica #2084 (29c48883d, 2026-07-18) — predates the feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268 branch.
• Present byte-for-byte in both deployed images: bb-block sha-dd222a5, bb-public sha-22be8bb (verified via git show <sha>:crates/buzz-relay/src/main.rs).
• Both production ESOs template READ_DATABASE_URL → the Aurora cluster-ro endpoint (bb-block: externalsecret-buzz-secrets.yaml:28, added in bb-block fix(desktop): prevent-sleep spawn convention + expiry badge #487, survived fix: classify agent JSON-RPC errors as application errors, not transport #539; bb-public: equivalent line, cluster-ro-cv206gsyeugr). The rendered bb-block deployment-buzz.yaml:98-103 wires it into the pod. Relay logs in prod will show Search service ready ... replica=true.

Measured failure modes (Max's live Helm redteam at dd26caa9f, PG17 physical streaming; evidence: RESEARCH/PR3268_LIVE_REPLICA_REDTEAM.md, thread cf5deba4 in #buzz-read-only-replica-usage)

1. Reader loss stalls live search ~30s — NIP-50 request exceeded the 10s client deadline; SQLx pool timeout surfaced at ~30s instead of the fenced paths' 150ms writer fallback.
2. Replica lag makes acknowledged writes unsearchable — with WAL replay paused, search returned zero hits for a message the writer had just accepted, while the fenced channel paths correctly fell back writer/stale. Resuming replay fixed the identical query.
3. Reader loss blocks pod startup/rollout — the eager .connect().await? is fatal; with the standby absent a replacement pod stayed 0/1 past 50s while the fenced lazy pool correctly logged serving all-writer.
4. B=0 does not disable replica FTS — routed hydration reported writer/disabled while FTS still logged replica=true and queried the reader.

Aurora nuance (why prod hasn't burned yet)

Max's rig used vanilla physical streaming, where a dead standby = failed connect. Aurora's cluster-ro endpoint resolves to the writer when no readers exist and re-points on failover, so modes 1 and 3 are softened to DNS-failover windows rather than hard outages. Mode 2 is not softened: observed bb-block replica lag is 5-25ms baseline but spiked to 3,976ms at 2026-07-29 16:11Z (aws.rds.aurora_replica_lag, fine-grained query) — during any such window, freshly accepted messages are silently unsearchable. Frequency ~1-2 one-minute windows/day.

Fix options

• Minimal (recommended first): pin the FTS pool to database_url unconditionally. One-line semantic change; search load returns to the writer (it lived there before Route lag-tolerant reads to an optional Postgres read replica #2084).
• Proper (follow-up): route the initial FTS query through the same bounded/fenced writer-fallback contract as the routed seams in feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268.

Interim config-only mitigation if search symptoms appear before the fix deploys: remove READ_DATABASE_URL from the ESO (Max verified this rollback rolls out cleanly).

Verification gate (per Max — required before any future reader URL enables replica FTS)

1. Reader-down traffic latency (search must fall back within budget, not 30s).
2. Reader-down cold boot / scale-out (pod must become ready).
3. Paused-replay read-your-own-write search (accepted message must be findable).

Note: #3622 (CI enforcement of the buzz-db fixtures) would not catch this — it is a separate pool outside buzz-db.

Refs: #3268, #2084, #3622. Found by Max's live redteam; provenance verified by Eva.

[tlongwell-block]

Reopening: this was auto-closed by GitHub keyword parsing — PR #3268's body contained the phrase "→ fix #3643 →" in the rollout-ladder prose, which GitHub treated as a closing keyword when the PR merged (closed 2026-07-30T00:35:17Z, one second after the merge at 00:35:16Z).

The bug is NOT fixed. Verified at 3b8567a05 (current origin/main and the sha-3b8567a image in squareup/bb-block#578 / squareup/bb-public#325), crates/buzz-relay/src/main.rs:402-410: the NIP-50 search pool is still built raw from read_database_url with an eager .connect(), no fence, no fallback, no acquire budget. All failure modes in this issue remain live in the image both deploy PRs are about to roll.