Summary
Once a routed read has acquired its reader connection, nothing bounds any subsequent statement. A replica path that goes dark mid-transaction (LB failover, security-group change, standby promotion, Aurora failover) converts a routed read into an indefinitely hung request — the writer-fallback code never regains control. Dawn reproduced a hang that ran to her 15s probe cap with no bound in sight (thread 65279d98 in #buzz-read-only-replica-usage, 2026-07-30).
Found during the #3268 activation review. Not a #3268 merge blocker (seams are dark until BUZZ_REPLICA_READ_MAX_AGE_MS is set, and the writer pool has the same missing timeout today), but it is an activation gate: #3268's one-snapshot-across-page+aux design holds reader transactions open longer than any pre-existing read, widening the exposure window.
Evidence (all at dd26caa9f)
What IS bounded — Dawn measured the acquire path exhaustively with a silent TCP listener (accepts, never writes — the hard blackhole shape):
- Cold connect through acquire: 151–152ms
PoolTimedOut, 3/3 (sqlx inner.rs:252-255 wraps the whole acquire future in rt::timeout; connect is re-bounded inside at :338/:350).
- Idle-connection-gone-stale: 152ms (sqlx
test_before_acquire defaults true, ping runs inside the acquire timeout).
What is NOT bounded — after read_pool.acquire() succeeds:
MIDTX_STATEMENT elapsed_ms=15002 result=STILL_HANGING_at_15s_cap (UNBOUNDED) — BEGIN succeeded on a healthy replica, path blackholed, next statement hung past the probe cap.
- No bound exists anywhere (verified independently by Dawn and Eva at the SHA):
rg statement_timeout crates/ -g '*.rs' → zero hits repo-wide; the reader pool builder (lib.rs:723-731) sets only max/min/acquire/lifetime/idle, no after_connect; no tokio::time::timeout wraps any routed read. Everything in proved_reader after acquire — Aurora probe (:908), Transaction::begin (:909), observe_heartbeat (:955) — and every page/aux statement on the held snapshot is unbounded.
Fix (small)
Add .after_connect to connect_read_pool issuing SET statement_timeout = '2s' (value debatable; comfortably above any routed read, far below a hung request). The writer pool already uses the after_connect pattern for the floor guard (lib.rs:686), so this matches in-tree conventions. Reader-side is strictly safer than writer-side: replica sessions are read-only.
Bonus: this also bounds the recovery-conflict-cancellation family — a standby cancellation that never arrives is the same hang, and statement_timeout converts it into a reader_validation_error writer fallback.
Consider idle_in_transaction_session_timeout on the same after_connect while there (guards the snapshot-held-but-idle shape between page and aux queries).
Writer-pool statement_timeout is a separate, riskier discussion (writes mid-statement) — explicitly out of scope here.
Acceptance
- Dawn's mid-tx blackhole probe bounded at the configured timeout with a
reader_validation_error → writer fallback serving a complete, correct page (never partial).
- Live-rig repeat (Max/Wren topology): blackhole the reader path after a routed read acquires; request completes from the writer within timeout + one budget.
Refs: #3268 (activation gate item 5), #3643 (separate unfenced-FTS-pool family), #3622. Found by Dawn; provenance/negatives co-verified by Eva.
Summary
Once a routed read has acquired its reader connection, nothing bounds any subsequent statement. A replica path that goes dark mid-transaction (LB failover, security-group change, standby promotion, Aurora failover) converts a routed read into an indefinitely hung request — the writer-fallback code never regains control. Dawn reproduced a hang that ran to her 15s probe cap with no bound in sight (thread 65279d98 in #buzz-read-only-replica-usage, 2026-07-30).
Found during the #3268 activation review. Not a #3268 merge blocker (seams are dark until
BUZZ_REPLICA_READ_MAX_AGE_MSis set, and the writer pool has the same missing timeout today), but it is an activation gate: #3268's one-snapshot-across-page+aux design holds reader transactions open longer than any pre-existing read, widening the exposure window.Evidence (all at
dd26caa9f)What IS bounded — Dawn measured the acquire path exhaustively with a silent TCP listener (accepts, never writes — the hard blackhole shape):
PoolTimedOut, 3/3 (sqlxinner.rs:252-255wraps the whole acquire future inrt::timeout; connect is re-bounded inside at:338/:350).test_before_acquiredefaults true, ping runs inside the acquire timeout).What is NOT bounded — after
read_pool.acquire()succeeds:MIDTX_STATEMENT elapsed_ms=15002 result=STILL_HANGING_at_15s_cap (UNBOUNDED)— BEGIN succeeded on a healthy replica, path blackholed, next statement hung past the probe cap.rg statement_timeout crates/ -g '*.rs'→ zero hits repo-wide; the reader pool builder (lib.rs:723-731) sets only max/min/acquire/lifetime/idle, noafter_connect; notokio::time::timeoutwraps any routed read. Everything inproved_readerafter acquire — Aurora probe (:908),Transaction::begin(:909),observe_heartbeat(:955) — and every page/aux statement on the held snapshot is unbounded.Fix (small)
Add
.after_connecttoconnect_read_poolissuingSET statement_timeout = '2s'(value debatable; comfortably above any routed read, far below a hung request). The writer pool already uses theafter_connectpattern for the floor guard (lib.rs:686), so this matches in-tree conventions. Reader-side is strictly safer than writer-side: replica sessions are read-only.Bonus: this also bounds the recovery-conflict-cancellation family — a standby cancellation that never arrives is the same hang, and
statement_timeoutconverts it into areader_validation_errorwriter fallback.Consider
idle_in_transaction_session_timeouton the sameafter_connectwhile there (guards the snapshot-held-but-idle shape between page and aux queries).Writer-pool
statement_timeoutis a separate, riskier discussion (writes mid-statement) — explicitly out of scope here.Acceptance
reader_validation_error→ writer fallback serving a complete, correct page (never partial).Refs: #3268 (activation gate item 5), #3643 (separate unfenced-FTS-pool family), #3622. Found by Dawn; provenance/negatives co-verified by Eva.