Skip to content

Routed reader statements are unbounded after acquire — mid-transaction blackhole hangs requests indefinitely (needs reader statement_timeout) #3651

Description

@tlongwell-block

Summary

Once a routed read has acquired its reader connection, nothing bounds any subsequent statement. A replica path that goes dark mid-transaction (LB failover, security-group change, standby promotion, Aurora failover) converts a routed read into an indefinitely hung request — the writer-fallback code never regains control. Dawn reproduced a hang that ran to her 15s probe cap with no bound in sight (thread 65279d98 in #buzz-read-only-replica-usage, 2026-07-30).

Found during the #3268 activation review. Not a #3268 merge blocker (seams are dark until BUZZ_REPLICA_READ_MAX_AGE_MS is set, and the writer pool has the same missing timeout today), but it is an activation gate: #3268's one-snapshot-across-page+aux design holds reader transactions open longer than any pre-existing read, widening the exposure window.

Evidence (all at dd26caa9f)

What IS bounded — Dawn measured the acquire path exhaustively with a silent TCP listener (accepts, never writes — the hard blackhole shape):

  • Cold connect through acquire: 151–152ms PoolTimedOut, 3/3 (sqlx inner.rs:252-255 wraps the whole acquire future in rt::timeout; connect is re-bounded inside at :338/:350).
  • Idle-connection-gone-stale: 152ms (sqlx test_before_acquire defaults true, ping runs inside the acquire timeout).

What is NOT bounded — after read_pool.acquire() succeeds:

  • MIDTX_STATEMENT elapsed_ms=15002 result=STILL_HANGING_at_15s_cap (UNBOUNDED) — BEGIN succeeded on a healthy replica, path blackholed, next statement hung past the probe cap.
  • No bound exists anywhere (verified independently by Dawn and Eva at the SHA): rg statement_timeout crates/ -g '*.rs' → zero hits repo-wide; the reader pool builder (lib.rs:723-731) sets only max/min/acquire/lifetime/idle, no after_connect; no tokio::time::timeout wraps any routed read. Everything in proved_reader after acquire — Aurora probe (:908), Transaction::begin (:909), observe_heartbeat (:955) — and every page/aux statement on the held snapshot is unbounded.

Fix (small)

Add .after_connect to connect_read_pool issuing SET statement_timeout = '2s' (value debatable; comfortably above any routed read, far below a hung request). The writer pool already uses the after_connect pattern for the floor guard (lib.rs:686), so this matches in-tree conventions. Reader-side is strictly safer than writer-side: replica sessions are read-only.

Bonus: this also bounds the recovery-conflict-cancellation family — a standby cancellation that never arrives is the same hang, and statement_timeout converts it into a reader_validation_error writer fallback.

Consider idle_in_transaction_session_timeout on the same after_connect while there (guards the snapshot-held-but-idle shape between page and aux queries).

Writer-pool statement_timeout is a separate, riskier discussion (writes mid-statement) — explicitly out of scope here.

Acceptance

  • Dawn's mid-tx blackhole probe bounded at the configured timeout with a reader_validation_error → writer fallback serving a complete, correct page (never partial).
  • Live-rig repeat (Max/Wren topology): blackhole the reader path after a routed read acquires; request completes from the writer within timeout + one budget.

Refs: #3268 (activation gate item 5), #3643 (separate unfenced-FTS-pool family), #3622. Found by Dawn; provenance/negatives co-verified by Eva.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions