Benchmark harness for the outbox transport#130
Merged
Conversation
Places the library in Variant 2 of Tsvettsikh's outbox taxonomy and records the measurements that shape the harness: per-row terminal DELETE is the entire round-trip budget (1.044 rt/msg), WAL and wall-clock disagree by two orders of magnitude on whether batching matters, and fillfactor cannot help the claim UPDATE (HOT is structurally impossible). Gate tiers are set by measured determinism, not assumption: calls is exact (0.0% spread), wal_records tolerant (3.3%), wal_bytes ungated (65.8%, FPI-dominated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Snapshots pg_stat_statements/pg_stat_wal/pg_stat_user_tables around a workload and returns a ProbeResult delta. Filters its own queries out of the round-trip count by SQL text (NOT ILIKE '%pg_stat%'), subtracts cumulative table/WAL counters from a start snapshot, and settles on pg_stat_user_tables before reading (the stats collector flushes lazily). assert_owns_database guards against a foreign backend polluting the database-wide statement counters.
The settle loop polled n_tup_upd + n_tup_del, which never moves for an insert-only workload, and its 0.75s stability window was shorter than Postgres' 1s PGSTAT_MIN_INTERVAL, so the seed's stats could still be unflushed when the origin was captured and leaked into the delta. Replace it with engine.dispose() before each snapshot: Postgres force-flushes a backend's pending stats when the backend exits, so the flush is exact rather than eventual. Run every probe statement on an AUTOCOMMIT connection. SQLAlchemy's implicit BEGIN/COMMIT/ROLLBACK are tracked by pg_stat_statements and contain no 'pg_stat' substring, so they slipped past the self-exclusion filter and inflated the exactly-gated 'calls' metric by a run-dependent amount. The workload's own transaction control is still counted. Also pin schemaname in the catalog lookups and fail fast at probe open on a missing table, instead of dying with NoResultFound at the end of a long run.
engine.dispose() only closes checked-in connections. A workload that still holds a connection when the probe block exits never flushes that backend's stats, so tup_ins/tup_upd/tup_del/wal_records silently under-report (or read zero) while calls stays correct and looks plausible. Assert the pool has zero checked-out connections on the exit path and raise RuntimeError naming the count, matching the existing assert_owns_database standard of failing loudly instead of reporting a quietly wrong number. Also document the other footgun review found: constructing an AsyncEngine inside the probe window inflates calls by 6 via SQLAlchemy's one-time dialect-init queries, none of which match the pg_stat filter.
…and exact The benchmark gate must assert structural, per-message quantities, not totals that scale with wall-clock. Five measurement defects, all reproduced under load: - probes: break `calls` down by SQL operation into `ProbeResult.calls_by_op` (a second aggregate grouped by the leading keyword, on the same AUTOCOMMIT connection, self-excluding via `pg_stat` in its text). The terminal DELETE is exactly 1/message on the happy path, so `calls_by_op['delete']` isolates a load-independent gate metric; the total `calls` stays informational. - workload: pass min/max_fetch_interval=0.001 so drain is DB-bound, not sleep-bound (the seed emits no pg_notify, so the fetch loop otherwise slept its full max_fetch_interval and max_workers had zero throughput effect). - workload: count distinct row identities (unique seed payload per row) for drain detection so a lease-expiry redelivery can't stop the broker early; assert count(*)==0 outside the probe block to catch a partial drain. - workload: disable autovacuum on the bench table so a mid-window autovacuum can't reap dead tuples; dead_tup becomes an exact, deterministic garbage count. - workload: silence the broker (logger=None) so per-message INFO logging stays out of the window; reuse probes._autocommit in _checkpoint; drop the no-op future=True from make_engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gate raw structural totals (delete_calls + tuple counters exact, wal_records within a 10% band), not the load-dependent total calls or FPI-noisy WAL bytes. Adds pure-function unit tests that import benchmarks.* to prove the coverage omit keeps the Postgres-only modules out of the 100% gate.
Adds a bench job that runs just bench-check via docker compose (not a services: container, since GitHub Actions service containers can't take command args and therefore can't set shared_preload_libraries for pg_stat_statements).
The plan proposed gating total pg_stat_statements.calls on exact equality; implementation falsified that (fetch-loop polling makes it load-dependent, 519 idle vs 894 loaded). Record the pivot to per-operation + tuple-counter gating, correct the discovery numbers (dead_tup=2x messages, ~7 WAL records/msg not 11), and document the force-flush probe and the descoped publish_batch.
The bench job is the only CI job that builds the docker compose application image, whose Dockerfile does COPY uv.lock + uv sync --frozen. uv.lock is git-ignored, so a clean CI checkout lacks it and docker build fails on COPY uv.lock. Generate the lock in the job first. Verified by building from a clean git-archive checkout with and without the step. Also cap the job at 20 minutes so a pathological drain hang fails fast.
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
benchmarks/— a harness that drives the real FastStream outbox broker against Postgres and reports per-message DB counters plus throughput, with a CI gate on the structural subset. Establishes the performance baseline for the outbox work. Ships the instrument, not a fix.Design and the measurements behind it (the change's truth home):
planning/changes/2026-07-14.01-benchmark-harness.md.What it measures
Two scenarios against a fresh per-run table, 256-byte payloads (commensurable with Tsvettsikh's talk):
max_workers ∈ {1,2,4}×fetch_batch_size ∈ {10,100}.publish()in one transaction.The probe snapshots Postgres catalogs (
pg_stat_statements,pg_stat_wal,pg_stat_user_tables) around the workload and returns per-op statement counts + tuple/WAL deltas. It force-flushes stats viaengine.dispose()and runs every probe query on AUTOCOMMIT so it never counts itself (empty workload →calls == 0).The gate
just bench-checkcompares a run against the committedbenchmarks/baseline.jsonand fails a PR on drift. It gates only structural, load-independent per-message work:delete_calls/insert_calls/select_calls(fromcalls_by_op) and the tuple counterstup_upd/tup_del/tup_inswal_recordscalls,wal_bytes,wall-clock(timing/FPI noise)The original plan gated total
pg_stat_statements.callsexactly. Implementation falsified that: the subscriber's fetch loop polls on a wall-clock timer, so totalcallswas 519 idle vs 894 under CPU load. The gate moved to per-operation + tuple counts, which are identical idle and under load. The spec records this pivot in full.Headline baseline numbers (today's code)
dead_tup == 2 × messages, exact — a direct measurement of the talk's "2× mutations → 2× autovacuum" cost for this outbox variant.tup_hot_upd == 0— HOT is structurally impossible (the claim UPDATE mutates both partial indexes' key columns).publish()(INSERT … RETURNING+SELECT pg_notify).Verification
bench-checkfail onselect_calls(0 → 5000) across all consumer points; reverted, it returns green.--cov-fail-under=100(benchmarks/*is coverage-omitted; a test imports it).benchmarks/is dev tooling, never shipped in the wheel.No runtime behavior changes; no
architecture/capability is affected.🤖 Generated with Claude Code