Per-table autovacuum tuning helper + probe#133
Merged
Conversation
Exports outbox_autovacuum_ddl (importable ALTER TABLE statement for an Alembic migration) + a separate warn-level check_outbox_autovacuum probe. Recommend-not-require: package applies nothing, never raises. Records the load-bearing constraint (SQLAlchemy Table can't carry reloptions, so autogenerate can't emit them), the scale_factor=0 fix, fillfactor excluded on evidence, and benefit documented not benchmarked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
check_outbox_autovacuum is schema-aware but outbox_autovacuum_ddl always rendered an unqualified ALTER TABLE, so a user with the outbox in a named Postgres schema had no way to target the same table the probe checks. Add an optional keyword-only schema param (default None, byte-identical to current output); when set, quote schema and table independently and join with '.' rather than passing a dotted string to a single quote() call.
Fold the standalone warn-level check_outbox_autovacuum into an opt-in validate_schema(check_autovacuum=True) that enforces (raises), giving one schema-health entry point. Records the accepted consequences: autovacuum becomes enforceable rather than advisory, and the check couples to the [validate] (Alembic) extra.
Remove the standalone warn-level check_outbox_autovacuum(engine, table) free
function and fold the reloptions check into validate_schema behind an opt-in
check_autovacuum flag that enforces (raises) rather than advises.
autovacuum.py keeps a pure autovacuum_findings(table_name, reloptions) helper
(the old probe minus the DB query) that OutboxClient.validate_schema calls after
running _RELOPTIONS_QUERY on its existing connection. Autovacuum errors are
labeled distinctly ("Outbox autovacuum not tuned: ") from a schema mismatch so
an operator can tell an untuned-but-working table from a wrong schema.
With dlq_table=None and check_autovacuum=False (the default) every path is
bit-for-bit identical to before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hemas) _run_validate configured Alembic's MigrationContext without include_schemas=True, so compare_metadata never reflected tables outside the connection's default schema. Any outbox table in a non-default MetaData(schema=...) therefore failed validate_schema() with "table 'outbox' does not exist", regardless of whether the schema was actually correct. Add include_schemas=True and narrow _include_name's schema branch to the target schema (name == table.schema; Alembic reports the default schema as None, so this matches None-vs-None for a default-schema table and the literal name otherwise) so unrelated schemas never reflect into the diff as false drift. Unblocks the named-schema autovacuum round-trip: the autovacuum check now routes through validate_schema(check_autovacuum=True) on a named-schema table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hema With include_schemas=True, Alembic reports the connection's default schema to the include_name hook as None. The prior `name == table.schema` check therefore excluded a table that explicitly names the default schema (MetaData(schema="public"), or a named schema on the connection's search_path): the table never reflected and a CORRECT table falsely raised "table 'outbox' does not exist" — a false positive that would hard-fail a correct deployment's startup gate. Normalize table.schema against connection.dialect.default_schema_name: when the table names the default schema, treat it as None so it matches Alembic's None. Non-default named schemas are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds per-table autovacuum tuning for the high-churn outbox table, and — as this work surfaced — fixes a pre-existing bug where
validate_schemanever worked for tables in a named Postgres schema.Design and its evolution:
planning/changes/2026-07-15.02-autovacuum-tuning.md. Third in the outbox-performance series (harness #130, batched flush #131).Why
Every message is one
INSERT+ one leaseUPDATE+ one terminalDELETE, so the benchmark measureddead_tup ≈ 2 × messages. Postgres' defaultautovacuum_vacuum_scale_factor = 0.2fires vacuum only after a fraction of the table is dead — on a queue table that lets bloat grow, and if it bloats the fraction is of the bloated size, so vacuum fires ever less often (a death spiral). SQLAlchemy'sTablecan't carry reloptions, so Alembic autogenerate can never emit them; the settings must be applied by an explicit statement.What ships
outbox_autovacuum_ddl(table_name="outbox", *, schema=None, vacuum_threshold=1000, insert_threshold=1000) -> str(public) — renders the recommendedALTER TABLE … SET (autovacuum_*)for an Alembic migration (op.execute(...)).scale_factor=0is the structural fix; thresholds tunable; schema-aware. Nofillfactor(HOT is structurally impossible here — excluded on evidence).validate_schema(*, check_autovacuum=False)— opt-in enforcement. WhenTrue,validate_schemaalso probespg_class.reloptionsand raises (labeled distinctly from a schema mismatch) if the table lacks the aggressive settings. One schema-health entry point; requires the[validate]extra. (Design note: this started as a standalonecheck_outbox_autovacuumprobe and was folded intovalidate_schemaon review — see the spec's revision note.)validate_schemanow works for named-schema tables. It previously configured Alembic withoutinclude_schemas=True, so aMetaData(schema="app")table falsely reported "table does not exist." Fixed withinclude_schemas=True+ a schema-scopedinclude_namefilter that normalizes an explicitly-named default schema against the connection'sdefault_schema_name. This fixes validation for all named-schema deployments, not just autovacuum.Testing
Deterministic. Unit tests pin the rendered SQL (default, overrides, quoting, schema-qualified). Integration tests cover
validate_schema(check_autovacuum=True)(raises when untuned / labeled distinctly / not on custom threshold / not when off), the named-schema round-trip throughvalidate_schema, and the schema-normalization cases (idiomatic default, explicitly-named default, named non-default) with no false or missed drift. Full suite 589 passed at 100% coverage. The bloat-control benefit is documented (the death-spiral mechanism), not benchmarked — it's a per-time emergent property that autovacuum's async schedule makes un-gateable.🤖 Generated with Claude Code