Skip to content

fix: Slow lower search functionality and cost.py fix#22

Merged
gkennos merged 11 commits into
mainfrom
21-indices
Jun 25, 2026
Merged

fix: Slow lower search functionality and cost.py fix#22
gkennos merged 11 commits into
mainfrom
21-indices

Conversation

@nicoloesch

@nicoloesch nicoloesch commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Fix

Added two functional btree indexes:

  • ix_concept_concept_name_lower on concept (lower(concept_name))
  • ix_concept_synonym_concept_synonym_name_lower on concept_synonym (lower(concept_synonym_name))

Declared via omop_index() in the Concept / Concept_Synonym ORM __table_args__, applied through the existing omop-alchemy indexes enable --vocab maintenance CLI. ANALYZE now runs automatically right after index creation so the planner picks up new indexes immediately (previously only autovacuum's load-time analyze covered existing indexes; a newly added index gets no stats until something explicitly analyzes it). Also added a --cluster/--no-cluster flag to indexes enable to skip the full heap rewrite on large vocabulary tables when disk headroom is limited, and hardened index creation against SQLite's inability to reflect expression-based indexes (a duplicate-create attempt on an existing functional index is now reported as skipped instead of raising).

Why OMOP_Alchemy

These are general-purpose, exact-match indexes useful to anything querying concept/synonym names directly, not specific to a single consumer, so they belong with the rest of the CDM schema/index definitions here.

Result

EXPLAIN ANALYZE confirms the plan changed from sequential scan to Index Scan: ~24,000ms → ~0.05ms.


Due to the importance of it, this PR also includes the following bug fixes found:

  • manage_indexes() raised KeyError for cost.cost_type_concept_id under any schema-qualified connection: cost was the only model using an implicitly-named column index (required_concept_fk(index=True)), whose auto-generated name embeds the schema and drifts from the unqualified name the maintenance CLI looks up by. Fixed by switching to an explicit omop_index(...), matching every other model.

  • fact_relationship's primary key was silently missing fact_id_1/fact_id_2: merge_table_args() dropped the explicit PrimaryKeyConstraint because it's falsy (zero-length .columns) until attached to a table, so the constraint never took effect and the table's real PK was only 3 of its intended 5 columns. Fixed by declaring primary_key=True per column, matching every other composite-PK table.

  • Hardened merge_table_args() to filter on is None rather than truthiness, so a Constraint/Index built from unresolved column-name strings can no longer be silently dropped from __table_args__: this is what let the fact_relationship bug above go unnoticed.

  • Breaking: removed the index: bool parameter from required_concept_fk(), optional_concept_fk(), optional_fk(): it produced schema-unsafe implicit index names and had no remaining call sites; index FK columns via an explicit omop_index(...) in __table_args__ instead.

  • Fixes cost.py does not use omop_index() resulting in wrong index naming convention with schemas #23

  • Fixes Pin oa-configurator to an exact version due to expected breaking changes #24

@nicoloesch nicoloesch marked this pull request as ready for review June 22, 2026 00:01
@nicoloesch nicoloesch requested a review from gkennos June 22, 2026 00:01
@nicoloesch nicoloesch changed the title fix: Slow lower search functionality fix: Slow lower search functionality and cost.py fix Jun 23, 2026
Comment thread omop_alchemy/maintenance/cli_indexes.py
Comment thread omop_alchemy/maintenance/cli_indexes.py
Comment thread omop_alchemy/maintenance/cli_indexes.py Outdated
Comment thread omop_alchemy/maintenance/cli_indexes.py
@nicoloesch nicoloesch requested a review from gkennos June 25, 2026 01:09
@gkennos gkennos merged commit a4cc5d7 into main Jun 25, 2026
4 checks passed
@gkennos gkennos deleted the 21-indices branch June 25, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants