Skip to content

Fix Hive metastore partition sensor initialization#69612

Merged
henry3260 merged 2 commits into
apache:mainfrom
henry3260:fix-hive-metastore-partition-sensor-init
Jul 15, 2026
Merged

Fix Hive metastore partition sensor initialization#69612
henry3260 merged 2 commits into
apache:mainfrom
henry3260:fix-hive-metastore-partition-sensor-init

Conversation

@henry3260

@henry3260 henry3260 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Why

MetastorePartitionSensor documents mysql_conn_id as the connection argument and is intended to generate the Hive metastore partition query internally. After inheriting from SqlSensor, the documented constructor path still required callers to pass the parent conn_id and sql arguments manually, so normal usage did not match the public API.

The sensor also interpolated templated table, schema, and partition values directly into the SQL string. Passing those values through SQL parameters keeps the query construction aligned with SqlSensor behavior.

What

  • Updated providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py so MetastorePartitionSensor passes mysql_conn_id to SqlSensor as conn_id and owns the metastore partition SQL.
  • Changed poke() to split schema.table input at runtime and pass table, schema, and partition_name through SqlSensor.parameters.
  • Expanded SqlSensor template fields while preserving partition_name, table, and schema templating.
  • Updated providers/apache/hive/tests/unit/apache/hive/sensors/test_metastore_partition.py to cover the documented constructor, schema/table splitting, and parameterized SQL call.
  • Updated the existing Hive metastore sensor test setup to use mysql_conn_id.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    Codex 5-5

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

…s/metastore_partition.py

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@henry3260
henry3260 force-pushed the fix-hive-metastore-partition-sensor-init branch from 7450e86 to 2682772 Compare July 14, 2026 13:52
@henry3260
henry3260 requested a review from Lee-W July 14, 2026 14:59
@henry3260
henry3260 merged commit 7fed0f0 into apache:main Jul 15, 2026
83 checks passed
@henry3260
henry3260 deleted the fix-hive-metastore-partition-sensor-init branch July 15, 2026 12:00
joshuabvarghese pushed a commit to joshuabvarghese/airflow that referenced this pull request Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants