Skip to content

Preserve Dataform workflow invocation config#69161

Merged
shahar1 merged 1 commit into
apache:mainfrom
deepujain:fix-53843-dataform-full-refresh-config
Jul 22, 2026
Merged

Preserve Dataform workflow invocation config#69161
shahar1 merged 1 commit into
apache:mainfrom
deepujain:fix-53843-dataform-full-refresh-config

Conversation

@deepujain

@deepujain deepujain commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Title: Preserve Dataform workflow invocation config

Summary

DataformHook.create_workflow_invocation now normalizes dict-based workflow invocation payloads into a WorkflowInvocation proto before passing the request to the generated Dataform client.

This keeps the public operator input unchanged while making nested proto fields explicit. In particular, invocation_config.fully_refresh_incremental_tables_enabled is now covered by a regression test so full-refresh workflow invocation configs are not dropped at the hook boundary.

Changes

  • providers/google/src/airflow/providers/google/cloud/hooks/dataform.py -- coerce dict workflow invocation payloads into the generated WorkflowInvocation proto before calling the Dataform client.
  • providers/google/tests/unit/google/cloud/hooks/test_dataform.py -- add a regression that passes the issue's dict-shaped invocation_config and verifies fully_refresh_incremental_tables_enabled=True reaches the client request.

Evidence it works

  • The regression follows the issue's reported operator input shape: a dict workflow invocation with nested invocation_config.
  • The test asserts the generated client receives a proto request with the nested full-refresh flag preserved.

Test plan

  • uvx --from uv==0.11.21 uv run --project providers/google ruff format --check providers/google/src/airflow/providers/google/cloud/hooks/dataform.py providers/google/tests/unit/google/cloud/hooks/test_dataform.py
  • uvx --from uv==0.11.21 uv run --project providers/google ruff check providers/google/src/airflow/providers/google/cloud/hooks/dataform.py providers/google/tests/unit/google/cloud/hooks/test_dataform.py
  • AIRFLOW_HOME=/private/tmp/airflow-test-53843 uvx --from uv==0.11.21 uv run --project providers/google pytest providers/google/tests/unit/google/cloud/hooks/test_dataform.py::TestDataformHook::test_create_workflow_invocation providers/google/tests/unit/google/cloud/hooks/test_dataform.py::TestDataformHook::test_create_workflow_invocation_preserves_invocation_config_from_dict -xvs --with-db-init
  • CI passes
Was generative AI tooling used to co-author this PR?
  • Yes (OpenAI Codex)

Generated-by: OpenAI Codex following the guidelines

Fixes #53843

@deepujain
deepujain requested a review from shahar1 as a code owner June 30, 2026 04:56
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Jun 30, 2026

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deepujain please note that you (or your operator) should include AI attribution in the PR as we state in the AI guidelines.
This PR, as well as others, are blocked from being merged until this is corrected.
Please be warned that creating additional PRs except for the 10 you've already created (listed below) without addressing the above might cause in a permanent block from Apache organization on GitHub.

List of PRs:

@shahar1
shahar1 dismissed their stale review June 30, 2026 06:54

AI attribution addressed

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full Refresh arg in invocation_config does not work via DataformCreateWorkflowInvocationOperator

3 participants