Status of testing Providers that were prepared on March 26, 2025

[eladkal]

Body

I have a kind request for all the contributors to the latest provider distributions release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comments, whether the issue is addressed.

These are providers that require testing as there were some substantial changes introduced:

Provider amazon: 9.5.0rc2

• Add a backup implementation in AWS MwaaHook for calling the MWAA API (#47035): @ramitkataria
• Add AWS SageMaker Unified Studio Workflow Operator (#45726): @agupta01
• Add error statuses check in RdsExportTaskExistenceSensor (#46917): @vargacypher
• Draft Common Message Queue (#46694): @vikramkoka
• add startTime to paginator.paginate when fetching logs in GlueJobHook (#46950): @rawwar
• Add MwaaDagRunSensor to Amazon Provider Package (#46945): @ramitkataria
• Add wait/defer support - MwaaTriggerDagRunOperator (#47528): @ramitkataria
• Add deferrable support for MwaaDagRunSensor (#47527): @ramitkataria
• Fix aws trigger tests, use get_async_conn for mock object (#47515): @gopidesupavan
• fix: don't use blocking property access for async purposes (#47326): @m1racoli
• Fix and simplify get_permitted_dag_ids in auth manager (#47458): @vincbeck
• Log state for EMR Containers sensor on failure (#47125): @o-nikolas
• Don't expect default conns in s3_to_redshift (#48363): @o-nikolas
  Linked issues:
  • Linked Issue #47968: @o-nikolas
• Don't expect default connections to be present (#47968): @o-nikolas
• fix PosixPath not working with file create_asset (#47880): @mobuchowski
• Add some typing and require kwargs for auth manager (#47455): @jedcunningham
• Base AWS classes - S3 (#47321): @o-nikolas
  Linked issues:
  • Linked Issue #35278: @Taragolis
• Remove unused methods from auth managers (#47316): @vincbeck
• Updating EC2 Operators and Sensors with AWS Base classes (#47931): @ellisms
• Fix mypy boto3 appflow version (#47912): @simonprydden
  Linked issues:
  • Linked Issue #47905: @eladkal

Provider apache.hive: 9.0.4rc1

• Setting Airflow context Environment variables for operators (#47644): @prabhusneha

Provider apache.kafka: 1.8.0rc1

• Create operators for working with Consumer Groups for GCP Apache Kafka (#47056): @MaksYermak
• Remove google provider dependencies from apache-kafka provider (#47563): @MaksYermak
• Remove exclusions of confluent-kafka (#47240): @potiuk
• Limit confluent-kafka temporarily to exclude 2.8.1 (#47204): @potiuk
• Add legacy namespace packages to airflow.providers (#47064): @potiuk

Provider apache.kylin: 3.8.2rc1

• Disallowing kylinpy 2.7.0 as it contains invalid pypi metadata (#48226): @amoghrajesh

Provider apache.spark: 5.1.1rc1

• feat: inject OpenLineage configuration to SparkSubmitOperator (#47508): @mobuchowski

Provider celery: 3.10.4rc1

• Scheduler shouldn't crash when 'executor_config' is passed for executors using task SDK (#47548): @amoghrajesh
  Linked issues:
  • Linked Issue #47349: @atul-astronomer
• AIP-81: Flatten core CLI commands (#48224): @jedcunningham

Provider cncf.kubernetes: 10.4.0rc1

• KubernetesPodOperator: add base_container_name to the templated fields (#47864): @brouberol
• cncf.kubernetes: make the base container status check polling interval configurable (#47774): @brouberol
• Switch to non-interactive mode where stdin is disabled (#47568): @majorosdonat
  Linked issues:
  • Linked Issue #39267: @paramjeet01
• Retry k8s API requests in KubernetesPodTrigger (#47187): @insomnes
  Linked issues:
  • Linked Issue #46908: @dstandish
• Remove extra method for async hook getting on KubernetesPodTrigger (#47313): @insomnes

Provider common.compat: 1.6.0rc1

• feat: Add helper for any provider version check (#47909): @kacpermuda
  Linked issues:
  • Linked Issue #47897: @kacpermuda
• feat: Add helper for OpenLineage version check (#47897): @kacpermuda
• AIP-84 Add Auth for DAG Versioning (#47553): @vatsrahul1001
• AIP-84 Add Auth for backfill (#47482): @vatsrahul1001
• AIP 84 - Add auth for asset alias (#47241): @rawwar

Provider common.io: 1.5.2rc1

• fix PosixPath not working with file create_asset (#47880): @mobuchowski
• openlineage, common.io: convert non-absolute file path to prevent namespace explosion (#47818): @mobuchowski
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil

Provider common.sql: 1.24.1rc1

• Fix get_hook() got an unexpected keyword argument 'hook_params' in SQLExecuteQueryTrigger (#47728): @dabla
• Do not allow semicolon in partition_clause (#48098): @amoghrajesh
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil
• AIP-72: Implement short circuit and branch operators (#46584): @shahar1
  Linked issues:
  • Linked Issue #44912: @shahar1
  • Linked Issue #43883: @Chais
  • Linked Issue #45823: @kaxil

Provider databricks: 7.2.2rc1

• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil

Provider dbt.cloud: 4.3.1rc1

• feat: Adjust DBT OpenLineage to Airflow 3 and improve logging (#47500): @kacpermuda
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil

Provider docker: 4.3.0rc1

• Make the docker swarm service name customizable (#47957): @niklasr22

Provider google: 14.1.0rc1

• Add DNS endpoint support for GKE Hooks and Operators (#48075): @MaksYermak
• Set task/dag labels for Dataproc Batch (#46781): @ntr
• Create operators for working with Consumer Groups for GCP Apache Kafka (#47056): @MaksYermak
• FIX: BigQuery traceability labels missing in TaskGroup (#47583): @punitchauhan771
  Linked issues:
  • Linked Issue #47297: @jacgeo
• Vendor-in json-merge-patch and add --no-dev to direct uv tool install (#48210): @potiuk
• serialize http transports contained in composite transport (#47444): @mobuchowski
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil
• Update API version of Google ADS (#47772): @Crowiant
• Introduce operation helper class and refactor Google provider hooks (#45354): @olegkachur-e
• Remove auth backends from core Airflow (#47399): @vincbeck
• Mark Google Oauth2 backend as Airflow 2 only (#47622): @ashb
• Add Dataflow Apache Beam Java streaming system test (#47209): @olegkachur-e
• Add legacy namespace packages to airflow.providers (#47064): @potiuk

Provider microsoft.azure: 12.2.2rc1

• Add lower-binding for microsoft azure/winrm providers (#47990): @potiuk
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil
• Remove default_view property from DAG class and model (#47616): @ashb
  Linked issues:
  • Linked Issue #43519: @bbovenzi

Provider microsoft.winrm: 3.9.2rc1

• fix inconsistence between the get_conn and run in WinRMHook (#47229): @darkag
  Linked issues:
  • Linked Issue #46668: @darkag
• Add lower-binding for microsoft azure/winrm providers (#47990): @potiuk

Provider openai: 1.5.3rc1

• Make open ai provider compatible with openai>=1.66.0 (#47726): @simonprydden
  Linked issues:
  • Linked Issue #47642: @kaxil

Provider openlineage: 2.1.2rc1

• fix: Adjust OpenLineage DefaultExtractor for RuntimeTaskInstance in Airflow 3 (#47673): @kacpermuda
• Stop trying to reconfigure the ORM in the OpenLineage workers (#47580): @ashb
• fix: Re-add configuring orm for OpenLineage's listener on scheduler (#48049): @kacpermuda
  Linked issues:
  • Linked Issue #47580: @ashb
• feat: inject OpenLineage configuration to SparkSubmitOperator (#47508): @mobuchowski
• feat: Add dagrun's end_date and duration to OL facet (#47901): @kacpermuda
• Use TaskInstance ID as FK in TaskReschedule instead of the multiple fields (#47459): @ephraimbuddy
• serialize http transports contained in composite transport (#47444): @mobuchowski
• Implement task-level “on” callbacks in sdk (#48002): @uranusjr
  Linked issues:
  • Linked Issue #45424: @kaxil
• Calculate retry eligibility before task runs (#47996): @uranusjr
  Linked issues:
  • Linked Issue #45106: @amoghrajesh

Provider presto: 5.8.3rc1

• Setting Airflow context Environment variables for operators (#47644): @prabhusneha

Provider standard: 0.2.0rc1

• AIP-72: Implement short circuit and branch operators (#46584): @shahar1
  Linked issues:
  • Linked Issue #44912: @shahar1
  • Linked Issue #43883: @Chais
  • Linked Issue #45823: @kaxil
• Handle null logical date in TimeDeltaSensorAsync (#47652): @dstandish
• Fix deprecation warning for BranchMixIn (#47856): @kaxil
• Fix DayOfWeekSensor use_task_logical_date condition (#47825): @gopidesupavan
  Linked issues:
  • Linked Issue #47717: @atul-astronomer
• Fix python operators errors when initialising plugins in virtualenv jinja script (#48035): @gopidesupavan
  Linked issues:
  • Linked Issue #47854: @kaxil
• AIP-72: Get TriggerDagRunOperator working with Task SDK (#47882): @kaxil
• Relocate utils/weekday.py to standard provider (#47892): @eladkal
• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil
• Rewrite asset event registration (#47677): @uranusjr
  Linked issues:
  • Linked Issue #46958: @atul-astronomer
• Implement pre- and post-execute hooks in sdk (#48230): @uranusjr

Provider trino: 6.1.1rc1

• Setting Airflow context Environment variables for operators (#47644): @prabhusneha

Provider yandex: 4.0.3rc1

• AIP-72: Handle Custom XCom Backend on Task SDK (#47339): @amoghrajesh
  Linked issues:
  • Linked Issue #45481: @kaxil

All users involved in the PRs:
@brouberol @kaxil @o-nikolas @dstandish @MaksYermak @uranusjr @amoghrajesh @niklasr22 @ntr @ashb @shahar1 @rawwar @vatsrahul1001 @vargacypher @potiuk @ramitkataria @vikramkoka @m1racoli @prabhusneha @kaxil

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ephraimbuddy]

@eladkal , I was not expecting this: #47459 to be included. There's a breaking change since Airflow 2 doesn't use UUID primary key for TIs

[eladkal]

@eladkal , I was not expecting this: #47459 to be included. There's a breaking change since Airflow 2 doesn't use UUID primary key for TIs

Providers are released from main. There is no cherry pick to release branch thus any merged code gets released.
The PR made changes to openlinage provider thus the openlinage part is released in this wave.

[eladkal]

openlinage provider is excluded from this wave, please keep testing just in case there are additional bugs that needs to be fixed.

[darkag]

tested #47229, works as expected

[brouberol]

I suspect that #47864 does not work as expected. I'm investigating.

[insomnes]

#47313 looks fine in package

Will try to take a look on #47187 in package a bit later today

[niklasr22]

#47957 works as expected

[olegkachur-e]

The #45354, #47209 works as expected. Thank you!

[kacpermuda]

For #47673, there is a small fix needed #48456, but the OpenLineage will already have rc2 so hopefully that's not a problem. All my other PRs are working fine.

[amoghrajesh]

Tested my changes in addition to running some example DAGs that i work with on a regular basis. All of them work as expected.

CI fixes:
#48226

Recursive changes due to #47339 these changes are mainly changes made to the tests of the providers to adjust the changes made in the linked PR.

Regular changes
#47548
#48098
#47996

I have also ticked my changes

[gopidesupavan]

Verified changes all are looking good.
Fixes for 3.0.0 changes.
#47825
#48035

Test fixes:
#47515

[potiuk]

Checked that all my changes are in ! All good!