Status of testing Providers that were prepared on March 07, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.1.0rc1

• Added AWS RDS sensors (#21231): @kazanzhy
• Added AWS RDS operators (#20907): @kazanzhy
• Add RedshiftDataHook (#19137): @john-jac
• Feature: Add invoke lambda function operator (#21686): @schirag1993
• Add JSON output on SqlToS3Operator (#21779): @mariotaddeucci
• Implement a Sagemaker DeleteModelOperator and Delete model hook. (#21673): @hsrocks
• Added Hook for Amazon RDS (#20642): @kazanzhy
• Added SNS example DAG and rst (#21475): @ferruzzi
• retry on very specific eni provision failures (#22002): @zachliu
• Configurable AWS Session Factory (#21778): @ac1997
• S3KeySensor to use S3Hook url parser (#21500): @dstandish
• ECSOperator: Get log events after sleep to get all logs (#21574): @kanga333
• Use temporary file in GCSToS3Operator (#21295): @rafalh
• Fix the Type Hints in RedshiftSQLOperator (#21885): @kaxil
• Bug Fix - S3DeleteObjectsOperator will try and delete all keys (#21458): @njrs92
• Fix Amazon SES emailer signature (#21681): @potiuk
• Fix EcsOperatorError, so it can be loaded from a picklefile (#21441): @ngwallace
• [doc] Improve s3 operator example by adding task upload_keys (#21422): @zhongjiajie
• Add docs and sample dags for AWS Batch Operator/Sensor (#22010): @o-nikolas
• Cleanup RedshiftSQLOperator documentation (#21976): @vincbeck
• Move S3ToRedshiftOperator documentation to transfer dir (#21975): @vincbeck
• Rename 'S3' hook name to 'Amazon S3' (#21988): @treyyi
• Update ECS sample DAG and Docs to new standards (#21828): @ferruzzi
• Update EKS sample DAGs and docs (#21523): @ferruzzi

Provider apache.beam: 3.2.0rc1

• Add support for BeamGoPipelineOperator (#20386): @pierrejeambrun

Provider apache.hdfs: 2.2.1rc1

• Add how-to guide for WebHDFS operators (#21393): @Melodie97

Provider apache.hive: 2.3.0rc1

• Set larger limit get_partitions_by_filter in HiveMetastoreHook (#21504): @pingzh
• Add Python 3.9 support to Hive (#21893): @potiuk
• Fix key typo in template_fields_renderers for HiveOperator (#21525): @josh-fell
• add how-to guide for hive operator (#21590): @Melodie97

Provider apache.livy: 2.2.0rc1

• Added retries to LivyHook #19384 (#21550): @utkarsharma2
• add how-to guide for livy operator (#21529): @Melodie97

Provider apache.spark: 2.1.1rc1

• fix param rendering in docs of SparkSubmitHook (#21788): @eladkal

Provider cncf.kubernetes: 3.1.0rc1

• Add map_index label to mapped KubernetesPodOperator (#21916): @ashb
• Change KubePodOperator labels from exeuction_date to run_id (#21960): @ashb
• Fix Kubernetes example with wrong operator casing (#21898): @potiuk
• Remove types from KPO docstring (#21826): @jedcunningham

Provider databricks: 2.3.0rc1

• Add-showing-runtime-error-feature-to-DatabricksSubmitRunOperator (#21709): @akifcakir
• Databricks: add support for triggering jobs by name (#21663): @eskarimov
• Added template_ext = ('.json') to databricks operators #18925 (#21530): @utkarsharma2
• Databricks SQL operators (#21363): @alexott
• Updated Databricks docs for correct jobs 2.1 API and links (#21494): @tfayyaz

Provider docker: 2.5.0rc1

• added docker network_mode options in documentation (#21986): @Bowrna

Provider elasticsearch: 3.0.0rc1

• Change default log filename template to include map_index (#21495): @ashb

Provider ftp: 2.1.0rc1

• [Feature] Updates FTPHook provider to have test_connection (#21997): @frankcash

Provider google: 6.5.0rc1

• Add Looker PDT operators (#20882): @alekseiloginov
• Add autodetect arg to external table creation in GCS-to-BQ operator (#21944): @shahkshitij15
• Dataproc assets (#21756): @wojsamjan
• Create Auto ML operators for Vertex AI service (#21470): @MaksYermak
• [Part 2] Calendar to GCS Operator (#20769): @rsg17
• Make project_id argument optional in all dataproc operators (#21866): @rafalh
• Allow templates in more DataprocUpdateClusterOperator fields (#21865): @rafalh
• Dataflow Assets (#21639): @lwyszomi
• Extract ClientInfo to module level (#21554): @pierrejeambrun
• Datafusion assets (#21518): @wojsamjan
• Dataproc metastore assets (#21267): @wojsamjan
• Normalize *_conn_id parameters in BigQuery sensors (#21430): @rafalh
• Fix bigquery_dts parameter docstring typo (#21786): @totogo
• Fixed PostgresToGCSOperator fail on empty resultset for use_server_side_cursor=True (#21307): @kazanzhy
• Fix multi query scenario in bigquery example DAG (#21575): @phanikumv
• Unpin google-cloud-memcache (#21912): @kaxil
• Unpin pandas-gbq and remove unused code (#21915): @kaxil
• Remove cursor methods calls from GCSToBigQueryOperator (#20119): @kazanzhy

Provider hashicorp: 2.1.2rc1

• Fix Vault default conn name (#20792): @ac1997

Provider http: 2.1.0rc1

• Add 'method' to attributes in HttpSensor (#21831): @h2cone

Provider microsoft.azure: 3.7.0rc1

• Add test_connection method to AzureDataFactoryHook (#21924): @josh-fell : Note: It won't work until Update test connection functionality to use custom form fields #21330 is released.
• Add pre-commit check for docstring param types (#21398): @dstandish
• Make container creation configurable when uploading files via WasbHook (#20510): @josh-fell
• (AzureCosmosDBHook) Update to latest Cosmos API (#21514): @BenGatewood

Provider mysql: 2.2.1rc1

• Make dbapi inherit get_uri from connection (#21764): @bolkedebruin
• Task names should be consistent with variable names (drop vs create) in example dags (#21434): @vinooganesh

Provider oracle: 2.2.1rc1

• Fix oracle test connection (#21699): @hubert-pietron

Provider postgres: 4.0.0rc1

• Replaces the usage of postgres:// with postgresql:// (#21205): @potiuk
• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell

Provider presto: 2.1.0rc1

• Add GCSToPrestoOperator (#21084): @rsg17
• Quick Update GCS Presto (#21855): @rsg17
• Replaced hql references to sql in TrinoHook and PrestoHook #21597 (#21630): @utkarsharma2
• Fix K8S changelog to be PyPI-compatible (#20614): @potiuk
• Fix MyPy Errors for Presto provider (#20244): @kazanzhy

Provider sftp: 2.5.0rc1

• [Feature] Updates FTPHook provider to have test_connection (#21997): @frankcash

Provider trino: 2.1.0rc1

• Add GCSToTrinoOperator (#21704): @rsg17

Provider zendesk: 3.0.1rc1

• Update Zendesk example DAG to use TaskFlow API (#21411): @josh-fell

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vinooganesh]

#21434 looks good (docs change)

[kazanzhy]

#21307 - tested before PR
#20119 - seems OK
#20244 - OK

#20642 - tested
#20907 - tested
#21231 - tested
Fix PR #22125

[treyyi]

#21988 Looks good (hook name change)

[zhongjiajie]

#21422 Looking good, and it just docs change BTW

[alexott]

Checked #21363 from the main branch, works.
#21494 looks good.

The rest for Databricks operator are also looks good from the source code perspective

[hsrocks]

#21673 Tested it. Worked fine

[h2cone]

#21831 Tested it, LGTM.

[akifcakir]

#21709 tested it , works fine.

[potiuk]

Thanks for the tests so far :) .

FYI. For those who struggle on how to test, here is a helpful guideline:

https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors

UPDATE: Actually the docs were a bit outdated. Better instructions here:

---

Installing in your local virtualenv

You have to make sure you have Airflow 2* installed in your PIP virtualenv
(the version you want to install providers with).

pip install apache-airflow-providers-<provider>==<VERSION>rc<X>

Installing with Breeze

./breeze start-airflow --use-airflow-version 2.2.4 --python 3.7 --backend postgres \
    --load-example-dags --load-default-connections

After you are in Breeze:

pip install apache-airflow-providers-<provider>==<VERSION>rc<X>

NOTE! You should Ctrl-C and restart the connections to restart airflow components and make sure new
provider packages is used.

---

I will add it to the issue template for next wave :)

[frankcash]

Confirmed #21997 via the UI utility and running a task that calls .test_connection() for both FTP and SFTP hooks

[schirag1993]

Tested #21686 - works as expected

[vincbeck]

#21976 LGTM. Only docs change

[vincbeck]

#21975 is good too. Docs change only