Status of testing Providers that were prepared on October 30, 2021

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 2.4.0rc1

• MySQLToS3Operator add support for parquet format (#18755): @guotongfei
• Add RedshiftSQLHook, RedshiftSQLOperator (#18447): @Brooke-white
• Remove extra postgres dependency from AWS Provider (#18844): @mariotaddeucci
• Remove duplicated code on S3ToRedshiftOperator (#18671): @mariotaddeucci
• Fixing ses email backend (#18042): @ignaski
• Fixup string concatenations (#19099): @blag
• Update S3PrefixSensor to support checking multiple prefixes within a bucket (#18807): @anaynayak
• Move validation of templated input params to run after the context init (#19048): @eskarimov

Provider apache.hive: 2.0.3rc1

• fix get_connections deprecation warn in hivemetastore hook (#18854): @Aakcht

Provider cncf.kubernetes: 2.1.0rc1

• Add more type hints to PodLauncher (#18928): @jameslamb
• add more information to PodLauncher timeout error (#17953): @jameslamb

Provider docker: 2.3.0rc1

• Add support of placement in the DockerSwarmOperator (#18990): @enima2684
• Fixup string concatenations (#19099): @blag
• Remove the docker timeout workaround (#18872): @RyanSiu1995
• Move docker decorator example dag to docker provider (#18739): @ephraimbuddy

Provider elasticsearch: 2.1.0rc1

• AIP 39: Documentation (#17552): @uranusjr
• Adds example showing the ES_hook (#17944): @frankcash

Provider facebook: 2.1.0rc1

• Align the default version with Facebook business SDK (#18883): @RyanSiu1995

Provider google: 6.1.0rc1

• Add value to "namespaceId" of query (#19163): @deedmitrij
• Add pre-commit hook for common misspelling check in files (#18964): @ephraimbuddy
• Support query timeout as an argument in CassandraToGCSOperator (#18927): @xuan616
• Update BigQueryCreateExternalTableOperator doc and parameters (#18676): @tnyz
• Replacing non-attribute template_fields for BigQueryToMsSqlOperator (#19052): @josh-fell
• Upgrade the dataproc package to 3.0.0 and migrate from v1beta2 to v1 api (#18879): @lwyszomi
• Use google cloud credentials when executing beam commands in subprocesses (#18992): @SamWheating
• Replace default api_version of FacebookAdsReportToGcsOperator (#18996): @eladkal
• Dataflow Operators - use project and location from job in on_kill method. (#18699): @lwyszomi
• Fix hard-coded /tmp directory in CloudSQL Hook (#19229): @potiuk
• Fix bug in Dataflow hook when no jobs are returned (#18981): @keze
• Fix BigQueryToMsSqlOperator documentation (#18995): @potatochip
• Move validation of templated input params to run after the context init (#19048): @eskarimov
• Google provider catch invalid secret name (#18790): @raphaelauv

Provider jenkins: 2.0.3rc1

• Remove distutils usages for Python 3.10 (#19064): @uranusjr

Provider microsoft.azure: 3.3.0rc1

• update azure cosmos to latest version (#18695): @eladkal
• Added sas_token var to BlobServiceClient return. Updated tests (#19234): @ReadytoRocc
• Add pre-commit hook for common misspelling check in files (#18964): @ephraimbuddy
• Fix changelog for Azure Provider (#18736): @potiuk
• Expanding docs on client auth for AzureKeyVaultBackend (#18659): @alexbegg
• Static start_date and default arg cleanup for Microsoft providers example DAGs (#19062): @josh-fell

Provider mongo: 2.2.0rc1

• #19223 add mongo_db param to MongoSensor (#19276): @bhavaniravi

Provider pagerduty: 2.1.0rc1

• Add standard hook fields to pagerdutyHook to make hook show up in UI (#18763): @GuidoTournois
• Make a separate hook for interacting with the Pagerduty Events API (#18784): @GuidoTournois

Provider salesforce: 3.3.0rc1

• Add SalesforceApexRestOperator (#18819): @mariotaddeucci

Provider samba: 3.0.1rc1

• Handle leading slash in samba path (#18847): @fredthomsen
• Open src and dst in binary for samba copy (#18752): @fredthomsen

Provider sftp: 2.2.0rc1

• SFTP hook to prefer the SSH paramiko key over the key file path (#18988): @30blay

Provider snowflake: 2.3.0rc1

• Add test_connection method for Snowflake Hook (#19041): @msumit
• Add region to Snowflake URI. (#18650): @dimberman

Provider ssh: 2.3.0rc1

• Refactor SSHOperator so a subclass can run many commands (#10874) (#17378): @baolsen
• update sshtunnel version (#18684): @eladkal
• Correctly handle get_pty attribute if command passed as XComArg or template (#19323): @josh-fell

Provider tableau: 2.1.2rc1

• Remove distutils usages for Python 3.10 (#19064): @uranusjr

Provider trino: 2.0.2rc1

• Properly handle verify parameter in TrinoHook (#18791): @danarwix

Providers that do not need testing

Those are providers that were either doc-only or had changes that do not require testing.

• Provider airbyte: 2.1.1rc1
• Provider alibaba: 1.0.0rc1
• Provider apache.drill: 1.0.1rc1
• Provider apache.druid: 2.0.2rc1
• Provider apache.kylin: 2.0.1rc1
• Provider apache.pig: 2.0.1rc1
• Provider apache.pinot: 2.0.1rc1
• Provider apache.spark: 2.0.1rc1
• Provider apache.sqoop: 2.0.2rc1
• Provider celery: 2.1.0rc1
• Provider cloudant: 2.0.1rc1
• Provider datadog: 2.0.1rc1
• Provider dingding: 2.0.1rc1
• Provider discord: 2.0.1rc1
• Provider ftp: 2.0.1rc1
• Provider grpc: 2.0.1rc1
• Provider http: 2.0.1rc1
• Provider imap: 2.0.1rc1
• Provider influxdb: 1.0.0rc1
• Provider jira: 2.0.1rc1
• Provider microsoft.mssql: 2.0.1rc1
• Provider microsoft.winrm: 2.0.1rc1
• Provider mysql: 2.1.1rc1
• Provider odbc: 2.0.1rc1
• Provider opsgenie: 2.0.1rc1
• Provider oracle: 2.0.1rc1
• Provider plexus: 2.0.1rc1
• Provider presto: 2.0.1rc1
• Provider qubole: 2.0.1rc1
• Provider redis: 2.0.1rc1
• Provider segment: 2.0.1rc1
• Provider sendgrid: 2.0.1rc1
• Provider singularity: 2.0.1rc1
• Provider telegram: 2.0.1rc1
• Provider vertica: 2.0.1rc1
• Provider zendesk: 2.0.1rc1

Thanks to all who contributed to those provider's release

@sreenath-kamath @jarfgit @frankcash @deedmitrij @enima2684 @uranusjr @peter-volkov @mariotaddeucci @potatochip @malthe @msumit @dimberman @jameslamb @Brooke-white @GuidoTournois @minu7 @ashb @shadrus @ignaski @eladkal @ephraimbuddy @eskarimov @JavierLopezT @keze @josh-fell @raphaelauv @blag @Aakcht @guotongfei @SamWheating @danarwix
@subkanthi @alexbegg @bhavaniravi @tnyz @Goodkat @fredthomsen @SayTen @kaxil @lwyszomi @ReadytoRocc @baolsen @30blay @nathadfield @xuan616 @RyanSiu1995

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

for cncf.kubernetes: 2.1.0rc1 it's all good

add more information to PodLauncher timeout error (#17953) -> is working

  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 374, in execute
   raise AirflowException(f'Pod Launching failed: {ex}')
airflow.exceptions.AirflowException: Pod Launching failed: Pod took longer than 120 seconds to start. Check the pod events in kubernetes to determine why.
[2021-10-30, 14:21:41 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2021-10-30, 14:21:41 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

Add more type hints to PodLauncher (#18928) > just typing , so OK

[pavelhlushchanka]

I think #18733 was released in 2.3.0

[raphaelauv]

for google: 6.1.0rc1

Google provider catch invalid secret name (#18790) -> is working

[2021-10-30 14:56:31,254] {logging_mixin.py:104} INFO - Running <TaskInstance: a_nice_dag.also_run_this 2021-10-20T00:00:00+00:00 [running]> on host 0dbece050201
[2021-10-30 14:56:31,797] {secret_manager_client.py:100} ERROR - Google Cloud API Call Error (InvalidArgument): Invalid secret ID XXXXXXXXXXXXXX-variable-toto.tata.
                Only ASCII alphabets (a-Z), numbers (0-9), dashes (-), and underscores (_)
                are allowed in the secret ID.
                
[2021-10-30 14:56:31,858] {taskinstance.py:1300} INFO - Exporting the following env vars:
...

the error about secrets with a non valid char is now catch

[potiuk]

I think #18733 was released in 2.3.0

Good spot @codenamestif ! It turned out that after last month's release of 2.3.0 as rc2 I set the "2.3.0" tag wrongly to rc1 - so the changes added between rc1 and rc2 were duplicated in the issue and changelog (and I missed that when I prepared it today). Those are the duplicated issues:

• Add AWS Fargate profile support (#18645)
• Add emr cluster link (#18691)
• AwsGlueJobOperator: add wait_for_completion to Glue job run (#18814)
• AwsGlueJobOperator: add run_job_kwargs to Glue job run (#16796)
• Add additional dependency for postgres extra for amazon provider (#18737)
• Adds an s3 list prefixes operator (#17145)
• ECSOperator: airflow exception on edge case when cloudwatch log stream is not found (#18733)
• Amazon Athena Example (#18785)
• Amazon SQS Example (#18760)

I already corrected the tag, the issue and I will also correct the Changelog (and in docs those entries will be missing), but unless there are other, more serious changes that make it rc1, it will remain in the package README (and tecchnically those changes ARE in 2.4.0 as well as 2.3.0), In the future packages changelog will corrected.

Sorry everyone involved for spamming!

[Aakcht]

Tested #18854 - all good.

#18331 is already present in 2.1.1 ( it was tested in #18638) , so I don't think hdfs 2.1.1rc1 should be present here.

[JavierLopezT]

#17397 and #18764 were already tested in the last wave

…

On Sat, 30 Oct 2021 at 20:39, Aakcht ***@***.***> wrote: Tested #18854 <#18854> - all good. #18331 <#18331> is already present in 2.1.1 ( it was tested in #18638 <#18638>) , so I don't think hdfs 2.1.1rc1 should be present here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19328 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWQK7CHD5D4NJINYNXVWSLUJQ3WBANCNFSM5HBDLQLQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[mariotaddeucci]

#18027 is already present in 2.3.0, tested in #18638
#18671 and #18819 are all good
#18844 got errors using RedshiftSQLHook when executing multiple commands into a prepared statement. I'll fix it.

[enima2684]

#18990 tested and working

[enima2684]

#18872 I tested this PR and it is not working as expected.
The DockerSwarmOperator never returns and keep hanging.
I used for the test the following Job :

    task = DockerSwarmOperator(
        task_id="task",
        image="python:3.9",
        enable_logging=True,
        tty=True,
        command=["echo", "hello world !"]
    )

[potiuk]

#18331 is already present in 2.1.1 ( it was tested in #18638) , so I don't think hdfs 2.1.1rc1 should be present here.

Right - the issue generation script has a bug apparently in this case - hdfs is not being released. I Will fix it. Removed it from the issue.

[potiuk]

#17397 and #18764 were already tested in the last wave

Correct - already removed (that was amazon's later release and wrong tag set - already corrected :)

[potiuk]

#18844 got errors using RedshiftSQLHook when executing multiple commands into a prepared statement. I'll fix it.

How serious / how much of a regression change is it @mariotadeucci ?

[Goodkat]

#17850 was already tested within 2.0.0rc2 release
#15016 (comment)