Databricks provider and numpy constraint for python 3.11 and 3.12 for Airflow 3.3.0 got downgraded

[n-badtke-cg]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.3.0

What happened and how to reproduce it?

The contstraints for Airflow 3.3.0 and python 3.11 (https://github.com/apache/airflow/blob/constraints-3-3/constraints-3.11.txt#L128) and 3.12 (https://github.com/apache/airflow/blob/constraints-3-3/constraints-3.12.txt#L128) are not properly managed, at least for the databricks provider.

There is listed apache-airflow-providers-databricks==6.13.0 which is a version from October 2024.

apache-airflow-providers-databricks==6.13.0 has import errors for 3.3.0

With Airflow 3.2.x python 3.12 constraints (https://github.com/apache/airflow/blob/constraints-3-2/constraints-3.12.txt#L128), there is apache-airflow-providers-databricks==7.15.0

What you think should happen instead?

I think there should be something near the latest version 7.16.1. I don't know if this intended or if there might be a deeper issue because with Airflow 3.2.x, a proper version is included.

Operating System

container

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

databricks

Versions of Apache Airflow Providers

apache-airflow-providers-databricks==6.13.0

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

1.35

Helm Chart configuration

airflowVersion: 3.3.0
images:
  airflow:
    tag: 3.3.0

Docker Image customizations

not applicable

Anything else?

none

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[n-badtke-cg]

I compared
Airflow 3.2.x python 3.12 constraints (https://github.com/apache/airflow/blob/constraints-3-2/constraints-3.12.txt) with
Airflow 3.3.x python 3.12 constraints (https://github.com/apache/airflow/blob/constraints-3-3/constraints-3.12.txt)

Downgrades I found (left is constraints-3-2, right is constraints-3-3):

• apache-airflow-providers-databricks==7.15.0 -> apache-airflow-providers-databricks==6.13.0
• databricks-sql-connector==4.2.5 -> databricks-sql-connector==2.0.2
  • databricks-sql-connector is a dependency of apache-airflow-providers-databricks
• numpy==2.4.6 -> numpy==1.26.4
• pandas==2.3.3 -> pandas==2.1.4

Apart from that, the diffs for python 3.11 and 3.12 look more weird in general: https://github.com/apache/airflow/compare/constraints-3-2..constraints-3-3

[n-badtke-cg]

Ah btw, the databricks provider and numpy constraints for python 3.10, 3.13 and 3.14 of Airflow 3.3.0 seems to be up-to-date.

It is really only about python 3.11 and 3.12 specifically

[DonHaul]

Also pandas is now pandas==2.1.4 was pandas==2.3.3

[n-badtke-cg]

Also pandas is now pandas==2.1.4 was pandas==2.3.3

presumably because of numpy?

EDIT: hmm, I am tempted to assume that the dependencies go like this:

1. apache-airflow-providers-databricks==7.15.0 -> apache-airflow-providers-databricks==6.13.0
2. databricks-sql-connector is a dependency of apache-airflow-providers-databricks: https://github.com/apache/airflow/blob/main/providers/databricks/pyproject.toml#L66
3. pandas is a dependency of apache-airflow-providers-databricks: https://github.com/apache/airflow/blob/main/providers/databricks/pyproject.toml#L69
4. numpy is a dependency of pandas: https://github.com/pandas-dev/pandas/blob/main/pyproject.toml#L11

[eladkal]

I think there should be something near the latest version 7.16.1. I don't know if this intended or if there might be a deeper issue because with Airflow 3.2.x, a proper version is included.

There is no way to do that. The versions are decided by PIP resolution algorithm. As long as versions are compatible that is acceptable.

You can always generate your own constraints file. You don't have to use the one provided by Airflow.

[n-badtke-cg]

@imkp1 wrong issue?

[imkp1]

@imkp1 wrong issue?

@n-badtke-cg Yep, valid reference for a validation case in my repo, but accidentally used GitHub's cross-reference syntax in the commit message. Fixed.