SQLAlchemy / Pandas incompatibility

[aa-matthias]

Apache Airflow version

2.8.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After upgrading to Airflow 2.8.2 one of our DAGs started failing. It seems to come from updating Pandas to 2.2.1 while SQLAlchemy is still on 1.4.X. According to Pandas docs v2.2.1 needs at least 2.0.

https://pandas.pydata.org/docs/whatsnew/v2.2.0.html#increased-minimum-versions-for-dependencies

The issue we get is from within pandas

File "./virtualenvs/airflow/lib/python3.11/site-packages/pandas/io/sql.py", line 2670, in execute
cur = self.con.cursor()
^^^^^^^^^^^^^^^
AttributeError: 'Connection' object has no attribute 'cursor'

What you think should happen instead?

No response

How to reproduce

Upgrade to 2.8.2 using the Constraint File for Python 3.11

Operating System

Ubuntu 22.04.4 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[aa-matthias]

pandas-dev/pandas#57049

[potiuk]

We updated constraints, and re-built the images. When users will be using reference image or install airflow 2.8.2 following our recommendation (with constraints https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html ) - they should get Pandas 2.1.4 when installing Airflow 2.8.2. Thansk @aa-matthias for notifying us! 1

[aa-matthias]

@potiuk - looks like the issue reappeared in Airflow 2.8.3 ?

[potiuk]

@potiuk - looks like the issue reappeared in Airflow 2.8.3 ?

Can you please elaborate how it reappeared? What makes you think so ?

[Taragolis]

Can you please elaborate how it reappeared? What makes you think so ?

We exclude #37748 from 2.8.3 and originally I did not add this tests #37779 as 2.8.3 milestone ¯\_(ツ)_/¯.
The things happen, just need to think how we might prevent this into the future.

[Taragolis]

So for Python > 3.8 we generate constraints, build docker image with pandas greater or equal than 2.2

[potiuk]

Let me see - I can check and regenerate the images again. I think the way to do it in the future is to "remember" :) . There is no other way