I think this is the problem hit by #18684. I am not entirely familiar with the CI script, but this is my best guess reading the CI logs. https://github.com/apache/airflow/pull/18684/checks?check_run_id=3788521722
cc @eladkal @potiuk
The problem
Currently the Installing all packages with eager upgrade job installs airflow[devel_ci] with --upgrade-strategy=eager, which installs the latest versions of all Airflow providers. However, those providers are specified by name, so the "latest version" is defined as the version on PyPI, with whatever the dependencies defined in those released versions. This is usually not a problem, but if we change any of the providers’ dependencies to a new range mutually exclusive with the released provider version, this eager-upgrade job would fail because it’s impossible to both satisfy the new version range included as a part of the devel_ci extra, and the old version range included by the provider release from PyPI.
This becomes an issue in e.g. #18684, where a version range (sshtunnel>=0.1.4,<0.2) is changed to an entirely non-overlapping one (sshtunnel>=0.3.2,<0.5).
Proposed solution
Before running the eager upgrade job, CI should first build all local providers into installable distributions (perferrably wheels), and constrain the pip install command like this:
# provider-constraints.txt
apache-airflow-providers-airbyte @ ./provider-wheels/apache_airflow_providers_airbyte-...whl
# ... all the wheel paths
pip install \
--upgrade \
--upgrade-strategy eager \
'.[devel_ci]' \
'lazy-object-proxy<1.5.0' \
'pyjwt<2.0.0' \
'dill<0.3.3' \
'certifi<2021.0.0' \
-c provider-constraints.txt # This line is new.So providers are always fetched locally instead of from PyPI, thus always include the most up-to-date dependencies.
I think this is the problem hit by #18684. I am not entirely familiar with the CI script, but this is my best guess reading the CI logs. https://github.com/apache/airflow/pull/18684/checks?check_run_id=3788521722
cc @eladkal @potiuk
The problem
Currently the Installing all packages with eager upgrade job installs
airflow[devel_ci]with--upgrade-strategy=eager, which installs the latest versions of all Airflow providers. However, those providers are specified by name, so the "latest version" is defined as the version on PyPI, with whatever the dependencies defined in those released versions. This is usually not a problem, but if we change any of the providers’ dependencies to a new range mutually exclusive with the released provider version, this eager-upgrade job would fail because it’s impossible to both satisfy the new version range included as a part of thedevel_ciextra, and the old version range included by the provider release from PyPI.This becomes an issue in e.g. #18684, where a version range (
sshtunnel>=0.1.4,<0.2) is changed to an entirely non-overlapping one (sshtunnel>=0.3.2,<0.5).Proposed solution
Before running the eager upgrade job, CI should first build all local providers into installable distributions (perferrably wheels), and constrain the
pip installcommand like this:pip install \ --upgrade \ --upgrade-strategy eager \ '.[devel_ci]' \ 'lazy-object-proxy<1.5.0' \ 'pyjwt<2.0.0' \ 'dill<0.3.3' \ 'certifi<2021.0.0' \ -c provider-constraints.txt # This line is new.So providers are always fetched locally instead of from PyPI, thus always include the most up-to-date dependencies.