Dockerfile fails to build with default set of extras

[ecerulm]

Apache Airflow version: master

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
Trying to build the docker image for 2.0.1 with default extras:

git clone --depth 1 https://github.com/apache/airflow.git airflow-repo
cd airflow-repo
git log -1 # 03d3c7db931df34a724a3f254bf13197a9063576 Kamil Breguła Wed Mar 17 07:15:18 2021 +0100
docker build . \
  --tag baseairflow:2.0.1 \
  --build-arg AIRFLOW_VERSION="2.0.1" \
  --build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
  --build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
  --build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
  2>&1 | tee ../docker-baseairflow.log

fails with

#26 147.5   Attempting uninstall: google-cloud-bigquery-storage
#26 147.5     Found existing installation: google-cloud-bigquery-storage 2.3.0
#26 147.5     Uninstalling google-cloud-bigquery-storage-2.3.0:
#26 147.9       Successfully uninstalled google-cloud-bigquery-storage-2.3.0
#26 148.1 ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
#26 148.1
#26 148.1 We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
#26 148.1
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-automl<3.0.0,>=2.1.0, but you'll have google-cloud-automl 1.0.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-bigquery-datatransfer<4.0.0,>=3.0.0, but you'll have google-cloud-bigquery-datatransfer 1.1.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-datacatalog<4.0.0,>=3.0.0, but you'll have google-cloud-datacatalog 0.7.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-dataproc<3.0.0,>=2.2.0, but you'll have google-cloud-dataproc 1.1.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-kms<3.0.0,>=2.0.0, but you'll have google-cloud-kms 1.4.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-logging<3.0.0,>=2.1.1, but you'll have google-cloud-logging 1.15.1 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-monitoring<3.0.0,>=2.0.0, but you'll have google-cloud-monitoring 1.1.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-os-login<3.0.0,>=2.0.0, but you'll have google-cloud-os-login 1.0.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-pubsub<3.0.0,>=2.0.0, but you'll have google-cloud-pubsub 1.7.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-redis<3.0.0,>=2.0.0, but you'll have google-cloud-redis 1.0.0 which is incompatible.
#26 148.1 apache-airflow-providers-google 2.1.0 requires google-cloud-tasks<3.0.0,>=2.0.0, but you'll have google-cloud-tasks 1.5.0 which is incompatible.
#26 148.1 Successfully installed flask-appbuilder-3.1.1 google-cloud-automl-1.0.1 google-cloud-bigquery-datatransfer-1.1.1 google-cloud-bigquery-storage-1.1.0 google-cloud-datacatalog-0.7.0 google-cloud-dataproc-1.1.1 google-cloud-kms-1.4.0 google-cloud-logging-1.15.1 google-cloud-monitoring-1.1.0 google-cloud-os-login-1.0.0 google-cloud-pubsub-1.7.0 google-cloud-redis-1.0.0 google-cloud-tasks-1.5.0 pyarrow-1.0.1
#26 148.3 WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
#26 148.3 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#26 148.4 + pip install --user --upgrade pip==20.2.4
#26 149.3 Requirement already up-to-date: pip==20.2.4 in /usr/local/lib/python3.8/site-packages (20.2.4)
#26 155.6 WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
#26 155.6 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#26 155.7 + pip check
#26 162.0 apache-airflow-providers-google 2.1.0 has requirement google-cloud-automl<3.0.0,>=2.1.0, but you have google-cloud-automl 1.0.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-bigquery-datatransfer<4.0.0,>=3.0.0, but you have google-cloud-bigquery-datatransfer 1.1.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-datacatalog<4.0.0,>=3.0.0, but you have google-cloud-datacatalog 0.7.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-dataproc<3.0.0,>=2.2.0, but you have google-cloud-dataproc 1.1.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-kms<3.0.0,>=2.0.0, but you have google-cloud-kms 1.4.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-logging<3.0.0,>=2.1.1, but you have google-cloud-logging 1.15.1.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-monitoring<3.0.0,>=2.0.0, but you have google-cloud-monitoring 1.1.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-os-login<3.0.0,>=2.0.0, but you have google-cloud-os-login 1.0.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-pubsub<3.0.0,>=2.0.0, but you have google-cloud-pubsub 1.7.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-redis<3.0.0,>=2.0.0, but you have google-cloud-redis 1.0.0.
#26 162.1 apache-airflow-providers-google 2.1.0 has requirement google-cloud-tasks<3.0.0,>=2.0.0, but you have google-cloud-tasks 1.5.0.
#26 162.1 + false
#26 ERROR: executor failed running [/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi;     if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then         bash /scripts/docker/install_from_docker_context_files.sh;     fi;     if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi;     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ;     find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true]: exit code: 1
------
 > [airflow-build-image 15/18] RUN if [[ true == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi;     if [[  == "true" ]]; then         bash /scripts/docker/install_from_docker_context_files.sh;     fi;     if [[ -n "" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi;     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ;     find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true:
------
executor failed running [/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi;     if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then         bash /scripts/docker/install_from_docker_context_files.sh;     fi;     if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi;     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || tr

What you expected to happen:

I expected to build the image,

How to reproduce it:

git clone --depth 1 https://github.com/apache/airflow.git airflow-repo
cd airflow-repo
git log -1 # 03d3c7db931df34a724a3f254bf13197a9063576 Kamil Breguła Wed Mar 17 07:15:18 2021 +0100
docker build . \
  --tag baseairflow:2.0.1 \
  --build-arg AIRFLOW_VERSION="2.0.1" \
  --build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
  --build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
  --build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
  2>&1 | tee ../docker-baseairflow.log

Anything else we need to know:

If instead of using the default extras "async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv" I narrow it down to 'async,cncf.kubernetes,microsoft.mssql' then it will build successfully

docker build . \
  --tag baseairflow:2.0.1 \
  --build-arg AIRFLOW_VERSION="2.0.1" \
  --build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
  --build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
  --build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
  --build-arg AIRFLOW_EXTRAS='async,cncf.kubernetes,microsoft.mssql' \
  2>&1 | tee ../docker-baseairflow.log

So I'm guessing that some of the extras is causing the issue

[ecerulm]

I think the google extra is the one causing the issue. I took the default AIRFLOW_EXTRAS ("async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv") and removed google from it :

--build-arg AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"

and this builds

[ecerulm]

After conversation with @potiuk, the problem is that it will take the constraints from master, you need to make sure that it takes the constraints for 2.0.1 (with AIRFLOW_CONSTRAINTS_REFERENCE ) so that the right versions of the provider packages are installed:

docker build . \
  --tag baseairflow:2.0.1 \
  --build-arg AIRFLOW_VERSION="2.0.1" \
  --build-arg AIRFLOW_INSTALL_VERSION="==2.0.1" \
  --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.1"
  --build-arg PYTHON_BASE_IMAGE='python:3.8-buster' \
  --build-arg PYTHON_MAJOR_MINOR_VERSION='3.8' \
  2>&1 | tee ../docker-baseairflow.log