+ 
+
+
+Then you can setup breakpoints and debug breeze as any other Python script or test.
+
+Similar configuration can be done for VSCode.
diff --git a/CI.rst b/CI.rst
index f1e7512fbe596..469768251df5d 100644
--- a/CI.rst
+++ b/CI.rst
@@ -356,13 +356,17 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+=================================+==========================================================+==========+==========+===========+===================+
| Build info | Prints detailed information about the build | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Build CI/PROD images | Builds images in-workflow (not in the build images one) | - | Yes | Yes (1) | Yes (4) |
+| Push early cache & images | Pushes early cache/images to GitHub Registry and test | - | Yes | - | - |
+| | speed of building breeze images from scratch | | | | |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Check that image builds quickly | Checks that image builds quickly without taking a lot of | - | Yes | - | Yes |
| | time for ``pip`` to figure out the right set of deps. | | | | |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Push early cache & images | Pushes early cache/images to GitHub Registry and test | - | Yes | - | - |
-| | speed of building breeze images from scratch | | | | |
+| Build CI images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Generate constraints | Generates constraints that were updated in this build | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Build PROD images | Builds images in-workflow (not in the ``build images``) | - | Yes | Yes (1) | Yes (4) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Run breeze tests | Run unit tests for Breeze | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
@@ -370,61 +374,67 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| React WWW tests | React UI tests for new Airflow UI | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Test image building | Tests if PROD image build examples work | Yes | Yes | Yes | Yes |
+| Test examples image building | Tests if PROD image build examples work | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Test git clone on Windows | Tests if Git clone for for Windows | Yes (5) | - | - | Yes (5) |
+| Test git clone on Windows | Tests if Git clone for for Windows | Yes (5) | Yes (5) | Yes (5) | Yes (5) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Waits for CI Images | Waits for and verify CI Images (2) | Yes | Yes | Yes | Yes |
+| Waits for CI Images | Waits for and verify CI Images | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Static checks | Performs full static checks | Yes (6) | Yes | Yes | Yes (7) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Basic static checks | Performs basic static checks | Yes (6) | - | - | Yes (7) |
+| Basic static checks | Performs basic static checks (no image) | Yes (6) | - | - | - |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Build docs | Builds and tests publishing of the documentation | Yes | Yes | Yes | Yes |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Tests wheel provider packages | Tests if provider packages can be built and released | Yes | Yes | Yes | - |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Build docs | Builds documentation | Yes | Yes | Yes | Yes |
+| Tests Airflow compatibility | Compatibility of provider packages with older Airflow | Yes | Yes | Yes | - |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Test Pytest collection | Tests if pytest collection works | Yes | Yes | Yes | Yes |
+| Tests dist provider packages | Tests if dist provider packages can be built | - | Yes | Yes | - |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Tests | Run the Pytest unit tests (Backend/Python matrix) | Yes | Yes | Yes | Yes (8) |
+| Tests airflow release commands | Tests if airflow release command works | - | Yes | Yes | - |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Tests (Backend/Python matrix) | Run the Pytest unit DB tests (Backend/Python matrix) | Yes | Yes | Yes | Yes (8) |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| No DB tests | Run the Pytest unit Non-DB tests (with pytest-xdist) | Yes | Yes | Yes | Yes (8) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Integration tests | Runs integration tests (Postgres/Mysql) | Yes | Yes | Yes | Yes (9) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Quarantined tests | Runs quarantined tests (with flakiness and side-effects) | Yes | Yes | Yes | Yes (8) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Tests provider packages | Tests if provider packages can be built and released | Yes | Yes | Yes | - |
-+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Test airflow packages | Tests that Airflow package can be built and released | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Helm tests | Run the Helm integration tests | Yes | Yes | Yes | - |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Summarize warnings | Summarizes warnings from all other tests | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Wait for PROD Images | Waits for and verify PROD Images (2) | Yes | Yes | Yes | Yes |
-+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Tests Kubernetes | Run Kubernetes test | Yes | Yes | Yes | - |
+| Wait for PROD Images | Waits for and verify PROD Images | Yes (2) | Yes (2) | Yes (2) | Yes (2) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
| Test docker-compose | Tests if quick-start docker compose works | Yes | Yes | Yes | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Constraints | Upgrade constraints to latest ones (3) | - | Yes | Yes | Yes |
+| Tests Kubernetes | Run Kubernetes test | Yes | Yes | Yes | - |
++---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
+| Update constraints | Upgrade constraints to latest ones | Yes (3) | Yes (3) | Yes (3) | Yes (3) |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Push cache & images | Pushes cache/images to GitHub Registry (3) | - | Yes | Yes | Yes |
+| Push cache & images | Pushes cache/images to GitHub Registry (3) | - | Yes (3) | - | Yes |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
-| Build CI ARM images | Builds CI images for ARM to detect any problems which | Yes (10) | - | Yes | Yes |
+| Build CI ARM images | Builds CI images for ARM to detect any problems which | Yes (10) | - | Yes | - |
| | would only appear if we install all dependencies on ARM | | | | |
+---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+
``(1)`` Scheduled jobs builds images from scratch - to test if everything works properly for clean builds
-``(2)`` The jobs wait for CI images to be available.
+``(2)`` The jobs wait for CI images to be available. It only actually runs when build image is needed (in
+ case of simpler PRs that do not change dependencies or source code, images are not build)
``(3)`` PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are
upgraded only if all tests are successful. The images are rebuilt in this step using constraints pushed
-in the previous step.
+in the previous step. Constraints are only actually pushed in the ``canary`` runs.
``(4)`` In main, PROD image uses locally build providers using "latest" version of the provider code. In the
non-main version of the build, the latest released providers from PyPI are used.
-``(5)`` Only runs those tests for the builds where public runners are used (so either when non-committer
-runs it or when ``use public runner`` label is assigned to the PR.
+``(5)`` Always run with public runners to test if Git clone works on Windows.
``(6)`` Run full set of static checks when selective-checks determine that they are needed (basically, when
Python code has been modified).
@@ -438,7 +448,7 @@ Python code has been modified).
``(9)`` On non-main builds the integration tests for providers are skipped via ``skip-provider-tests`` selective
check output.
-``(10)`` Only run the builds in case dependencies are changed (``upgrade-to-newer-dependencies`` is set).
+``(10)`` Only run the builds in case PR is run by a committer from "apache" repository and in scheduled build.
CodeQL scan
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index a47c62cc70b7e..357be6623d446 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -189,7 +189,7 @@ sequenceDiagram
par
opt
GitHub Registry ->> Tests: Pull CI Images
+ [COMMIT_SHA] - Note over Tests: Unit Tests
Python/DB matrix + Note over Tests: Unit Tests
Python/DB matrix/No DB end and opt diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 15bba2fc9125e..aa5f44e5c6700 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -676,10 +676,10 @@ deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql, -mysql, neo4j, odbc, openfaas, openlineage, opsgenie, oracle, otel, pagerduty, pandas, papermill, -password, pinot, plexus, postgres, presto, rabbitmq, redis, s3, salesforce, samba, segment, -sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, -tabular, telegram, trino, vertica, virtualenv, webhdfs, winrm, zendesk +mysql, neo4j, odbc, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty, pandas, +papermill, password, pinot, plexus, postgres, presto, rabbitmq, redis, s3, salesforce, samba, +segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, +tableau, tabular, telegram, trino, vertica, virtualenv, webhdfs, winrm, zendesk .. END EXTRAS HERE Provider packages diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst index 7ccfac30d09fb..4f6936b481879 100644 --- a/CONTRIBUTORS_QUICK_START.rst +++ b/CONTRIBUTORS_QUICK_START.rst @@ -263,11 +263,26 @@ Setting up Breeze 1. Install ``pipx`` - follow the instructions in `Install pipx
diff --git a/Dockerfile b/Dockerfile
index bc2ac389d3530..898c319f25317 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -44,11 +44,11 @@ ARG AIRFLOW_UID="50000"
ARG AIRFLOW_USER_HOME_DIR=/home/airflow
# latest released version here
-ARG AIRFLOW_VERSION="2.7.0"
+ARG AIRFLOW_VERSION="2.7.2"
ARG PYTHON_BASE_IMAGE="python:3.8-slim-bullseye"
-ARG AIRFLOW_PIP_VERSION=23.2.1
+ARG AIRFLOW_PIP_VERSION=23.3.1
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
@@ -75,7 +75,7 @@ FROM scratch as scripts
COPY <<"EOF" /install_os_dependencies.sh
set -euo pipefail
-DOCKER_CLI_VERSION=20.10.9
+DOCKER_CLI_VERSION=24.0.6
if [[ "$#" != 1 ]]; then
echo "ERROR! There should be 'runtime' or 'dev' parameter passed as argument.".
@@ -178,53 +178,68 @@ COPY <<"EOF" /install_mysql.sh
set -euo pipefail
declare -a packages
-MYSQL_VERSION="8.0"
-readonly MYSQL_VERSION
-MARIADB_VERSION="10.5"
-readonly MARIADB_VERSION
+MYSQL_LTS_VERSION="8.0"
+MARIADB_LTS_VERSION="10.11"
+readonly MYSQL_LTS_VERSION
+readonly MARIADB_LTS_VERSION
COLOR_BLUE=$'\e[34m'
readonly COLOR_BLUE
COLOR_YELLOW=$'\e[1;33m'
readonly COLOR_YELLOW
+COLOR_RED=$'\e[1;31m'
+readonly COLOR_RED
COLOR_RESET=$'\e[0m'
readonly COLOR_RESET
: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+: "${INSTALL_MYSQL_CLIENT_TYPE:-mysql}"
+
+export_key() {
+ local key="${1}"
+ local name="${2:-mysql}"
+
+ echo "${COLOR_BLUE}Verify and export GPG public key ${key}${COLOR_RESET}"
+ GNUPGHOME="$(mktemp -d)"
+ export GNUPGHOME
+ set +e
+ for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
+ keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
+ do
+ gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
+ done
+ set -e
+ gpg --export "${key}" > "/etc/apt/trusted.gpg.d/${name}.gpg"
+ gpgconf --kill all
+ rm -rf "${GNUPGHOME}"
+ unset GNUPGHOME
+}
install_mysql_client() {
if [[ "${1}" == "dev" ]]; then
packages=("libmysqlclient-dev" "mysql-client")
elif [[ "${1}" == "prod" ]]; then
+ # `libmysqlclientXX` where XX is number, and it should be increased every new GA MySQL release, for example
+ # 18 - MySQL 5.6.48
+ # 20 - MySQL 5.7.42
+ # 21 - MySQL 8.0.34
+ # 22 - MySQL 8.1
packages=("libmysqlclient21" "mysql-client")
else
echo
- echo "Specify either prod or dev"
+ echo "${COLOR_RED}Specify either prod or dev${COLOR_RESET}"
echo
exit 1
fi
+ export_key "467B942D3A79BD29" "mysql"
+
echo
- echo "${COLOR_BLUE}Installing mysql client version ${MYSQL_VERSION}: ${1}${COLOR_RESET}"
+ echo "${COLOR_BLUE}Installing Oracle MySQL client version ${MYSQL_LTS_VERSION}: ${1}${COLOR_RESET}"
echo
- local key="467B942D3A79BD29"
- readonly key
-
- GNUPGHOME="$(mktemp -d)"
- export GNUPGHOME
- set +e
- for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
- keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
- do
- gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
- done
- set -e
- gpg --export "${key}" > /etc/apt/trusted.gpg.d/mysql.gpg
- gpgconf --kill all
- rm -rf "${GNUPGHOME}"
- unset GNUPGHOME
- echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_VERSION}" > /etc/apt/sources.list.d/mysql.list
+ echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_LTS_VERSION}" > \
+ /etc/apt/sources.list.d/mysql.list
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
apt-get autoremove -yqq --purge
@@ -232,21 +247,38 @@ install_mysql_client() {
}
install_mariadb_client() {
+ # List of compatible package Oracle MySQL -> MariaDB:
+ # `mysql-client` -> `mariadb-client` or `mariadb-client-compat` (11+)
+ # `libmysqlclientXX` (where XX is a number) -> `libmariadb3-compat`
+ # `libmysqlclient-dev` -> `libmariadb-dev-compat`
+ #
+ # Different naming against Debian repo which we used before
+ # that some of packages might contains `-compat` suffix, Debian repo -> MariaDB repo:
+ # `libmariadb-dev` -> `libmariadb-dev-compat`
+ # `mariadb-client-core` -> `mariadb-client` or `mariadb-client-compat` (11+)
if [[ "${1}" == "dev" ]]; then
- packages=("libmariadb-dev" "mariadb-client-core-${MARIADB_VERSION}")
+ packages=("libmariadb-dev-compat" "mariadb-client")
elif [[ "${1}" == "prod" ]]; then
- packages=("mariadb-client-core-${MARIADB_VERSION}")
+ packages=("libmariadb3-compat" "mariadb-client")
else
echo
- echo "Specify either prod or dev"
+ echo "${COLOR_RED}Specify either prod or dev${COLOR_RESET}"
echo
exit 1
fi
+ export_key "0xF1656F24C74CD1D8" "mariadb"
+
echo
- echo "${COLOR_BLUE}Installing MariaDB client version ${MARIADB_VERSION}: ${1}${COLOR_RESET}"
- echo "${COLOR_YELLOW}MariaDB client binary compatible with MySQL client.${COLOR_RESET}"
+ echo "${COLOR_BLUE}Installing MariaDB client version ${MARIADB_LTS_VERSION}: ${1}${COLOR_RESET}"
+ echo "${COLOR_YELLOW}MariaDB client protocol-compatible with MySQL client.${COLOR_RESET}"
echo
+
+ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
+ echo "deb [arch=amd64,arm64] https://archive.mariadb.org/mariadb-${MARIADB_LTS_VERSION}/repo/debian/ $(lsb_release -cs) main" > \
+ /etc/apt/sources.list.d/mariadb.list
+ # Make sure that dependencies from MariaDB repo are preferred over Debian dependencies
+ printf "Package: *\nPin: release o=MariaDB\nPin-Priority: 999\n" > /etc/apt/preferences.d/mariadb
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
apt-get autoremove -yqq --purge
@@ -255,9 +287,18 @@ install_mariadb_client() {
if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ INSTALL_MYSQL_CLIENT_TYPE="mariadb"
+ fi
+
+ if [[ "${INSTALL_MYSQL_CLIENT_TYPE}" == "mysql" ]]; then
+ install_mysql_client "${@}"
+ elif [[ "${INSTALL_MYSQL_CLIENT_TYPE}" == "mariadb" ]]; then
install_mariadb_client "${@}"
else
- install_mysql_client "${@}"
+ echo
+ echo "${COLOR_RED}Specify either mysql or mariadb, got ${INSTALL_MYSQL_CLIENT_TYPE}${COLOR_RESET}"
+ echo
+ exit 1
fi
fi
EOF
@@ -471,7 +512,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.2.1"
+ export AIRFLOW_PIP_VERSION="23.3.1"
fi
fi
}
@@ -505,9 +546,9 @@ function common::install_pip_version() {
echo "${COLOR_BLUE}Installing pip version ${AIRFLOW_PIP_VERSION}${COLOR_RESET}"
echo
if [[ ${AIRFLOW_PIP_VERSION} =~ .*https.* ]]; then
- pip install --disable-pip-version-check --no-cache-dir "pip @ ${AIRFLOW_PIP_VERSION}"
+ pip install --disable-pip-version-check "pip @ ${AIRFLOW_PIP_VERSION}"
else
- pip install --disable-pip-version-check --no-cache-dir "pip==${AIRFLOW_PIP_VERSION}"
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}"
fi
mkdir -p "${HOME}/.local/bin"
}
@@ -1073,7 +1114,7 @@ if [[ -n "${_PIP_ADDITIONAL_REQUIREMENTS=}" ]] ; then
>&2 echo " the container starts, so it is only useful for testing and trying out"
>&2 echo " of adding dependencies."
>&2 echo
- pip install --root-user-action ignore --no-cache-dir ${_PIP_ADDITIONAL_REQUIREMENTS}
+ pip install --root-user-action ignore ${_PIP_ADDITIONAL_REQUIREMENTS}
fi
@@ -1149,7 +1190,8 @@ SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "n
ARG PYTHON_BASE_IMAGE
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
- LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
+ LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 \
+ PIP_CACHE_DIR=/tmp/.cache/pip
ARG DEV_APT_DEPS=""
ARG ADDITIONAL_DEV_APT_DEPS=""
@@ -1167,6 +1209,7 @@ COPY --from=scripts install_os_dependencies.sh /scripts/docker/
RUN bash /scripts/docker/install_os_dependencies.sh dev
ARG INSTALL_MYSQL_CLIENT="true"
+ARG INSTALL_MYSQL_CLIENT_TYPE="mysql"
ARG INSTALL_MSSQL_CLIENT="true"
ARG INSTALL_POSTGRES_CLIENT="true"
ARG AIRFLOW_REPO=apache/airflow
@@ -1217,6 +1260,7 @@ ARG AIRFLOW_USER_HOME_DIR
ARG AIRFLOW_UID
ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+ INSTALL_MYSQL_CLIENT_TYPE=${INSTALL_MYSQL_CLIENT_TYPE} \
INSTALL_MSSQL_CLIENT=${INSTALL_MSSQL_CLIENT} \
INSTALL_POSTGRES_CLIENT=${INSTALL_POSTGRES_CLIENT}
@@ -1226,7 +1270,7 @@ COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scrip
# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
-ARG AIRFLOW_PIP_VERSION=23.2.1
+ARG AIRFLOW_PIP_VERSION=23.3.1
ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
COPY --from=scripts common.sh /scripts/docker/
@@ -1332,8 +1376,15 @@ WORKDIR ${AIRFLOW_HOME}
COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh \
install_additional_dependencies.sh /scripts/docker/
-# hadolint ignore=SC2086, SC2010
-RUN if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
+# Useful for creating a cache id based on the underlying architecture, preventing the use of cached python packages from
+# an incorrect architecture.
+ARG TARGETARCH
+# Value to be able to easily change cache id and therefore use a bare new cache
+ARG PIP_CACHE_EPOCH="0"
+
+# hadolint ignore=SC2086, SC2010, DL3042
+RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
+ if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
bash /scripts/docker/install_from_docker_context_files.sh; \
fi; \
if ! airflow version 2>/dev/null >/dev/null; then \
@@ -1351,8 +1402,10 @@ RUN if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
# In case there is a requirements.txt file in "docker-context-files" it will be installed
# during the build additionally to whatever has been installed so far. It is recommended that
# the requirements.txt contains only dependencies with == version specification
-RUN if [[ -f /docker-context-files/requirements.txt ]]; then \
- pip install --no-cache-dir --user -r /docker-context-files/requirements.txt; \
+# hadolint ignore=DL3042
+RUN --mount=type=cache,id=additional-requirements-$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
+ if [[ -f /docker-context-files/requirements.txt ]]; then \
+ pip install --user -r /docker-context-files/requirements.txt; \
fi
##############################################################################################
@@ -1388,6 +1441,7 @@ ARG RUNTIME_APT_COMMAND="echo"
ARG ADDITIONAL_RUNTIME_APT_COMMAND=""
ARG ADDITIONAL_RUNTIME_APT_ENV=""
ARG INSTALL_MYSQL_CLIENT="true"
+ARG INSTALL_MYSQL_CLIENT_TYPE="mysql"
ARG INSTALL_MSSQL_CLIENT="true"
ARG INSTALL_POSTGRES_CLIENT="true"
@@ -1396,6 +1450,7 @@ ENV RUNTIME_APT_DEPS=${RUNTIME_APT_DEPS} \
RUNTIME_APT_COMMAND=${RUNTIME_APT_COMMAND} \
ADDITIONAL_RUNTIME_APT_COMMAND=${ADDITIONAL_RUNTIME_APT_COMMAND} \
INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
+ INSTALL_MYSQL_CLIENT_TYPE=${INSTALL_MYSQL_CLIENT_TYPE} \
INSTALL_MSSQL_CLIENT=${INSTALL_MSSQL_CLIENT} \
INSTALL_POSTGRES_CLIENT=${INSTALL_POSTGRES_CLIENT} \
GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm" \
@@ -1420,7 +1475,7 @@ ENV PATH="${AIRFLOW_USER_HOME_DIR}/.local/bin:${PATH}" \
# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
-ARG AIRFLOW_PIP_VERSION=23.2.1
+ARG AIRFLOW_PIP_VERSION=23.3.1
ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
COPY --from=scripts common.sh /scripts/docker/
diff --git a/Dockerfile.ci b/Dockerfile.ci
index aba7e6913b3c0..0404036fd4979 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -35,7 +35,7 @@ FROM ${PYTHON_BASE_IMAGE} as scripts
COPY <<"EOF" /install_os_dependencies.sh
set -euo pipefail
-DOCKER_CLI_VERSION=20.10.9
+DOCKER_CLI_VERSION=24.0.6
if [[ "$#" != 1 ]]; then
echo "ERROR! There should be 'runtime' or 'dev' parameter passed as argument.".
@@ -138,53 +138,68 @@ COPY <<"EOF" /install_mysql.sh
set -euo pipefail
declare -a packages
-MYSQL_VERSION="8.0"
-readonly MYSQL_VERSION
-MARIADB_VERSION="10.5"
-readonly MARIADB_VERSION
+MYSQL_LTS_VERSION="8.0"
+MARIADB_LTS_VERSION="10.11"
+readonly MYSQL_LTS_VERSION
+readonly MARIADB_LTS_VERSION
COLOR_BLUE=$'\e[34m'
readonly COLOR_BLUE
COLOR_YELLOW=$'\e[1;33m'
readonly COLOR_YELLOW
+COLOR_RED=$'\e[1;31m'
+readonly COLOR_RED
COLOR_RESET=$'\e[0m'
readonly COLOR_RESET
: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+: "${INSTALL_MYSQL_CLIENT_TYPE:-mysql}"
+
+export_key() {
+ local key="${1}"
+ local name="${2:-mysql}"
+
+ echo "${COLOR_BLUE}Verify and export GPG public key ${key}${COLOR_RESET}"
+ GNUPGHOME="$(mktemp -d)"
+ export GNUPGHOME
+ set +e
+ for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
+ keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
+ do
+ gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
+ done
+ set -e
+ gpg --export "${key}" > "/etc/apt/trusted.gpg.d/${name}.gpg"
+ gpgconf --kill all
+ rm -rf "${GNUPGHOME}"
+ unset GNUPGHOME
+}
install_mysql_client() {
if [[ "${1}" == "dev" ]]; then
packages=("libmysqlclient-dev" "mysql-client")
elif [[ "${1}" == "prod" ]]; then
+ # `libmysqlclientXX` where XX is number, and it should be increased every new GA MySQL release, for example
+ # 18 - MySQL 5.6.48
+ # 20 - MySQL 5.7.42
+ # 21 - MySQL 8.0.34
+ # 22 - MySQL 8.1
packages=("libmysqlclient21" "mysql-client")
else
echo
- echo "Specify either prod or dev"
+ echo "${COLOR_RED}Specify either prod or dev${COLOR_RESET}"
echo
exit 1
fi
+ export_key "467B942D3A79BD29" "mysql"
+
echo
- echo "${COLOR_BLUE}Installing mysql client version ${MYSQL_VERSION}: ${1}${COLOR_RESET}"
+ echo "${COLOR_BLUE}Installing Oracle MySQL client version ${MYSQL_LTS_VERSION}: ${1}${COLOR_RESET}"
echo
- local key="467B942D3A79BD29"
- readonly key
-
- GNUPGHOME="$(mktemp -d)"
- export GNUPGHOME
- set +e
- for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
- keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
- do
- gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
- done
- set -e
- gpg --export "${key}" > /etc/apt/trusted.gpg.d/mysql.gpg
- gpgconf --kill all
- rm -rf "${GNUPGHOME}"
- unset GNUPGHOME
- echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_VERSION}" > /etc/apt/sources.list.d/mysql.list
+ echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_LTS_VERSION}" > \
+ /etc/apt/sources.list.d/mysql.list
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
apt-get autoremove -yqq --purge
@@ -192,21 +207,38 @@ install_mysql_client() {
}
install_mariadb_client() {
+ # List of compatible package Oracle MySQL -> MariaDB:
+ # `mysql-client` -> `mariadb-client` or `mariadb-client-compat` (11+)
+ # `libmysqlclientXX` (where XX is a number) -> `libmariadb3-compat`
+ # `libmysqlclient-dev` -> `libmariadb-dev-compat`
+ #
+ # Different naming against Debian repo which we used before
+ # that some of packages might contains `-compat` suffix, Debian repo -> MariaDB repo:
+ # `libmariadb-dev` -> `libmariadb-dev-compat`
+ # `mariadb-client-core` -> `mariadb-client` or `mariadb-client-compat` (11+)
if [[ "${1}" == "dev" ]]; then
- packages=("libmariadb-dev" "mariadb-client-core-${MARIADB_VERSION}")
+ packages=("libmariadb-dev-compat" "mariadb-client")
elif [[ "${1}" == "prod" ]]; then
- packages=("mariadb-client-core-${MARIADB_VERSION}")
+ packages=("libmariadb3-compat" "mariadb-client")
else
echo
- echo "Specify either prod or dev"
+ echo "${COLOR_RED}Specify either prod or dev${COLOR_RESET}"
echo
exit 1
fi
+ export_key "0xF1656F24C74CD1D8" "mariadb"
+
echo
- echo "${COLOR_BLUE}Installing MariaDB client version ${MARIADB_VERSION}: ${1}${COLOR_RESET}"
- echo "${COLOR_YELLOW}MariaDB client binary compatible with MySQL client.${COLOR_RESET}"
+ echo "${COLOR_BLUE}Installing MariaDB client version ${MARIADB_LTS_VERSION}: ${1}${COLOR_RESET}"
+ echo "${COLOR_YELLOW}MariaDB client protocol-compatible with MySQL client.${COLOR_RESET}"
echo
+
+ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
+ echo "deb [arch=amd64,arm64] https://archive.mariadb.org/mariadb-${MARIADB_LTS_VERSION}/repo/debian/ $(lsb_release -cs) main" > \
+ /etc/apt/sources.list.d/mariadb.list
+ # Make sure that dependencies from MariaDB repo are preferred over Debian dependencies
+ printf "Package: *\nPin: release o=MariaDB\nPin-Priority: 999\n" > /etc/apt/preferences.d/mariadb
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
apt-get autoremove -yqq --purge
@@ -215,9 +247,18 @@ install_mariadb_client() {
if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ INSTALL_MYSQL_CLIENT_TYPE="mariadb"
+ fi
+
+ if [[ "${INSTALL_MYSQL_CLIENT_TYPE}" == "mysql" ]]; then
+ install_mysql_client "${@}"
+ elif [[ "${INSTALL_MYSQL_CLIENT_TYPE}" == "mariadb" ]]; then
install_mariadb_client "${@}"
else
- install_mysql_client "${@}"
+ echo
+ echo "${COLOR_RED}Specify either mysql or mariadb, got ${INSTALL_MYSQL_CLIENT_TYPE}${COLOR_RESET}"
+ echo
+ exit 1
fi
fi
EOF
@@ -431,7 +472,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.2.1"
+ export AIRFLOW_PIP_VERSION="23.3.1"
fi
fi
}
@@ -465,9 +506,9 @@ function common::install_pip_version() {
echo "${COLOR_BLUE}Installing pip version ${AIRFLOW_PIP_VERSION}${COLOR_RESET}"
echo
if [[ ${AIRFLOW_PIP_VERSION} =~ .*https.* ]]; then
- pip install --disable-pip-version-check --no-cache-dir "pip @ ${AIRFLOW_PIP_VERSION}"
+ pip install --disable-pip-version-check "pip @ ${AIRFLOW_PIP_VERSION}"
else
- pip install --disable-pip-version-check --no-cache-dir "pip==${AIRFLOW_PIP_VERSION}"
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}"
fi
mkdir -p "${HOME}/.local/bin"
}
@@ -719,10 +760,22 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
echo " * ${COLOR_BLUE}Airflow core SQL connection:${COLOR_RESET} ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"
echo
+ if [[ ${STANDALONE_DAG_PROCESSOR=} == "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Running forcing scheduler/standalone_dag_processor to be True${COLOR_RESET}"
+ echo
+ export AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR=True
+ fi
+ if [[ ${DATABASE_ISOLATION=} == "true" ]]; then
+ echo "${COLOR_BLUE}Force database isolation configuration:${COLOR_RESET}"
+ export AIRFLOW__CORE__DATABASE_ACCESS_ISOLATION=True
+ export AIRFLOW__CORE__INTERNAL_API_URL=http://localhost:8080
+ export AIRFLOW__WEBSERVER_RUN_INTERNAL_API=True
+ fi
+
RUN_TESTS=${RUN_TESTS:="false"}
CI=${CI:="false"}
USE_AIRFLOW_VERSION="${USE_AIRFLOW_VERSION:=""}"
- TEST_TIMEOUT=${TEST_TIMEOUT:="60"}
if [[ ${USE_AIRFLOW_VERSION} == "" ]]; then
export PYTHONPATH=${AIRFLOW_SOURCES}
@@ -937,11 +990,12 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
fi
rm -f "${AIRFLOW_SOURCES}/pytest.ini"
+
if [[ ${UPGRADE_BOTO=} == "true" ]]; then
echo
echo "${COLOR_BLUE}Upgrading boto3, botocore to latest version to run Amazon tests with them${COLOR_RESET}"
echo
- pip uninstall --root-user-action ignore aiobotocore -y || true
+ pip uninstall --root-user-action ignore aiobotocore s3fs -y || true
pip install --root-user-action ignore --upgrade boto3 botocore
pip check
fi
@@ -954,91 +1008,14 @@ if [[ ${DOWNGRADE_SQLALCHEMY=} == "true" ]]; then
pip check
fi
+pip uninstall --root-user-action ignore "pytest-capture-warnings" -y >/dev/null 2>&1 || true
+
set +u
if [[ "${RUN_TESTS}" != "true" ]]; then
exec /bin/bash "${@}"
fi
set -u
-if [[ ${HELM_TEST_PACKAGE=} != "" ]]; then
- export RESULT_LOG_FILE="/files/test_result-${TEST_TYPE/\[*\]/}-${HELM_TEST_PACKAGE}-${BACKEND}.xml"
- export WARNINGS_FILE="/files/warnings-${TEST_TYPE/\[*\]/}-${HELM_TEST_PACKAGE}-${BACKEND}.txt"
-else
- export RESULT_LOG_FILE="/files/test_result-${TEST_TYPE/\[*\]/}-${BACKEND}.xml"
- export WARNINGS_FILE="/files/warnings-${TEST_TYPE/\[*\]/}-${BACKEND}.txt"
-fi
-
-EXTRA_PYTEST_ARGS=(
- "--verbosity=0"
- "--strict-markers"
- "--durations=100"
- "--maxfail=50"
- "--color=yes"
- "--junitxml=${RESULT_LOG_FILE}"
- # timeouts in seconds for individual tests
- "--timeouts-order"
- "moi"
- "--setup-timeout=${TEST_TIMEOUT}"
- "--execution-timeout=${TEST_TIMEOUT}"
- "--teardown-timeout=${TEST_TIMEOUT}"
- "--output=${WARNINGS_FILE}"
- "--disable-warnings"
- # Only display summary for non-expected cases
- #
- # f - failed
- # E - error
- # X - xpassed (passed even if expected to fail)
- # s - skipped
- #
- # The following cases are not displayed:
- # x - xfailed (expected to fail and failed)
- # p - passed
- # P - passed with output
- #
- "-rfEXs"
-)
-
-if [[ ${SUSPENDED_PROVIDERS_FOLDERS=} != "" ]]; then
- for provider in ${SUSPENDED_PROVIDERS_FOLDERS=}; do
- echo "Skipping tests for suspended provider: ${provider}"
- EXTRA_PYTEST_ARGS+=(
- "--ignore=tests/providers/${provider}"
- "--ignore=tests/system/providers/${provider}"
- "--ignore=tests/integration/providers/${provider}"
- )
- done
-fi
-
-if [[ "${TEST_TYPE}" == "Helm" ]]; then
- _cpus="$(grep -c 'cpu[0-9]' /proc/stat)"
- echo "Running tests with ${_cpus} CPUs in parallel"
- # Enable parallelism and disable coverage
- EXTRA_PYTEST_ARGS+=(
- "-n" "${_cpus}"
- "--no-cov"
- )
-else
- EXTRA_PYTEST_ARGS+=(
- "--with-db-init"
- )
-fi
-
-if [[ ${ENABLE_TEST_COVERAGE:="false"} == "true" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "--cov=airflow"
- "--cov-config=.coveragerc"
- "--cov-report=xml:/files/coverage-${TEST_TYPE/\[*\]/}-${BACKEND}.xml"
- )
-fi
-
-if [[ ${COLLECT_ONLY:="false"} == "true" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "--collect-only"
- "-qqqq"
- "--disable-warnings"
- )
-fi
-
if [[ ${REMOVE_ARM_PACKAGES:="false"} == "true" ]]; then
# Test what happens if we do not have ARM packages installed.
# This is useful to see if pytest collection works without ARM packages which is important
@@ -1046,177 +1023,16 @@ if [[ ${REMOVE_ARM_PACKAGES:="false"} == "true" ]]; then
python "${IN_CONTAINER_DIR}/remove_arm_packages.py"
fi
-declare -a SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
- ALL_TESTS ALL_PRESELECTED_TESTS ALL_OTHER_TESTS
-
-function find_all_other_tests() {
- local all_tests_dirs
- # The output of the find command should be sorted to make sure that the order is always the same
- # when we run the tests, to avoid cross-package side effects causing different test results
- # in different environments. See https://github.com/apache/airflow/pull/30588 for example.
- all_tests_dirs=$(find "tests" -type d ! -name '__pycache__' | sort)
- all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests$/d" )
- all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests\/dags/d" )
- local path
- for path in "${ALL_PRESELECTED_TESTS[@]}"
- do
- escaped_path="${path//\//\\\/}"
- all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/${escaped_path}/d" )
- done
- for path in ${all_tests_dirs}
- do
- ALL_OTHER_TESTS+=("${path}")
- done
-}
-
-if [[ ${#@} -gt 0 && -n "$1" ]]; then
- SELECTED_TESTS=("${@}")
-else
- CLI_TESTS=("tests/cli")
- API_TESTS=("tests/api_experimental" "tests/api_connexion" "tests/api_internal")
- PROVIDERS_TESTS=("tests/providers")
- ALWAYS_TESTS=("tests/always")
- CORE_TESTS=(
- "tests/core"
- "tests/executors"
- "tests/jobs"
- "tests/models"
- "tests/serialization"
- "tests/ti_deps"
- "tests/utils"
- )
- WWW_TESTS=("tests/www")
- HELM_CHART_TESTS=("helm_tests")
- INTEGRATION_TESTS=("tests/integration")
- SYSTEM_TESTS=("tests/system")
- ALL_TESTS=("tests")
- ALL_PRESELECTED_TESTS=(
- "${CLI_TESTS[@]}"
- "${API_TESTS[@]}"
- "${HELM_CHART_TESTS[@]}"
- "${INTEGRATION_TESTS[@]}"
- "${PROVIDERS_TESTS[@]}"
- "${CORE_TESTS[@]}"
- "${ALWAYS_TESTS[@]}"
- "${WWW_TESTS[@]}"
- "${SYSTEM_TESTS[@]}"
- )
-
- NO_PROVIDERS_INTEGRATION_TESTS=(
- "tests/integration/api_experimental"
- "tests/integration/cli"
- "tests/integration/executors"
- "tests/integration/security"
- )
-
- if [[ ${TEST_TYPE:=""} == "CLI" ]]; then
- SELECTED_TESTS=("${CLI_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "API" ]]; then
- SELECTED_TESTS=("${API_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "Providers" ]]; then
- SELECTED_TESTS=("${PROVIDERS_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "Core" ]]; then
- SELECTED_TESTS=("${CORE_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "Always" ]]; then
- SELECTED_TESTS=("${ALWAYS_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "WWW" ]]; then
- SELECTED_TESTS=("${WWW_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "Helm" ]]; then
- if [[ ${HELM_TEST_PACKAGE=} != "" ]]; then
- SELECTED_TESTS=("helm_tests/${HELM_TEST_PACKAGE}")
- else
- SELECTED_TESTS=("${HELM_CHART_TESTS[@]}")
- fi
- elif [[ ${TEST_TYPE:=""} == "Integration" ]]; then
- if [[ ${SKIP_PROVIDER_TESTS:=""} == "true" ]]; then
- SELECTED_TESTS=("${NO_PROVIDERS_INTEGRATION_TESTS[@]}")
- else
- SELECTED_TESTS=("${INTEGRATION_TESTS[@]}")
- fi
- elif [[ ${TEST_TYPE:=""} == "Other" ]]; then
- find_all_other_tests
- SELECTED_TESTS=("${ALL_OTHER_TESTS[@]}")
- elif [[ ${TEST_TYPE:=""} == "All" || ${TEST_TYPE} == "Quarantined" || \
- ${TEST_TYPE} == "Always" || \
- ${TEST_TYPE} == "Postgres" || ${TEST_TYPE} == "MySQL" || \
- ${TEST_TYPE} == "Long" ]]; then
- SELECTED_TESTS=("${ALL_TESTS[@]}")
- elif [[ ${TEST_TYPE} =~ Providers\[\-(.*)\] ]]; then
- # When providers start with `-` it means that we should run all provider tests except those
- SELECTED_TESTS=("${PROVIDERS_TESTS[@]}")
- for provider in ${BASH_REMATCH[1]//,/ }
- do
- providers_dir="tests/providers/${provider//./\/}"
- if [[ -d ${providers_dir} ]]; then
- echo "${COLOR_BLUE}Ignoring ${providers_dir} as it has been deselected.${COLOR_RESET}"
- EXTRA_PYTEST_ARGS+=("--ignore=tests/providers/${provider//./\/}")
- else
- echo "${COLOR_YELLOW}Skipping ${providers_dir} as the directory does not exist.${COLOR_RESET}"
- fi
- done
- elif [[ ${TEST_TYPE} =~ Providers\[(.*)\] ]]; then
- SELECTED_TESTS=()
- for provider in ${BASH_REMATCH[1]//,/ }
- do
- providers_dir="tests/providers/${provider//./\/}"
- if [[ -d ${providers_dir} ]]; then
- SELECTED_TESTS+=("${providers_dir}")
- else
- echo "${COLOR_YELLOW}Skip ${providers_dir} as the directory does not exist.${COLOR_RESET}"
- fi
- done
- elif [[ ${TEST_TYPE} =~ PlainAsserts ]]; then
- # Those tests fail when --asert=rewrite is set, therefore we run them separately
- # with --assert=plain to make sure they pass.
- SELECTED_TESTS=(
- # this on is mysteriously failing dill serialization. It could be removed once
- # https://github.com/pytest-dev/pytest/issues/10845 is fixed
- "tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context"
- )
- EXTRA_PYTEST_ARGS+=("--assert=plain")
- export PYTEST_PLAIN_ASSERTS="true"
- else
- echo
- echo "${COLOR_RED}ERROR: Wrong test type ${TEST_TYPE} ${COLOR_RESET}"
- echo
- exit 1
- fi
+if [[ ${TEST_TYPE} == "PlainAsserts" ]]; then
+ # Plain asserts should be converted to env variable to make sure they are taken into account
+ # otherwise they will not be effective during test collection when plain assert is breaking collection
+ export PYTEST_PLAIN_ASSERTS="true"
fi
-readonly SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
- ALL_TESTS ALL_PRESELECTED_TESTS
-
-if [[ ${TEST_TYPE:=""} == "Long" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "-m" "long_running"
- "--include-long-running"
- )
-elif [[ ${TEST_TYPE:=""} == "Postgres" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "--backend"
- "postgres"
- )
-elif [[ ${TEST_TYPE:=""} == "MySQL" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "--backend"
- "mysql"
- )
-elif [[ ${TEST_TYPE:=""} == "Quarantined" ]]; then
- EXTRA_PYTEST_ARGS+=(
- "-m" "quarantined"
- "--include-quarantined"
- )
-fi
-
-echo
-echo "Running tests ${SELECTED_TESTS[*]}"
-echo
-
-ARGS=("${EXTRA_PYTEST_ARGS[@]}" "${SELECTED_TESTS[@]}")
if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then
- "${IN_CONTAINER_DIR}/run_system_tests.sh" "${ARGS[@]}"
+ "${IN_CONTAINER_DIR}/run_system_tests.sh" "${@}"
else
- "${IN_CONTAINER_DIR}/run_ci_tests.sh" "${ARGS[@]}"
+ "${IN_CONTAINER_DIR}/run_ci_tests.sh" "${@}"
fi
EOF
@@ -1273,7 +1089,7 @@ RUN bash /scripts/docker/install_os_dependencies.sh dev
# THE 3 LINES ARE ONLY NEEDED IN ORDER TO MAKE PYMSSQL BUILD WORK WITH LATEST CYTHON
# AND SHOULD BE REMOVED WHEN WORKAROUND IN install_mssql.sh IS REMOVED
-ARG AIRFLOW_PIP_VERSION=23.2.1
+ARG AIRFLOW_PIP_VERSION=23.3.1
ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
COPY --from=scripts common.sh /scripts/docker/
@@ -1284,10 +1100,12 @@ COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scrip
ARG HOME=/root
ARG AIRFLOW_HOME=/root/airflow
ARG AIRFLOW_SOURCES=/opt/airflow
+ARG INSTALL_MYSQL_CLIENT_TYPE="mysql"
ENV HOME=${HOME} \
AIRFLOW_HOME=${AIRFLOW_HOME} \
- AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+ AIRFLOW_SOURCES=${AIRFLOW_SOURCES} \
+ INSTALL_MYSQL_CLIENT_TYPE=${INSTALL_MYSQL_CLIENT_TYPE}
# We run scripts with bash here to make sure we can execute the scripts. Changing to +x might have an
# unexpected result - the cache for Dockerfiles might get invalidated in case the host system
@@ -1296,7 +1114,13 @@ ENV HOME=${HOME} \
RUN bash /scripts/docker/install_mysql.sh prod \
&& bash /scripts/docker/install_mysql.sh dev \
&& bash /scripts/docker/install_mssql.sh dev \
- && bash /scripts/docker/install_postgres.sh dev
+ && bash /scripts/docker/install_postgres.sh dev \
+ # The user is added to allow ssh debugging (you can connect with airflow/airflow over ssh)
+ && adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
+ --quiet "airflow" --home "/home/airflow" \
+ && echo -e "airflow\nairflow" | passwd airflow 2>&1 \
+ && echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow \
+ && chmod 0440 /etc/sudoers.d/airflow
# Install Helm
ARG HELM_VERSION="v3.9.4"
@@ -1326,11 +1150,11 @@ ARG AIRFLOW_CONSTRAINTS_LOCATION=""
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# By changing the epoch we can force reinstalling Airflow and pip all dependencies
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
-ARG AIRFLOW_CI_BUILD_EPOCH="5"
+ARG AIRFLOW_CI_BUILD_EPOCH="6"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG AIRFLOW_PIP_VERSION=23.2.1
+ARG AIRFLOW_PIP_VERSION=23.3.1
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
@@ -1369,6 +1193,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
# * install airflow in editable mode
# * install always current version of airflow
INSTALL_MYSQL_CLIENT="true" \
+ INSTALL_MYSQL_CLIENT_TYPE=${INSTALL_MYSQL_CLIENT_TYPE} \
INSTALL_MSSQL_CLIENT="true" \
INSTALL_POSTGRES_CLIENT="true" \
AIRFLOW_INSTALLATION_METHOD="." \
@@ -1385,8 +1210,11 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# Those are additional constraints that are needed for some extras but we do not want to
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
-# aiobotocore is limited temporarily until it stops backtracking pip
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=""
+#
+# Without grpcio-status limit, pip gets into very long backtracking
+# We should attempt to remove it in the future
+#
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="grpcio-status>=1.55.0 aiobotocore>=2.7.0"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ARG VERSION_SUFFIX_FOR_PYPI=""
diff --git a/IMAGES.rst b/IMAGES.rst
index 9c34fce80d6ea..ef2d568daa603 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -462,7 +462,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.2.1`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.3.1`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/INSTALL b/INSTALL
index bd1e0006b85b3..d2bc970313a74 100644
--- a/INSTALL
+++ b/INSTALL
@@ -103,10 +103,10 @@ deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc
druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
-mysql, neo4j, odbc, openfaas, openlineage, opsgenie, oracle, otel, pagerduty, pandas, papermill,
-password, pinot, plexus, postgres, presto, rabbitmq, redis, s3, salesforce, samba, segment,
-sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau,
-tabular, telegram, trino, vertica, virtualenv, webhdfs, winrm, zendesk
+mysql, neo4j, odbc, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty, pandas,
+papermill, password, pinot, plexus, postgres, presto, rabbitmq, redis, s3, salesforce, samba,
+segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd,
+tableau, tabular, telegram, trino, vertica, virtualenv, webhdfs, winrm, zendesk
# END EXTRAS HERE
# For installing Airflow in development environments - see CONTRIBUTING.rst
diff --git a/README.md b/README.md
index cd8c597a44e00..b637232efce85 100644
--- a/README.md
+++ b/README.md
@@ -89,7 +89,7 @@ Airflow is not a streaming solution, but it is often used to process real-time d
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.7.2) |
+| | Main version (dev) | Stable version (2.7.3) |
|-------------|------------------------------|------------------------|
| Python | 3.8, 3.9, 3.10, 3.11 | 3.8, 3.9, 3.10, 3.11 |
| Platform | AMD64/ARM64(\*) | AMD64/ARM64(\*) |
@@ -97,7 +97,7 @@ Apache Airflow is tested with:
| PostgreSQL | 11, 12, 13, 14, 15 | 11, 12, 13, 14, 15 |
| MySQL | 5.7, 8.0, 8.1 | 5.7, 8.0 |
| SQLite | 3.15.0+ | 3.15.0+ |
-| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*), 2019(\*) |
+| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*\*), 2019(\*\*) |
\* Experimental
@@ -172,15 +172,15 @@ them to the appropriate format and workflow that your tool requires.
```bash
-pip install 'apache-airflow==2.7.2' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt"
+pip install 'apache-airflow==2.7.3' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
```
2. Installing with extras (i.e., postgres, google)
```bash
-pip install 'apache-airflow[postgres,google]==2.7.2' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt"
+pip install 'apache-airflow[postgres,google]==2.7.3' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
```
For information on installing provider packages, check
@@ -292,7 +292,7 @@ Apache Airflow version life cycle:
| Version | Current Patch/Minor | State | First Release | Limited Support | EOL/Terminated |
|-----------|-----------------------|-----------|-----------------|-------------------|------------------|
-| 2 | 2.7.2 | Supported | Dec 17, 2020 | TBD | TBD |
+| 2 | 2.7.3 | Supported | Dec 17, 2020 | TBD | TBD |
| 1.10 | 1.10.15 | EOL | Aug 27, 2018 | Dec 17, 2020 | June 17, 2021 |
| 1.9 | 1.9.0 | EOL | Jan 03, 2018 | Aug 27, 2018 | Aug 27, 2018 |
| 1.8 | 1.8.2 | EOL | Mar 19, 2017 | Jan 03, 2018 | Jan 03, 2018 |
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 6430d19ded585..07ceee7e97dae 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -21,6 +21,70 @@
.. towncrier release notes start
+Airflow 2.7.3 (2023-11-04)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
+
+Bug Fixes
+"""""""""
+- Fix pre-mature evaluation of tasks in mapped task group (#34337)
+- Add TriggerRule missing value in rest API (#35194)
+- Fix Scheduler crash looping when dagrun creation fails (#35135)
+- Fix test connection with ``codemirror`` and extra (#35122)
+- Fix usage of cron-descriptor since BC in v1.3.0 (#34836)
+- Fix ``get_plugin_info`` for class based listeners. (#35022)
+- Some improvements/fixes for dag_run and task_instance endpoints (#34942)
+- Fix the dags count filter in webserver home page (#34944)
+- Return only the TIs of the readable dags when ~ is provided as a dag_id (#34939)
+- Fix triggerer thread crash in daemon mode (#34931)
+- Fix wrong plugin schema (#34858)
+- Use DAG timezone in TimeSensorAsync (#33406)
+- Mark tasks with ``all_skipped`` trigger rule as ``skipped`` if any task is in ``upstream_failed`` state (#34392)
+- Add read only validation to read only fields (#33413)
+
+Misc/Internal
+"""""""""""""
+- Improve testing harness to separate DB and non-DB tests (#35160, #35333)
+- Add pytest db_test markers to our tests (#35264)
+- Add pip caching for faster build (#35026)
+- Upper bound ``pendulum`` requirement to ``<3.0`` (#35336)
+- Limit ``sentry_sdk`` to ``1.33.0`` (#35298)
+- Fix subtle bug in mocking processor_agent in our tests (#35221)
+- Bump ``@babel/traverse`` from ``7.16.0 to 7.23.2`` in ``/airflow/www`` (#34988)
+- Bump ``undici`` from ``5.19.1 to 5.26.3`` in ``/airflow/www`` (#34971)
+- Remove unused set from ``SchedulerJobRunner`` (#34810)
+- Remove warning about ``max_tis per query > parallelism`` (#34742)
+- Improve modules import in Airflow core by moving some of them into a type-checking block (#33755)
+- Fix tests to respond to Python 3.12 handling of utcnow in sentry-sdk (#34946)
+- Add ``connexion<3.0`` upper bound (#35218)
+- Limit Airflow to ``< 3.12`` (#35123)
+- update moto version (#34938)
+- Limit WTForms to below ``3.1.0`` (#34943)
+
+Doc Only Changes
+""""""""""""""""
+- Fix variables substitution in Airflow Documentation (#34462)
+- Added example for defaults in ``conn.extras`` (#35165)
+- Update datasets.rst issue with running example code (#35035)
+- Remove ``mysql-connector-python`` from recommended MySQL driver (#34287)
+- Fix syntax error in task dependency ``set_downstream`` example (#35075)
+- Update documentation to enable test connection (#34905)
+- Update docs errors.rst - Mention sentry "transport" configuration option (#34912)
+- Update dags.rst to put SubDag deprecation note right after the SubDag section heading (#34925)
+- Add info on getting variables and config in custom secrets backend (#34834)
+- Document BaseExecutor interface in more detail to help users in writing custom executors (#34324)
+- Fix broken link to ``airflow_local_settings.py`` template (#34826)
+- Fixes python_callable function assignment context kwargs example in params.rst (#34759)
+- Add missing multiple_outputs=True param in the TaskFlow example (#34812)
+- Remove extraneous ``'>'`` in provider section name (#34813)
+- Fix imports in extra link documentation (#34547)
+
+
+
Airflow 2.7.2 (2023-10-12)
--------------------------
diff --git a/TESTING.rst b/TESTING.rst
index 9b1276382ad4f..1cc95aa7de523 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -56,12 +56,6 @@ Follow the guidelines when writing unit tests:
overrides ``recwarn`` fixture behaviour.
-.. note::
-
- We are in the process of converting all unit tests to standard "asserts" and pytest fixtures
- so if you find some tests that are still using classic setUp/tearDown approach or unittest asserts, feel
- free to convert them to pytest.
-
Airflow configuration for unit tests
------------------------------------
@@ -72,56 +66,646 @@ fixture. This in turn makes Airflow load test configuration from the file
defaults from ``airflow/config_templates/config.yml``. If you want to add some test-only configuration,
as default for all tests you should add the value to this file.
-You can also of course override the values in individual test by patching environment variables following
+You can also - of course - override the values in individual test by patching environment variables following
the usual ``AIRFLOW__SECTION__KEY`` pattern or ``conf_vars`` context manager.
-.. note:: Previous way of setting the test configuration
+Airflow test types
+------------------
- The test configuration for Airflow before July 2023 was automatically generated in a file named
- ``AIRFLOW_HOME/unittest.cfg``. The template for it was stored in "config_templates" next to the yaml file.
- However writing the file was only done for the first time you run airflow and you had to manually
- maintain the file. It was pretty arcane knowledge, and this generated file in {AIRFLOW_HOME}
- has been overwritten in the Breeze environment with another CI-specific file. Using ``unit_tests.cfg``
- as a single source of the configuration for tests - coming from Airflow sources
- rather than from {AIRFLOW_HOME} is much more convenient and it is automatically used by pytest.
+Airflow tests in the CI environment are split into several test types. You can narrow down which
+test types you want to use in various ``breeze testing`` sub-commands in three ways:
- The unittest.cfg file generated in {AIRFLOW_HOME} will no longer be used and can be removed.
+* via specifying the ``--test-type`` when you run single test type in ``breeze testing tests`` command
+* via specifying space separating list of test types via ``--paralleltest-types`` or
+ ``--exclude-parallel-test-types`` options when you run tests in parallel (in several testing commands)
+Those test types are defined:
-Airflow test types
-------------------
+* ``Always`` - those are tests that should be always executed (always sub-folder)
+* ``API`` - Tests for the Airflow API (api, api_connexion, api_experimental and api_internal sub-folders)
+* ``CLI`` - Tests for the Airflow CLI (cli folder)
+* ``Core`` - for the core Airflow functionality (core, executors, jobs, models, ti_deps, utils sub-folders)
+* ``Operators`` - tests for the operators (operators folder with exception of Virtualenv Operator tests and
+ External Python Operator tests that have their own test type). They are skipped by the
+``virtualenv_operator`` and ``external_python_operator`` test markers that the tests are marked with.
+* ``WWW`` - Tests for the Airflow webserver (www folder)
+* ``Providers`` - Tests for all Providers of Airflow (providers folder)
+* ``PlainAsserts`` - tests that require disabling ``assert-rewrite`` feature of Pytest (usually because
+ a buggy/complex implementation of an imported library) (``plain_asserts`` marker)
+* ``Other`` - all other tests remaining after the above tests are selected
+
+There are also Virtualenv/ExternalPython operator test types that are excluded from ``Operators`` test type
+and run as separate test types. Those are :
+
+* ``PythonVenv`` - tests for PythonVirtualenvOperator - selected directly as TestPythonVirtualenvOperator
+* ``ExternalPython`` - tests for ExternalPythonOperator - selected directly as TestExternalPythonOperator
+* ``BranchExternalPython`` - tests for BranchExternalPythonOperator - selected directly as TestBranchExternalPythonOperator
+
+We have also tests that run "all" tests (so they do not look at the folder, but at the ``pytest`` markers
+the tests are marked with to run with some filters applied.
+
+* ``All-Postgres`` - tests that require Postgres database. They are only run when backend is Postgres (``backend("postgres")`` marker)
+* ``All-MySQL`` - tests that require MySQL database. They are only run when backend is MySQL (``backend("mysql")`` marker)
+* ``All-Quarantined`` - tests that are flaky and need to be fixed (``quarantined`` marker)
+* ``All`` - all tests are run (this is the default)
-Airflow tests in the CI environment are split into several test types:
-* Always - those are tests that should be always executed (always folder)
-* Core - for the core Airflow functionality (core folder)
-* API - Tests for the Airflow API (api and api_connexion folders)
-* CLI - Tests for the Airflow CLI (cli folder)
-* WWW - Tests for the Airflow webserver (www folder)
-* Providers - Tests for all Providers of Airflow (providers folder)
-* Other - all other tests (all other folders that are not part of any of the above)
+We also have ``Integration`` tests that are running Integration tests with external software that is run
+via ``--integration`` flag in ``breeze`` environment - via ``breeze testing integration-tests``.
+
+* ``Integration`` - tests that require external integration images running in docker-compose
This is done for three reasons:
1. in order to selectively run only subset of the test types for some PRs
-2. in order to allow parallel execution of the tests on Self-Hosted runners
+2. in order to allow efficient parallel test execution of the tests on Self-Hosted runners
For case 2. We can utilise memory and CPUs available on both CI and local development machines to run
-test in parallel. This way we can decrease the time of running all tests in self-hosted runners from
-60 minutes to ~15 minutes.
+test in parallel, but we cannot use pytest xdist plugin for that - we need to split the tests into test
+types and run each test type with their own instance of database and separate container where the tests
+in each type are run with exclusive access to their database and each test within test type runs sequentially.
+By the nature of those tests - they rely on shared databases - and they update/reset/cleanup data in the
+databases while they are executing.
-.. note::
- We need to split tests manually into separate suites rather than utilise
- ``pytest-xdist`` or ``pytest-parallel`` which could be a simpler and much more "native" parallelization
- mechanism. Unfortunately, we cannot utilise those tools because our tests are not truly ``unit`` tests that
- can run in parallel. A lot of our tests rely on shared databases - and they update/reset/cleanup the
- databases while they are executing. They are also exercising features of the Database such as locking which
- further increases cross-dependency between tests. Until we make all our tests truly unit tests (and not
- touching the database or until we isolate all such tests to a separate test type, we cannot really rely on
- frameworks that run tests in parallel. In our solution each of the test types is run in parallel with its
- own database (!) so when we have 8 test types running in parallel, there are in fact 8 databases run
- behind the scenes to support them and each of the test types executes its own tests sequentially.
+DB and non-DB tests
+-------------------
+
+There are two kinds of unit tests in Airflow - DB and non-DB tests.
+
+Some of the tests of Airflow (around 7000 of them on October 2023)
+require a database to connect to in order to run. Those tests store and read data from Airflow DB using
+Airflow's core code and it's crucial to run the tests against all real databases that Airflow supports in order
+to check if the SQLAlchemy queries are correct and if the database schema is correct.
+
+Those tests should be marked with ``@pytest.mark.db`` decorator on one of the levels:
+
+* test method can be marked with ``@pytest.mark.db`` decorator
+* test class can be marked with ``@pytest.mark.db`` decorator
+* test module can be marked with ``pytestmark = pytest.mark.db`` at the top level of the module
+
+Airflow's CI runs different test kinds separately.
+
+For the DB tests, they are run against the multiple databases Airflow support, multiple versions of those
+and multiple Python versions it supports. In order to save time for testing not all combinations are
+tested but enough various combinations are tested to detect potential problems.
+
+As of October 2023, Airflow has ~9000 Non-DB tests and around 7000 DB tests.
+
+Airflow non-DB tests
+--------------------
+
+For the Non-DB tests, they are run once for each tested Python version with ``none`` database backend (which
+causes any database access to fail. Those tests are run with ``pytest-xdist`` plugin in parallel which
+means that we can efficiently utilised multi-processor machines (including ``self-hosted`` runners with
+8 CPUS we have to run the tests with maximum parallelism).
+
+It's usually straightforward to run those tests in local virtualenv because they do not require any
+setup or running database. They also run much faster than DB tests. You can run them with ``pytest`` command
+or with ``breeze`` that has all the dependencies needed to run all tests automatically installed. Of course
+you can also select just specific test or folder or module for the Pytest to collect/run tests from there,
+the example below shows how to run all tests, parallelising them with ``pytest-xdist``
+(by specifying ``tests`` folder):
+
+.. code-block:: bash
+
+ pytest tests --skip-db-tests -n auto
+
+
+The ``--skip-db-tests`` flag will only run tests that are not marked as DB tests.
+
+
+You can also run ``breeze`` command to run all the tests (they will run in a separate container,
+the selected python version and without access to any database). Adding ``--use-xdist`` flag will run all
+tests in parallel using ``pytest-xdist`` plugin.
+
+We have a dedicated, opinionated ``breeze testing non-db-tests`` command as well that runs non-DB tests
+(it is also used in CI to run the non-DB tests, where you do not have to specify extra flags for
+parallel running and you can run all the Non-DB tests
+(or just a subset of them with ``--parallel-test-types`` or ``--exclude-parallel-test-types``) in parallel:
+
+.. code-block:: bash
+
+ breeze testing non-db-tests
+
+You can pass ``--parallel-test-type`` list of test types to execute or ``--exclude--parallel-test-types``
+to exclude them from the default set:.
+
+.. code-block:: bash
+
+ breeze testing non-db-tests --parallel-test-types "Providers API CLI"
+
+
+.. code-block:: bash
+
+ breeze testing non-db-tests --exclude-parallel-test-types "Providers API CLI"
+
+You can also run the same commands via ``breeze testing tests`` - by adding the necessary flags manually:
+
+.. code-block:: bash
+
+ breeze testing tests --skip-db-tests --backend none --use-xdist
+
+Also you can enter interactive shell with ``breeze`` and run tests from there if you want to iterate
+with the tests. Source files in ``breeze`` are mounted as volumes so you can modify them locally and
+rerun in Breeze as you will (``-n auto`` will parallelize tests using ``pytest-xdist`` plugin):
+
+.. code-block:: bash
+
+ breeze shell --backend none --python 3.8
+ > pytest tests --skip-db-tests -n auto
+
+
+Airflow DB tests
+----------------
+
+Airflow DB tests require database to run. It can be any of the supported Airflow Databases and they can
+be run either using local virtualenv or Breeze
+
+
+
+By default, the DB tests will use sqlite and the "airflow.db" database created and populated in the
+``${AIRFLOW_HOME}`` folder. You do not need to do anything to get the database created and initialized,
+but if you need to clean and restart the db, you can run tests with ``-with-db-init`` flag - then the
+database will be re-initialized. You can also set ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` environment
+variable to point to supported database (Postgres, MySQL, etc.) and the tests will use that database. You
+might need to run ``airflow db reset`` to initialize the database in that case.
+
+The "non-DB" tests are perfectly fine to run when you have database around but if you want to just run
+DB tests (as happens in our CI for the ``Database`` runs) you can use ``--run-db-tests-only`` flag to filter
+out non-DB tests (and obviously you can specify not only on the whole ``tests`` directory but on any
+folders/files/tests selection, ``pytest`` supports).
+
+.. code-block:: bash
+
+ pytest tests/ --run-db-tests-only
+
+You can also run DB tests with ``breeze`` dockerized environment. You can choose backend to use with
+``--backend`` flag. The default is ``sqlite`` but you can also use others such as ``postgres`` or ``mysql``.
+You can also select backend version and Python version to use. You can specify the ``test-type`` to run -
+breeze will list the test types you can run with ``--help`` and provide auto-complete for them. Example
+below runs the ``Core`` tests with ``postgres`` backend and ``3.8`` Python version:
+
+We have a dedicated, opinionated ``breeze testing db-tests`` command as well that runs DB tests
+(it is also used in CI to run the DB tests, where you do not have to specify extra flags for
+parallel running and you can run all the DB tests
+(or just a subset of them with ``--parallel-test-types`` or ``--exclude-parallel-test-types``) in parallel:
+
+.. code-block:: bash
+
+ breeze testing non-db-tests --backent postgres
+
+You can pass ``--parallel-test-type`` list of test types to execute or ``--exclude--parallel-test-types``
+to exclude them from the default set:.
+
+.. code-block:: bash
+
+ breeze testing db-tests --parallel-test-types "Providers API CLI"
+
+
+.. code-block:: bash
+
+ breeze testing db-tests --exclude-parallel-test-types "Providers API CLI"
+
+You can also run the same commands via ``breeze testing tests`` - by adding the necessary flags manually:
+
+.. code-block:: bash
+
+ breeze testing tests --run-db-tests-only --backend postgres --run-tests-in-parallel
+
+
+Also - if you want to iterate with the tests you can enter interactive shell and run the tests iteratively -
+either by package/module/test or by test type - whatever ``pytest`` supports.
+
+.. code-block:: bash
+
+ breeze shell --backend postgres --python 3.8
+ > pytest tests --run-db-tests-only
+
+As explained before, you cannot run DB tests in parallel using ``pytest-xdist`` plugin, but ``breeze`` has
+support to split all the tests into test-types to run in separate containers and with separate databases
+and you can run the tests using ``--run-tests-in-parallel`` flag (which is automatically enabled when
+you use ``breeze testing db-tests`` command):
+
+.. code-block:: bash
+
+ breeze testing tests --run-db-tests-only --backend postgres --python 3.8 --run-tests-in-parallel
+
+
+Best practices for DB tests
+===========================
+
+Usually when you add new tests you add tests "similar" to the ones that are already there. In most cases,
+therefore you do not have to worry about the test type - it will be automatically selected for you by the
+fact that the Test Class that you add the tests or the whole module will be marked with ``db_test`` marker.
+
+You should strive to write "pure" non-db unit tests (i.e. DB tests) but sometimes it's just better to plug-in
+the existing framework of DagRuns, Dags, Connections and Variables to use the Database directly rather
+than having to mock the DB access for example. It's up to you to decide.
+
+However, if you choose to write DB tests you have to make sure you add the ``db_test`` marker - either to
+the test method, class (with decorator) or whole module (with pytestmark at the top level of the module).
+
+In most cases when you add tests to existing modules or classes, you follow similar tests so you do not
+have to do anything, but in some cases you need to decide if your test should be marked as DB test or
+whether it should be changed to not use the database at all.
+
+If your test accesses the database but is not marked properly the Non-DB test in CI will fail with this message:
+
+.. code ::
+
+ "Your test accessed the DB but `_AIRFLOW_SKIP_DB_TESTS` is set.
+ Either make sure your test does not use database or mark your test with `@pytest.mark.db_test`.
+
+Marking test as DB test
+-----------------------
+
+You can apply the marker on method/function/class level with ``@pytest.mark.db_test`` decorator or
+at the module level with ``pytestmark = pytest.mark.db_test`` at the top level of the module.
+
+It's up to the author to decide whether to mark the test, class, or module as "DB-test" - generally the
+less DB tests - the better and if we can clearly separate the parts that are DB from non-DB, we should,
+but also it's ok if few tests are marked as DB tests when they are not but they are part of the class
+or module that is "mostly-DB".
+
+Sometimes, when your class can be clearly split to DB and non-DB parts, it's better to split the class
+into two separate classes and mark only the DB class as DB test.
+
+Method level:
+
+.. code-block:: python
+
+ import pytest
+
+
+ @pytest.mark.db_test
+ def test_add_tagging(self, sentry, task_instance):
+ ...
+
+Class level:
+
+
+.. code-block:: python
+
+ import pytest
+
+
+ @pytest.mark.db_test
+ class TestDatabricksHookAsyncAadTokenSpOutside:
+ ...
+
+Module level (at the top of the module):
+
+.. code-block:: python
+
+ import pytest
+
+ from airflow.models.baseoperator import BaseOperator
+ from airflow.models.dag import DAG
+ from airflow.ti_deps.dep_context import DepContext
+ from airflow.ti_deps.deps.task_concurrency_dep import TaskConcurrencyDep
+
+ pytestmark = pytest.mark.db_test
+
+
+How to verify if DB test is correctly classified
+------------------------------------------------
+
+When you add if you want to see if your DB test is correctly classified, you can run the test or group
+of tests with ``--skip-db-tests`` flag.
+
+You can run the all (or subset of) test types if you want to make sure all ot the problems are fixed
+
+ .. code-block:: bash
+
+ breeze testing tests --skip-db-tests tests/your_test.py
+
+For the whole test suite you can run:
+
+ .. code-block:: bash
+
+ breeze testing non-db-tests
+
+For selected test types (example - the tests will run for Providers/API/CLI code only:
+
+ .. code-block:: bash
+
+ breeze testing non-db-tests --parallel-test-types "Providers API CLI"
+
+
+How to make your test not depend on DB
+--------------------------------------
+
+This is tricky and there is no single solution. Sometimes we can mock-out the methods that require
+DB access or objects that normally require database. Sometimes we can decide to test just sinle method
+of class rather than more complex set of steps. Generally speaking it's good to have as many "pure"
+unit tests that require no DB as possible comparing to DB tests. They are usually faster an more
+reliable as well.
+
+
+Special cases
+-------------
+
+There are some tricky test cases that require special handling. Here are some of them:
+
+
+Parameterized tests stability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The parameterized tests require stable order of parameters if they are run via xdist - because the parameterized
+tests are distributed among multiple processes and handled separately. In some cases the parameterized tests
+have undefined / random order (or parameters are not hashable - for example set of enums). In such cases
+the xdist execution of the tests will fail and you will get an error mentioning "Known Limitations of xdist".
+You can see details about the limitation `here `_
+
+The error in this case will look similar to:
+
+ .. code-block::
+
+ Different tests were collected between gw0 and gw7. The difference is:
+
+
+The fix for that is to sort the parameters in ``parametrize``. For example instead of this:
+
+ .. code-block:: python
+
+ @pytest.mark.parametrize("status", ALL_STATES)
+ def test_method():
+ ...
+
+
+do that:
+
+
+ .. code-block:: python
+
+ @pytest.mark.parametrize("status", sorted(ALL_STATES))
+ def test_method():
+ ...
+
+Similarly if your parameters are defined as result of utcnow() or other dynamic method - you should
+avoid that, or assign unique IDs for those parametrized tests. Instead of this:
+
+ .. code-block:: python
+
+ @pytest.mark.parametrize(
+ "url, expected_dag_run_ids",
+ [
+ (
+ f"api/v1/dags/TEST_DAG_ID/dagRuns?end_date_gte="
+ f"{urllib.parse.quote((timezone.utcnow() + timedelta(days=1)).isoformat())}",
+ [],
+ ),
+ (
+ f"api/v1/dags/TEST_DAG_ID/dagRuns?end_date_lte="
+ f"{urllib.parse.quote((timezone.utcnow() + timedelta(days=1)).isoformat())}",
+ ["TEST_DAG_RUN_ID_1", "TEST_DAG_RUN_ID_2"],
+ ),
+ ],
+ )
+ def test_end_date_gte_lte(url, expected_dag_run_ids):
+ ...
+
+Do this:
+
+ .. code-block:: python
+
+ @pytest.mark.parametrize(
+ "url, expected_dag_run_ids",
+ [
+ pytest.param(
+ f"api/v1/dags/TEST_DAG_ID/dagRuns?end_date_gte="
+ f"{urllib.parse.quote((timezone.utcnow() + timedelta(days=1)).isoformat())}",
+ [],
+ id="end_date_gte",
+ ),
+ pytest.param(
+ f"api/v1/dags/TEST_DAG_ID/dagRuns?end_date_lte="
+ f"{urllib.parse.quote((timezone.utcnow() + timedelta(days=1)).isoformat())}",
+ ["TEST_DAG_RUN_ID_1", "TEST_DAG_RUN_ID_2"],
+ id="end_date_lte",
+ ),
+ ],
+ )
+ def test_end_date_gte_lte(url, expected_dag_run_ids):
+ ...
+
+
+
+Problems with Non-DB test collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes, even if whole module is marked as ``@pytest.mark.db_test`` even parsing the file and collecting
+tests will fail when ``--skip-db-tests`` is used because some of the imports od objects created in the
+module will read the database.
+
+Usually what helps is to move such initialization code to inside the tests or pytest fixtures (and pass
+objects needed by tests as fixtures rather than importing them from the module). Similarly you might
+use DB - bound objects (like Connection) in your ``parametrize`` specification - this will also fail pytest
+collection. Move creation of such objects to inside the tests:
+
+Moving object creation from top-level to inside tests. This code will break collection of tests even if
+the test is marked as DB test:
+
+
+ .. code-block:: python
+
+ pytestmark = pytest.mark.db_test
+
+ TI = TaskInstance(
+ task=BashOperator(task_id="test", bash_command="true", dag=DAG(dag_id="id"), start_date=datetime.now()),
+ run_id="fake_run",
+ state=State.RUNNING,
+ )
+
+
+ class TestCallbackRequest:
+ @pytest.mark.parametrize(
+ "input,request_class",
+ [
+ (CallbackRequest(full_filepath="filepath", msg="task_failure"), CallbackRequest),
+ (
+ TaskCallbackRequest(
+ full_filepath="filepath",
+ simple_task_instance=SimpleTaskInstance.from_ti(ti=TI),
+ processor_subdir="/test_dir",
+ is_failure_callback=True,
+ ),
+ TaskCallbackRequest,
+ ),
+ (
+ DagCallbackRequest(
+ full_filepath="filepath",
+ dag_id="fake_dag",
+ run_id="fake_run",
+ processor_subdir="/test_dir",
+ is_failure_callback=False,
+ ),
+ DagCallbackRequest,
+ ),
+ (
+ SlaCallbackRequest(
+ full_filepath="filepath",
+ dag_id="fake_dag",
+ processor_subdir="/test_dir",
+ ),
+ SlaCallbackRequest,
+ ),
+ ],
+ )
+ def test_from_json(self, input, request_class):
+ ...
+
+
+Instead - this will not break collection. The TaskInstance is not initialized when the module is parsed,
+it will only be initialized when the test gets executed because we moved initialization of it from
+top level / parametrize to inside the test:
+
+ .. code-block:: python
+
+ pytestmark = pytest.mark.db_test
+
+
+ class TestCallbackRequest:
+ @pytest.mark.parametrize(
+ "input,request_class",
+ [
+ (CallbackRequest(full_filepath="filepath", msg="task_failure"), CallbackRequest),
+ (
+ None, # to be generated when test is run
+ TaskCallbackRequest,
+ ),
+ (
+ DagCallbackRequest(
+ full_filepath="filepath",
+ dag_id="fake_dag",
+ run_id="fake_run",
+ processor_subdir="/test_dir",
+ is_failure_callback=False,
+ ),
+ DagCallbackRequest,
+ ),
+ (
+ SlaCallbackRequest(
+ full_filepath="filepath",
+ dag_id="fake_dag",
+ processor_subdir="/test_dir",
+ ),
+ SlaCallbackRequest,
+ ),
+ ],
+ )
+ def test_from_json(self, input, request_class):
+ if input is None:
+ ti = TaskInstance(
+ task=BashOperator(
+ task_id="test", bash_command="true", dag=DAG(dag_id="id"), start_date=datetime.now()
+ ),
+ run_id="fake_run",
+ state=State.RUNNING,
+ )
+
+ input = TaskCallbackRequest(
+ full_filepath="filepath",
+ simple_task_instance=SimpleTaskInstance.from_ti(ti=ti),
+ processor_subdir="/test_dir",
+ is_failure_callback=True,
+ )
+
+
+Sometimes it is difficult to rewrite the tests, so you might add conditional handling and mock out some
+database-bound methods or objects to avoid hitting the database during test collection. The code below
+will hit the Database while parsing the tests, because this is what Variable.setdefault does when
+parametrize specification is being parsed - even if test is marked as DB test.
+
+
+ .. code-block:: python
+
+ from airflow.models.variable import Variable
+
+ pytestmark = pytest.mark.db_test
+
+ initial_db_init()
+
+
+ @pytest.mark.parametrize(
+ "env, expected",
+ [
+ pytest.param(
+ {"plain_key": "plain_value"},
+ "{'plain_key': 'plain_value'}",
+ id="env-plain-key-val",
+ ),
+ pytest.param(
+ {"plain_key": Variable.setdefault("plain_var", "banana")},
+ "{'plain_key': 'banana'}",
+ id="env-plain-key-plain-var",
+ ),
+ pytest.param(
+ {"plain_key": Variable.setdefault("secret_var", "monkey")},
+ "{'plain_key': '***'}",
+ id="env-plain-key-sensitive-var",
+ ),
+ pytest.param(
+ {"plain_key": "{{ var.value.plain_var }}"},
+ "{'plain_key': '{{ var.value.plain_var }}'}",
+ id="env-plain-key-plain-tpld-var",
+ ),
+ ],
+ )
+ def test_rendered_task_detail_env_secret(patch_app, admin_client, request, env, expected):
+ ...
+
+
+You can make the code conditional and mock out the Variable to avoid hitting the database.
+
+
+ .. code-block:: python
+
+ from airflow.models.variable import Variable
+
+ pytestmark = pytest.mark.db_test
+
+
+ if os.environ.get("_AIRFLOW_SKIP_DB_TESTS") == "true":
+ # Handle collection of the test by non-db case
+ Variable = mock.MagicMock() # type: ignore[misc] # noqa: F811
+ else:
+ initial_db_init()
+
+
+ @pytest.mark.parametrize(
+ "env, expected",
+ [
+ pytest.param(
+ {"plain_key": "plain_value"},
+ "{'plain_key': 'plain_value'}",
+ id="env-plain-key-val",
+ ),
+ pytest.param(
+ {"plain_key": Variable.setdefault("plain_var", "banana")},
+ "{'plain_key': 'banana'}",
+ id="env-plain-key-plain-var",
+ ),
+ pytest.param(
+ {"plain_key": Variable.setdefault("secret_var", "monkey")},
+ "{'plain_key': '***'}",
+ id="env-plain-key-sensitive-var",
+ ),
+ pytest.param(
+ {"plain_key": "{{ var.value.plain_var }}"},
+ "{'plain_key': '{{ var.value.plain_var }}'}",
+ id="env-plain-key-plain-tpld-var",
+ ),
+ ],
+ )
+ def test_rendered_task_detail_env_secret(patch_app, admin_client, request, env, expected):
+ ...
+
+
+
+Running Unit tests
+==================
Running Unit Tests from PyCharm IDE
-----------------------------------
@@ -159,7 +743,6 @@ this in two clicks.
:align: center
:alt: Installing Python extension
-
2. Add the tool to the context menu:
a. From the settings menu, navigate to Appearance & Behavior > Menus & Toolbars > Project View Popup Menu
diff --git a/airflow/__init__.py b/airflow/__init__.py
index 182b71dc2feb9..ae82e562aec7e 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -26,7 +26,7 @@
"""
from __future__ import annotations
-__version__ = "2.7.2"
+__version__ = "2.7.3"
# flake8: noqa: F401
diff --git a/airflow/__main__.py b/airflow/__main__.py
index e49f7e9bdf6a0..6f82f7536026b 100644
--- a/airflow/__main__.py
+++ b/airflow/__main__.py
@@ -45,7 +45,6 @@ def main():
parser = cli_parser.get_parser()
argcomplete.autocomplete(parser)
args = parser.parse_args()
-
if args.subcommand not in ["lazy_loaded", "version"]:
# Here we ensure that the default configuration is written if needed before running any command
# that might need it. This used to be done during configuration initialization but having it
@@ -55,7 +54,6 @@ def main():
conf = write_default_airflow_configuration_if_needed()
if args.subcommand in ["webserver", "internal-api", "worker"]:
write_webserver_configuration_if_needed(conf)
-
args.func(args)
diff --git a/airflow/api_connexion/endpoints/dag_run_endpoint.py b/airflow/api_connexion/endpoints/dag_run_endpoint.py
index 377b79aa113bb..a42e2079f67aa 100644
--- a/airflow/api_connexion/endpoints/dag_run_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_run_endpoint.py
@@ -376,7 +376,7 @@ def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:
@security.requires_access(
[
- (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+ (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG_RUN),
],
)
@@ -408,7 +408,7 @@ def update_dag_run_state(*, dag_id: str, dag_run_id: str, session: Session = NEW
@security.requires_access(
[
- (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+ (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG_RUN),
],
)
@@ -459,7 +459,7 @@ def clear_dag_run(*, dag_id: str, dag_run_id: str, session: Session = NEW_SESSIO
@security.requires_access(
[
- (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+ (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG_RUN),
],
)
diff --git a/airflow/api_connexion/endpoints/task_instance_endpoint.py b/airflow/api_connexion/endpoints/task_instance_endpoint.py
index b9a5ac9777279..53ffd9896e80f 100644
--- a/airflow/api_connexion/endpoints/task_instance_endpoint.py
+++ b/airflow/api_connexion/endpoints/task_instance_endpoint.py
@@ -42,6 +42,7 @@
task_instance_reference_schema,
task_instance_schema,
)
+from airflow.api_connexion.security import get_readable_dags
from airflow.api_connexion.types import APIResponse
from airflow.models import SlaMiss
from airflow.models.dagrun import DagRun as DR
@@ -338,6 +339,8 @@ def get_task_instances(
if dag_id != "~":
base_query = base_query.where(TI.dag_id == dag_id)
+ else:
+ base_query = base_query.where(TI.dag_id.in_(get_readable_dags()))
if dag_run_id != "~":
base_query = base_query.where(TI.run_id == dag_run_id)
base_query = _apply_range_filter(
@@ -460,7 +463,7 @@ def get_task_instances_batch(session: Session = NEW_SESSION) -> APIResponse:
@security.requires_access(
[
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
- (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG_RUN),
+ (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG_RUN),
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_TASK_INSTANCE),
],
)
@@ -663,7 +666,6 @@ def patch_mapped_task_instance(
[
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
(permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG_RUN),
- (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_TASK_INSTANCE),
],
)
diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
index b94fe05b59602..7b02cb365b559 100644
--- a/airflow/api_connexion/openapi/v1.yaml
+++ b/airflow/api_connexion/openapi/v1.yaml
@@ -231,7 +231,7 @@ info:
This means that the server encountered an unexpected condition that prevented it from
fulfilling the request.
- version: '2.7.2'
+ version: '2.7.3'
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
@@ -369,6 +369,15 @@ paths:
description: |
Test a connection.
+ For security reasons, the test connection functionality is disabled by default across Airflow UI, API and CLI.
+ For more information on capabilities of users, see the documentation:
+ https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users.
+ It is strongly advised to not enable the feature until you make sure that only
+ highly trusted UI/API users have "edit connection" permissions.
+
+ Set the "test_connection" flag to "Enabled" in the "core" section of Airflow configuration (airflow.cfg) to enable testing of collections.
+ It can also be controlled by the environment variable `AIRFLOW__CORE__TEST_CONNECTION`.
+
*New in version 2.2.0*
x-openapi-router-controller: airflow.api_connexion.endpoints.connection_endpoint
operationId: test_connection
@@ -3770,13 +3779,13 @@ components:
macros:
type: array
items:
- type: object
+ type: string
nullable: true
description: The plugin macros
flask_blueprints:
type: array
items:
- type: object
+ type: string
nullable: true
description: The flask blueprints
appbuilder_views:
@@ -3794,13 +3803,13 @@ components:
global_operator_extra_links:
type: array
items:
- type: object
+ type: string
nullable: true
description: The global operator extra links
operator_extra_links:
type: array
items:
- type: object
+ type: string
nullable: true
description: Operator extra links
source:
@@ -4713,19 +4722,29 @@ components:
description: |
Trigger rule.
- *Changed in version 2.2.0*: 'none_failed_min_one_success' is added as a possible value.
+ *Changed in version 2.2.0*: 'none_failed_min_one_success' is added as a possible value. Deprecated 'dummy' and 'always' is added as a possible value
+
+ *Changed in version 2.3.0*: 'all_skipped' is added as a possible value.
+
+ *Changed in version 2.5.0*: 'one_done' is added as a possible value.
+
+ *Changed in version 2.7.0*: 'all_done_setup_success' is added as a possible value.
type: string
enum:
- all_success
- all_failed
- all_done
+ - all_done_setup_success
- one_success
- one_failed
+ - one_done
- none_failed
- none_skipped
- none_failed_or_skipped
- none_failed_min_one_success
- dummy
+ - all_skipped
+ - always
WeightRule:
description: Weight rule.
diff --git a/airflow/api_connexion/schemas/plugin_schema.py b/airflow/api_connexion/schemas/plugin_schema.py
index 780fef17bf76a..4b62111482075 100644
--- a/airflow/api_connexion/schemas/plugin_schema.py
+++ b/airflow/api_connexion/schemas/plugin_schema.py
@@ -27,12 +27,12 @@ class PluginSchema(Schema):
name = fields.String()
hooks = fields.List(fields.String())
executors = fields.List(fields.String())
- macros = fields.List(fields.Dict())
- flask_blueprints = fields.List(fields.Dict())
+ macros = fields.List(fields.String())
+ flask_blueprints = fields.List(fields.String())
appbuilder_views = fields.List(fields.Dict())
appbuilder_menu_items = fields.List(fields.Dict())
- global_operator_extra_links = fields.List(fields.Dict())
- operator_extra_links = fields.List(fields.Dict())
+ global_operator_extra_links = fields.List(fields.String())
+ operator_extra_links = fields.List(fields.String())
source = fields.String()
diff --git a/airflow/api_connexion/security.py b/airflow/api_connexion/security.py
index b108adc2c36b5..b19f15257c18a 100644
--- a/airflow/api_connexion/security.py
+++ b/airflow/api_connexion/security.py
@@ -19,7 +19,7 @@
from functools import wraps
from typing import Callable, Sequence, TypeVar, cast
-from flask import Response
+from flask import Response, g
from airflow.api_connexion.exceptions import PermissionDenied, Unauthenticated
from airflow.utils.airflow_flask_app import get_airflow_app
@@ -55,3 +55,11 @@ def decorated(*args, **kwargs):
return cast(T, decorated)
return requires_access_decorator
+
+
+def get_readable_dags() -> list[str]:
+ return get_airflow_app().appbuilder.sm.get_accessible_dag_ids(g.user)
+
+
+def can_read_dag(dag_id: str) -> bool:
+ return get_airflow_app().appbuilder.sm.can_read_dag(dag_id, g.user)
diff --git a/airflow/cli/commands/triggerer_command.py b/airflow/cli/commands/triggerer_command.py
index 2288f1537fbb9..5ddb4e23b6633 100644
--- a/airflow/cli/commands/triggerer_command.py
+++ b/airflow/cli/commands/triggerer_command.py
@@ -58,7 +58,6 @@ def triggerer(args):
settings.MASK_SECRETS_IN_LOGS = True
print(settings.HEADER)
triggerer_heartrate = conf.getfloat("triggerer", "JOB_HEARTBEAT_SEC")
- triggerer_job_runner = TriggererJobRunner(job=Job(heartrate=triggerer_heartrate), capacity=args.capacity)
if args.daemon:
pid, stdout, stderr, log_file = setup_locations(
@@ -77,10 +76,16 @@ def triggerer(args):
umask=int(settings.DAEMON_UMASK, 8),
)
with daemon_context, _serve_logs(args.skip_serve_logs):
+ triggerer_job_runner = TriggererJobRunner(
+ job=Job(heartrate=triggerer_heartrate), capacity=args.capacity
+ )
run_job(job=triggerer_job_runner.job, execute_callable=triggerer_job_runner._execute)
else:
signal.signal(signal.SIGINT, sigint_handler)
signal.signal(signal.SIGTERM, sigint_handler)
signal.signal(signal.SIGQUIT, sigquit_handler)
with _serve_logs(args.skip_serve_logs):
+ triggerer_job_runner = TriggererJobRunner(
+ job=Job(heartrate=triggerer_heartrate), capacity=args.capacity
+ )
run_job(job=triggerer_job_runner.job, execute_callable=triggerer_job_runner._execute)
diff --git a/airflow/configuration.py b/airflow/configuration.py
index fb3af35852f55..7ff3dd3574ecc 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -36,22 +36,25 @@
from contextlib import contextmanager
from copy import deepcopy
from json.decoder import JSONDecodeError
-from typing import IO, Any, Dict, Generator, Iterable, Pattern, Set, Tuple, Union
+from typing import IO, TYPE_CHECKING, Any, Dict, Generator, Iterable, Pattern, Set, Tuple, Union
from urllib.parse import urlsplit
import re2
from packaging.version import parse as parse_version
from typing_extensions import overload
-from airflow.auth.managers.base_auth_manager import BaseAuthManager
from airflow.exceptions import AirflowConfigException
-from airflow.secrets import DEFAULT_SECRETS_SEARCH_PATH, BaseSecretsBackend
+from airflow.secrets import DEFAULT_SECRETS_SEARCH_PATH
from airflow.utils import yaml
from airflow.utils.empty_set import _get_empty_set_for_configuration
from airflow.utils.module_loading import import_string
from airflow.utils.providers_configuration_loader import providers_configuration_loaded
from airflow.utils.weight_rule import WeightRule
+if TYPE_CHECKING:
+ from airflow.auth.managers.base_auth_manager import BaseAuthManager
+ from airflow.secrets import BaseSecretsBackend
+
log = logging.getLogger(__name__)
# show Airflow's deprecation warnings
@@ -718,7 +721,6 @@ def restore_core_default_configuration(self) -> None:
def validate(self):
self._validate_sqlite3_version()
self._validate_enums()
- self._validate_max_tis_per_query()
for section, replacement in self.deprecated_values.items():
for name, info in replacement.items():
@@ -740,26 +742,6 @@ def validate(self):
self._upgrade_postgres_metastore_conn()
self.is_validated = True
- def _validate_max_tis_per_query(self) -> None:
- """
- Check if config ``scheduler.max_tis_per_query`` is not greater than ``core.parallelism``.
-
- If not met, a warning message is printed to guide the user to correct it.
-
- More info: https://github.com/apache/airflow/pull/32572
- """
- max_tis_per_query = self.getint("scheduler", "max_tis_per_query")
- parallelism = self.getint("core", "parallelism")
-
- if max_tis_per_query > parallelism:
- warnings.warn(
- f"Config scheduler.max_tis_per_query (value: {max_tis_per_query}) "
- f"should NOT be greater than core.parallelism (value: {parallelism}). "
- "Will now use core.parallelism as the max task instances per query "
- "instead of specified value.",
- UserWarning,
- )
-
def _upgrade_auth_backends(self):
"""
Ensure a custom auth_backends setting contains session.
@@ -2080,19 +2062,6 @@ def make_group_other_inaccessible(file_path: str):
)
-# Historical convenience functions to access config entries
-def load_test_config():
- """Historical load_test_config."""
- warnings.warn(
- "Accessing configuration method 'load_test_config' directly from the configuration module is "
- "deprecated. Please access the configuration from the 'configuration.conf' object via "
- "'conf.load_test_config'",
- DeprecationWarning,
- stacklevel=2,
- )
- conf.load_test_config()
-
-
def get(*args, **kwargs) -> ConfigType | None:
"""Historical get."""
warnings.warn(
diff --git a/airflow/example_dags/example_python_operator.py b/airflow/example_dags/example_python_operator.py
index 7c8b27de7660e..99fbcd9719d1d 100644
--- a/airflow/example_dags/example_python_operator.py
+++ b/airflow/example_dags/example_python_operator.py
@@ -23,7 +23,6 @@
import logging
import sys
-import tempfile
import time
from pprint import pprint
@@ -37,8 +36,6 @@
PATH_TO_PYTHON_BINARY = sys.executable
-BASE_DIR = tempfile.gettempdir()
-
def x():
pass
diff --git a/airflow/exceptions.py b/airflow/exceptions.py
index b471297cd9593..0840e801a1a85 100644
--- a/airflow/exceptions.py
+++ b/airflow/exceptions.py
@@ -20,7 +20,6 @@
"""Exceptions used by Airflow."""
from __future__ import annotations
-import datetime
import warnings
from http import HTTPStatus
from typing import TYPE_CHECKING, Any, NamedTuple, Sized
@@ -28,6 +27,8 @@
from airflow.utils.trigger_rule import TriggerRule
if TYPE_CHECKING:
+ import datetime
+
from airflow.models import DAG, DagRun
diff --git a/airflow/executors/base_executor.py b/airflow/executors/base_executor.py
index b9c2991d24c58..d364d2315b16b 100644
--- a/airflow/executors/base_executor.py
+++ b/airflow/executors/base_executor.py
@@ -17,18 +17,16 @@
"""Base executor - this is the base class for all the implemented executors."""
from __future__ import annotations
-import argparse
import logging
import sys
import warnings
from collections import defaultdict
from dataclasses import dataclass, field
-from datetime import datetime
from typing import TYPE_CHECKING, Any, List, Optional, Sequence, Tuple
import pendulum
-from airflow.cli.cli_config import DefaultHelpParser, GroupCommand
+from airflow.cli.cli_config import DefaultHelpParser
from airflow.configuration import conf
from airflow.exceptions import RemovedInAirflow3Warning
from airflow.stats import Stats
@@ -38,8 +36,12 @@
PARALLELISM: int = conf.getint("core", "PARALLELISM")
if TYPE_CHECKING:
+ import argparse
+ from datetime import datetime
+
from airflow.callbacks.base_callback_sink import BaseCallbackSink
from airflow.callbacks.callback_requests import CallbackRequest
+ from airflow.cli.cli_config import GroupCommand
from airflow.models.taskinstance import TaskInstance
from airflow.models.taskinstancekey import TaskInstanceKey
diff --git a/airflow/executors/local_executor.py b/airflow/executors/local_executor.py
index 8621aa9b131f8..16aa649ee7fb6 100644
--- a/airflow/executors/local_executor.py
+++ b/airflow/executors/local_executor.py
@@ -30,8 +30,7 @@
import subprocess
from abc import abstractmethod
from multiprocessing import Manager, Process
-from multiprocessing.managers import SyncManager
-from queue import Empty, Queue
+from queue import Empty
from typing import TYPE_CHECKING, Any, Optional, Tuple
from setproctitle import getproctitle, setproctitle
@@ -43,6 +42,9 @@
from airflow.utils.state import TaskInstanceState
if TYPE_CHECKING:
+ from multiprocessing.managers import SyncManager
+ from queue import Queue
+
from airflow.executors.base_executor import CommandType
from airflow.models.taskinstance import TaskInstanceStateType
from airflow.models.taskinstancekey import TaskInstanceKey
diff --git a/airflow/jobs/scheduler_job_runner.py b/airflow/jobs/scheduler_job_runner.py
index c92ff805cb42c..3d06d509a472a 100644
--- a/airflow/jobs/scheduler_job_runner.py
+++ b/airflow/jobs/scheduler_job_runner.py
@@ -227,7 +227,6 @@ def __init__(
self.processor_agent: DagFileProcessorAgent | None = None
self.dagbag = DagBag(dag_folder=self.subdir, read_dags_from_db=True, load_op_links=False)
- self._paused_dag_without_running_dagruns: set = set()
@provide_session
def heartbeat_callback(self, session: Session = NEW_SESSION) -> None:
@@ -810,7 +809,7 @@ def _execute(self) -> int | None:
processor_timeout_seconds: int = conf.getint("core", "dag_file_processor_timeout")
processor_timeout = timedelta(seconds=processor_timeout_seconds)
- if not self._standalone_dag_processor:
+ if not self._standalone_dag_processor and not self.processor_agent:
self.processor_agent = DagFileProcessorAgent(
dag_directory=Path(self.subdir),
max_runs=self.num_times_parse_dags,
@@ -1171,17 +1170,25 @@ def _create_dag_runs(self, dag_models: Collection[DagModel], session: Session) -
# create a new one. This is so that in the next Scheduling loop we try to create new runs
# instead of falling in a loop of Integrity Error.
if (dag.dag_id, dag_model.next_dagrun) not in existing_dagruns:
- dag.create_dagrun(
- run_type=DagRunType.SCHEDULED,
- execution_date=dag_model.next_dagrun,
- state=DagRunState.QUEUED,
- data_interval=data_interval,
- external_trigger=False,
- session=session,
- dag_hash=dag_hash,
- creating_job_id=self.job.id,
- )
- active_runs_of_dags[dag.dag_id] += 1
+ try:
+ dag.create_dagrun(
+ run_type=DagRunType.SCHEDULED,
+ execution_date=dag_model.next_dagrun,
+ state=DagRunState.QUEUED,
+ data_interval=data_interval,
+ external_trigger=False,
+ session=session,
+ dag_hash=dag_hash,
+ creating_job_id=self.job.id,
+ )
+ active_runs_of_dags[dag.dag_id] += 1
+ # Exceptions like ValueError, ParamValidationError, etc. are raised by
+ # dag.create_dagrun() when dag is misconfigured. The scheduler should not
+ # crash due to misconfigured dags. We should log any exception encountered
+ # and continue to the next dag.
+ except Exception:
+ self.log.exception("Failed creating DagRun for %s", dag.dag_id)
+ continue
if self._should_update_dag_next_dagruns(
dag,
dag_model,
diff --git a/airflow/kubernetes/pre_7_4_0_compatibility/k8s_model.py b/airflow/kubernetes/pre_7_4_0_compatibility/k8s_model.py
index 8280a3265f09d..cff12d057bcad 100644
--- a/airflow/kubernetes/pre_7_4_0_compatibility/k8s_model.py
+++ b/airflow/kubernetes/pre_7_4_0_compatibility/k8s_model.py
@@ -19,8 +19,10 @@
from abc import ABC, abstractmethod
from functools import reduce
+from typing import TYPE_CHECKING
-from kubernetes.client import models as k8s
+if TYPE_CHECKING:
+ from kubernetes.client import models as k8s
class K8SModel(ABC):
diff --git a/airflow/kubernetes/pre_7_4_0_compatibility/pod_generator.py b/airflow/kubernetes/pre_7_4_0_compatibility/pod_generator.py
index 021dd36687108..73671fd25a241 100644
--- a/airflow/kubernetes/pre_7_4_0_compatibility/pod_generator.py
+++ b/airflow/kubernetes/pre_7_4_0_compatibility/pod_generator.py
@@ -27,13 +27,13 @@
from __future__ import annotations
import copy
-import datetime
import logging
import os
import secrets
import string
import warnings
from functools import reduce
+from typing import TYPE_CHECKING
import re2
from dateutil import parser
@@ -54,6 +54,9 @@
from airflow.utils.hashlib_wrapper import md5
from airflow.version import version as airflow_version
+if TYPE_CHECKING:
+ import datetime
+
log = logging.getLogger(__name__)
MAX_LABEL_LEN = 63
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index 3e27b6a795f99..cf2cf606583b9 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -222,7 +222,7 @@ def create_timetable(interval: ScheduleIntervalArg, timezone: Timezone) -> Timet
return DeltaDataIntervalTimetable(interval)
if isinstance(interval, str):
return CronDataIntervalTimetable(interval, timezone)
- raise ValueError(f"{interval!r} is not a valid schedule_interval.")
+ raise ValueError(f"{interval!r} is not a valid interval.")
def get_last_dagrun(dag_id, session, include_externally_triggered=False):
diff --git a/airflow/plugins_manager.py b/airflow/plugins_manager.py
index ef51175b532e3..7275588d52e07 100644
--- a/airflow/plugins_manager.py
+++ b/airflow/plugins_manager.py
@@ -29,19 +29,18 @@
from pathlib import Path
from typing import TYPE_CHECKING, Any, Iterable
-try:
- import importlib_metadata
-except ImportError:
- from importlib import metadata as importlib_metadata # type: ignore[no-redef]
-
-from types import ModuleType
-
from airflow import settings
from airflow.utils.entry_points import entry_points_with_dist
from airflow.utils.file import find_path_from_directory
from airflow.utils.module_loading import import_string, qualname
if TYPE_CHECKING:
+ try:
+ import importlib_metadata
+ except ImportError:
+ from importlib import metadata as importlib_metadata # type: ignore[no-redef]
+ from types import ModuleType
+
from airflow.hooks.base import BaseHook
from airflow.listeners.listener import ListenerManager
from airflow.timetables.base import Timetable
@@ -558,8 +557,10 @@ def get_plugin_info(attrs_to_dump: Iterable[str] | None = None) -> list[dict[str
elif attr in ("macros", "timetables", "hooks", "executors"):
info[attr] = [qualname(d) for d in getattr(plugin, attr)]
elif attr == "listeners":
- # listeners are always modules
- info[attr] = [d.__name__ for d in getattr(plugin, attr)]
+ # listeners may be modules or class instances
+ info[attr] = [
+ d.__name__ if inspect.ismodule(d) else qualname(d) for d in getattr(plugin, attr)
+ ]
elif attr == "appbuilder_views":
info[attr] = [
{**d, "view": qualname(d["view"].__class__) if "view" in d else None}
diff --git a/airflow/providers/amazon/aws/hooks/s3.py b/airflow/providers/amazon/aws/hooks/s3.py
index c932d6f059fb8..4af1336a13142 100644
--- a/airflow/providers/amazon/aws/hooks/s3.py
+++ b/airflow/providers/amazon/aws/hooks/s3.py
@@ -221,12 +221,12 @@ def parse_s3_url(s3url: str) -> tuple[str, str]:
elif format[0] == "https:":
temp_split = format[1].split(".")
if temp_split[0] == "s3":
- split_url = format[1].split("/")
- bucket_name = split_url[1]
- key = "/".join(split_url[2:])
+ # "https://s3.region-code.amazonaws.com/bucket-name/key-name"
+ _, bucket_name, key = format[1].split("/", 2)
elif temp_split[1] == "s3":
+ # "https://bucket-name.s3.region-code.amazonaws.com/key-name"
bucket_name = temp_split[0]
- key = "/".join(format[1].split("/")[1:])
+ key = format[1].partition("/")[-1]
else:
raise S3HookUriParseFailure(
"Please provide a bucket name using a valid virtually hosted format which should "
diff --git a/scripts/ci/installed_providers.txt b/airflow/providers/installed_providers.txt
similarity index 100%
rename from scripts/ci/installed_providers.txt
rename to airflow/providers/installed_providers.txt
diff --git a/airflow/providers/microsoft/azure/CHANGELOG.rst b/airflow/providers/microsoft/azure/CHANGELOG.rst
index 99366e3332b15..dc637f98acf54 100644
--- a/airflow/providers/microsoft/azure/CHANGELOG.rst
+++ b/airflow/providers/microsoft/azure/CHANGELOG.rst
@@ -27,6 +27,142 @@
Changelog
---------
+8.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+.. warning::
+ In this version of the provider, we have removed network_profile param from AzureContainerInstancesOperator and
+ AzureDataFactoryHook methods and AzureDataFactoryRunPipelineOperator arguments resource_group_name and factory_name
+ is now required instead of kwargs
+
+* resource_group_name and factory_name is now required argument in AzureDataFactoryHook method get_factory, update_factory,
+ create_factory, delete_factory, get_linked_service, delete_linked_service, get_dataset, delete_dataset, get_dataflow,
+ update_dataflow, create_dataflow, delete_dataflow, get_pipeline, delete_pipeline, run_pipeline, get_pipeline_run,
+ get_trigger, get_pipeline_run_status, cancel_pipeline_run, create_trigger, delete_trigger, start_trigger,
+ stop_trigger, get_adf_pipeline_run_status, cancel_pipeline_run
+* resource_group_name and factory_name is now required in AzureDataFactoryRunPipelineOperator
+* Remove class ``PipelineRunInfo`` from ``airflow.providers.microsoft.azure.hooks.data_factory``
+* Remove ``network_profile`` param from ``AzureContainerInstancesOperator``
+* Remove deprecated ``extra__azure__tenantId`` from azure_container_instance connection extras
+* Remove deprecated ``extra__azure__subscriptionId`` from azure_container_instance connection extras
+
+
+7.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+.. warning::
+ In this version of the provider, we have changed AzureFileShareHook to use azure-storage-file-share library instead
+ of azure-storage-file this change has impact on existing hook method see below for details, removed deprecated
+ extra__azure_fileshare__ prefix from connection extras param and removed protocol param from connection extras
+
+* get_conn from AzureFileShareHook return None instead FileService
+* Remove protocol param from Azure fileshare connection extras
+* Remove deprecated extra__azure_fileshare__ prefix from Azure fileshare connection extras, list_files
+* Remove share_name, directory_name param from AzureFileShareHook method check_for_directory,
+ list_directories_and_files, create_directory in favor of AzureFileShareHook share_name and directory_path param
+* AzureFileShareHook method create_share and delete_share accept kwargs from ShareServiceClient.create_share
+ and ShareServiceClient.delete_share
+* Remove share_name, directory_name, file_name param from AzureFileShareHook method get_file, get_file_to_stream
+ and load_file in favor of AzureFileShareHook share_name and file_path
+* Remove AzureFileShareHook.check_for_file method
+* Remove AzureFileShareHook.load_string, AzureFileShareHook.load_stream in favor of AzureFileShareHook.load_data
+
+.. note::
+ ``LocalToAzureDataLakeStorageOperator`` class has been removed in favor of ``LocalFilesystemToADLSOperator``
+ ``AzureDataFactoryPipelineRunStatusAsyncSensor`` class has been removed in favor of ``AzureDataFactoryPipelineRunStatusSensor``
+
+* ``Update Azure fileshare hook to use azure-storage-file-share instead of azure-storage-file (#33904)``
+* ``Remove 'AzureDataFactoryPipelineRunStatusAsyncSensor' class (#34036)``
+* ``Remove 'LocalToAzureDataLakeStorageOperator' class (#34035)``
+
+Features
+~~~~~~~~
+
+* ``feat(providers/microsoft): add AzureContainerInstancesOperator.volume as template field (#34070)``
+* ``Add DefaultAzureCredential support to AzureContainerRegistryHook (#33825)``
+* ``feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerVolumeHook (#33822)``
+
+Misc
+~~~~
+
+* ``Refactor regex in providers (#33898)``
+* ``Improve docs on AzureBatchHook DefaultAzureCredential support (#34098)``
+* ``Remove azure-storage-common from microsoft azure providers (#34038)``
+* ``Remove useless string join from providers (#33968)``
+* ``Refactor unneeded jumps in providers (#33833)``
+
+
+6.3.0
+.....
+
+Features
+~~~~~~~~
+
+* ``Add AzureBatchOperator example (#33716)``
+* ``feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerInstanceHook (#33467)``
+* ``Add DefaultAzureCredential auth for ADX service (#33627)``
+* ``feat(providers/microsoft): add DefaultAzureCredential to data_lake (#33433)``
+* ``Allow passing fully_qualified_namespace and credential to initialize Azure Service Bus Client (#33493)``
+* ``Add DefaultAzureCredential support to cosmos (#33436)``
+* ``Add DefaultAzureCredential support to AzureBatchHook (#33469)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix updating account url for WasbHook (#33457)``
+* ``Fix Azure Batch Hook instantation (#33731)``
+* ``Truncate Wasb storage account name if it's more than 24 characters (#33851)``
+* ``Remove duplicated message commit in Azure MessageHook (#33776)``
+* ``fix(providers/azure): remove json.dumps when querying AzureCosmosDBHook (#33653)``
+
+Misc
+~~~~
+
+* ``Refactor: Remove useless str() calls (#33629)``
+* ``Bump azure-kusto-data>=4.1.0 (#33598)``
+* ``Simplify conditions on len() in providers/microsoft (#33566)``
+* ``Set logging level to WARNING (#33314)``
+* ``Simplify 'X for X in Y' to 'Y' where applicable (#33453)``
+* ``Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 (#33696)``
+* ``Improve modules import in Airflow providers by some of them into a type-checking block (#33754)``
+* ``Use a single statement with multiple contexts instead of nested statements in providers (#33768)``
+* ``remove unnecessary and rewrite it using list in providers (#33763)``
+* ``Optimise Airflow DB backend usage in Azure Provider (#33750)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``Fix typos (double words and it's/its) (#33623)``
+ * ``Further improvements for provider verification (#33670)``
+ * ``Prepare docs for Aug 2023 3rd wave of Providers (#33730)``
+ * ``Move Azure examples into system tests (#33727)``
+
+6.2.4
+.....
+
+Misc
+~~~~~
+
+* ``Clean microsoft azure provider by deleting the custom prefix from conn extra fields (#30558)``
+
+6.2.3
+.....
+
+Misc
+~~~~
+
+* ``Refactor account_url use in WasbHook (#32980)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``Delete azure cosmos DB sensor example_dag (#32906)``
+ * ``Add issue link for TODO wrt Azure integration pinned dependencies (#33064)``
+
6.2.2
.....
diff --git a/airflow/providers/microsoft/azure/hooks/container_instance.py b/airflow/providers/microsoft/azure/hooks/container_instance.py
index 9a0d0ec21067f..34edbf74bb673 100644
--- a/airflow/providers/microsoft/azure/hooks/container_instance.py
+++ b/airflow/providers/microsoft/azure/hooks/container_instance.py
@@ -18,13 +18,24 @@
from __future__ import annotations
import warnings
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, cast
+from azure.common.client_factory import get_client_from_auth_file, get_client_from_json_dict
+from azure.identity import ClientSecretCredential, DefaultAzureCredential
from azure.mgmt.containerinstance import ContainerInstanceManagementClient
-from azure.mgmt.containerinstance.models import ContainerGroup
-from airflow.exceptions import AirflowProviderDeprecationWarning
+from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
+if TYPE_CHECKING:
+ from azure.mgmt.containerinstance.models import (
+ ContainerGroup,
+ ContainerPropertiesInstanceView,
+ ContainerState,
+ Event,
+ )
+
class AzureContainerInstanceHook(AzureBaseHook):
"""
@@ -47,7 +58,47 @@ class AzureContainerInstanceHook(AzureBaseHook):
def __init__(self, azure_conn_id: str = default_conn_name) -> None:
super().__init__(sdk_client=ContainerInstanceManagementClient, conn_id=azure_conn_id)
- self.connection = self.get_conn()
+
+ @cached_property
+ def connection(self):
+ return self.get_conn()
+
+ def get_conn(self) -> Any:
+ """
+ Authenticates the resource using the connection id passed during init.
+
+ :return: the authenticated client.
+ """
+ conn = self.get_connection(self.conn_id)
+ tenant = conn.extra_dejson.get("tenantId")
+
+ key_path = conn.extra_dejson.get("key_path")
+ if key_path:
+ if not key_path.endswith(".json"):
+ raise AirflowException("Unrecognised extension for key file.")
+ self.log.info("Getting connection using a JSON key file.")
+ return get_client_from_auth_file(client_class=self.sdk_client, auth_path=key_path)
+
+ key_json = conn.extra_dejson.get("key_json")
+ if key_json:
+ self.log.info("Getting connection using a JSON config.")
+ return get_client_from_json_dict(client_class=self.sdk_client, config_dict=key_json)
+
+ credential: ClientSecretCredential | DefaultAzureCredential
+ if all([conn.login, conn.password, tenant]):
+ self.log.info("Getting connection using specific credentials and subscription_id.")
+ credential = ClientSecretCredential(
+ client_id=conn.login, client_secret=conn.password, tenant_id=cast(str, tenant)
+ )
+ else:
+ self.log.info("Using DefaultAzureCredential as credential")
+ credential = DefaultAzureCredential()
+
+ subscription_id = cast(str, conn.extra_dejson.get("subscriptionId"))
+ return ContainerInstanceManagementClient(
+ credential=credential,
+ subscription_id=subscription_id,
+ )
def create_or_update(self, resource_group: str, name: str, container_group: ContainerGroup) -> None:
"""
@@ -57,7 +108,7 @@ def create_or_update(self, resource_group: str, name: str, container_group: Cont
:param name: the name of the container group
:param container_group: the properties of the container group
"""
- self.connection.container_groups.create_or_update(resource_group, name, container_group)
+ self.connection.container_groups.begin_create_or_update(resource_group, name, container_group)
def get_state_exitcode_details(self, resource_group: str, name: str) -> tuple:
"""
@@ -74,8 +125,10 @@ def get_state_exitcode_details(self, resource_group: str, name: str) -> tuple:
stacklevel=2,
)
cg_state = self.get_state(resource_group, name)
- c_state = cg_state.containers[0].instance_view.current_state
- return (c_state.state, c_state.exit_code, c_state.detail_status)
+ container = cg_state.containers[0]
+ instance_view: ContainerPropertiesInstanceView = container.instance_view # type: ignore[assignment]
+ c_state: ContainerState = instance_view.current_state # type: ignore[assignment]
+ return c_state.state, c_state.exit_code, c_state.detail_status
def get_messages(self, resource_group: str, name: str) -> list:
"""
@@ -91,8 +144,10 @@ def get_messages(self, resource_group: str, name: str) -> list:
stacklevel=2,
)
cg_state = self.get_state(resource_group, name)
- instance_view = cg_state.containers[0].instance_view
- return [event.message for event in instance_view.events]
+ container = cg_state.containers[0]
+ instance_view: ContainerPropertiesInstanceView = container.instance_view # type: ignore[assignment]
+ events: list[Event] = instance_view.events # type: ignore[assignment]
+ return [event.message for event in events]
def get_state(self, resource_group: str, name: str) -> ContainerGroup:
"""
@@ -102,7 +157,7 @@ def get_state(self, resource_group: str, name: str) -> ContainerGroup:
:param name: the name of the container group
:return: ContainerGroup
"""
- return self.connection.container_groups.get(resource_group, name, raw=False)
+ return self.connection.container_groups.get(resource_group, name)
def get_logs(self, resource_group: str, name: str, tail: int = 1000) -> list:
"""
@@ -113,7 +168,7 @@ def get_logs(self, resource_group: str, name: str, tail: int = 1000) -> list:
:param tail: the size of the tail
:return: A list of log messages
"""
- logs = self.connection.container.list_logs(resource_group, name, name, tail=tail)
+ logs = self.connection.containers.list_logs(resource_group, name, name, tail=tail)
return logs.content.splitlines(True)
def delete(self, resource_group: str, name: str) -> None:
@@ -123,7 +178,7 @@ def delete(self, resource_group: str, name: str) -> None:
:param resource_group: the name of the resource group
:param name: the name of the container group
"""
- self.connection.container_groups.delete(resource_group, name)
+ self.connection.container_groups.begin_delete(resource_group, name)
def exists(self, resource_group: str, name: str) -> bool:
"""
diff --git a/airflow/providers/microsoft/azure/operators/container_instances.py b/airflow/providers/microsoft/azure/operators/container_instances.py
index 8b5fba26cdcb1..0e8d140cda345 100644
--- a/airflow/providers/microsoft/azure/operators/container_instances.py
+++ b/airflow/providers/microsoft/azure/operators/container_instances.py
@@ -25,12 +25,12 @@
from azure.mgmt.containerinstance.models import (
Container,
ContainerGroup,
- ContainerGroupNetworkProfile,
ContainerPort,
EnvironmentVariable,
IpAddress,
ResourceRequests,
ResourceRequirements,
+ Volume as _AzureVolume,
VolumeMount,
)
from msrestazure.azure_exceptions import CloudError
@@ -44,13 +44,11 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
-
Volume = namedtuple(
"Volume",
["conn_id", "account_name", "share_name", "mount_path", "read_only"],
)
-
DEFAULT_ENVIRONMENT_VARIABLES: dict[str, str] = {}
DEFAULT_SECURED_VARIABLES: Sequence[str] = []
DEFAULT_VOLUMES: Sequence[Volume] = []
@@ -90,7 +88,6 @@ class AzureContainerInstancesOperator(BaseOperator):
:param restart_policy: Restart policy for all containers within the container group.
Possible values include: 'Always', 'OnFailure', 'Never'
:param ip_address: The IP address type of the container group.
- :param network_profile: The network profile information for a container group.
**Example**::
@@ -145,7 +142,6 @@ def __init__(
restart_policy: str = "Never",
ip_address: IpAddress | None = None,
ports: list[ContainerPort] | None = None,
- network_profile: ContainerGroupNetworkProfile | None = None,
**kwargs,
) -> None:
super().__init__(**kwargs)
@@ -183,7 +179,6 @@ def __init__(
)
self.ip_address = ip_address
self.ports = ports
- self.network_profile = network_profile
def execute(self, context: Context) -> int:
# Check name again in case it was templated.
@@ -212,7 +207,7 @@ def execute(self, context: Context) -> int:
e = EnvironmentVariable(name=key, value=value)
environment_variables.append(e)
- volumes: list[Volume | Volume] = []
+ volumes: list[_AzureVolume] = []
volume_mounts: list[VolumeMount | VolumeMount] = []
for conn_id, account_name, share_name, mount_path, read_only in self.volumes:
hook = AzureContainerVolumeHook(conn_id)
@@ -256,7 +251,6 @@ def execute(self, context: Context) -> int:
os_type=self.os_type,
tags=self.tags,
ip_address=self.ip_address,
- network_profile=self.network_profile,
)
self._ci_hook.create_or_update(self.resource_group, self.name, container_group)
diff --git a/airflow/providers/microsoft/azure/provider.yaml b/airflow/providers/microsoft/azure/provider.yaml
index d34336e49ab6a..1a45881afaefe 100644
--- a/airflow/providers/microsoft/azure/provider.yaml
+++ b/airflow/providers/microsoft/azure/provider.yaml
@@ -76,10 +76,10 @@ dependencies:
- azure-synapse-spark
- adal>=1.2.7
- azure-storage-file-datalake>=12.9.1
- # TODO: upgrade to newer versions of all the below libraries
- - azure-mgmt-containerinstance>=1.5.0,<2.0
- - azure-mgmt-datafactory>=1.0.0,<2.0
- - azure-kusto-data>=0.0.43,<0.1
+ - azure-kusto-data>=4.1.0
+ - azure-mgmt-datafactory>=2.0.0
+ - azure-mgmt-containerregistry>=8.0.0
+ - azure-mgmt-containerinstance>=9.0.0
integrations:
- integration-name: Microsoft Azure Batch
diff --git a/airflow/providers/opensearch/CHANGELOG.rst b/airflow/providers/opensearch/CHANGELOG.rst
new file mode 100644
index 0000000000000..4cb4526149fa0
--- /dev/null
+++ b/airflow/providers/opensearch/CHANGELOG.rst
@@ -0,0 +1,27 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+.. NOTE TO CONTRIBUTORS:
+ Please, only add notes to the Changelog just below the "Changelog" header when there are some breaking changes
+ and you want to add an explanation to the users on how they are supposed to deal with them.
+ The changelog is updated and maintained semi-automatically by release manager.
+
+``apache-airflow-providers-opensearch``
+
+Changelog
+---------
diff --git a/scripts/in_container/filter_out_warnings.py b/airflow/providers/opensearch/__init__.py
similarity index 68%
rename from scripts/in_container/filter_out_warnings.py
rename to airflow/providers/opensearch/__init__.py
index ebd5b0aa7cf20..13a83393a9124 100644
--- a/scripts/in_container/filter_out_warnings.py
+++ b/airflow/providers/opensearch/__init__.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env python
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,19 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-from __future__ import annotations
-
-import fileinput
-import sys
-
-suppress = False
-
-for line in fileinput.input():
- if line.startswith("warnings summary:"):
- suppress = True
- if line.startswith("All Warning errors can be found in"):
- suppress = False
- if not suppress:
- print(line, end="")
- sys.stdout.flush()
diff --git a/tests/providers/amazon/aws/triggers/test_athena.py b/airflow/providers/opensearch/hooks/__init__.py
similarity index 59%
rename from tests/providers/amazon/aws/triggers/test_athena.py
rename to airflow/providers/opensearch/hooks/__init__.py
index d18bdc1553f78..13a83393a9124 100644
--- a/tests/providers/amazon/aws/triggers/test_athena.py
+++ b/airflow/providers/opensearch/hooks/__init__.py
@@ -14,22 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from __future__ import annotations
-
-from airflow.providers.amazon.aws.triggers.athena import AthenaTrigger
-
-
-class TestAthenaTrigger:
- def test_serialize_recreate(self):
- trigger = AthenaTrigger("query_id", 1, 5, "aws connection")
-
- class_path, args = trigger.serialize()
-
- class_name = class_path.split(".")[-1]
- clazz = globals()[class_name]
- instance = clazz(**args)
-
- class_path2, args2 = instance.serialize()
-
- assert class_path == class_path2
- assert args == args2
diff --git a/airflow/providers/opensearch/hooks/opensearch.py b/airflow/providers/opensearch/hooks/opensearch.py
new file mode 100644
index 0000000000000..97907e54d52ca
--- /dev/null
+++ b/airflow/providers/opensearch/hooks/opensearch.py
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+from functools import cached_property
+from typing import Any
+
+from opensearchpy import OpenSearch, RequestsHttpConnection
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+
+
+class OpenSearchHook(BaseHook):
+ """
+ Provide a thin wrapper around the OpenSearch client.
+
+ :param: open_search_conn_id: Connection to use with Open Search
+ :param: log_query: Whether to log the query used for Open Search
+ """
+
+ conn_name_attr = "opensearch_conn_id"
+ default_conn_name = "opensearch_default"
+ conn_type = "opensearch"
+ hook_name = "OpenSearch Hook"
+
+ def __init__(self, open_search_conn_id: str, log_query: bool, **kwargs: Any):
+ super().__init__(**kwargs)
+ self.conn_id = open_search_conn_id
+ self.log_query = log_query
+
+ self.use_ssl = self.conn.extra_dejson.get("use_ssl", False)
+ self.verify_certs = self.conn.extra_dejson.get("verify_certs", False)
+ self.__SERVICE = "es"
+
+ @cached_property
+ def conn(self):
+ return self.get_connection(self.conn_id)
+
+ @cached_property
+ def client(self) -> OpenSearch:
+ """This function is intended for Operators that forward high level client objects."""
+ auth = (self.conn.login, self.conn.password)
+ client = OpenSearch(
+ hosts=[{"host": self.conn.host, "port": self.conn.port}],
+ http_auth=auth,
+ use_ssl=self.use_ssl,
+ verify_certs=self.verify_certs,
+ connection_class=RequestsHttpConnection,
+ )
+ return client
+
+ def search(self, query: dict, index_name: str, **kwargs: Any) -> Any:
+ """
+ Run a search query against the connected OpenSearch cluster.
+
+ :param: query: The query for the search against OpenSearch.
+ :param: index_name: The name of the index to search against
+ """
+ if self.log_query:
+ self.log.info("Searching %s with Query: %s", index_name, query)
+ return self.client.search(body=query, index=index_name, **kwargs)
+
+ def index(self, document: dict, index_name: str, doc_id: int, **kwargs: Any) -> Any:
+ """
+ Index a document on OpenSearch.
+
+ :param: document: A dictionary representation of the document
+ :param: index_name: the name of the index that this document will be associated with
+ :param: doc_id: the numerical identifier that will be used to identify the document on the index.
+ """
+ return self.client.index(index=index_name, id=doc_id, body=document, **kwargs)
+
+ def delete(self, index_name: str, query: dict | None = None, doc_id: int | None = None) -> Any:
+ """
+ Delete from an index by either a query or by the document id.
+
+ :param: index_name: the name of the index to delete from
+ :param: query: If deleting by query a dict representation of the query to run to
+ identify documents to delete.
+ :param: doc_id: The identifier of the document to delete.
+ """
+ if query is not None:
+ if self.log_query:
+ self.log.info("Deleting from %s using Query: %s", index_name, query)
+ return self.client.delete_by_query(index=index_name, body=query)
+ elif doc_id is not None:
+ return self.client.delete(index=index_name, id=doc_id)
+ else:
+ AirflowException("To delete a document you must include one of either a query or a document id. ")
+
+ @staticmethod
+ def get_ui_field_behaviour() -> dict[str, Any]:
+ """Returns custom UI field behaviour for Open Search Connection."""
+ return {
+ "hidden_fields": ["schema"],
+ "relabeling": {
+ "extra": "Open Search Configuration",
+ },
+ "placeholders": {
+ "extra": json.dumps(
+ {"use_ssl": True, "verify_certs": True},
+ indent=2,
+ ),
+ },
+ }
diff --git a/airflow/providers/opensearch/operators/__init__.py b/airflow/providers/opensearch/operators/__init__.py
new file mode 100644
index 0000000000000..13a83393a9124
--- /dev/null
+++ b/airflow/providers/opensearch/operators/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/opensearch/operators/opensearch.py b/airflow/providers/opensearch/operators/opensearch.py
new file mode 100644
index 0000000000000..08ab1dc0ce21b
--- /dev/null
+++ b/airflow/providers/opensearch/operators/opensearch.py
@@ -0,0 +1,194 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Any, Sequence
+
+from opensearchpy.exceptions import OpenSearchException
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.opensearch.hooks.opensearch import OpenSearchHook
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class OpenSearchQueryOperator(BaseOperator):
+ """
+ Runs a query search against a given index on an OpenSearch cluster and returns results.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:OpenSearchQueryOperator`
+
+ :param: query: A Dictionary Open Search DSL query.
+ :param: search_object: A Search object from opensearch-dsl.
+ :param: index_name: The name of the index to search for documents.
+ :param: opensearch_conn_id: opensearch connection to use
+ :param: log_query: Whether to log the query used. Defaults to True and logs query used.
+ """
+
+ template_fields: Sequence[str] = ["query"]
+
+ def __init__(
+ self,
+ *,
+ query: dict | None = None,
+ search_object: Any | None = None,
+ index_name: str | None = None,
+ opensearch_conn_id: str = "opensearch_default",
+ log_query: bool = True,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.query = query
+ self.index_name = index_name
+ self.opensearch_conn_id = opensearch_conn_id
+ self.log_query = log_query
+ self.search_object = search_object
+
+ @cached_property
+ def hook(self) -> OpenSearchHook:
+ """Gets an instance of an OpenSearchHook."""
+ return OpenSearchHook(open_search_conn_id=self.opensearch_conn_id, log_query=self.log_query)
+
+ def execute(self, context: Context) -> Any:
+ """Executes a search against a given index or a Search object on an OpenSearch Cluster."""
+ result = None
+
+ if self.query is not None:
+ if not self.query.get("query"):
+ raise AirflowException("Query input is missing required field Query in dictionary")
+ if self.index_name is None:
+ raise AirflowException("Index name is required when using the query input.")
+ try:
+ result = self.hook.search(index_name=self.index_name, query=self.query)
+ except OpenSearchException as e:
+ raise AirflowException(e)
+ elif self.search_object is not None:
+ try:
+ result = self.search_object.using(self.hook.client).execute()
+ except OpenSearchException as e:
+ raise AirflowException(e)
+ else:
+ raise AirflowException(
+ """Input missing required input of query or search_object.
+ Either query or search_object is required."""
+ )
+ return result
+
+
+class OpenSearchCreateIndexOperator(BaseOperator):
+ """
+ Create a new index on an Open Search cluster with a given index name.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:OpenSearchCreateIndexOperator`
+
+ :param: index_name: The name of the index to be created.
+ :param: index_body: A dictionary that defines index settings
+ :param: opensearch_conn_id: opensearch connection to use
+ """
+
+ def __init__(
+ self,
+ *,
+ index_name: str,
+ index_body: dict[str, Any],
+ opensearch_conn_id: str = "opensearch_default",
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.index_name = index_name
+ self.index_body = index_body
+ self.opensearch_conn_id = opensearch_conn_id
+
+ @cached_property
+ def hook(self) -> OpenSearchHook:
+ """Gets an instance of an OpenSearchHook."""
+ return OpenSearchHook(open_search_conn_id=self.opensearch_conn_id, log_query=False)
+
+ def execute(self, context: Context) -> Any:
+ """Creates an index on an Open Search cluster."""
+ try:
+ self.hook.client.indices.create(index=self.index_name, body=self.index_body)
+ except OpenSearchException as e:
+ raise AirflowException(e)
+
+
+class OpenSearchAddDocumentOperator(BaseOperator):
+ """
+ Add a new document to a given Index or overwrite an existing one.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:OpenSearchAddDocumentOperator`
+
+ :param: index_name: The name of the index to put the document.
+ :param: document: A dictionary representation of the document.
+ :param: document_id: The id for the document in the index.
+ :param: doc_class: A Document subclassed object using opensearch-dsl
+ :param: opensearch_conn_id: opensearch connection to use
+ """
+
+ def __init__(
+ self,
+ *,
+ index_name: str | None = None,
+ document: dict[str, Any] | None = None,
+ doc_id: int | None = None,
+ doc_class: Any | None = None,
+ opensearch_conn_id: str = "opensearch_default",
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.index_name = index_name
+ self.document = document
+ self.doc_id = doc_id
+ self.doc_class = doc_class
+ self.opensearch_conn_id = opensearch_conn_id
+
+ @cached_property
+ def hook(self) -> OpenSearchHook:
+ """Gets an instance of an OpenSearchHook."""
+ return OpenSearchHook(open_search_conn_id=self.opensearch_conn_id, log_query=False)
+
+ def execute(self, context: Context) -> Any:
+ """Saves a document to a given index on an OpenSearch cluster."""
+ if self.doc_class is not None:
+ try:
+ doc = self.doc_class.init(using=self.hook.client)
+ result = doc.save(using=self.hook.client)
+ except OpenSearchException as e:
+ raise AirflowException(e)
+ elif self.index_name is not None and self.document is not None and self.doc_id is not None:
+ try:
+ result = self.hook.index(
+ index_name=self.index_name, document=self.document, doc_id=self.doc_id
+ )
+ except OpenSearchException as e:
+ raise AirflowException(e)
+ else:
+ raise AirflowException(
+ "Index name, document dictionary and doc_id or a Document subclassed object is required."
+ )
+
+ return result
diff --git a/airflow/providers/opensearch/provider.yaml b/airflow/providers/opensearch/provider.yaml
new file mode 100644
index 0000000000000..270d7cd2085dd
--- /dev/null
+++ b/airflow/providers/opensearch/provider.yaml
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-opensearch
+name: Opensearch
+description: |
+ `Opensearch `.
-* Always - those are tests that should be always executed (always folder)
-* Core - for the core Airflow functionality (core folder)
-* API - Tests for the Airflow API (api and api_connexion folders)
-* CLI - Tests for the Airflow CLI (cli folder)
-* WWW - Tests for the Airflow webserver (www folder)
-* Providers - Tests for all Providers of Airflow (providers folder)
+We also have `Integration` tests that are running Integration tests with external software that is run
+via `--integration` flag in `breeze` environment.
-We also have several special kinds of tests that are not separated by packages, but they are marked with
-pytest markers. They can be found in any of those packages and they can be selected by the appropriate
-pytest custom command line options. See `TESTING.rst `_ for details but those are:
-
-* Integration - tests that require external integration images running in docker-compose
-* Quarantined - tests that are flaky and need to be fixed
-* Postgres - tests that require Postgres database. They are only run when backend is Postgres
-* MySQL - tests that require MySQL database. They are only run when backend is MySQL
+* `Integration` - tests that require external integration images running in docker-compose
Even if the types are separated, In case they share the same backend version/python version, they are
run sequentially in the same job, on the same CI machine. Each of them in a separate `docker run` command
@@ -95,15 +89,15 @@ The logic implements the following rules:
* if there are any changes to "common" provider code not belonging to any provider (usually system tests
or tests), then tests for all Providers are run
* The specific unit test type is enabled only if changed files match the expected patterns for each type
- (`API`, `CLI`, `WWW`, `Providers`). The `Always` test type is added always if any unit tests are run.
- `Providers` tests are removed if current branch is different than `main`
+ (`API`, `CLI`, `WWW`, `Providers`, `Operators` etc.). The `Always` test type is added always if any unit
+ tests are run. `Providers` tests are removed if current branch is different than `main`
* If there are no files left in sources after matching the test types and Kubernetes files,
then apparently some Core/Other files have been changed. This automatically adds all test
types to execute. This is done because changes in core might impact all the other test types.
* if `Image building` is disabled, only basic pre-commits are enabled - no 'image-depending` pre-commits
are enabled.
* If there are some setup files changed, `upgrade to newer dependencies` is enabled.
-* If docs are build, the `docs-filter-list-as-string` will determine which docs packages to build. This is based on
+* If docs are build, the `docs-list-as-string` will determine which docs packages to build. This is based on
several criteria: if any of the airflow core, charts, docker-stack, providers files or docs have changed,
then corresponding packages are build (including cross-dependent providers). If any of the core files
changed, also providers docs are built because all providers depend on airflow docs. If any of the docs
@@ -119,50 +113,50 @@ empty string means `everything`, where lack of the output means `nothing` and li
separated by spaces. This is to accommodate for the wau how outputs of this kind can be easily used by
Github Actions to pass the list of parameters to a command to execute
-| Output | Meaning of the output | Example value | List as string |
-|------------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------------------------|----------------|
-| affected-providers-list-as-string | List of providers affected when they are selectively affected. | airbyte http | * |
-| all-python-versions | List of all python versions there are available in the form of JSON array | ['3.8', '3.9', '3.10'] | |
-| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.8 3.9 3.10 | * |
-| basic-checks-only | Whether to run all static checks ("false") or only basic set of static checks ("true") | false | |
-| cache-directive | Which cache should be used for images ("registry", "local" , "disabled") | registry | |
-| debug-resources | Whether resources usage should be printed during parallel job execution ("true"/ "false") | false | |
-| default-branch | Which branch is default for the build ("main" for main branch, "v2-4-test" for 2.4 line etc.) | main | |
-| default-constraints-branch | Which branch is default for the build ("constraints-main" for main branch, "constraints-2-4" etc.) | constraints-main | |
-| default-helm-version | Which Helm version to use as default | v3.9.4 | |
-| default-kind-version | Which Kind version to use as default | v0.16.0 | |
-| default-kubernetes-version | Which Kubernetes version to use as default | v1.25.2 | |
-| default-mssql-version | Which MsSQL version to use as default | 2017-latest | |
-| default-mysql-version | Which MySQL version to use as default | 5.7 | |
-| default-postgres-version | Which Postgres version to use as default | 10 | |
-| default-python-version | Which Python version to use as default | 3.8 | |
-| docs-build | Whether to build documentation ("true"/"false") | true | |
-| docs-filter-list-as-string | What filter to apply to docs building - based on which documentation packages should be built | --package-filter apache-airflow -package-filter-helm-chart | |
-| full-tests-needed | Whether this build runs complete set of tests or only subset (for faster PR builds) | false | |
-| helm-version | Which Helm version to use for tests | v3.9.4 | |
-| image-build | Whether CI image build is needed | true | |
-| kind-version | Which Kind version to use for tests | v0.16.0 | |
-| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.8-v1.25.2 3.9-v1.26.4 | * |
-| kubernetes-versions | All Kubernetes versions to use for tests as JSON array | ['v1.25.2'] | |
-| kubernetes-versions-list-as-string | All Kubernetes versions to use for tests as space-separated string | v1.25.2 | * |
-| mssql-exclude | Which versions of MsSQL to exclude for tests as JSON array | [] | |
-| mssql-versions | Which versions of MsSQL to use for tests as JSON array | ['2017-latest'] | |
-| mysql-exclude | Which versions of MySQL to exclude for tests as JSON array | [] | |
-| mysql-versions | Which versions of MySQL to use for tests as JSON array | ['5.7'] | |
-| needs-api-codegen | Whether "api-codegen" are needed to run ("true"/"false") | true | |
-| needs-api-tests | Whether "api-tests" are needed to run ("true"/"false") | true | |
-| needs-helm-tests | Whether Helm tests are needed to run ("true"/"false") | true | |
-| needs-javascript-scans | Whether javascript CodeQL scans should be run ("true"/"false") | true | |
-| needs-python-scans | Whether Python CodeQL scans should be run ("true"/"false") | true | |
-| parallel-test-types-list-as-string | Which test types should be run for unit tests | API Always Providers\[amazon\] Providers\[-amazon\] | * |
-| postgres-exclude | Which versions of Postgres to exclude for tests as JSON array | [] | |
-| postgres-versions | Which versions of Postgres to use for tests as JSON array | ['10'] | |
-| python-versions | Which versions of Python to use for tests as JSON array | ['3.8'] | |
-| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.8 | * |
-| run-kubernetes-tests | Whether Kubernetes tests should be run ("true"/"false") | true | |
-| run-tests | Whether unit tests should be run ("true"/"false") | true | |
-| run-www-tests | Whether WWW tests should be run ("true"/"false") | true | |
-| skip-pre-commits | Which pre-commits should be skipped during the static-checks run | true | |
-| skip-provider-tests | When provider tests should be skipped (on non-main branch or when no provider changes detected) | true | |
-| sqlite-exclude | Which versions of Sqlite to exclude for tests as JSON array | [] | |
-| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (might be true/false or commit hash) | false | |
+| Output | Meaning of the output | Example value | List as string |
+|------------------------------------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------------|----------------|
+| affected-providers-list-as-string | List of providers affected when they are selectively affected. | airbyte http | * |
+| all-python-versions | List of all python versions there are available in the form of JSON array | ['3.8', '3.9', '3.10'] | |
+| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.8 3.9 3.10 | * |
+| basic-checks-only | Whether to run all static checks ("false") or only basic set of static checks ("true") | false | |
+| cache-directive | Which cache should be used for images ("registry", "local" , "disabled") | registry | |
+| debug-resources | Whether resources usage should be printed during parallel job execution ("true"/ "false") | false | |
+| default-branch | Which branch is default for the build ("main" for main branch, "v2-4-test" for 2.4 line etc.) | main | |
+| default-constraints-branch | Which branch is default for the build ("constraints-main" for main branch, "constraints-2-4" etc.) | constraints-main | |
+| default-helm-version | Which Helm version to use as default | v3.9.4 | |
+| default-kind-version | Which Kind version to use as default | v0.16.0 | |
+| default-kubernetes-version | Which Kubernetes version to use as default | v1.25.2 | |
+| default-mssql-version | Which MsSQL version to use as default | 2017-latest | |
+| default-mysql-version | Which MySQL version to use as default | 5.7 | |
+| default-postgres-version | Which Postgres version to use as default | 10 | |
+| default-python-version | Which Python version to use as default | 3.8 | |
+| docs-build | Whether to build documentation ("true"/"false") | true | |
+| docs-list-as-string | What filter to apply to docs building - based on which documentation packages should be built | apache-airflow helm-chart google | |
+| full-tests-needed | Whether this build runs complete set of tests or only subset (for faster PR builds) | false | |
+| helm-version | Which Helm version to use for tests | v3.9.4 | |
+| image-build | Whether CI image build is needed | true | |
+| kind-version | Which Kind version to use for tests | v0.16.0 | |
+| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.8-v1.25.2 3.9-v1.26.4 | * |
+| kubernetes-versions | All Kubernetes versions to use for tests as JSON array | ['v1.25.2'] | |
+| kubernetes-versions-list-as-string | All Kubernetes versions to use for tests as space-separated string | v1.25.2 | * |
+| mssql-exclude | Which versions of MsSQL to exclude for tests as JSON array | [] | |
+| mssql-versions | Which versions of MsSQL to use for tests as JSON array | ['2017-latest'] | |
+| mysql-exclude | Which versions of MySQL to exclude for tests as JSON array | [] | |
+| mysql-versions | Which versions of MySQL to use for tests as JSON array | ['5.7'] | |
+| needs-api-codegen | Whether "api-codegen" are needed to run ("true"/"false") | true | |
+| needs-api-tests | Whether "api-tests" are needed to run ("true"/"false") | true | |
+| needs-helm-tests | Whether Helm tests are needed to run ("true"/"false") | true | |
+| needs-javascript-scans | Whether javascript CodeQL scans should be run ("true"/"false") | true | |
+| needs-python-scans | Whether Python CodeQL scans should be run ("true"/"false") | true | |
+| parallel-test-types-list-as-string | Which test types should be run for unit tests | API Always Providers\[amazon\] Providers\[-amazon\] | * |
+| postgres-exclude | Which versions of Postgres to exclude for tests as JSON array | [] | |
+| postgres-versions | Which versions of Postgres to use for tests as JSON array | ['10'] | |
+| python-versions | Which versions of Python to use for tests as JSON array | ['3.8'] | |
+| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.8 | * |
+| run-kubernetes-tests | Whether Kubernetes tests should be run ("true"/"false") | true | |
+| run-tests | Whether unit tests should be run ("true"/"false") | true | |
+| run-www-tests | Whether WWW tests should be run ("true"/"false") | true | |
+| skip-pre-commits | Which pre-commits should be skipped during the static-checks run | true | |
+| skip-provider-tests | When provider tests should be skipped (on non-main branch or when no provider changes detected) | true | |
+| sqlite-exclude | Which versions of Sqlite to exclude for tests as JSON array | [] | |
+| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (might be true/false or commit hash) | false | |
diff --git a/dev/breeze/setup.cfg b/dev/breeze/setup.cfg
index 95b8df62e32db..617e855659bf8 100644
--- a/dev/breeze/setup.cfg
+++ b/dev/breeze/setup.cfg
@@ -36,6 +36,7 @@ classifiers =
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
+ Programming Language :: Python :: 3.11
project_urls =
Documentation=https://github.com/apache/airflow/BREEZE.rst
Bug Tracker=https://github.com/apache/airflow/issues
@@ -47,7 +48,9 @@ project_urls =
[options]
zip_safe = False
include_package_data = True
-python_requires = ~=3.8
+# Mainly because of distutils deprecation and some packages not being compatible with it, we should
+# Limit airflow to < 3.12 until those dependencies are ready and until we can support Python 3.12
+python_requires = ~=3.8,<3.12
package_dir=
=src
packages = find:
@@ -65,8 +68,8 @@ install_requires =
pyyaml
PyGithub
requests
- rich>=12.6.0
- rich-click>=1.5
+ rich>=13.6.0
+ rich-click>=1.7.0
gitpython
twine
wheel
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index a71723f30bc5c..92c01501187ed 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -34,7 +34,6 @@
DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
DOCKER_DEFAULT_PLATFORM,
MOUNT_SELECTED,
- get_available_documentation_packages,
)
from airflow_breeze.params.build_ci_params import BuildCiParams
from airflow_breeze.params.doc_build_params import DocBuildParams
@@ -43,6 +42,7 @@
from airflow_breeze.utils.cache import read_from_cache_file
from airflow_breeze.utils.coertions import one_or_none_set
from airflow_breeze.utils.common_options import (
+ argument_short_doc_packages_with_providers_index,
option_airflow_constraints_reference,
option_airflow_extras,
option_answer,
@@ -50,6 +50,7 @@
option_builder,
option_celery_broker,
option_celery_flower,
+ option_database_isolation,
option_db_reset,
option_downgrade_sqlalchemy,
option_dry_run,
@@ -71,19 +72,23 @@
option_platform_single,
option_postgres_version,
option_python,
+ option_run_db_tests_only,
+ option_skip_db_tests,
+ option_standalone_dag_processor,
option_upgrade_boto,
option_use_airflow_version,
option_use_packages_from_dist,
option_verbose,
)
from airflow_breeze.utils.console import get_console
-from airflow_breeze.utils.custom_param_types import BetterChoice, NotVerifiedBetterChoice
+from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
check_docker_resources,
get_env_variables_for_docker_commands,
get_extra_docker_flags,
perform_environment_checks,
)
+from airflow_breeze.utils.general_utils import expand_all_providers
from airflow_breeze.utils.path_utils import (
AIRFLOW_SOURCES_ROOT,
cleanup_python_generated_files,
@@ -165,6 +170,8 @@ def run(self):
@option_include_mypy_volume
@option_upgrade_boto
@option_downgrade_sqlalchemy
+@option_run_db_tests_only
+@option_skip_db_tests
@option_verbose
@option_dry_run
@option_github_repository
@@ -172,12 +179,14 @@ def run(self):
@option_executor
@option_celery_broker
@option_celery_flower
+@option_standalone_dag_processor
+@option_database_isolation
@click.argument("extra-args", nargs=-1, type=click.UNPROCESSED)
def shell(
python: str,
backend: str,
builder: str,
- integration: tuple[str],
+ integration: tuple[str, ...],
postgres_version: str,
mysql_version: str,
mssql_version: str,
@@ -202,6 +211,10 @@ def shell(
extra_args: tuple,
upgrade_boto: bool,
downgrade_sqlalchemy: bool,
+ run_db_tests_only: bool,
+ skip_db_tests: bool,
+ standalone_dag_processor: bool,
+ database_isolation: bool,
):
"""Enter breeze environment. this is the default command use when no other is selected."""
if get_verbose() or get_dry_run():
@@ -241,6 +254,10 @@ def shell(
celery_flower=celery_flower,
upgrade_boto=upgrade_boto,
downgrade_sqlalchemy=downgrade_sqlalchemy,
+ standalone_dag_processor=standalone_dag_processor,
+ database_isolation=database_isolation,
+ run_db_tests_only=run_db_tests_only,
+ skip_db_tests=skip_db_tests,
)
sys.exit(result.returncode)
@@ -286,11 +303,13 @@ def shell(
@option_executor
@option_celery_broker
@option_celery_flower
+@option_standalone_dag_processor
+@option_database_isolation
def start_airflow(
python: str,
backend: str,
builder: str,
- integration: tuple[str],
+ integration: tuple[str, ...],
postgres_version: str,
load_example_dags: bool,
load_default_connections: bool,
@@ -314,6 +333,8 @@ def start_airflow(
executor: str,
celery_broker: str,
celery_flower: bool,
+ standalone_dag_processor: bool,
+ database_isolation: bool,
):
"""
Enter breeze environment and starts all Airflow components in the tmux session.
@@ -358,6 +379,8 @@ def start_airflow(
executor=executor,
celery_broker=celery_broker,
celery_flower=celery_flower,
+ standalone_dag_processor=standalone_dag_processor,
+ database_isolation=database_isolation,
)
sys.exit(result.returncode)
@@ -365,11 +388,12 @@ def start_airflow(
@main.command(name="build-docs")
@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
+@argument_short_doc_packages_with_providers_index
@option_builder
@click.option(
"--package-filter",
help="List of packages to consider.",
- type=NotVerifiedBetterChoice(get_available_documentation_packages()),
+ type=str,
multiple=True,
)
@click.option(
@@ -387,12 +411,13 @@ def start_airflow(
@option_verbose
@option_dry_run
def build_docs(
+ short_doc_packages: tuple[str, ...],
docs_only: bool,
spellcheck_only: bool,
builder: str,
clean_build: bool,
one_pass_only: bool,
- package_filter: tuple[str],
+ package_filter: tuple[str, ...],
github_repository: str,
):
"""
@@ -417,6 +442,7 @@ def build_docs(
spellcheck_only=spellcheck_only,
one_pass_only=one_pass_only,
skip_environment_initialization=True,
+ short_doc_packages=expand_all_providers(short_doc_packages),
)
extra_docker_flags = get_extra_docker_flags(MOUNT_SELECTED)
env = get_env_variables_for_docker_commands(params)
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
index 9ef4f8ca96d63..3ffc016474c92 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
@@ -32,39 +32,61 @@
DEVELOPER_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"breeze": [
{
- "name": "Basic flags",
+ "name": "Execution mode",
"options": [
"--python",
+ "--integration",
+ "--standalone-dag-processor",
+ "--database-isolation",
+ ],
+ },
+ {
+ "name": "Database",
+ "options": [
"--backend",
"--postgres-version",
"--mysql-version",
"--mssql-version",
- "--integration",
- "--forward-credentials",
"--db-reset",
- "--max-time",
+ ],
+ },
+ {
+ "name": "Build CI image (before entering shell)",
+ "options": [
"--github-repository",
"--builder",
],
},
+ {
+ "name": "Other options",
+ "options": [
+ "--forward-credentials",
+ "--max-time",
+ ],
+ },
],
"breeze shell": [
{
- "name": "Basic flags",
+ "name": "Execution mode",
"options": [
"--python",
+ "--integration",
+ "--standalone-dag-processor",
+ "--database-isolation",
+ ],
+ },
+ {
+ "name": "Database",
+ "options": [
"--backend",
"--postgres-version",
"--mysql-version",
"--mssql-version",
- "--integration",
- "--forward-credentials",
"--db-reset",
- "--max-time",
],
},
{
- "name": "Choosing executor",
+ "name": "Choose executor",
"options": [
"--executor",
"--celery-broker",
@@ -72,7 +94,7 @@
],
},
{
- "name": "Building image before entering shell",
+ "name": "Build CI image (before entering shell)",
"options": [
"--force-build",
"--platform",
@@ -106,6 +128,20 @@
"--downgrade-sqlalchemy",
],
},
+ {
+ "name": "DB test flags",
+ "options": [
+ "--run-db-tests-only",
+ "--skip-db-tests",
+ ],
+ },
+ {
+ "name": "Other options",
+ "options": [
+ "--forward-credentials",
+ "--max-time",
+ ],
+ },
],
"breeze compile-www-assets": [
{
@@ -117,18 +153,24 @@
],
"breeze start-airflow": [
{
- "name": "Basic flags",
+ "name": "Execution mode",
"options": [
"--python",
+ "--platform",
+ "--integration",
+ "--standalone-dag-processor",
+ "--database-isolation",
"--load-example-dags",
"--load-default-connections",
+ ],
+ },
+ {
+ "name": "Database",
+ "options": [
"--backend",
- "--platform",
"--postgres-version",
"--mysql-version",
"--mssql-version",
- "--integration",
- "--forward-credentials",
"--db-reset",
],
},
@@ -148,7 +190,7 @@
],
},
{
- "name": "Building image before entering shell",
+ "name": "Build CI image (before entering shell)",
"options": [
"--force-build",
"--image-tag",
@@ -172,6 +214,12 @@
"--package-format",
],
},
+ {
+ "name": "Other options",
+ "options": [
+ "--forward-credentials",
+ ],
+ },
],
"breeze exec": [
{"name": "Drops in the interactive shell of active airflow container"},
diff --git a/dev/breeze/src/airflow_breeze/commands/main_command.py b/dev/breeze/src/airflow_breeze/commands/main_command.py
index cf29cb3450072..a9340e07c75ac 100644
--- a/dev/breeze/src/airflow_breeze/commands/main_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/main_command.py
@@ -33,6 +33,7 @@
option_answer,
option_backend,
option_builder,
+ option_database_isolation,
option_db_reset,
option_dry_run,
option_forward_credentials,
@@ -43,6 +44,7 @@
option_mysql_version,
option_postgres_version,
option_python,
+ option_standalone_dag_processor,
option_verbose,
)
from airflow_breeze.utils.confirm import Answer, user_confirm
@@ -106,6 +108,8 @@ def get_command(self, ctx: Context, cmd_name: str):
@option_mysql_version
@option_mssql_version
@option_integration
+@option_database_isolation
+@option_standalone_dag_processor
@option_forward_credentials
@option_db_reset
@option_max_time
@@ -166,7 +170,7 @@ def check_for_python_emulation():
def check_for_rosetta_environment():
- if sys.platform != "darwin" or platform.processor() == "i386":
+ if platform != "darwin" or platform.processor() == "i386":
return
from inputimeout import TimeoutOccurred, inputimeout
@@ -276,7 +280,7 @@ def cleanup(all: bool):
)
elif given_answer == Answer.QUIT:
sys.exit(0)
- get_console().print(f"Removing build cache dir ${BUILD_CACHE_DIR}")
+ get_console().print(f"Removing build cache dir {BUILD_CACHE_DIR}")
given_answer = user_confirm("Are you sure with the removal?")
if given_answer == Answer.YES:
if not get_dry_run():
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 254697863183d..4421dddcfc2a5 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -26,7 +26,6 @@
from copy import deepcopy
from datetime import datetime
from pathlib import Path
-from re import Pattern, match
from typing import IO, Generator, NamedTuple
import click
@@ -43,7 +42,6 @@
MOUNT_ALL,
MOUNT_SELECTED,
MULTI_PLATFORM,
- get_available_documentation_packages,
)
from airflow_breeze.params.shell_params import ShellParams
from airflow_breeze.utils.add_back_references import (
@@ -52,10 +50,13 @@
from airflow_breeze.utils.ci_group import ci_group
from airflow_breeze.utils.common_options import (
argument_packages,
- argument_packages_plus_all_providers,
+ argument_short_doc_packages,
+ argument_short_doc_packages_with_providers_index,
option_airflow_constraints_mode_ci,
+ option_airflow_constraints_mode_update,
option_airflow_constraints_reference,
option_airflow_extras,
+ option_airflow_site_directory,
option_answer,
option_commit_sha,
option_debug_resources,
@@ -80,13 +81,14 @@
)
from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import Output, get_console
-from airflow_breeze.utils.custom_param_types import BetterChoice, NotVerifiedBetterChoice
+from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
check_remote_ghcr_io_commands,
get_env_variables_for_docker_commands,
get_extra_docker_flags,
perform_environment_checks,
)
+from airflow_breeze.utils.general_utils import expand_all_providers
from airflow_breeze.utils.github import download_constraints_file, get_active_airflow_versions
from airflow_breeze.utils.parallel import (
GenericRegexpProgressMatcher,
@@ -118,7 +120,7 @@
run_command,
run_compile_www_assets,
)
-from airflow_breeze.utils.shared_options import get_dry_run, get_forced_answer
+from airflow_breeze.utils.shared_options import get_dry_run, get_forced_answer, get_verbose
from airflow_breeze.utils.suspended_providers import get_suspended_provider_ids
option_debug_release_management = click.option(
@@ -513,7 +515,7 @@ def generate_constraints(
def _get_all_providers_in_dist(
- filename_prefix: str, filename_pattern: Pattern[str]
+ filename_prefix: str, filename_pattern: re.Pattern[str]
) -> Generator[str, None, None]:
for file in DIST_DIR.glob(f"{filename_prefix}*.tar.gz"):
matched = filename_pattern.match(file.name)
@@ -768,30 +770,99 @@ def alias_image(image_from: str, image_to: str):
)
+def run_docs_publishing(
+ package_name: str,
+ airflow_site_directory: str,
+ override_versioned: bool,
+ verbose: bool,
+ output: Output | None,
+) -> tuple[int, str]:
+ builder = PublishDocsBuilder(package_name=package_name, output=output, verbose=verbose)
+ builder.publish(override_versioned=override_versioned, airflow_site_dir=airflow_site_directory)
+ return (
+ 0,
+ f"Docs published: {package_name}",
+ )
+
+
+PUBLISHING_DOCS_PROGRESS_MATCHER = r"Publishing docs|Copy directory"
+
+
+def run_publish_docs_in_parallel(
+ package_list: list[str],
+ airflow_site_directory: str,
+ override_versioned: bool,
+ include_success_outputs: bool,
+ parallelism: int,
+ skip_cleanup: bool,
+ debug_resources: bool,
+):
+ """Run docs publishing in parallel"""
+ with ci_group("Publishing docs for packages"):
+ all_params = [f"Publishing docs {package_name}" for package_name in package_list]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=GenericRegexpProgressMatcher(
+ regexp=PUBLISHING_DOCS_PROGRESS_MATCHER, lines_to_search=6
+ ),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ run_docs_publishing,
+ kwds={
+ "package_name": package_name,
+ "airflow_site_directory": airflow_site_directory,
+ "override_versioned": override_versioned,
+ "output": outputs[index],
+ "verbose": get_verbose(),
+ },
+ )
+ for index, package_name in enumerate(package_list)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All package documentation published.",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ summarize_on_ci=SummarizeAfter.NO_SUMMARY,
+ )
+
+
@release_management.command(
name="publish-docs",
help="Command to publish generated documentation to airflow-site",
)
@click.option("-s", "--override-versioned", help="Overrides versioned directories.", is_flag=True)
-@click.option(
- "-a",
- "--airflow-site-directory",
- envvar="AIRFLOW_SITE_DIRECTORY",
- help="Local directory path of cloned airflow-site repo.",
- required=True,
-)
+@option_airflow_site_directory
+@argument_short_doc_packages_with_providers_index
@click.option(
"--package-filter",
- help="List of packages to consider.",
- type=NotVerifiedBetterChoice(get_available_documentation_packages()),
+ help="List of packages to consider. You can use the full names like apache-airflow-providers-, "
+ "the short hand names or the glob pattern matching the full package name. "
+ "The list of short hand names can be found in --help output",
+ type=str,
multiple=True,
)
+@option_run_in_parallel
+@option_parallelism
+@option_debug_resources
+@option_include_success_outputs
+@option_skip_cleanup
@option_verbose
@option_dry_run
def publish_docs(
override_versioned: bool,
airflow_site_directory: str,
- package_filter: tuple[str],
+ short_doc_packages: tuple[str, ...],
+ package_filter: tuple[str, ...],
+ run_in_parallel: bool,
+ parallelism: int,
+ debug_resources: bool,
+ include_success_outputs: bool,
+ skip_cleanup: bool,
):
"""Publishes documentation to airflow-site."""
if not os.path.isdir(airflow_site_directory):
@@ -800,37 +871,41 @@ def publish_docs(
"Provide the path of cloned airflow-site repo\n"
)
- available_packages = get_available_packages()
- package_filters = package_filter
-
- current_packages = process_package_filters(available_packages, package_filters)
+ current_packages = process_package_filters(
+ get_available_packages(), package_filter, expand_all_providers(short_doc_packages)
+ )
print(f"Publishing docs for {len(current_packages)} package(s)")
for pkg in current_packages:
print(f" - {pkg}")
print()
- for package_name in current_packages:
- builder = PublishDocsBuilder(package_name=package_name)
- builder.publish(override_versioned=override_versioned, airflow_site_dir=airflow_site_directory)
+ if run_in_parallel:
+ run_publish_docs_in_parallel(
+ package_list=current_packages,
+ parallelism=parallelism,
+ skip_cleanup=skip_cleanup,
+ debug_resources=debug_resources,
+ include_success_outputs=True,
+ airflow_site_directory=airflow_site_directory,
+ override_versioned=override_versioned,
+ )
+ else:
+ for package_name in current_packages:
+ run_docs_publishing(
+ package_name, airflow_site_directory, override_versioned, verbose=get_verbose(), output=None
+ )
@release_management.command(
name="add-back-references",
help="Command to add back references for documentation to make it backward compatible.",
)
-@click.option(
- "-a",
- "--airflow-site-directory",
- envvar="AIRFLOW_SITE_DIRECTORY",
- type=click.Path(exists=True, file_okay=False, dir_okay=True, resolve_path=True),
- help="Local directory path of cloned airflow-site repo.",
- required=True,
-)
-@argument_packages_plus_all_providers
+@option_airflow_site_directory
+@argument_short_doc_packages
@option_verbose
@option_dry_run
def add_back_references(
airflow_site_directory: str,
- packages_plus_all_providers: tuple[str],
+ short_doc_packages: tuple[str, ...],
):
"""Adds back references for documentation generated by build-docs and publish-docs"""
site_path = Path(airflow_site_directory)
@@ -840,16 +915,12 @@ def add_back_references(
"Provide the path of cloned airflow-site repo\n"
)
sys.exit(1)
- if not packages_plus_all_providers:
+ if not short_doc_packages:
get_console().print(
"\n[error]You need to specify at least one package to generate back references for\n"
)
sys.exit(1)
- packages = list(packages_plus_all_providers)
- if "all-providers" in packages_plus_all_providers:
- packages.remove("all-providers")
- packages.extend(get_available_documentation_packages(only_providers=True, short_version=True))
- start_generating_back_references(site_path, packages)
+ start_generating_back_references(site_path, list(expand_all_providers(short_doc_packages)))
@release_management.command(
@@ -902,7 +973,7 @@ def release_prod_images(
perform_environment_checks()
check_remote_ghcr_io_commands()
rebuild_or_pull_ci_image_if_needed(command_params=ShellParams(python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION))
- if not match(r"^\d*\.\d*\.\d*$", airflow_version):
+ if not re.match(r"^\d*\.\d*\.\d*$", airflow_version):
get_console().print(
f"[warning]Skipping latest image tagging as this is a pre-release version: {airflow_version}"
)
@@ -1070,12 +1141,10 @@ def get_prs_for_package(package_id: str) -> list[int]:
if skip_line:
# Skip first "....." header
skip_line = False
- continue
- if line.strip() == current_release_version:
+ elif line.strip() == current_release_version:
extract_prs = True
skip_line = True
- continue
- if extract_prs:
+ elif extract_prs:
if len(line) > 1 and all(c == "." for c in line.strip()):
# Header for next version reached
break
@@ -1158,9 +1227,7 @@ class ProviderPRInfo(NamedTuple):
pull_requests: dict[int, PullRequest.PullRequest | Issue.Issue] = {}
with Progress(console=get_console(), disable=disable_progress) as progress:
task = progress.add_task(f"Retrieving {len(all_prs)} PRs ", total=len(all_prs))
- pr_list = list(all_prs)
- for i in range(len(pr_list)):
- pr_number = pr_list[i]
+ for pr_number in all_prs:
progress.console.print(
f"Retrieving PR#{pr_number}: https://github.com/apache/airflow/pull/{pr_number}"
)
@@ -1312,14 +1379,35 @@ def checkout_constraint_tag_and_reset_branch(constraints_repo: Path, airflow_ver
get_console().print(f"[info]The hash commit of the tag:[/] {result.stdout}")
-def modify_single_file_constraints(constraints_file: Path, updated_constraints: tuple[str]) -> bool:
+def update_comment(content: str, comment_file: Path) -> str:
+ comment_text = comment_file.read_text()
+ if comment_text in content:
+ return content
+ comment_lines = comment_text.splitlines()
+ content_lines = content.splitlines()
+ updated_lines: list[str] = []
+ updated = False
+ for line in content_lines:
+ if not line.strip().startswith("#") and not updated:
+ updated_lines.extend(comment_lines)
+ updated = True
+ updated_lines.append(line)
+ return "".join(f"{line}\n" for line in updated_lines)
+
+
+def modify_single_file_constraints(
+ constraints_file: Path, updated_constraints: tuple[str, ...] | None, comment_file: Path | None
+) -> bool:
constraint_content = constraints_file.read_text()
original_content = constraint_content
- for constraint in updated_constraints:
- package, version = constraint.split("==")
- constraint_content = re.sub(
- rf"^{package}==.*$", f"{package}=={version}", constraint_content, flags=re.MULTILINE
- )
+ if comment_file:
+ constraint_content = update_comment(constraint_content, comment_file)
+ if updated_constraints:
+ for constraint in updated_constraints:
+ package, version = constraint.split("==")
+ constraint_content = re.sub(
+ rf"^{package}==.*$", f"{package}=={version}", constraint_content, flags=re.MULTILINE
+ )
if constraint_content != original_content:
if not get_dry_run():
constraints_file.write_text(constraint_content)
@@ -1330,12 +1418,26 @@ def modify_single_file_constraints(constraints_file: Path, updated_constraints:
return False
-def modify_all_constraint_files(constraints_repo: Path, updated_constraint: tuple[str]) -> bool:
+def modify_all_constraint_files(
+ constraints_repo: Path,
+ updated_constraint: tuple[str, ...] | None,
+ comit_file: Path | None,
+ airflow_constrains_mode: str | None,
+) -> bool:
get_console().print("[info]Updating constraints files:[/]")
modified = False
- for constraints_file in constraints_repo.glob("constraints-*.txt"):
+ select_glob = "constraints-*.txt"
+ if airflow_constrains_mode == "constraints":
+ select_glob = "constraints-[0-9.]*.txt"
+ elif airflow_constrains_mode == "constraints-source-providers":
+ select_glob = "constraints-source-providers-[0-9.]*.txt"
+ elif airflow_constrains_mode == "constraints-no-providers":
+ select_glob = "constraints-no-providers-[0-9.]*.txt"
+ else:
+ raise RuntimeError(f"Invalid airflow-constraints-mode: {airflow_constrains_mode}")
+ for constraints_file in constraints_repo.glob(select_glob):
get_console().print(f"[info]Updating {constraints_file.name}")
- if modify_single_file_constraints(constraints_file, updated_constraint):
+ if modify_single_file_constraints(constraints_file, updated_constraint, comit_file):
modified = True
return modified
@@ -1406,11 +1508,20 @@ def push_constraints_and_tag(constraints_repo: Path, remote_name: str, airflow_v
)
@click.option(
"--updated-constraint",
- required=True,
+ required=False,
envvar="UPDATED_CONSTRAINT",
multiple=True,
help="Constraints to be set - in the form of `package==version`. Can be repeated",
)
+@click.option(
+ "--comment-file",
+ required=False,
+ type=click.Path(file_okay=True, dir_okay=False, path_type=Path, exists=True),
+ envvar="COMMENT_FILE",
+ help="File containing comment to be added to the constraint "
+ "file before the first package (if not added yet).",
+)
+@option_airflow_constraints_mode_update
@option_verbose
@option_dry_run
@option_answer
@@ -1419,8 +1530,13 @@ def update_constraints(
remote_name: str,
airflow_versions: str,
commit_message: str,
- updated_constraint: tuple[str],
+ airflow_constraints_mode: str | None,
+ updated_constraint: tuple[str, ...] | None,
+ comment_file: Path | None,
) -> None:
+ if not updated_constraint and not comment_file:
+ get_console().print("[error]You have to provide one of --updated-constraint or --comment-file[/]")
+ sys.exit(1)
airflow_versions_array = airflow_versions.split(",")
if not airflow_versions_array:
get_console().print("[error]No airflow versions specified - you provided empty string[/]")
@@ -1435,7 +1551,9 @@ def update_constraints(
fetch_remote(constraints_repo, remote_name)
for airflow_version in airflow_versions_array:
checkout_constraint_tag_and_reset_branch(constraints_repo, airflow_version)
- if modify_all_constraint_files(constraints_repo, updated_constraint):
+ if modify_all_constraint_files(
+ constraints_repo, updated_constraint, comment_file, airflow_constraints_mode
+ ):
if confirm_modifications(constraints_repo):
commit_constraints_and_tag(constraints_repo, airflow_version, commit_message)
push_constraints_and_tag(constraints_repo, remote_name, airflow_version)
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
index 43321586b41be..a9ff84f5d6a9a 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py
@@ -170,6 +170,16 @@
"--airflow-site-directory",
],
},
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
],
"breeze release-management add-back-references": [
{
@@ -218,10 +228,22 @@
"name": "Update constraints flags",
"options": [
"--constraints-repo",
+ "--commit-message",
"--remote-name",
+ ],
+ },
+ {
+ "name": "Selection criteria",
+ "options": [
"--airflow-versions",
- "--commit-message",
+ "--airflow-constraints-mode",
+ ],
+ },
+ {
+ "name": "Action to perform",
+ "options": [
"--updated-constraint",
+ "--comment-file",
],
},
],
diff --git a/dev/breeze/src/airflow_breeze/commands/sbom_commands.py b/dev/breeze/src/airflow_breeze/commands/sbom_commands.py
index 10ac14755af1c..8fc30faf72237 100644
--- a/dev/breeze/src/airflow_breeze/commands/sbom_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/sbom_commands.py
@@ -17,18 +17,20 @@
from __future__ import annotations
+import json
+import sys
from pathlib import Path
import click
from airflow_breeze.global_constants import (
+ AIRFLOW_PYTHON_COMPATIBILITY_MATRIX,
ALL_HISTORICAL_PYTHON_VERSIONS,
- DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
PROVIDER_DEPENDENCIES,
)
from airflow_breeze.utils.cdxgen import (
SbomApplicationJob,
- build_providers_base_image,
+ build_all_airflow_versions_base_image,
get_cdxgen_port_mapping,
get_requirements_for_provider,
)
@@ -45,12 +47,17 @@
option_skip_cleanup,
option_verbose,
)
+from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import get_console
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import perform_environment_checks
-from airflow_breeze.utils.github import get_active_airflow_versions
-from airflow_breeze.utils.parallel import ShowLastLineProgressMatcher, check_async_run_results, run_with_pool
-from airflow_breeze.utils.path_utils import AIRFLOW_TMP_DIR_PATH
+from airflow_breeze.utils.parallel import (
+ DockerBuildxProgressMatcher,
+ ShowLastLineProgressMatcher,
+ check_async_run_results,
+ run_with_pool,
+)
+from airflow_breeze.utils.path_utils import AIRFLOW_TMP_DIR_PATH, PROVIDER_METADATA_JSON_FILE_PATH
from airflow_breeze.utils.shared_options import get_dry_run
@@ -80,10 +87,10 @@ def sbom():
@sbom.command(name="update-sbom-information", help="Update SBOM information in airflow-site project.")
@click.option(
- "--airflow-site-dir",
+ "--airflow-site-directory",
type=click.Path(file_okay=False, dir_okay=True, path_type=Path, exists=True),
required=True,
- envvar="AIRFLOW_SITE_DIR",
+ envvar="AIRFLOW_SITE_DIRECTORY",
help="Directory where airflow-site directory is located.",
)
@click.option(
@@ -113,7 +120,7 @@ def sbom():
@option_dry_run
@option_answer
def update_sbom_information(
- airflow_site_dir: Path,
+ airflow_site_directory: Path,
airflow_version: str | None,
python: str | None,
include_provider_dependencies: bool,
@@ -146,21 +153,23 @@ def update_sbom_information(
jobs_to_run: list[SbomApplicationJob] = []
- apache_airflow_dir = airflow_site_dir / "docs-archive" / "apache-airflow"
+ apache_airflow_directory = airflow_site_directory / "docs-archive" / "apache-airflow"
for airflow_v in airflow_versions:
- airflow_version_dir = apache_airflow_dir / airflow_v
+ airflow_version_dir = apache_airflow_directory / airflow_v
if not airflow_version_dir.exists():
get_console().print(f"[warning]The {airflow_version_dir} does not exist. Skipping")
continue
destination_dir = airflow_version_dir / "sbom"
- destination_dir.mkdir(parents=True, exist_ok=True)
if destination_dir.exists():
if not force:
get_console().print(f"[warning]The {destination_dir} already exists. Skipping")
continue
else:
get_console().print(f"[warning]The {destination_dir} already exists. Forcing update")
+
+ destination_dir.mkdir(parents=True, exist_ok=True)
+
get_console().print(f"[info]Attempting to update sbom for {airflow_v}.")
get_console().print(f"[success]The {destination_dir} exists. Proceeding.")
for python_version in python_versions:
@@ -185,7 +194,9 @@ def update_sbom_information(
parallelism = min(parallelism, len(jobs_to_run))
get_console().print(f"[info]Running {len(jobs_to_run)} jobs in parallel")
with ci_group(f"Generating SBoMs for {airflow_versions}:{python_versions}"):
- all_params = [f"CI {job.airflow_version}:{job.python_version}" for job in jobs_to_run]
+ all_params = [
+ f"Generate SBoMs for {job.airflow_version}:{job.python_version}" for job in jobs_to_run
+ ]
with run_with_pool(
parallelism=parallelism,
all_params=all_params,
@@ -216,7 +227,7 @@ def update_sbom_information(
produce_sbom_for_application_via_cdxgen_server(job, output=None)
for airflow_v in airflow_versions:
- airflow_version_dir = apache_airflow_dir / airflow_v
+ airflow_version_dir = apache_airflow_directory / airflow_v
destination_dir = airflow_version_dir / "sbom"
destination_index_path = destination_dir / "index.html"
get_console().print(f"[info]Generating index for {destination_dir}")
@@ -230,43 +241,218 @@ def update_sbom_information(
)
-@sbom.command(name="generate-provider-requirements", help="Generate requirements for selected provider.")
-@click.option(
- "--airflow-version", type=str, required=False, help="Airflow version to use to generate the requirements"
-)
-@click.option(
- "--python",
- type=BetterChoice(ALL_HISTORICAL_PYTHON_VERSIONS),
- default=DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
- required=False,
- help="Python version to generate the requirements for",
-)
+@sbom.command(name="build-all-airflow-images", help="Generate images with airflow versions pre-installed")
+@option_historical_python_version
+@option_verbose
+@option_dry_run
+@option_answer
+@option_run_in_parallel
+@option_parallelism
+@option_debug_resources
+@option_include_success_outputs
+@option_skip_cleanup
+def build_all_airflow_images(
+ python: str,
+ run_in_parallel: bool,
+ parallelism: int,
+ debug_resources: bool,
+ include_success_outputs: bool,
+ skip_cleanup: bool,
+):
+ if python is None:
+ python_versions = ALL_HISTORICAL_PYTHON_VERSIONS
+ else:
+ python_versions = [python]
+
+ if run_in_parallel:
+ parallelism = min(parallelism, len(python_versions))
+ get_console().print(f"[info]Running {len(python_versions)} jobs in parallel")
+ with ci_group(f"Building all airflow base images for python: {python_versions}"):
+ all_params = [
+ f"Building all airflow base image for python: {python_version}"
+ for python_version in python_versions
+ ]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=DockerBuildxProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ build_all_airflow_versions_base_image,
+ kwds={
+ "python_version": python_version,
+ "confirm": False,
+ "output": outputs[index],
+ },
+ )
+ for (index, python_version) in enumerate(python_versions)
+ ]
+ check_async_run_results(
+ results=results,
+ success="All airflow base images were built successfully",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+ else:
+ for python_version in python_versions:
+ build_all_airflow_versions_base_image(
+ python_version=python_version,
+ confirm=False,
+ output=None,
+ )
+
+
+@sbom.command(name="generate-providers-requirements", help="Generate requirements for selected provider.")
+@option_historical_python_version
@click.option(
"--provider-id",
type=BetterChoice(list(PROVIDER_DEPENDENCIES.keys())),
- required=True,
- help="Provider to generate the requirements for",
+ required=False,
+ help="Provider id to generate the requirements for",
)
@click.option(
- "--provider-version", type=str, required=False, help="Provider version to generate the requirements for"
+ "--provider-version",
+ type=str,
+ required=False,
+ help="Provider version to generate the requirements for i.e `2.1.0`. `latest` is also a supported value "
+ "to account for the most recent version of the provider",
)
@option_verbose
@option_dry_run
@option_answer
-def generate_provider_requirements(
- airflow_version: str | None,
+@option_run_in_parallel
+@option_parallelism
+@option_debug_resources
+@option_include_success_outputs
+@option_skip_cleanup
+@click.option(
+ "--force",
+ is_flag=True,
+ help="Force update providers requirements even if they already exist.",
+)
+def generate_providers_requirements(
python: str,
- provider_id: str,
+ provider_id: str | None,
provider_version: str | None,
+ run_in_parallel: bool,
+ parallelism: int,
+ debug_resources: bool,
+ include_success_outputs: bool,
+ skip_cleanup: bool,
+ force: bool,
):
perform_environment_checks()
- if airflow_version is None:
- airflow_version = get_active_airflow_versions(confirm=False)[-1]
- get_console().print(f"[info]Using {airflow_version} as airflow version")
- build_providers_base_image(airflow_version=airflow_version, python_version=python)
- get_requirements_for_provider(
- provider_id=provider_id,
- provider_version=provider_version,
- airflow_version=airflow_version,
- python_version=python,
- )
+
+ if python is None:
+ python_versions = ALL_HISTORICAL_PYTHON_VERSIONS
+ else:
+ python_versions = [python]
+
+ with open(PROVIDER_METADATA_JSON_FILE_PATH) as f:
+ provider_metadata = json.load(f)
+
+ if provider_id is None:
+ if provider_version is not None and provider_version != "latest":
+ get_console().print(
+ "[error] You cannot pin the version of the providers if you generate the requirements for "
+ "all historical or latest versions. --provider-version needs to be unset when you pass None "
+ "or latest to --provider-id"
+ )
+ sys.exit(1)
+ provider_ids = provider_metadata.keys()
+ else:
+ provider_ids = [provider_id]
+
+ if provider_version is None:
+ user_confirm(
+ f"You are about to generate providers requirements for all historical versions for "
+ f"{len(provider_ids)} provider(s) based on `provider_metadata.json` file. "
+ f"Do you want to proceed?",
+ quit_allowed=False,
+ default_answer=Answer.YES,
+ )
+
+ providers_info = []
+ for provider_id in provider_ids:
+ if provider_version is not None:
+ if provider_version == "latest":
+ # Only the latest version for each provider
+ p_version, info = list(provider_metadata[provider_id].items())[-1]
+ else:
+ # Specified providers version
+ info = provider_metadata[provider_id][provider_version]
+ p_version = provider_version
+
+ airflow_version = info["associated_airflow_version"]
+
+ providers_info += [
+ (provider_id, p_version, python_version, airflow_version)
+ for python_version in AIRFLOW_PYTHON_COMPATIBILITY_MATRIX[airflow_version]
+ if python_version in python_versions
+ ]
+ else:
+ # All historical providers' versions
+ providers_info += [
+ (
+ provider_id,
+ p_version,
+ python_version,
+ info["associated_airflow_version"],
+ )
+ for (p_version, info) in provider_metadata[provider_id].items()
+ for python_version in AIRFLOW_PYTHON_COMPATIBILITY_MATRIX[info["associated_airflow_version"]]
+ if python_version in python_versions
+ ]
+
+ if run_in_parallel:
+ parallelism = min(parallelism, len(providers_info))
+ get_console().print(f"[info]Running {len(providers_info)} jobs in parallel")
+ with ci_group(f"Generating provider requirements for {providers_info}"):
+ all_params = [
+ f"Generate provider requirements for {provider_id} version {provider_version} python "
+ f"{python_version}"
+ for (provider_id, provider_version, python_version, _) in providers_info
+ ]
+ with run_with_pool(
+ parallelism=parallelism,
+ all_params=all_params,
+ debug_resources=debug_resources,
+ progress_matcher=ShowLastLineProgressMatcher(),
+ ) as (pool, outputs):
+ results = [
+ pool.apply_async(
+ get_requirements_for_provider,
+ kwds={
+ "provider_id": provider_id,
+ "airflow_version": airflow_version,
+ "provider_version": provider_version,
+ "python_version": python_version,
+ "force": force,
+ "output": outputs[index],
+ },
+ )
+ for (
+ index,
+ (provider_id, provider_version, python_version, airflow_version),
+ ) in enumerate(providers_info)
+ ]
+ check_async_run_results(
+ results=results,
+ success="Providers requirements were generated successfully",
+ outputs=outputs,
+ include_success_outputs=include_success_outputs,
+ skip_cleanup=skip_cleanup,
+ )
+ else:
+ for provider_id, provider_version, python_version, airflow_version in providers_info:
+ get_requirements_for_provider(
+ provider_id=provider_id,
+ provider_version=provider_version,
+ airflow_version=airflow_version,
+ python_version=python_version,
+ force=force,
+ output=None,
+ )
diff --git a/dev/breeze/src/airflow_breeze/commands/sbom_commands_config.py b/dev/breeze/src/airflow_breeze/commands/sbom_commands_config.py
index 14dbcbff38e78..ce17e4a258e7b 100644
--- a/dev/breeze/src/airflow_breeze/commands/sbom_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/sbom_commands_config.py
@@ -20,7 +20,8 @@
"name": "SBOM commands",
"commands": [
"update-sbom-information",
- "generate-provider-requirements",
+ "build-all-airflow-images",
+ "generate-providers-requirements",
],
}
@@ -29,7 +30,7 @@
{
"name": "Update SBOM information flags",
"options": [
- "--airflow-site-dir",
+ "--airflow-site-directory",
"--airflow-version",
"--python",
"--include-provider-dependencies",
@@ -47,14 +48,42 @@
],
},
],
- "breeze sbom generate-provider-requirements": [
+ "breeze sbom build-all-airflow-images": [
+ {
+ "name": "Generate all airflow images flags",
+ "options": [
+ "--python",
+ ],
+ },
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
+ ],
+ "breeze sbom generate-providers-requirements": [
{
"name": "Generate provider requirements flags",
"options": [
- "--airflow-version",
"--python",
"--provider-id",
"--provider-version",
+ "--force",
+ ],
+ },
+ {
+ "name": "Parallel running",
+ "options": [
+ "--run-in-parallel",
+ "--parallelism",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
],
},
],
diff --git a/dev/breeze/src/airflow_breeze/commands/setup_commands.py b/dev/breeze/src/airflow_breeze/commands/setup_commands.py
index 61e1f5ac7b57c..24a3d2e8a2522 100644
--- a/dev/breeze/src/airflow_breeze/commands/setup_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/setup_commands.py
@@ -269,6 +269,32 @@ def dict_hash(dictionary: dict[str, Any]) -> str:
return dhash.hexdigest()
+def is_short_flag(opt):
+ return len(opt) == 2 and (not opt.startswith("--"))
+
+
+def validate_params_for_command(command_params, command):
+ options_command_map = {}
+ is_duplicate_found = False
+ if "params" in command_params:
+ for param in command_params["params"]:
+ name = param["name"]
+ for opt in param["opts"]:
+ if is_short_flag(opt):
+ if opt not in options_command_map:
+ options_command_map[opt] = [[command, name]]
+ else:
+ # same flag used in same command
+ get_console().print(
+ f"[error] {opt} short flag has duplicate short hand commands under command(s): "
+ f"{'breeze ' + command} for parameters "
+ f"{options_command_map[opt][0][1]} and {name}\n"
+ )
+ options_command_map[opt][0][1] = name
+ is_duplicate_found = True
+ return is_duplicate_found
+
+
def get_command_hash_export() -> str:
import rich_click
@@ -281,6 +307,9 @@ def get_command_hash_export() -> str:
commands_dict = the_context_dict["command"]["commands"]
options = rich_click.rich_click.OPTION_GROUPS
for command in sorted(commands_dict.keys()):
+ duplicate_found = validate_params_for_command(commands_dict[command], command)
+ if duplicate_found:
+ sys.exit(1)
current_command_dict = commands_dict[command]
current_command_hash_dict = {
"command": current_command_dict,
@@ -288,7 +317,13 @@ def get_command_hash_export() -> str:
}
if "commands" in current_command_dict:
subcommands = current_command_dict["commands"]
+ duplicate_found_subcommand = False
for subcommand in sorted(subcommands.keys()):
+ duplicate_found = validate_params_for_command(
+ commands_dict[command]["commands"][subcommand], command + " " + subcommand
+ )
+ if duplicate_found:
+ duplicate_found_subcommand = True
subcommand_click_dict = subcommands[subcommand]
try:
subcommand_rich_click_dict = options[f"breeze {command} {subcommand}"]
@@ -307,10 +342,12 @@ def get_command_hash_export() -> str:
"rich_click_options": subcommand_rich_click_dict,
}
hashes.append(f"{command}:{subcommand}:{dict_hash(final_dict)}")
+ if duplicate_found_subcommand:
+ sys.exit(1)
hashes.append(f"{command}:{dict_hash(current_command_hash_dict)}")
else:
hashes.append(f"{command}:{dict_hash(current_command_hash_dict)}")
- return "\n".join(hashes) + "\n"
+ return "".join(f"{h}\n" for h in hashes)
def write_to_shell(command_to_execute: str, script_path: str, force_setup: bool) -> bool:
@@ -367,11 +404,9 @@ def remove_autogenerated_code(script_path: str):
for line in lines:
if line == START_LINE:
pass_through = False
- continue
- if line.startswith(END_LINE):
+ elif line.startswith(END_LINE):
pass_through = True
- continue
- if pass_through:
+ elif pass_through:
new_lines.append(line)
Path(script_path).write_text("".join(new_lines))
@@ -392,9 +427,8 @@ def get_commands() -> list[str]:
content = COMMAND_HASH_FILE_PATH.read_text()
for line in content.splitlines():
strip_line = line.strip()
- if strip_line == "" or strip_line.startswith("#"):
- continue
- results.append(":".join(strip_line.split(":")[:-1]))
+ if strip_line and not strip_line.startswith("#"):
+ results.append(strip_line.rpartition(":")[0])
return results
@@ -410,11 +444,9 @@ def get_command_hash_dict(hash_file_content: str) -> dict[str, str]:
results = {}
for line in hash_file_content.splitlines():
strip_line = line.strip()
- if not strip_line or strip_line.startswith("#"):
- continue
- command = ":".join(strip_line.split(":")[:-1])
- the_hash = strip_line.split(":")[-1]
- results[command] = the_hash
+ if strip_line and not strip_line.startswith("#"):
+ command, the_hash = strip_line.rsplit(":", 1)
+ results[command] = the_hash
return results
@@ -484,17 +516,12 @@ def regenerate_help_images_for_all_commands(commands: tuple[str, ...], check_onl
env=env,
)
for command in commands_list:
- if command == "main":
- continue
-
- subcommands = command.split(":")
- env["RECORD_BREEZE_TITLE"] = f"Command: {' '.join(subcommands)}"
- env["RECORD_BREEZE_OUTPUT_FILE"] = str(BREEZE_IMAGES_DIR / f"output_{'_'.join(subcommands)}.svg")
- env["RECORD_BREEZE_UNIQUE_ID"] = f"breeze-{'-'.join(subcommands)}"
- run_command(
- ["breeze", *subcommands, "--help"],
- env=env,
- )
+ if command != "main":
+ subcommands = command.split(":")
+ env["RECORD_BREEZE_TITLE"] = f"Command: {' '.join(subcommands)}"
+ env["RECORD_BREEZE_OUTPUT_FILE"] = str(BREEZE_IMAGES_DIR / f"output_{'_'.join(subcommands)}.svg")
+ env["RECORD_BREEZE_UNIQUE_ID"] = f"breeze-{'-'.join(subcommands)}"
+ run_command(["breeze", *subcommands, "--help"], env=env)
if regenerate_all_commands:
COMMAND_HASH_FILE_PATH.write_text(new_hash_text_dump)
get_console().print(f"\n[info]New hash of breeze commands written in {COMMAND_HASH_FILE_PATH}\n")
diff --git a/dev/breeze/src/airflow_breeze/commands/setup_commands_config.py b/dev/breeze/src/airflow_breeze/commands/setup_commands_config.py
index 463305a716118..1da86bb56a9b7 100644
--- a/dev/breeze/src/airflow_breeze/commands/setup_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/setup_commands_config.py
@@ -23,6 +23,7 @@
"self-upgrade",
"cleanup",
"config",
+ "check-all-params-in-groups",
"regenerate-command-images",
"command-hash-export",
"version",
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index 606bdee7ed8bf..d26d77c4f86bc 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -27,8 +27,6 @@
from airflow_breeze.commands.ci_image_commands import rebuild_or_pull_ci_image_if_needed
from airflow_breeze.global_constants import (
ALLOWED_HELM_TEST_PACKAGES,
- ALLOWED_TEST_TYPE_CHOICES,
- all_selective_test_types,
)
from airflow_breeze.params.build_prod_params import BuildProdParams
from airflow_breeze.params.shell_params import ShellParams
@@ -36,10 +34,13 @@
from airflow_breeze.utils.click_utils import BreezeGroup
from airflow_breeze.utils.common_options import (
option_backend,
+ option_collect_only,
option_db_reset,
option_debug_resources,
option_downgrade_sqlalchemy,
option_dry_run,
+ option_enable_coverage,
+ option_excluded_parallel_test_types,
option_github_repository,
option_image_name,
option_image_tag_for_running,
@@ -48,17 +49,26 @@
option_mount_sources,
option_mssql_version,
option_mysql_version,
+ option_parallel_test_types,
option_parallelism,
option_postgres_version,
option_python,
+ option_remove_arm_packages,
+ option_run_db_tests_only,
option_run_in_parallel,
option_skip_cleanup,
+ option_skip_db_tests,
+ option_skip_docker_compose_down,
+ option_skip_provider_tests,
+ option_test_timeout,
+ option_test_type,
option_upgrade_boto,
option_use_airflow_version,
+ option_use_xdist,
option_verbose,
)
from airflow_breeze.utils.console import Output, get_console
-from airflow_breeze.utils.custom_param_types import BetterChoice, NotVerifiedBetterChoice
+from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.docker_command_utils import (
get_env_variables_for_docker_commands,
perform_environment_checks,
@@ -71,7 +81,11 @@
run_with_pool,
)
from airflow_breeze.utils.path_utils import FILES_DIR, cleanup_python_generated_files
-from airflow_breeze.utils.run_tests import file_name_from_test_type, run_docker_compose_tests
+from airflow_breeze.utils.run_tests import (
+ file_name_from_test_type,
+ generate_args_for_pytest,
+ run_docker_compose_tests,
+)
from airflow_breeze.utils.run_utils import get_filesystem_type, run_command
from airflow_breeze.utils.suspended_providers import get_suspended_providers_folders
@@ -138,7 +152,7 @@ def docker_compose_tests(
TEST_PROGRESS_REGEXP = r"tests/.*|.*=====.*"
-PERCENT_TEST_PROGRESS_REGEXP = r"^tests/.*\[[ \d%]*\].*"
+PERCENT_TEST_PROGRESS_REGEXP = r"^tests/.*\[[ \d%]*\].*|^\..*\[[ \d%]*\].*"
def _run_test(
@@ -152,14 +166,11 @@ def _run_test(
) -> tuple[int, str]:
env_variables = get_env_variables_for_docker_commands(exec_shell_params)
env_variables["RUN_TESTS"] = "true"
- if test_timeout:
- env_variables["TEST_TIMEOUT"] = str(test_timeout)
if db_reset:
env_variables["DB_RESET"] = "true"
env_variables["TEST_TYPE"] = exec_shell_params.test_type
env_variables["COLLECT_ONLY"] = str(exec_shell_params.collect_only).lower()
env_variables["REMOVE_ARM_PACKAGES"] = str(exec_shell_params.remove_arm_packages).lower()
- env_variables["SKIP_PROVIDER_TESTS"] = str(exec_shell_params.skip_provider_tests).lower()
env_variables["SUSPENDED_PROVIDERS_FOLDERS"] = " ".join(get_suspended_providers_folders()).strip()
if "[" in exec_shell_params.test_type and not exec_shell_params.test_type.startswith("Providers"):
get_console(output=output).print(
@@ -177,6 +188,7 @@ def _run_test(
compose_project_name,
"down",
"--remove-orphans",
+ "--volumes",
]
run_command(down_cmd, env=env_variables, output=output, check=False)
run_cmd = [
@@ -190,6 +202,22 @@ def _run_test(
"--rm",
"airflow",
]
+ run_cmd.extend(
+ generate_args_for_pytest(
+ test_type=exec_shell_params.test_type,
+ test_timeout=test_timeout,
+ skip_provider_tests=exec_shell_params.skip_provider_tests,
+ skip_db_tests=exec_shell_params.skip_db_tests,
+ run_db_tests_only=exec_shell_params.run_db_tests_only,
+ backend=exec_shell_params.backend,
+ use_xdist=exec_shell_params.use_xdist,
+ enable_coverage=exec_shell_params.enable_coverage,
+ collect_only=exec_shell_params.collect_only,
+ parallelism=exec_shell_params.parallelism,
+ parallel_test_types_list=exec_shell_params.parallel_test_types_list,
+ helm_test_package=None,
+ )
+ )
run_cmd.extend(list(extra_pytest_args))
try:
remove_docker_networks(networks=[f"{compose_project_name}_default"])
@@ -259,6 +287,27 @@ def _run_tests_in_pool(
skip_cleanup: bool,
skip_docker_compose_down: bool,
):
+ if not tests_to_run:
+ return
+ # this should be hard-coded as we want to have very specific sequence of tests
+ # Heaviest tests go first and lightest tests go last. This way we can maximise parallelism as the
+ # lightest tests will continue to complete and new light tests will get added while the heavy
+ # tests are still running. We are only adding here test types that take more than 2 minutes to run
+ # on a fast machine in parallel
+ sorting_order = [
+ "Providers",
+ "Providers[-amazon,google]",
+ "Core",
+ "WWW",
+ "CLI",
+ "Other",
+ "Serialization",
+ "Always",
+ "PythonVenv",
+ ]
+ sort_key = {item: i for i, item in enumerate(sorting_order)}
+ # Put the test types in the order we want them to run
+ tests_to_run = sorted(tests_to_run, key=lambda x: (sort_key.get(x, len(sorting_order)), x))
escaped_tests = [test.replace("[", "\\[") for test in tests_to_run]
with ci_group(f"Testing {' '.join(escaped_tests)}"):
all_params = [f"{test_type}" for test_type in tests_to_run]
@@ -300,7 +349,6 @@ def _run_tests_in_pool(
def run_tests_in_parallel(
exec_shell_params: ShellParams,
- parallel_test_types_list: list[str],
extra_pytest_args: tuple,
db_reset: bool,
test_timeout: int,
@@ -311,7 +359,7 @@ def run_tests_in_parallel(
skio_docker_compose_down: bool,
) -> None:
_run_tests_in_pool(
- tests_to_run=parallel_test_types_list,
+ tests_to_run=exec_shell_params.parallel_test_types_list,
parallelism=parallelism,
exec_shell_params=exec_shell_params,
extra_pytest_args=extra_pytest_args,
@@ -324,9 +372,28 @@ def run_tests_in_parallel(
)
+def _verify_parallelism_parameters(
+ excluded_parallel_test_types: str, run_db_tests_only: bool, run_in_parallel: bool, use_xdist: bool
+):
+ if excluded_parallel_test_types and not (run_in_parallel or use_xdist):
+ get_console().print(
+ "\n[error]You can only specify --excluded-parallel-test-types when --run-in-parallel or "
+ "--use-xdist are set[/]\n"
+ )
+ sys.exit(1)
+ if use_xdist and run_in_parallel:
+ get_console().print("\n[error]You can only specify one of --use-xdist, --run-in-parallel[/]\n")
+ sys.exit(1)
+ if use_xdist and run_db_tests_only:
+ get_console().print("\n[error]You can only specify one of --use-xdist, --run-db-tests-only[/]\n")
+ sys.exit(1)
+
+
@group_for_testing.command(
name="tests",
- help="Run the specified unit test targets.",
+ help="Run the specified unit tests. This is a low level testing command that allows you to run "
+ "various kind of tests subset with a number of options. You can also use dedicated commands such"
+ "us db_tests, non_db_tests, integration_tests for more opinionated test suite execution.",
context_settings=dict(
ignore_unknown_options=True,
allow_extra_args=True,
@@ -341,89 +408,174 @@ def run_tests_in_parallel(
@option_image_tag_for_running
@option_use_airflow_version
@option_mount_sources
-@click.option(
- "--test-type",
- help="Type of test to run. With Providers, you can specify tests of which providers "
- "should be run: `Providers[airbyte,http]` or "
- "excluded from the full test suite: `Providers[-amazon,google]`",
- default="All",
- envvar="TEST_TYPE",
- type=NotVerifiedBetterChoice(ALLOWED_TEST_TYPE_CHOICES),
-)
-@click.option(
- "--test-timeout",
- help="Test timeout. Set the pytest setup, execution and teardown timeouts to this value",
- default=60,
- envvar="TEST_TIMEOUT",
- type=IntRange(min=0),
- show_default=True,
-)
+@option_test_type
+@option_test_timeout
+@option_run_db_tests_only
+@option_skip_db_tests
@option_db_reset
@option_run_in_parallel
@option_parallelism
@option_skip_cleanup
@option_debug_resources
@option_include_success_outputs
-@click.option(
- "--parallel-test-types",
- help="Space separated list of test types used for testing in parallel.",
- default=" ".join(all_selective_test_types()) + " PlainAsserts",
- show_default=True,
- envvar="PARALLEL_TEST_TYPES",
-)
+@option_parallel_test_types
+@option_excluded_parallel_test_types
@option_upgrade_boto
@option_downgrade_sqlalchemy
-@click.option(
- "--collect-only",
- help="Collect tests only, do not run them.",
- is_flag=True,
- envvar="COLLECT_ONLY",
-)
-@click.option(
- "--remove-arm-packages",
- help="Removes arm packages from the image to test if ARM collection works",
- is_flag=True,
- envvar="REMOVE_ARM_PACKAGES",
+@option_collect_only
+@option_remove_arm_packages
+@option_skip_docker_compose_down
+@option_use_xdist
+@option_skip_provider_tests
+@option_enable_coverage
+@option_verbose
+@option_dry_run
+@option_github_repository
+@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
+def command_for_tests(**kwargs):
+ _run_test_command(**kwargs)
+
+
+@group_for_testing.command(
+ name="db-tests",
+ help="Run all (default) or specified DB-bound unit tests. This is a dedicated command that only runs "
+ "DB tests and it runs them in parallel via splitting tests by test types into separate "
+ "containers with separate database started for each container.",
+ context_settings=dict(
+ ignore_unknown_options=False,
+ allow_extra_args=False,
+ ),
)
-@click.option(
- "--skip-docker-compose-down",
- help="Skips running docker-compose down after tests",
- is_flag=True,
- envvar="SKIP_DOCKER_COMPOSE_DOWN",
+@option_python
+@option_backend
+@option_postgres_version
+@option_mysql_version
+@option_mssql_version
+@option_image_tag_for_running
+@option_use_airflow_version
+@option_mount_sources
+@option_test_timeout
+@option_parallelism
+@option_skip_cleanup
+@option_debug_resources
+@option_include_success_outputs
+@option_parallel_test_types
+@option_excluded_parallel_test_types
+@option_upgrade_boto
+@option_downgrade_sqlalchemy
+@option_collect_only
+@option_remove_arm_packages
+@option_skip_docker_compose_down
+@option_skip_provider_tests
+@option_enable_coverage
+@option_verbose
+@option_dry_run
+@option_github_repository
+def command_for_db_tests(**kwargs):
+ _run_test_command(
+ integration=(),
+ run_in_parallel=True,
+ use_xdist=False,
+ skip_db_tests=False,
+ run_db_tests_only=True,
+ test_type="Default",
+ db_reset=True,
+ extra_pytest_args=(),
+ **kwargs,
+ )
+
+
+@group_for_testing.command(
+ name="non-db-tests",
+ help="Run all (default) or specified Non-DB unit tests. This is a dedicated command that only"
+ "runs Non-DB tests and it runs them in parallel via pytest-xdist in single container, "
+ "with `none` backend set.",
+ context_settings=dict(
+ ignore_unknown_options=False,
+ allow_extra_args=False,
+ ),
)
+@option_python
+@option_image_tag_for_running
+@option_use_airflow_version
+@option_mount_sources
+@option_test_timeout
+@option_parallelism
+@option_skip_cleanup
+@option_debug_resources
+@option_include_success_outputs
+@option_parallel_test_types
+@option_excluded_parallel_test_types
+@option_upgrade_boto
+@option_downgrade_sqlalchemy
+@option_collect_only
+@option_remove_arm_packages
+@option_skip_docker_compose_down
+@option_skip_provider_tests
+@option_enable_coverage
@option_verbose
@option_dry_run
@option_github_repository
-@click.argument("extra_pytest_args", nargs=-1, type=click.UNPROCESSED)
-def command_for_tests(
- python: str,
+def command_for_non_db_tests(**kwargs):
+ _run_test_command(
+ integration=(),
+ run_in_parallel=False,
+ use_xdist=True,
+ skip_db_tests=True,
+ run_db_tests_only=False,
+ test_type="Default",
+ db_reset=False,
+ backend="none",
+ extra_pytest_args=(),
+ **kwargs,
+ )
+
+
+def _run_test_command(
+ *,
backend: str,
- postgres_version: str,
- mysql_version: str,
- mssql_version: str,
- integration: tuple,
- test_type: str,
- test_timeout: int,
+ collect_only: bool,
db_reset: bool,
- image_tag: str | None,
- use_airflow_version: str | None,
- run_in_parallel: bool,
- parallelism: int,
- skip_cleanup: bool,
debug_resources: bool,
+ downgrade_sqlalchemy: bool,
+ enable_coverage: bool,
+ excluded_parallel_test_types: str,
+ extra_pytest_args: tuple,
+ github_repository: str,
+ image_tag: str | None,
include_success_outputs: bool,
- parallel_test_types: str,
+ integration: tuple[str, ...],
mount_sources: str,
- extra_pytest_args: tuple,
- upgrade_boto: bool,
- downgrade_sqlalchemy: bool,
- collect_only: bool,
+ parallel_test_types: str,
+ parallelism: int,
+ python: str,
remove_arm_packages: bool,
- github_repository: str,
+ run_db_tests_only: bool,
+ run_in_parallel: bool,
+ skip_cleanup: bool,
+ skip_db_tests: bool,
skip_docker_compose_down: bool,
+ skip_provider_tests: bool,
+ test_timeout: int,
+ test_type: str,
+ upgrade_boto: bool,
+ use_airflow_version: str | None,
+ use_xdist: bool,
+ mssql_version: str = "",
+ mysql_version: str = "",
+ postgres_version: str = "",
):
docker_filesystem = get_filesystem_type("/var/lib/docker")
get_console().print(f"Docker filesystem: {docker_filesystem}")
+ _verify_parallelism_parameters(
+ excluded_parallel_test_types, run_db_tests_only, run_in_parallel, use_xdist
+ )
+ test_list = parallel_test_types.split(" ")
+ excluded_test_list = excluded_parallel_test_types.split(" ")
+ if excluded_test_list:
+ test_list = [test for test in test_list if test not in excluded_test_list]
+ if skip_provider_tests or "Providers" in excluded_test_list:
+ test_list = [test for test in test_list if not test.startswith("Providers")]
exec_shell_params = ShellParams(
python=python,
backend=backend,
@@ -441,15 +593,26 @@ def command_for_tests(
collect_only=collect_only,
remove_arm_packages=remove_arm_packages,
github_repository=github_repository,
+ run_db_tests_only=run_db_tests_only,
+ skip_db_tests=skip_db_tests,
+ use_xdist=use_xdist,
+ enable_coverage=enable_coverage,
+ parallelism=parallelism,
+ skip_provider_tests=skip_provider_tests,
+ parallel_test_types_list=test_list,
)
rebuild_or_pull_ci_image_if_needed(command_params=exec_shell_params)
cleanup_python_generated_files()
perform_environment_checks()
if run_in_parallel:
- test_list = parallel_test_types.split(" ")
+ if test_type != "Default":
+ get_console().print(
+ "[error]You should not specify --test-type when --run-in-parallel is set[/]. "
+ f"Your test type = {test_type}\n"
+ )
+ sys.exit(1)
run_tests_in_parallel(
exec_shell_params=exec_shell_params,
- parallel_test_types_list=test_list,
extra_pytest_args=extra_pytest_args,
db_reset=db_reset,
test_timeout=test_timeout,
@@ -460,6 +623,13 @@ def command_for_tests(
skio_docker_compose_down=skip_docker_compose_down,
)
else:
+ if exec_shell_params.test_type == "Default":
+ if any([arg.startswith("tests") for arg in extra_pytest_args]):
+ # in case some tests are specified as parameters, do not pass "tests" as default
+ exec_shell_params.test_type = "None"
+ exec_shell_params.parallel_test_types_list = []
+ else:
+ exec_shell_params.test_type = "All"
returncode, _ = _run_test(
exec_shell_params=exec_shell_params,
extra_pytest_args=extra_pytest_args,
@@ -488,6 +658,7 @@ def command_for_tests(
@option_image_tag_for_running
@option_mount_sources
@option_integration
+@option_enable_coverage
@option_github_repository
@click.option(
"--test-timeout",
@@ -496,12 +667,7 @@ def command_for_tests(
type=IntRange(min=0),
show_default=True,
)
-@click.option(
- "--skip-provider-tests",
- help="Skip provider tests",
- is_flag=True,
- envvar="SKIP_PROVIDER_TESTS",
-)
+@option_skip_provider_tests
@option_db_reset
@option_verbose
@option_dry_run
@@ -520,6 +686,7 @@ def integration_tests(
image_tag: str | None,
mount_sources: str,
extra_pytest_args: tuple,
+ enable_coverage: bool,
):
docker_filesystem = get_filesystem_type("/var/lib/docker")
get_console().print(f"Docker filesystem: {docker_filesystem}")
@@ -536,6 +703,7 @@ def integration_tests(
test_type="Integration",
skip_provider_tests=skip_provider_tests,
github_repository=github_repository,
+ enable_coverage=enable_coverage,
)
cleanup_python_generated_files()
perform_environment_checks()
@@ -561,6 +729,9 @@ def integration_tests(
@option_image_tag_for_running
@option_mount_sources
@option_github_repository
+@option_test_timeout
+@option_parallelism
+@option_use_xdist
@option_verbose
@option_dry_run
@click.option(
@@ -576,6 +747,9 @@ def helm_tests(
mount_sources: str,
helm_test_package: str,
github_repository: str,
+ test_timeout: int,
+ parallelism: int,
+ use_xdist: bool,
):
exec_shell_params = ShellParams(
image_tag=image_tag,
@@ -589,6 +763,20 @@ def helm_tests(
env_variables["HELM_TEST_PACKAGE"] = helm_test_package
perform_environment_checks()
cleanup_python_generated_files()
- cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", *extra_pytest_args]
+ pytest_args = generate_args_for_pytest(
+ test_type="Helm",
+ test_timeout=test_timeout,
+ skip_provider_tests=True,
+ skip_db_tests=False,
+ run_db_tests_only=False,
+ backend="none",
+ use_xdist=use_xdist,
+ enable_coverage=False,
+ collect_only=False,
+ parallelism=parallelism,
+ parallel_test_types_list=[],
+ helm_test_package=helm_test_package,
+ )
+ cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", *pytest_args, *extra_pytest_args]
result = run_command(cmd, env=env_variables, check=False, output_outside_the_group=True)
sys.exit(result.returncode)
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py b/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
index 84d0d2b6ec0c8..4f5f348ae76f8 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands_config.py
@@ -23,27 +23,93 @@
TESTING_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"breeze testing tests": [
{
- "name": "Basic flag for tests command",
+ "name": "Select test types to run (tests can also be selected by command args individually)",
"options": [
"--test-type",
+ "--parallel-test-types",
+ "--excluded-parallel-test-types",
+ ],
+ },
+ {
+ "name": "Test options",
+ "options": [
"--test-timeout",
+ "--enable-coverage",
"--collect-only",
"--db-reset",
+ "--skip-provider-tests",
+ ],
+ },
+ {
+ "name": "Selectively run DB or non-DB tests",
+ "options": [
+ "--run-db-tests-only",
+ "--skip-db-tests",
+ ],
+ },
+ {
+ "name": "Test environment",
+ "options": [
+ "--integration",
"--backend",
"--python",
"--postgres-version",
"--mysql-version",
"--mssql-version",
- "--integration",
- "--github-repository",
],
},
{
"name": "Options for parallel test commands",
"options": [
"--run-in-parallel",
+ "--use-xdist",
"--parallelism",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
+ {
+ "name": "Advanced flag for tests command",
+ "options": [
+ "--image-tag",
+ "--github-repository",
+ "--use-airflow-version",
+ "--mount-sources",
+ "--upgrade-boto",
+ "--downgrade-sqlalchemy",
+ "--remove-arm-packages",
+ "--skip-docker-compose-down",
+ ],
+ },
+ ],
+ "breeze testing non-db-tests": [
+ {
+ "name": "Select test types to run",
+ "options": [
"--parallel-test-types",
+ "--excluded-parallel-test-types",
+ ],
+ },
+ {
+ "name": "Test options",
+ "options": [
+ "--test-timeout",
+ "--enable-coverage",
+ "--collect-only",
+ "--skip-provider-tests",
+ ],
+ },
+ {
+ "name": "Test environment",
+ "options": [
+ "--python",
+ ],
+ },
+ {
+ "name": "Options for parallel test commands",
+ "options": [
+ "--parallelism",
"--skip-cleanup",
"--debug-resources",
"--include-success-outputs",
@@ -53,6 +119,57 @@
"name": "Advanced flag for tests command",
"options": [
"--image-tag",
+ "--github-repository",
+ "--use-airflow-version",
+ "--mount-sources",
+ "--upgrade-boto",
+ "--downgrade-sqlalchemy",
+ "--remove-arm-packages",
+ "--skip-docker-compose-down",
+ ],
+ },
+ ],
+ "breeze testing db-tests": [
+ {
+ "name": "Select tests to run",
+ "options": [
+ "--parallel-test-types",
+ "--excluded-parallel-test-types",
+ ],
+ },
+ {
+ "name": "Test options",
+ "options": [
+ "--test-timeout",
+ "--enable-coverage",
+ "--collect-only",
+ "--skip-provider-tests",
+ ],
+ },
+ {
+ "name": "Test environment",
+ "options": [
+ "--backend",
+ "--python",
+ "--postgres-version",
+ "--mysql-version",
+ "--mssql-version",
+ ],
+ },
+ {
+ "name": "Options for parallel test commands",
+ "options": [
+ "--parallelism",
+ "--skip-cleanup",
+ "--debug-resources",
+ "--include-success-outputs",
+ ],
+ },
+ {
+ "name": "Advanced flag for tests command",
+ "options": [
+ "--image-tag",
+ "--github-repository",
"--use-airflow-version",
"--mount-sources",
"--upgrade-boto",
@@ -64,17 +181,23 @@
],
"breeze testing integration-tests": [
{
- "name": "Basic flag for integration tests command",
+ "name": "Test options",
"options": [
- "--integration",
"--test-timeout",
+ "--enable-coverage",
"--db-reset",
+ "--skip-provider-tests",
+ ],
+ },
+ {
+ "name": "Test environment",
+ "options": [
+ "--integration",
"--backend",
"--python",
"--postgres-version",
"--mysql-version",
"--mssql-version",
- "--github-repository",
],
},
{
@@ -82,17 +205,25 @@
"options": [
"--image-tag",
"--mount-sources",
- "--skip-provider-tests",
+ "--github-repository",
],
},
],
"breeze testing helm-tests": [
{
- "name": "Advanced flag for helms-tests command",
+ "name": "Flags for helms-tests command",
+ "options": [
+ "--helm-test-package",
+ "--test-timeout",
+ "--use-xdist",
+ "--parallelism",
+ ],
+ },
+ {
+ "name": "Advanced flags for helms-tests command",
"options": [
"--image-tag",
"--mount-sources",
- "--helm-test-package",
"--github-repository",
],
},
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index f4f6c9b7e2044..e8877ae66340b 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -28,8 +28,12 @@
from airflow_breeze.utils.host_info_utils import Architecture
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, PROVIDER_DEPENDENCIES_JSON_FILE_PATH
-RUNS_ON_PUBLIC_RUNNER = "ubuntu-22.04"
-RUNS_ON_SELF_HOSTED_RUNNER = "self-hosted"
+RUNS_ON_PUBLIC_RUNNER = '["ubuntu-22.04"]'
+# we should get more sophisticated logic here in the future, but for now we just check if
+# we use self airflow, vm-based, amd hosted runner as a default
+# TODO: when we have it properly set-up with labels we should change it to
+# RUNS_ON_SELF_HOSTED_RUNNER = '["self-hosted", "airflow-runner", "vm-runner", "X64"]'
+RUNS_ON_SELF_HOSTED_RUNNER = '["self-hosted", "Linux", "X64"]'
SELF_HOSTED_RUNNERS_CPU_COUNT = 8
ANSWER = ""
@@ -40,11 +44,13 @@
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.8", "3.9", "3.10", "3.11"]
DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
ALLOWED_ARCHITECTURES = [Architecture.X86_64, Architecture.ARM]
-ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "mssql"]
+# Database Backends used when starting Breeze. The "none" value means that invalid configuration
+# Is set and no database started - access to a database will fail.
+ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "mssql", "none"]
ALLOWED_PROD_BACKENDS = ["mysql", "postgres", "mssql"]
DEFAULT_BACKEND = ALLOWED_BACKENDS[0]
TESTABLE_INTEGRATIONS = ["cassandra", "celery", "kerberos", "mongo", "pinot", "trino", "kafka"]
-OTHER_INTEGRATIONS = ["statsd"]
+OTHER_INTEGRATIONS = ["statsd", "otel", "openlineage"]
ALL_INTEGRATIONS = sorted(
[
*TESTABLE_INTEGRATIONS,
@@ -55,8 +61,6 @@
[
"all-testable",
"all",
- "otel",
- "statsd",
*ALL_INTEGRATIONS,
]
)
@@ -87,7 +91,17 @@
ALLOWED_MYSQL_VERSIONS = ["5.7", "8"]
ALLOWED_MSSQL_VERSIONS = ["2017-latest", "2019-latest"]
-PIP_VERSION = "23.2.1"
+PIP_VERSION = "23.3.1"
+
+# key used for generating providers index
+PROVIDERS_INDEX_KEY = "providers-index"
+# keys for generated non providers docs
+NON_PROVIDERS_DOC_KEYS = ["apache-airflow", "docker-stack", "helm-chart"]
+# Mapping which store short-key:full-key
+ALL_SPECIAL_DOC_KEYS = {
+ PROVIDERS_INDEX_KEY: "apache-airflow-providers",
+ **dict(zip(NON_PROVIDERS_DOC_KEYS, NON_PROVIDERS_DOC_KEYS)),
+}
@lru_cache(maxsize=None)
@@ -98,20 +112,30 @@ def all_selective_test_types() -> tuple[str, ...]:
class SelectiveUnitTestTypes(Enum):
ALWAYS = "Always"
API = "API"
+ EXTERNAL_PYTHON = "ExternalPython"
+ EXTERNAL_BRANCH_PYTHON = "BranchExternalPython"
CLI = "CLI"
CORE = "Core"
+ SERIALIZATION = "Serialization"
OTHER = "Other"
+ OPERATORS = "Operators"
+ PLAIN_ASSERTS = "PlainAsserts"
PROVIDERS = "Providers"
+ PYTHON_VENV = "PythonVenv"
WWW = "WWW"
ALLOWED_TEST_TYPE_CHOICES = [
"All",
+ "Default",
+ *all_selective_test_types(),
+ "All-Postgres",
+ "All-MySQL",
+ "All-Quarantined",
+]
+
+ALLOWED_PARALLEL_TEST_TYPE_CHOICES = [
*all_selective_test_types(),
- "PlainAsserts",
- "Postgres",
- "MySQL",
- "Quarantine",
]
@@ -151,7 +175,7 @@ def get_available_documentation_packages(short_version=False, only_providers: bo
doc_provider_names = [provider_name.replace(".", "-") for provider_name in provider_names]
available_packages = []
if not only_providers:
- available_packages.extend(["apache-airflow", "docker-stack", "helm-chart"])
+ available_packages.extend(NON_PROVIDERS_DOC_KEYS)
all_providers = [f"apache-airflow-providers-{doc_provider}" for doc_provider in doc_provider_names]
all_providers.sort()
available_packages.extend(all_providers)
@@ -196,6 +220,44 @@ def get_default_platform_machine() -> str:
CURRENT_MSSQL_VERSIONS = ["2017-latest", "2019-latest"]
DEFAULT_MSSQL_VERSION = CURRENT_MSSQL_VERSIONS[0]
+
+AIRFLOW_PYTHON_COMPATIBILITY_MATRIX = {
+ "2.0.0": ["3.6", "3.7", "3.8"],
+ "2.0.1": ["3.6", "3.7", "3.8"],
+ "2.0.2": ["3.6", "3.7", "3.8"],
+ "2.1.0": ["3.6", "3.7", "3.8"],
+ "2.1.1": ["3.6", "3.7", "3.8"],
+ "2.1.2": ["3.6", "3.7", "3.8", "3.9"],
+ "2.1.3": ["3.6", "3.7", "3.8", "3.9"],
+ "2.1.4": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.0": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.1": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.2": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.3": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.4": ["3.6", "3.7", "3.8", "3.9"],
+ "2.2.5": ["3.6", "3.7", "3.8", "3.9"],
+ "2.3.0": ["3.7", "3.8", "3.9", "3.10"],
+ "2.3.1": ["3.7", "3.8", "3.9", "3.10"],
+ "2.3.2": ["3.7", "3.8", "3.9", "3.10"],
+ "2.3.3": ["3.7", "3.8", "3.9", "3.10"],
+ "2.3.4": ["3.7", "3.8", "3.9", "3.10"],
+ "2.4.0": ["3.7", "3.8", "3.9", "3.10"],
+ "2.4.1": ["3.7", "3.8", "3.9", "3.10"],
+ "2.4.2": ["3.7", "3.8", "3.9", "3.10"],
+ "2.4.3": ["3.7", "3.8", "3.9", "3.10"],
+ "2.5.0": ["3.7", "3.8", "3.9", "3.10"],
+ "2.5.1": ["3.7", "3.8", "3.9", "3.10"],
+ "2.5.2": ["3.7", "3.8", "3.9", "3.10"],
+ "2.5.3": ["3.7", "3.8", "3.9", "3.10"],
+ "2.6.0": ["3.7", "3.8", "3.9", "3.10"],
+ "2.6.1": ["3.7", "3.8", "3.9", "3.10"],
+ "2.6.2": ["3.7", "3.8", "3.9", "3.10", "3.11"],
+ "2.6.3": ["3.7", "3.8", "3.9", "3.10", "3.11"],
+ "2.7.0": ["3.8", "3.9", "3.10", "3.11"],
+ "2.7.1": ["3.8", "3.9", "3.10", "3.11"],
+ "2.7.2": ["3.8", "3.9", "3.10", "3.11"],
+}
+
DB_RESET = False
START_AIRFLOW = "false"
LOAD_EXAMPLES = False
diff --git a/dev/breeze/src/airflow_breeze/params/__init__.py b/dev/breeze/src/airflow_breeze/params/__init__.py
new file mode 100644
index 0000000000000..13a83393a9124
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/params/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/dev/breeze/src/airflow_breeze/params/doc_build_params.py b/dev/breeze/src/airflow_breeze/params/doc_build_params.py
index 961291b7a0ea9..34da7fc231b49 100644
--- a/dev/breeze/src/airflow_breeze/params/doc_build_params.py
+++ b/dev/breeze/src/airflow_breeze/params/doc_build_params.py
@@ -20,13 +20,17 @@
from dataclasses import dataclass
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
+from airflow_breeze.utils.general_utils import get_docs_filter_name_from_short_hand
+
+providers_prefix = "apache-airflow-providers-"
@dataclass
class DocBuildParams:
- package_filter: tuple[str]
+ package_filter: tuple[str, ...]
docs_only: bool
spellcheck_only: bool
+ short_doc_packages: tuple[str, ...]
skip_environment_initialization: bool = False
one_pass_only: bool = False
github_actions = os.environ.get("GITHUB_ACTIONS", "false")
@@ -42,7 +46,10 @@ def args_doc_builder(self) -> list[str]:
doc_args.append("--one-pass-only")
if AIRFLOW_BRANCH != "main":
doc_args.append("--disable-provider-checks")
+ if self.short_doc_packages:
+ for filter_from_short_doc in get_docs_filter_name_from_short_hand(self.short_doc_packages):
+ doc_args.extend(["--package-filter", filter_from_short_doc])
if self.package_filter:
- for single_filter in self.package_filter:
- doc_args.extend(["--package-filter", single_filter])
+ for filter in self.package_filter:
+ doc_args.extend(["--package-filter", filter])
return doc_args
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py b/dev/breeze/src/airflow_breeze/params/shell_params.py
index b2240a42c991c..182befccdda7d 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -18,7 +18,7 @@
import os
from copy import deepcopy
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from pathlib import Path
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
@@ -104,6 +104,8 @@ class ShellParams:
mssql_version: str = ALLOWED_MSSQL_VERSIONS[0]
mysql_version: str = ALLOWED_MYSQL_VERSIONS[0]
num_runs: str = ""
+ run_db_tests_only: bool = False
+ skip_db_tests: bool = False
package_format: str = ALLOWED_INSTALLATION_PACKAGE_FORMATS[0]
platform: str = DOCKER_DEFAULT_PLATFORM
postgres_version: str = ALLOWED_POSTGRES_VERSIONS[0]
@@ -127,6 +129,12 @@ class ShellParams:
only_min_version_update: bool = False
regenerate_missing_docs: bool = False
skip_provider_dependencies_check: bool = False
+ standalone_dag_processor: bool = False
+ database_isolation: bool = False
+ use_xdist: bool = False
+ enable_coverage: bool = False
+ parallelism: int = 0
+ parallel_test_types_list: list[str] = field(default_factory=list)
def clone_with_test(self, test_type: str) -> ShellParams:
new_params = deepcopy(self)
@@ -206,7 +214,7 @@ def print_badge_info(self):
def get_backend_compose_files(self, backend: str) -> list[Path]:
backend_docker_compose_file = DOCKER_COMPOSE_DIR / f"backend-{backend}.yml"
- if backend == "sqlite" or not self.forward_ports:
+ if backend in ("sqlite", "none") or not self.forward_ports:
return [backend_docker_compose_file]
return [backend_docker_compose_file, DOCKER_COMPOSE_DIR / f"backend-{backend}-port.yml"]
diff --git a/dev/breeze/src/airflow_breeze/utils/add_back_references.py b/dev/breeze/src/airflow_breeze/utils/add_back_references.py
index ebc0032acdd5e..ffb46bf6c65cc 100644
--- a/dev/breeze/src/airflow_breeze/utils/add_back_references.py
+++ b/dev/breeze/src/airflow_breeze/utils/add_back_references.py
@@ -109,6 +109,9 @@ def create_back_reference_html(back_ref_url: str, target_path: Path):
def generate_back_references(link: str, base_path: Path):
+ if not base_path.exists():
+ get_console().print("[blue]The provider is not yet released.Skipping.")
+ return
is_downloaded, file_name = download_file(link)
if not is_downloaded:
old_to_new: list[tuple[str, str]] = []
diff --git a/dev/breeze/src/airflow_breeze/utils/cdxgen.py b/dev/breeze/src/airflow_breeze/utils/cdxgen.py
index b1a64efac8f97..977c20c0ac326 100644
--- a/dev/breeze/src/airflow_breeze/utils/cdxgen.py
+++ b/dev/breeze/src/airflow_breeze/utils/cdxgen.py
@@ -29,9 +29,15 @@
import yaml
-from airflow_breeze.global_constants import DEFAULT_PYTHON_MAJOR_MINOR_VERSION
+from airflow_breeze.global_constants import (
+ AIRFLOW_PYTHON_COMPATIBILITY_MATRIX,
+ DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
+)
from airflow_breeze.utils.console import Output, get_console
-from airflow_breeze.utils.github import download_constraints_file, download_file_from_github
+from airflow_breeze.utils.github import (
+ download_constraints_file,
+ download_file_from_github,
+)
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, FILES_DIR
from airflow_breeze.utils.run_utils import run_command
from airflow_breeze.utils.shared_options import get_dry_run
@@ -119,25 +125,8 @@ def get_cdxgen_port_mapping(parallelism: int, pool: Pool) -> dict[str, int]:
return port_map
-def get_provider_requirement_image_name(airflow_version: str, python_version: str) -> str:
- return f"apache/airflow-dev/base_requirements/{airflow_version}/python{python_version}"
-
-
-def build_providers_base_image(airflow_version: str, python_version: str):
- image_name = get_provider_requirement_image_name(
- airflow_version=airflow_version, python_version=python_version
- )
- dockerfile = f"""
-FROM ghcr.io/apache/airflow/main/ci/python{python_version}
-RUN pip install --upgrade pip
-# Remove all packages
-RUN python -m venv /opt/airflow/providers
-RUN /opt/airflow/providers/bin/pip install --upgrade pip
-RUN /opt/airflow/providers/bin/pip install apache-airflow=={airflow_version} \
- --constraint https://raw.githubusercontent.com/apache/airflow/\
-constraints-{airflow_version}/constraints-{python_version}.txt
-"""
- run_command(["docker", "build", "--tag", image_name, "-"], input=dockerfile, text=True, check=True)
+def get_all_airflow_versions_image_name(python_version: str) -> str:
+ return f"apache/airflow-dev/all_airflow_versions/python{python_version}"
TARGET_DIR_NAME = "provider_requirements"
@@ -147,27 +136,59 @@ def build_providers_base_image(airflow_version: str, python_version: str):
def get_requirements_for_provider(
provider_id: str,
airflow_version: str,
+ output: Output | None,
provider_version: str | None = None,
python_version: str = DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
-):
+ force: bool = False,
+) -> tuple[int, str]:
provider_path_array = provider_id.split(".")
if not provider_version:
provider_file = (AIRFLOW_SOURCES_ROOT / "airflow" / "providers").joinpath(
*provider_path_array
) / "provider.yaml"
provider_version = yaml.safe_load(provider_file.read_text())["versions"][0]
- airflow_file_name = f"provider-{provider_id}-{provider_version}-base-requirements.txt"
- provider_with_airflow_file_name = f"provider-{provider_id}-{provider_version}-airflow-requirements.txt"
- provider_file_name = f"provider-{provider_id}-{provider_version}-requirements.txt"
+
+ target_dir = FILES_DIR / TARGET_DIR_NAME
+ airflow_core_file_name = f"airflow-{airflow_version}-python{python_version}-requirements.txt"
+ airflow_core_path = target_dir / airflow_core_file_name
+
+ provider_with_core_file_name = f"python{python_version}-with-core-requirements.txt"
+ provider_without_core_file_name = f"python{python_version}-without-core-requirements.txt"
+
+ provider_folder_name = f"provider-{provider_id}-{provider_version}"
+ provider_folder_path = target_dir / provider_folder_name
+ provider_with_core_path = provider_folder_path / provider_with_core_file_name
+ provider_without_core_file = provider_folder_path / provider_without_core_file_name
+
+ docker_file_provider_folder_prefix = f"{DOCKER_FILE_PREFIX}/{provider_folder_name}/"
+
+ if (
+ os.path.exists(provider_with_core_path)
+ and os.path.exists(provider_without_core_file)
+ and force is False
+ ):
+ get_console(output=output).print(
+ f"[warning] Requirements for provider {provider_id} version {provider_version} python "
+ f"{python_version} already exist, skipping. Set force=True to force generation."
+ )
+ return (
+ 0,
+ f"Provider requirements already existed, skipped generation for {provider_id} version "
+ f"{provider_version} python {python_version}",
+ )
+ else:
+ provider_folder_path.mkdir(exist_ok=True)
+
command = f"""
mkdir -pv {DOCKER_FILE_PREFIX}
-/opt/airflow/providers/bin/pip freeze | sort > {DOCKER_FILE_PREFIX}{airflow_file_name}
-/opt/airflow/providers/bin/pip install apache-airflow=={airflow_version} \
+/opt/airflow/airflow-{airflow_version}/bin/pip freeze | sort > {DOCKER_FILE_PREFIX}{airflow_core_file_name}
+/opt/airflow/airflow-{airflow_version}/bin/pip install apache-airflow=={airflow_version} \
apache-airflow-providers-{provider_id}=={provider_version}
-/opt/airflow/providers/bin/pip freeze | sort > {DOCKER_FILE_PREFIX}{provider_with_airflow_file_name}
-chown --recursive {os.getuid()}:{os.getgid()} {DOCKER_FILE_PREFIX}
+/opt/airflow/airflow-{airflow_version}/bin/pip freeze | sort > \
+ {docker_file_provider_folder_prefix}{provider_with_core_file_name}
+chown --recursive {os.getuid()}:{os.getgid()} {DOCKER_FILE_PREFIX}{provider_with_core_file_name}
"""
- run_command(
+ provider_command_result = run_command(
[
"docker",
"run",
@@ -178,37 +199,76 @@ def get_requirements_for_provider(
f"HOST_GROUP_ID={os.getgid()}",
"-v",
f"{AIRFLOW_SOURCES_ROOT}/files:/files",
- get_provider_requirement_image_name(
- airflow_version=airflow_version, python_version=python_version
- ),
+ get_all_airflow_versions_image_name(python_version=python_version),
"-c",
- ";".join(command.split("\n")[1:-1]),
- ]
+ ";".join(command.splitlines()[1:-1]),
+ ],
+ output=output,
)
- target_dir = FILES_DIR / TARGET_DIR_NAME
- airflow_file = target_dir / airflow_file_name
- provider_with_airflow_file = target_dir / provider_with_airflow_file_name
- get_console().print(f"[info]Airflow requirements in {airflow_file}")
- get_console().print(f"[info]Provider requirements in {provider_with_airflow_file}")
- base_packages = set([package.split("==")[0] for package in airflow_file.read_text().split("\n")])
+ get_console(output=output).print(f"[info]Airflow requirements in {airflow_core_path}")
+ get_console(output=output).print(f"[info]Provider requirements in {provider_with_core_path}")
+ base_packages = {package.split("==")[0] for package in airflow_core_path.read_text().splitlines()}
base_packages.add("apache-airflow-providers-" + provider_id.replace(".", "-"))
provider_packages = sorted(
[
line
- for line in provider_with_airflow_file.read_text().split("\n")
+ for line in provider_with_core_path.read_text().splitlines()
if line.split("==")[0] not in base_packages
]
)
- get_console().print(
+ get_console(output=output).print(
f"[info]Provider {provider_id} has {len(provider_packages)} transitively "
f"dependent packages (excluding airflow and its dependencies)"
)
- get_console().print(provider_packages)
- provider_file = target_dir / provider_file_name
- provider_file.write_text("\n".join(provider_packages) + "\n")
- get_console().print(
- f"[success]Generated {provider_id}:{provider_version} requirements in {provider_file}"
+ get_console(output=output).print(provider_packages)
+ provider_without_core_file.write_text("".join(f"{p}\n" for p in provider_packages))
+ get_console(output=output).print(
+ f"[success]Generated {provider_id}:{provider_version}:{python_version} requirements in "
+ f"{provider_without_core_file}"
+ )
+
+ return (
+ provider_command_result.returncode,
+ f"Provider requirements generated for {provider_id}:{provider_version}:{python_version}",
+ )
+
+
+def build_all_airflow_versions_base_image(
+ python_version: str,
+ output: Output | None,
+ confirm: bool = True,
+) -> tuple[int, str]:
+ """
+ Build an image with all airflow versions pre-installed in separate virtualenvs.
+ """
+ image_name = f"apache/airflow-dev/all_airflow_versions/python{python_version}"
+ image_name = get_all_airflow_versions_image_name(python_version=python_version)
+ dockerfile = f"""
+FROM ghcr.io/apache/airflow/main/ci/python{python_version}
+RUN pip install --upgrade pip --no-cache-dir
+# Prevent setting sources in PYTHONPATH to not interfere with virtualenvs
+ENV USE_AIRFLOW_VERSION=none
+ENV START_AIRFLOW=none
+ """
+ compatible_airflow_versions = [
+ airflow_version
+ for airflow_version, python_versions in AIRFLOW_PYTHON_COMPATIBILITY_MATRIX.items()
+ if python_version in python_versions
+ ]
+
+ for airflow_version in compatible_airflow_versions:
+ dockerfile += f"""
+# Create the virtualenv and install the proper airflow version in it
+RUN python -m venv /opt/airflow/airflow-{airflow_version} && \
+/opt/airflow/airflow-{airflow_version}/bin/pip install --no-cache-dir --upgrade pip && \
+/opt/airflow/airflow-{airflow_version}/bin/pip install apache-airflow=={airflow_version} \
+ --constraint https://raw.githubusercontent.com/apache/airflow/\
+constraints-{airflow_version}/constraints-{python_version}.txt
+"""
+ build_command = run_command(
+ ["docker", "build", "--tag", image_name, "-"], input=dockerfile, text=True, check=True, output=output
)
+ return build_command.returncode, f"All airflow image built for python {python_version}"
@dataclass
diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py b/dev/breeze/src/airflow_breeze/utils/common_options.py
index 5e8e50da61d99..6deee8bf295d8 100644
--- a/dev/breeze/src/airflow_breeze/utils/common_options.py
+++ b/dev/breeze/src/airflow_breeze/utils/common_options.py
@@ -19,6 +19,7 @@
import multiprocessing as mp
import click
+from click import IntRange
from airflow_breeze.branch_defaults import DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import (
@@ -34,13 +35,16 @@
ALLOWED_MSSQL_VERSIONS,
ALLOWED_MYSQL_VERSIONS,
ALLOWED_PACKAGE_FORMATS,
+ ALLOWED_PARALLEL_TEST_TYPE_CHOICES,
ALLOWED_PLATFORMS,
ALLOWED_POSTGRES_VERSIONS,
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
+ ALLOWED_TEST_TYPE_CHOICES,
ALLOWED_USE_AIRFLOW_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
AUTOCOMPLETE_INTEGRATIONS,
DEFAULT_CELERY_BROKER,
+ PROVIDERS_INDEX_KEY,
SINGLE_PLATFORMS,
START_AIRFLOW_ALLOWED_EXECUTORS,
START_AIRFLOW_DEFAULT_ALLOWED_EXECUTORS,
@@ -52,10 +56,12 @@
CacheableChoice,
CacheableDefault,
DryRunOption,
+ NotVerifiedBetterChoice,
UseAirflowVersionType,
VerboseOption,
)
from airflow_breeze.utils.recording import generating_command_images
+from airflow_breeze.utils.selective_checks import ALL_CI_SELECTIVE_TEST_TYPES
def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option, value):
@@ -131,7 +137,8 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
type=CacheableChoice(ALLOWED_BACKENDS),
default=CacheableDefault(value=ALLOWED_BACKENDS[0]),
show_default=True,
- help="Database backend to use.",
+ help="Database backend to use. If 'none' is selected, breeze starts with invalid DB configuration "
+ "and no database and any attempts to connect to Airflow DB will fail.",
envvar="BACKEND",
)
option_integration = click.option(
@@ -228,7 +235,6 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
envvar="IMAGE_TAG",
)
option_image_tag_for_building = click.option(
- "-t",
"--image-tag",
help="Tag the image after building it.",
show_default=True,
@@ -236,7 +242,6 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
envvar="IMAGE_TAG",
)
option_image_tag_for_running = click.option(
- "-t",
"--image-tag",
help="Tag of the image which is used to run the image (implies --mount-sources=skip).",
show_default=True,
@@ -427,7 +432,7 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
)
option_run_in_parallel = click.option(
"--run-in-parallel",
- help="Run the operation in parallel on all or selected subset of Python versions.",
+ help="Run the operation in parallel on all or selected subset of parameters.",
is_flag=True,
envvar="RUN_IN_PARALLEL",
)
@@ -445,12 +450,22 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
required=False,
type=BetterChoice(get_available_documentation_packages(short_version=True)),
)
-argument_packages_plus_all_providers = click.argument(
- "packages_plus_all_providers",
+argument_short_doc_packages = click.argument(
+ "short_doc_packages",
+ nargs=-1,
+ required=False,
+ type=BetterChoice(["all-providers", *get_available_documentation_packages(short_version=True)]),
+)
+
+argument_short_doc_packages_with_providers_index = click.argument(
+ "short_doc_packages",
nargs=-1,
required=False,
- type=BetterChoice(["all-providers"] + get_available_documentation_packages(short_version=True)),
+ type=BetterChoice(
+ ["all-providers", PROVIDERS_INDEX_KEY, *get_available_documentation_packages(short_version=True)]
+ ),
)
+
option_airflow_constraints_reference = click.option(
"--airflow-constraints-reference",
help="Constraint reference to use. Useful with --use-airflow-version parameter to specify "
@@ -472,7 +487,12 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
help="Constraint reference to use when building the image.",
envvar="AIRFLOW_CONSTRAINTS_REFERENCE",
)
-
+option_airflow_constraints_mode_update = click.option(
+ "--airflow-constraints-mode",
+ type=BetterChoice(ALLOWED_CONSTRAINTS_MODES_CI),
+ required=False,
+ help="Limit constraint update to only selected constraint mode - if selected.",
+)
option_airflow_constraints_mode_ci = click.option(
"--airflow-constraints-mode",
type=BetterChoice(ALLOWED_CONSTRAINTS_MODES_CI),
@@ -560,6 +580,18 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
show_default=True,
)
option_celery_flower = click.option("--celery-flower", help="Start celery flower", is_flag=True)
+option_standalone_dag_processor = click.option(
+ "--standalone-dag-processor",
+ help="Run standalone dag processor for start-airflow.",
+ is_flag=True,
+ envvar="STANDALONE_DAG_PROCESSOR",
+)
+option_database_isolation = click.option(
+ "--database-isolation",
+ help="Run airflow in database isolation mode.",
+ is_flag=True,
+ envvar="DATABASE_ISOLATION",
+)
option_install_selected_providers = click.option(
"--install-selected-providers",
help="Comma-separated list of providers selected to be installed (implies --use-packages-from-dist).",
@@ -601,6 +633,14 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
help="Optional additional requirements to upgrade eagerly to avoid backtracking "
"(see `breeze ci find-backtracking-candidates`).",
)
+option_airflow_site_directory = click.option(
+ "-a",
+ "--airflow-site-directory",
+ envvar="AIRFLOW_SITE_DIRECTORY",
+ type=click.Path(exists=True, file_okay=False, dir_okay=True, resolve_path=True),
+ help="Local directory path of cloned airflow-site repo.",
+ required=True,
+)
option_upgrade_boto = click.option(
"--upgrade-boto",
help="Remove aiobotocore and upgrade botocore and boto to the latest version.",
@@ -613,3 +653,85 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
is_flag=True,
envvar="DOWNGRADE_SQLALCHEMY",
)
+option_run_db_tests_only = click.option(
+ "--run-db-tests-only",
+ help="Only runs tests that require a database",
+ is_flag=True,
+ envvar="run_db_tests_only",
+)
+option_skip_db_tests = click.option(
+ "--skip-db-tests",
+ help="Skip tests that require a database",
+ is_flag=True,
+ envvar="SKIP_DB_TESTS",
+)
+option_test_timeout = click.option(
+ "--test-timeout",
+ help="Test timeout in seconds. Set the pytest setup, execution and teardown timeouts to this value",
+ default=60,
+ envvar="TEST_TIMEOUT",
+ type=IntRange(min=0),
+ show_default=True,
+)
+option_enable_coverage = click.option(
+ "--enable-coverage",
+ help="Enable coverage capturing for tests in the form of XML files",
+ is_flag=True,
+ envvar="ENABLE_COVERAGE",
+)
+option_skip_provider_tests = click.option(
+ "--skip-provider-tests",
+ help="Skip provider tests",
+ is_flag=True,
+ envvar="SKIP_PROVIDER_TESTS",
+)
+option_use_xdist = click.option(
+ "--use-xdist",
+ help="Use xdist plugin for pytest",
+ is_flag=True,
+ envvar="USE_XDIST",
+)
+option_test_type = click.option(
+ "--test-type",
+ help="Type of test to run. With Providers, you can specify tests of which providers "
+ "should be run: `Providers[airbyte,http]` or "
+ "excluded from the full test suite: `Providers[-amazon,google]`",
+ default="Default",
+ envvar="TEST_TYPE",
+ show_default=True,
+ type=NotVerifiedBetterChoice(ALLOWED_TEST_TYPE_CHOICES),
+)
+option_parallel_test_types = click.option(
+ "--parallel-test-types",
+ help="Space separated list of test types used for testing in parallel",
+ default=ALL_CI_SELECTIVE_TEST_TYPES,
+ show_default=True,
+ envvar="PARALLEL_TEST_TYPES",
+ type=NotVerifiedBetterChoice(ALLOWED_PARALLEL_TEST_TYPE_CHOICES),
+)
+option_excluded_parallel_test_types = click.option(
+ "--excluded-parallel-test-types",
+ help="Space separated list of test types that will be excluded from parallel tes runs.",
+ default="",
+ show_default=True,
+ envvar="EXCLUDED_PARALLEL_TEST_TYPES",
+ type=NotVerifiedBetterChoice(ALLOWED_PARALLEL_TEST_TYPE_CHOICES),
+)
+option_collect_only = click.option(
+ "--collect-only",
+ help="Collect tests only, do not run them.",
+ is_flag=True,
+ envvar="COLLECT_ONLY",
+)
+option_remove_arm_packages = click.option(
+ "--remove-arm-packages",
+ help="Removes arm packages from the image to test if ARM collection works",
+ is_flag=True,
+ envvar="REMOVE_ARM_PACKAGES",
+)
+option_skip_docker_compose_down = click.option(
+ "--skip-docker-compose-down",
+ help="Skips running docker-compose down after tests",
+ is_flag=True,
+ envvar="SKIP_DOCKER_COMPOSE_DOWN",
+)
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index 1b85afea5ab22..73865d32975c0 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -582,7 +582,6 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
set_value_to_default_if_not_set(env, "DEFAULT_BRANCH", AIRFLOW_BRANCH)
set_value_to_default_if_not_set(env, "DOCKER_IS_ROOTLESS", "false")
set_value_to_default_if_not_set(env, "ENABLED_SYSTEMS", "")
- set_value_to_default_if_not_set(env, "ENABLE_TEST_COVERAGE", "false")
set_value_to_default_if_not_set(env, "HELM_TEST_PACKAGE", "")
set_value_to_default_if_not_set(env, "HOST_GROUP_ID", get_host_group_id())
set_value_to_default_if_not_set(env, "HOST_OS", get_host_os())
@@ -600,11 +599,9 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
set_value_to_default_if_not_set(env, "RUN_SYSTEM_TESTS", "false")
set_value_to_default_if_not_set(env, "RUN_TESTS", "false")
set_value_to_default_if_not_set(env, "SKIP_ENVIRONMENT_INITIALIZATION", "false")
- set_value_to_default_if_not_set(env, "SKIP_PROVIDER_TESTS", "false")
set_value_to_default_if_not_set(env, "SKIP_SSH_SETUP", "false")
set_value_to_default_if_not_set(env, "SUSPENDED_PROVIDERS_FOLDERS", "")
set_value_to_default_if_not_set(env, "TEST_TYPE", "")
- set_value_to_default_if_not_set(env, "TEST_TIMEOUT", "60")
set_value_to_default_if_not_set(env, "UPGRADE_BOTO", "false")
set_value_to_default_if_not_set(env, "DOWNGRADE_SQLALCHEMY", "false")
set_value_to_default_if_not_set(env, "UPGRADE_TO_NEWER_DEPENDENCIES", "false")
@@ -616,6 +613,8 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
DERIVE_ENV_VARIABLES_FROM_ATTRIBUTES = {
+ "_AIRFLOW_RUN_DB_TESTS_ONLY": "run_db_tests_only",
+ "_AIRFLOW_SKIP_DB_TESTS": "skip_db_tests",
"AIRFLOW_CI_IMAGE": "airflow_image_name",
"AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_image_name_with_tag",
"AIRFLOW_CONSTRAINTS_MODE": "airflow_constraints_mode",
@@ -629,6 +628,7 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
"BACKEND": "backend",
"BASE_BRANCH": "base_branch",
"COMPOSE_FILE": "compose_file",
+ "DATABASE_ISOLATION": "database_isolation",
"DB_RESET": "db_reset",
"DEV_MODE": "dev_mode",
"DEFAULT_CONSTRAINTS_BRANCH": "default_constraints_branch",
@@ -644,23 +644,25 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
"MYSQL_VERSION": "mysql_version",
"NUM_RUNS": "num_runs",
"ONLY_MIN_VERSION_UPDATE": "only_min_version_update",
- "REGENERATE_MISSING_DOCS": "regenerate_missing_docs",
"PACKAGE_FORMAT": "package_format",
"POSTGRES_VERSION": "postgres_version",
"PYTHON_MAJOR_MINOR_VERSION": "python",
+ "REGENERATE_MISSING_DOCS": "regenerate_missing_docs",
"SKIP_CONSTRAINTS": "skip_constraints",
"SKIP_ENVIRONMENT_INITIALIZATION": "skip_environment_initialization",
- "SKIP_PROVIDER_TESTS": "skip_provider_tests",
"SQLITE_URL": "sqlite_url",
"START_AIRFLOW": "start_airflow",
"UPGRADE_BOTO": "upgrade_boto",
+ "USE_XDIST": "use_xdist",
"DOWNGRADE_SQLALCHEMY": "downgrade_sqlalchemy",
"USE_AIRFLOW_VERSION": "use_airflow_version",
"USE_PACKAGES_FROM_DIST": "use_packages_from_dist",
"VERSION_SUFFIX_FOR_PYPI": "version_suffix_for_pypi",
"CELERY_FLOWER": "celery_flower",
+ "STANDALONE_DAG_PROCESSOR": "standalone_dag_processor",
}
+
DOCKER_VARIABLE_CONSTANTS = {
"FLOWER_HOST_PORT": FLOWER_HOST_PORT,
"MSSQL_HOST_PORT": MSSQL_HOST_PORT,
diff --git a/dev/breeze/src/airflow_breeze/utils/general_utils.py b/dev/breeze/src/airflow_breeze/utils/general_utils.py
new file mode 100644
index 0000000000000..d5f64057a1399
--- /dev/null
+++ b/dev/breeze/src/airflow_breeze/utils/general_utils.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow_breeze.global_constants import ALL_SPECIAL_DOC_KEYS, get_available_documentation_packages
+
+providers_prefix = "apache-airflow-providers-"
+
+
+def get_docs_filter_name_from_short_hand(short_form_providers: tuple[str]):
+ providers = []
+ for short_form_provider in short_form_providers:
+ if specific_doc := ALL_SPECIAL_DOC_KEYS.get(short_form_provider):
+ providers.append(specific_doc)
+ continue
+
+ short_form_provider.split(".")
+ parts = "-".join(short_form_provider.split("."))
+ providers.append(providers_prefix + parts)
+ return tuple(providers)
+
+
+def expand_all_providers(short_doc_packages: tuple[str, ...]) -> tuple[str, ...]:
+ if "all-providers" in short_doc_packages:
+ packages = [package for package in short_doc_packages if package != "all-providers"]
+ packages.extend(get_available_documentation_packages(only_providers=True, short_version=True))
+ short_doc_packages = tuple(set(packages))
+ return short_doc_packages
diff --git a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
index 65458e6208642..c63ff8134e1b2 100644
--- a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
@@ -31,13 +31,12 @@
from typing import Any, NamedTuple
from urllib import request
-from airflow_breeze.branch_defaults import DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import ALLOWED_ARCHITECTURES, HELM_VERSION, KIND_VERSION, PIP_VERSION
from airflow_breeze.utils.console import Output, get_console
from airflow_breeze.utils.host_info_utils import Architecture, get_host_architecture, get_host_os
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, BUILD_CACHE_DIR
from airflow_breeze.utils.run_utils import RunCommandResult, run_command
-from airflow_breeze.utils.shared_options import get_dry_run
+from airflow_breeze.utils.shared_options import get_dry_run, get_verbose
K8S_ENV_PATH = BUILD_CACHE_DIR / ".k8s-env"
K8S_CLUSTERS_PATH = BUILD_CACHE_DIR / ".k8s-clusters"
@@ -274,7 +273,7 @@ def make_sure_kubernetes_tools_are_installed():
def _requirements_changed() -> bool:
if not CACHED_K8S_REQUIREMENTS.exists():
get_console().print(
- f"\n[warning]The K8S venv in {K8S_ENV_PATH}. has never been created. Installing it.\n"
+ f"\n[warning]The K8S venv in {K8S_ENV_PATH} has never been created. Installing it.\n"
)
return True
requirements_file_content = K8S_REQUIREMENTS.read_text()
@@ -288,7 +287,7 @@ def _requirements_changed() -> bool:
return False
-def _install_packages_in_k8s_virtualenv(with_constraints: bool):
+def _install_packages_in_k8s_virtualenv():
install_command = [
str(PYTHON_BIN_PATH),
"-m",
@@ -297,20 +296,19 @@ def _install_packages_in_k8s_virtualenv(with_constraints: bool):
"-r",
str(K8S_REQUIREMENTS.resolve()),
]
- if with_constraints:
- install_command.extend(
- [
- "--constraint",
- f"https://raw.githubusercontent.com/apache/airflow/{DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH}/"
- f"constraints-{sys.version_info.major}.{sys.version_info.minor}.txt",
- ]
- )
- install_packages_result = run_command(install_command, check=False, capture_output=True, text=True)
+ env = os.environ.copy()
+ env["INSTALL_PROVIDERS_FROM_SOURCES"] = "true"
+ capture_output = True
+ if get_verbose():
+ capture_output = False
+ install_packages_result = run_command(
+ install_command, check=False, capture_output=capture_output, text=True, env=env
+ )
if install_packages_result.returncode != 0:
- get_console().print(
- f"[error]Error when installing packages from : {K8S_REQUIREMENTS.resolve()}[/]\n"
- f"{install_packages_result.stdout}\n{install_packages_result.stderr}"
- )
+ get_console().print(f"[error]Error when installing packages from : {K8S_REQUIREMENTS.resolve()}[/]\n")
+ if not get_verbose():
+ get_console().print(install_packages_result.stdout)
+ get_console().print(install_packages_result.stderr)
return install_packages_result
@@ -332,7 +330,10 @@ def create_virtualenv(force_venv_setup: bool) -> RunCommandResult:
get_console().print(f"[info]Forcing initializing K8S virtualenv in {K8S_ENV_PATH}")
else:
get_console().print(f"[info]Initializing K8S virtualenv in {K8S_ENV_PATH}")
- shutil.rmtree(K8S_ENV_PATH, ignore_errors=True)
+ if get_dry_run():
+ get_console().print(f"[info]Dry run - would be removing {K8S_ENV_PATH}")
+ else:
+ shutil.rmtree(K8S_ENV_PATH, ignore_errors=True)
venv_command_result = run_command(
[sys.executable, "-m", "venv", str(K8S_ENV_PATH)],
check=False,
@@ -358,12 +359,12 @@ def create_virtualenv(force_venv_setup: bool) -> RunCommandResult:
return pip_reinstall_result
get_console().print(f"[info]Installing necessary packages in {K8S_ENV_PATH}")
- install_packages_result = _install_packages_in_k8s_virtualenv(with_constraints=True)
- if install_packages_result.returncode != 0:
- # if the first installation fails, attempt to install it without constraints
- install_packages_result = _install_packages_in_k8s_virtualenv(with_constraints=False)
+ install_packages_result = _install_packages_in_k8s_virtualenv()
if install_packages_result.returncode == 0:
- CACHED_K8S_REQUIREMENTS.write_text(K8S_REQUIREMENTS.read_text())
+ if get_dry_run():
+ get_console().print(f"[info]Dry run - would be saving {K8S_REQUIREMENTS} to cache")
+ else:
+ CACHED_K8S_REQUIREMENTS.write_text(K8S_REQUIREMENTS.read_text())
return install_packages_result
diff --git a/dev/breeze/src/airflow_breeze/utils/path_utils.py b/dev/breeze/src/airflow_breeze/utils/path_utils.py
index c9627c5ffb27e..1abdddac97dd0 100644
--- a/dev/breeze/src/airflow_breeze/utils/path_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/path_utils.py
@@ -249,7 +249,10 @@ def find_airflow_sources_root_to_operate_on() -> Path:
get_console().print(
"\n[error]Breeze should only be installed with -e flag[/]\n\n"
"[warning]Please go to Airflow sources and run[/]\n\n"
- f" {NAME} self-upgrade --force\n"
+ f" {NAME} setup self-upgrade --use-current-airflow-sources\n"
+ '[warning]If during installation you see warning starting "Ignoring --editable install",[/]\n'
+ '[warning]make sure you first downgrade "packaging" package to <23.2, for example by:[/]\n\n'
+ f' pip install "packaging<23.2"\n\n'
)
sys.exit(1)
airflow_sources = get_used_airflow_sources()
diff --git a/dev/breeze/src/airflow_breeze/utils/publish_docs_builder.py b/dev/breeze/src/airflow_breeze/utils/publish_docs_builder.py
index cb0fba3d8157f..e5dcc1be22ab0 100644
--- a/dev/breeze/src/airflow_breeze/utils/publish_docs_builder.py
+++ b/dev/breeze/src/airflow_breeze/utils/publish_docs_builder.py
@@ -24,12 +24,11 @@
from pathlib import Path
from subprocess import run
-from rich.console import Console
-
from airflow_breeze.global_constants import get_airflow_version
+from airflow_breeze.utils.console import Output, get_console
from airflow_breeze.utils.docs_errors import DocBuildError, parse_sphinx_warnings
from airflow_breeze.utils.helm_chart_utils import chart_version
-from airflow_breeze.utils.publish_docs_helpers import CONSOLE_WIDTH, load_package_data, pretty_format_path
+from airflow_breeze.utils.publish_docs_helpers import load_package_data, pretty_format_path
from airflow_breeze.utils.spelling_checks import SpellingError, parse_spelling_warnings
PROCESS_TIMEOUT = 15 * 60
@@ -37,14 +36,14 @@
ROOT_PROJECT_DIR = Path(__file__).parents[5].resolve()
DOCS_DIR = os.path.join(ROOT_PROJECT_DIR, "docs")
-console = Console(force_terminal=True, color_system="standard", width=CONSOLE_WIDTH)
-
class PublishDocsBuilder:
"""Documentation builder for Airflow Docs Publishing."""
- def __init__(self, package_name: str):
+ def __init__(self, package_name: str, output: Output | None, verbose: bool):
self.package_name = package_name
+ self.output = output
+ self.verbose = verbose
@property
def _doctree_dir(self) -> str:
@@ -153,11 +152,13 @@ def check_spelling(self, verbose: bool) -> list[SpellingError]:
env = os.environ.copy()
env["AIRFLOW_PACKAGE_NAME"] = self.package_name
if verbose:
- console.print(
+ get_console(output=self.output).print(
f"[info]{self.package_name:60}:[/] Executing cmd: ",
" ".join(shlex.quote(c) for c in build_cmd),
)
- console.print(f"[info]{self.package_name:60}:[/] The output is hidden until an error occurs.")
+ get_console(output=self.output).print(
+ f"[info]{self.package_name:60}:[/] The output is hidden until an error occurs."
+ )
with open(self.log_spelling_filename, "w") as output:
completed_proc = run(
build_cmd,
@@ -186,14 +187,16 @@ def check_spelling(self, verbose: bool) -> list[SpellingError]:
warning_text += spelling_file.read()
spelling_errors.extend(parse_spelling_warnings(warning_text, self._src_dir))
- console.print(f"[info]{self.package_name:60}:[/] [red]Finished spell-checking with errors[/]")
+ get_console(output=self.output).print(
+ f"[info]{self.package_name:60}:[/] [red]Finished spell-checking with errors[/]"
+ )
else:
if spelling_errors:
- console.print(
+ get_console(output=self.output).print(
f"[info]{self.package_name:60}:[/] [yellow]Finished spell-checking with warnings[/]"
)
else:
- console.print(
+ get_console(output=self.output).print(
f"[info]{self.package_name:60}:[/] [green]Finished spell-checking successfully[/]"
)
return spelling_errors
@@ -226,12 +229,12 @@ def build_sphinx_docs(self, verbose: bool) -> list[DocBuildError]:
env = os.environ.copy()
env["AIRFLOW_PACKAGE_NAME"] = self.package_name
if verbose:
- console.print(
+ get_console(output=self.output).print(
f"[info]{self.package_name:60}:[/] Executing cmd: ",
" ".join(shlex.quote(c) for c in build_cmd),
)
else:
- console.print(
+ get_console(output=self.output).print(
f"[info]{self.package_name:60}:[/] Running sphinx. "
f"The output is hidden until an error occurs."
)
@@ -259,32 +262,36 @@ def build_sphinx_docs(self, verbose: bool) -> list[DocBuildError]:
warning_text = re.sub(r"\x1B[@-_][0-?]*[ -/]*[@-~]", "", warning_text)
build_errors.extend(parse_sphinx_warnings(warning_text, self._src_dir))
if build_errors:
- console.print(f"[info]{self.package_name:60}:[/] [red]Finished docs building with errors[/]")
+ get_console(output=self.output).print(
+ f"[info]{self.package_name:60}:[/] [red]Finished docs building with errors[/]"
+ )
else:
- console.print(f"[info]{self.package_name:60}:[/] [green]Finished docs building successfully[/]")
+ get_console(output=self.output).print(
+ f"[info]{self.package_name:60}:[/] [green]Finished docs building successfully[/]"
+ )
return build_errors
def publish(self, override_versioned: bool, airflow_site_dir: str):
"""Copy documentation packages files to airflow-site repository."""
- console.print(f"Publishing docs for {self.package_name}")
+ get_console(output=self.output).print(f"Publishing docs for {self.package_name}")
output_dir = os.path.join(airflow_site_dir, self._publish_dir)
pretty_source = pretty_format_path(self._build_dir, os.getcwd())
pretty_target = pretty_format_path(output_dir, airflow_site_dir)
- console.print(f"Copy directory: {pretty_source} => {pretty_target}")
+ get_console(output=self.output).print(f"Copy directory: {pretty_source} => {pretty_target}")
if os.path.exists(output_dir):
if self.is_versioned:
if override_versioned:
- console.print(f"Overriding previously existing {output_dir}! ")
+ get_console(output=self.output).print(f"Overriding previously existing {output_dir}! ")
else:
- console.print(
+ get_console(output=self.output).print(
f"Skipping previously existing {output_dir}! "
f"Delete it manually if you want to regenerate it!"
)
- console.print()
+ get_console(output=self.output).print()
return
shutil.rmtree(output_dir)
shutil.copytree(self._build_dir, output_dir)
if self.is_versioned:
with open(os.path.join(output_dir, "..", "stable.txt"), "w") as stable_file:
stable_file.write(self._current_version)
- console.print()
+ get_console(output=self.output).print()
diff --git a/dev/breeze/src/airflow_breeze/utils/publish_docs_helpers.py b/dev/breeze/src/airflow_breeze/utils/publish_docs_helpers.py
index 1d7887f96f00b..e43eca408d43a 100644
--- a/dev/breeze/src/airflow_breeze/utils/publish_docs_helpers.py
+++ b/dev/breeze/src/airflow_breeze/utils/publish_docs_helpers.py
@@ -26,6 +26,8 @@
import yaml
+from airflow_breeze.utils.general_utils import get_docs_filter_name_from_short_hand
+
CONSOLE_WIDTH = 180
ROOT_DIR = Path(__file__).parents[5].resolve()
@@ -99,14 +101,18 @@ def get_available_packages():
]
-def process_package_filters(available_packages: list[str], package_filters: list[str] | None):
+def process_package_filters(
+ available_packages: list[str], package_filters: list[str] | None, packages_short_form: tuple[str]
+):
"""Filters the package list against a set of filters.
A packet is returned if it matches at least one filter. The function keeps the order of the packages.
"""
- if not package_filters:
+ if not package_filters and not packages_short_form:
return available_packages
+ package_filters = list(package_filters + get_docs_filter_name_from_short_hand(packages_short_form))
+
invalid_filters = [
f for f in package_filters if not any(fnmatch.fnmatch(p, f) for p in available_packages)
]
@@ -138,7 +144,7 @@ def prepare_code_snippet(file_path: str, line_no: int, context_lines_count: int
with open(file_path) as text_file:
# Highlight code
code = text_file.read()
- code_lines = code.split("\n")
+ code_lines = code.splitlines()
# Prepend line number
code_lines = [
f">{lno:3} | {line}" if line_no == lno else f"{lno:4} | {line}"
diff --git a/dev/breeze/src/airflow_breeze/utils/recording.py b/dev/breeze/src/airflow_breeze/utils/recording.py
index 7125b3d7252c5..879f20f34605a 100644
--- a/dev/breeze/src/airflow_breeze/utils/recording.py
+++ b/dev/breeze/src/airflow_breeze/utils/recording.py
@@ -19,12 +19,14 @@
import atexit
import os
import sys
-
-import rich
-from rich.console import Console
+from copy import deepcopy
+from typing import IO, TYPE_CHECKING
from airflow_breeze.utils.path_utils import in_autocomplete
+if TYPE_CHECKING:
+ from rich.console import Console
+
help_console: Console | None = None
DEFAULT_COLUMNS = 129
@@ -48,20 +50,28 @@ def save_ouput_as_svg():
if help_console:
help_console.save_svg(path=path, title=title, unique_id=unique_id)
- class RecordingConsole(rich.console.Console):
- def __init__(self, **kwargs):
- kwargs["force_terminal"] = True
- kwargs["width"] = width_int
- super().__init__(record=True, **kwargs)
- global help_console
- help_console = self
-
atexit.register(save_ouput_as_svg)
click.rich_click.MAX_WIDTH = width_int
click.formatting.FORCED_WIDTH = width_int - 2 # type: ignore[attr-defined]
click.rich_click.COLOR_SYSTEM = "standard"
- # monkeypatch rich_click console to record help (rich_click does not allow passing extra args to console)
- click.rich_click.Console = RecordingConsole # type: ignore[misc]
+ # monkeypatch rich_click console to record help
+ import rich_click
+
+ original_create_console = rich_click.rich_help_formatter.create_console
+
+ from rich_click import RichHelpConfiguration
+
+ def create_recording_console(config: RichHelpConfiguration, file: IO[str] | None = None) -> Console:
+ recording_config = deepcopy(config)
+ recording_config.width = width_int
+ recording_config.force_terminal = True
+ recording_console = original_create_console(recording_config, file)
+ recording_console.record = True
+ global help_console
+ help_console = recording_console
+ return recording_console
+
+ rich_click.rich_help_formatter.create_console = create_recording_console
output_file = os.environ.get("RECORD_BREEZE_OUTPUT_FILE")
diff --git a/dev/breeze/src/airflow_breeze/utils/reinstall.py b/dev/breeze/src/airflow_breeze/utils/reinstall.py
index 6dd459784d42e..68a0310eaa294 100644
--- a/dev/breeze/src/airflow_breeze/utils/reinstall.py
+++ b/dev/breeze/src/airflow_breeze/utils/reinstall.py
@@ -51,6 +51,9 @@ def warn_non_editable():
"\n[error]It should only be installed in editable mode[/]\n\n"
"[info]Please go to Airflow sources and run[/]\n\n"
f" {NAME} setup self-upgrade --use-current-airflow-sources\n"
+ '[warning]If during installation, you see warning with "Ignoring --editable install",[/]\n'
+ '[warning]make sure you first downgrade "packaging" package to <23.2, for example by:[/]\n\n'
+ f' pip install "packaging<23.2"\n\n'
)
diff --git a/dev/breeze/src/airflow_breeze/utils/run_tests.py b/dev/breeze/src/airflow_breeze/utils/run_tests.py
index c4311558421eb..b7bf38b6e681d 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_tests.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_tests.py
@@ -19,11 +19,13 @@
import os
import re
import sys
+from itertools import chain
from subprocess import DEVNULL
from airflow_breeze.utils.console import Output, get_console
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
from airflow_breeze.utils.run_utils import run_command
+from airflow_breeze.utils.suspended_providers import get_suspended_providers_folders
def verify_an_image(
@@ -89,3 +91,277 @@ def run_docker_compose_tests(
def file_name_from_test_type(test_type: str):
test_type_no_brackets = test_type.lower().replace("[", "_").replace("]", "")
return re.sub("[,.]", "_", test_type_no_brackets)[:30]
+
+
+def test_paths(test_type: str, backend: str, helm_test_package: str | None) -> tuple[str, str, str]:
+ file_friendly_test_type = file_name_from_test_type(test_type)
+ extra_package = f"-{helm_test_package}" if helm_test_package else ""
+ random_suffix = os.urandom(4).hex()
+ result_log_file = f"/files/test_result-{file_friendly_test_type}{extra_package}-{backend}.xml"
+ warnings_file = f"/files/warnings-{file_friendly_test_type}{extra_package}-{backend}.txt"
+ coverage_file = f"/files/coverage-{file_friendly_test_type}-{backend}-{random_suffix}.xml"
+ return result_log_file, warnings_file, coverage_file
+
+
+def get_suspended_provider_args() -> list[str]:
+ pytest_args = []
+ suspended_folders = get_suspended_providers_folders()
+ for providers in suspended_folders:
+ pytest_args.extend(
+ [
+ "--ignore",
+ f"tests/providers/{providers}",
+ "--ignore",
+ f"tests/system/providers/{providers}",
+ "--ignore",
+ f"tests/integration/providers/{providers}",
+ ]
+ )
+ return pytest_args
+
+
+TEST_TYPE_MAP_TO_PYTEST_ARGS: dict[str, list[str]] = {
+ "Always": ["tests/always"],
+ "API": ["tests/api", "tests/api_experimental", "tests/api_connexion", "tests/api_internal"],
+ "BranchExternalPython": [
+ "tests/operators/test_python.py::TestBranchExternalPythonOperator",
+ ],
+ "CLI": ["tests/cli"],
+ "Core": [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ ],
+ "ExternalPython": [
+ "tests/operators/test_python.py::TestExternalPythonOperator",
+ ],
+ "Integration": ["tests/integration"],
+ # Operators test type excludes Virtualenv/External tests - they have their own test types
+ "Operators": ["tests/operators", "--exclude-virtualenv-operator", "--exclude-external-python-operator"],
+ # this one is mysteriously failing dill serialization. It could be removed once
+ # https://github.com/pytest-dev/pytest/issues/10845 is fixed
+ "PlainAsserts": [
+ "tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context",
+ "--assert=plain",
+ ],
+ "Providers": ["tests/providers"],
+ "PythonVenv": [
+ "tests/operators/test_python.py::TestPythonVirtualenvOperator",
+ ],
+ "Serialization": [
+ "tests/serialization",
+ ],
+ "System": ["tests/system"],
+ "WWW": [
+ "tests/www",
+ ],
+}
+
+HELM_TESTS = "helm_tests"
+INTEGRATION_TESTS = "tests/integration"
+SYSTEM_TESTS = "tests/system"
+
+# Those directories are already ignored vu pyproject.toml. We want to exclude them here as well.
+NO_RECURSE_DIRS = [
+ "tests/dags_with_system_exit",
+ "tests/test_utils",
+ "tests/dags_corrupted",
+ "tests/dags",
+ "tests/system/providers/google/cloud/dataproc/resources",
+ "tests/system/providers/google/cloud/gcs/resources",
+]
+
+
+def find_all_other_tests() -> list[str]:
+ all_named_test_folders = list(chain.from_iterable(TEST_TYPE_MAP_TO_PYTEST_ARGS.values()))
+ all_named_test_folders.append(HELM_TESTS)
+ all_named_test_folders.append(INTEGRATION_TESTS)
+ all_named_test_folders.append(SYSTEM_TESTS)
+ all_named_test_folders.extend(NO_RECURSE_DIRS)
+
+ all_curent_test_folders = [
+ str(path.relative_to(AIRFLOW_SOURCES_ROOT))
+ for path in AIRFLOW_SOURCES_ROOT.glob("tests/*")
+ if path.is_dir() and path.name != "__pycache__"
+ ]
+ for named_test_folder in all_named_test_folders:
+ if named_test_folder in all_curent_test_folders:
+ all_curent_test_folders.remove(named_test_folder)
+ return sorted(all_curent_test_folders)
+
+
+PROVIDERS_LIST_PREFIX = "Providers["
+PROVIDERS_LIST_EXCLUDE_PREFIX = "Providers[-"
+
+ALL_TEST_SUITES: dict[str, tuple[str, ...]] = {
+ "All": ("tests",),
+ "All-Long": ("tests", "-m", "long_running", "--include-long-running"),
+ "All-Quarantined": ("tests", "-m", "quarantined", "--include-quarantined"),
+ "All-Postgres": ("tests", "--backend", "postgres"),
+ "All-MySQL": ("tests", "--backend", "mysql"),
+}
+
+
+def convert_test_type_to_pytest_args(
+ *,
+ test_type: str,
+ skip_provider_tests: bool,
+ helm_test_package: str | None = None,
+) -> list[str]:
+ if test_type == "None":
+ return []
+ if test_type in ALL_TEST_SUITES:
+ return [
+ *ALL_TEST_SUITES[test_type],
+ ]
+ if test_type == "Helm":
+ if helm_test_package and helm_test_package != "all":
+ return [f"helm_tests/{helm_test_package}"]
+ else:
+ return [HELM_TESTS]
+ if test_type == "Integration":
+ if skip_provider_tests:
+ return [
+ "tests/integration/api_experimental",
+ "tests/integration/cli",
+ "tests/integration/executors",
+ "tests/integration/security",
+ ]
+ else:
+ return [INTEGRATION_TESTS]
+ if test_type == "System":
+ return [SYSTEM_TESTS]
+ if skip_provider_tests and test_type.startswith("Providers"):
+ return []
+ if test_type.startswith(PROVIDERS_LIST_EXCLUDE_PREFIX):
+ excluded_provider_list = test_type[len(PROVIDERS_LIST_EXCLUDE_PREFIX) : -1].split(",")
+ providers_with_exclusions = TEST_TYPE_MAP_TO_PYTEST_ARGS["Providers"].copy()
+ for excluded_provider in excluded_provider_list:
+ providers_with_exclusions.append(
+ "--ignore=tests/providers/" + excluded_provider.replace(".", "/")
+ )
+ return providers_with_exclusions
+ if test_type.startswith(PROVIDERS_LIST_PREFIX):
+ provider_list = test_type[len(PROVIDERS_LIST_PREFIX) : -1].split(",")
+ providers_to_test = []
+ for provider in provider_list:
+ provider_path = "tests/providers/" + provider.replace(".", "/")
+ if (AIRFLOW_SOURCES_ROOT / provider_path).is_dir():
+ providers_to_test.append(provider_path)
+ else:
+ get_console().print(f"[warning]Provider {provider} does not exist. Ignoring it.")
+ return providers_to_test
+ if test_type == "Other":
+ return find_all_other_tests()
+ test_dirs = TEST_TYPE_MAP_TO_PYTEST_ARGS.get(test_type)
+ if test_dirs:
+ return test_dirs
+ get_console().print(f"[error]Unknown test type: {test_type}[/]")
+ sys.exit(1)
+
+
+def generate_args_for_pytest(
+ *,
+ test_type: str,
+ test_timeout: int,
+ skip_provider_tests: bool,
+ skip_db_tests: bool,
+ run_db_tests_only: bool,
+ backend: str,
+ use_xdist: bool,
+ enable_coverage: bool,
+ collect_only: bool,
+ parallelism: int,
+ parallel_test_types_list: list[str],
+ helm_test_package: str | None,
+):
+ result_log_file, warnings_file, coverage_file = test_paths(test_type, backend, helm_test_package)
+ if skip_db_tests:
+ if parallel_test_types_list:
+ args = convert_parallel_types_to_folders(parallel_test_types_list, skip_provider_tests)
+ else:
+ args = ["tests"] if test_type != "None" else []
+ else:
+ args = convert_test_type_to_pytest_args(
+ test_type=test_type, skip_provider_tests=skip_provider_tests, helm_test_package=helm_test_package
+ )
+ args.extend(
+ [
+ "--verbosity=0",
+ "--strict-markers",
+ "--durations=100",
+ "--maxfail=50",
+ "--color=yes",
+ f"--junitxml={result_log_file}",
+ # timeouts in seconds for individual tests
+ "--timeouts-order",
+ "moi",
+ f"--setup-timeout={test_timeout}",
+ f"--execution-timeout={test_timeout}",
+ f"--teardown-timeout={test_timeout}",
+ "--disable-warnings",
+ # Only display summary for non-expected cases
+ #
+ # f - failed
+ # E - error
+ # X - xpassed (passed even if expected to fail)
+ #
+ # The following cases are not displayed:
+ # x - xfailed (expected to fail and failed)
+ # p - passed
+ # P - passed with output
+ #
+ "-rfEX",
+ ]
+ )
+ if skip_db_tests:
+ args.append("--skip-db-tests")
+ if run_db_tests_only:
+ args.append("--run-db-tests-only")
+ if test_type != "System":
+ args.append(f"--ignore={SYSTEM_TESTS}")
+ if test_type != "Integration":
+ args.append(f"--ignore={INTEGRATION_TESTS}")
+ if test_type != "Helm":
+ # do not produce warnings output for helm tests
+ args.append(f"--warning-output-path={warnings_file}")
+ args.append(f"--ignore={HELM_TESTS}")
+ if test_type not in ("Helm", "System"):
+ args.append("--with-db-init")
+ args.extend(get_suspended_provider_args())
+ if use_xdist:
+ args.extend(["-n", str(parallelism) if parallelism else "auto"])
+ if enable_coverage:
+ args.extend(
+ [
+ "--cov=airflow",
+ "--cov-config=pyproject.toml",
+ f"--cov-report=xml:{coverage_file}",
+ ]
+ )
+ else:
+ args.append("--no-cov")
+ if collect_only:
+ args.extend(
+ [
+ "--collect-only",
+ "-qqqq",
+ "--disable-warnings",
+ ]
+ )
+ return args
+
+
+def convert_parallel_types_to_folders(parallel_test_types_list: list[str], skip_provider_tests: bool):
+ args = []
+ for _test_type in parallel_test_types_list:
+ args.extend(
+ convert_test_type_to_pytest_args(
+ test_type=_test_type, skip_provider_tests=skip_provider_tests, helm_test_package=None
+ )
+ )
+ # leave only folders, strip --pytest-args
+ return [arg for arg in args if arg.startswith("test")]
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index a60a4d1fdc2dd..65c1a0aed2db2 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -18,17 +18,12 @@
import json
import os
+import re
import sys
from enum import Enum
from functools import cached_property, lru_cache
-from re import match
from typing import Any, Dict, List, TypeVar
-if sys.version_info >= (3, 9):
- from typing import Literal
-else:
- from typing import Literal
-
from airflow_breeze.global_constants import (
ALL_PYTHON_MAJOR_MINOR_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
@@ -70,6 +65,13 @@
USE_PUBLIC_RUNNERS_LABEL = "use public runners"
UPGRADE_TO_NEWER_DEPENDENCIES_LABEL = "upgrade to newer dependencies"
+ALL_CI_SELECTIVE_TEST_TYPES = (
+ "API Always BranchExternalPython "
+ "CLI Core ExternalPython Operators Other PlainAsserts "
+ "Providers[-amazon,google] Providers[amazon] Providers[google] "
+ "PythonVenv Serialization WWW"
+)
+
class FileGroupForCi(Enum):
ENVIRONMENT_FILES = "environment_files"
@@ -190,11 +192,18 @@ def __hash__(self):
r"^airflow/cli",
r"^tests/cli",
],
+ SelectiveUnitTestTypes.OPERATORS: [
+ r"^airflow/operators",
+ r"^tests/operators",
+ ],
SelectiveUnitTestTypes.PROVIDERS: [
r"^airflow/providers/",
r"^tests/system/providers/",
r"^tests/providers/",
],
+ SelectiveUnitTestTypes.PYTHON_VENV: [
+ r"^tests/operators/test_python.py",
+ ],
SelectiveUnitTestTypes.WWW: [r"^airflow/www", r"^tests/www"],
}
)
@@ -232,7 +241,7 @@ def find_provider_affected(changed_file: str, include_docs: bool) -> str | None:
def find_all_providers_affected(
changed_files: tuple[str, ...], include_docs: bool, fail_if_suspended_providers_affected: bool
-) -> list[str] | Literal["ALL_PROVIDERS"] | None:
+) -> list[str] | str | None:
all_providers: set[str] = set()
all_providers_affected = False
@@ -473,10 +482,8 @@ def kubernetes_combos_list_as_string(self) -> str:
def _match_files_with_regexps(self, matched_files, regexps):
for file in self._files:
- for regexp in regexps:
- if match(regexp, file):
- matched_files.append(file)
- break
+ if any(re.match(regexp, file) for regexp in regexps):
+ matched_files.append(file)
@lru_cache(maxsize=None)
def _matching_files(self, match_group: T, match_dict: dict[T, list[str]]) -> list[str]:
@@ -589,6 +596,9 @@ def _get_test_types_to_run(self) -> list[str]:
matched_files.update(
self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.CLI)
)
+ matched_files.update(
+ self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.OPERATORS)
+ )
matched_files.update(
self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.API)
)
@@ -621,6 +631,7 @@ def _get_test_types_to_run(self) -> list[str]:
get_console().print(
"[warning]There are no core/other files. Only tests relevant to the changed files are run.[/]"
)
+ # sort according to predefined order
sorted_candidate_test_types = sorted(candidate_test_types)
get_console().print("[warning]Selected test type candidates to run:[/]")
get_console().print(sorted_candidate_test_types)
@@ -673,12 +684,7 @@ def parallel_test_types_list_as_string(self) -> str | None:
current_test_types = current_test_types - test_types_to_remove
self._extract_long_provider_tests(current_test_types)
-
- # this should be hard-coded as we want to have very specific sequence of tests
- sorting_order = ["Core", "Providers[-amazon,google]", "Other", "Providers[amazon]", "WWW"]
- sort_key = {item: i for i, item in enumerate(sorting_order)}
- # Put the test types in the order we want them to run
- return " ".join(sorted(current_test_types, key=lambda x: (sort_key.get(x, len(sorting_order)), x)))
+ return " ".join(sorted(current_test_types))
@cached_property
def basic_checks_only(self) -> bool:
@@ -693,12 +699,12 @@ def upgrade_to_newer_dependencies(self) -> bool:
)
@cached_property
- def docs_filter_list_as_string(self) -> str | None:
+ def docs_list_as_string(self) -> str | None:
_ALL_DOCS_LIST = ""
if not self.docs_build:
return None
if self._default_branch != "main":
- return "--package-filter apache-airflow --package-filter docker-stack"
+ return "apache-airflow docker-stack"
if self.full_tests_needed:
return _ALL_DOCS_LIST
providers_affected = find_all_providers_affected(
@@ -717,15 +723,15 @@ def docs_filter_list_as_string(self) -> str | None:
if any(file.startswith(("airflow/", "docs/apache-airflow/")) for file in self._files):
packages.append("apache-airflow")
if any(file.startswith("docs/apache-airflow-providers/") for file in self._files):
- packages.append("apache-airflow-providers")
+ packages.append("providers-index")
if any(file.startswith(("chart/", "docs/helm-chart")) for file in self._files):
packages.append("helm-chart")
if any(file.startswith("docs/docker-stack/") for file in self._files):
packages.append("docker-stack")
if providers_affected:
for provider in providers_affected:
- packages.append(f"apache-airflow-providers-{provider.replace('.', '-')}")
- return " ".join([f"--package-filter {package}" for package in packages])
+ packages.append(provider.replace("-", "."))
+ return " ".join(packages)
@cached_property
def skip_pre_commits(self) -> str:
@@ -799,7 +805,84 @@ def runs_on(self) -> str:
return RUNS_ON_SELF_HOSTED_RUNNER
return RUNS_ON_PUBLIC_RUNNER
+ @cached_property
+ def is_self_hosted_runner(self) -> bool:
+ """
+ True if the job has runs_on labels indicating It should run on "self-hosted" runner.
+
+ All self-hosted runners have "self-hosted" label.
+ """
+ return "self-hosted" in json.loads(self.runs_on)
+
+ @cached_property
+ def is_airflow_runner(self) -> bool:
+ """
+ True if the job has runs_on labels indicating It should run on Airflow managed runner.
+
+ All Airflow team-managed runners will have "airflow-runner" label.
+ """
+ # TODO: when we have it properly set-up with labels we should just check for
+ # "airflow-runner" presence in runs_on
+ runs_on_array = json.loads(self.runs_on)
+ return "Linux" in runs_on_array and "X64" in runs_on_array and "self-hosted" in runs_on_array
+
+ @cached_property
+ def is_amd_runner(self) -> bool:
+ """
+ True if the job has runs_on labels indicating AMD architecture.
+
+ Matching amd label, asf-runner, and any ubuntu that does not contain arm
+ The last case is just in case - currently there are no public runners that have ARM
+ instances, but they can add them in the future. It might be that for compatibility
+ they will just add arm in the runner name - because currently GitHub users use just
+ one label "ubuntu-*" for all their work and depend on them being AMD ones.
+ """
+ return any(
+ [
+ "amd" == label.lower()
+ or "amd64" == label.lower()
+ or "x64" == label.lower()
+ or "asf-runner" == label
+ or ("ubuntu" in label and "arm" not in label.lower())
+ for label in json.loads(self.runs_on)
+ ]
+ )
+
+ @cached_property
+ def is_arm_runner(self) -> bool:
+ """
+ True if the job has runs_on labels indicating ARM architecture.
+
+ Matches any label containing arm - including ASF-specific "asf-arm" label.
+
+ # See https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=INFRA&title=ASF+Infra+provided+self-hosted+runners
+ """
+ return any(
+ [
+ "arm" == label.lower() or "arm64" == label.lower() or "asf-arm" == label
+ for label in json.loads(self.runs_on)
+ ]
+ )
+
+ @cached_property
+ def is_vm_runner(self) -> bool:
+ """Whether the runner is VM runner (managed by airflow)."""
+ # TODO: when we have it properly set-up with labels we should just check for
+ # "airflow-runner" presence in runs_on
+ return self.is_airflow_runner
+
+ @cached_property
+ def is_k8s_runner(self) -> bool:
+ """Whether the runner is K8s runner (managed by airflow)."""
+ # TODO: when we have it properly set-up with labels we should just check for
+ # "k8s-runner" presence in runs_on
+ return False
+
@cached_property
def mssql_parallelism(self) -> int:
# Limit parallelism for MSSQL to 1 for public runners due to race conditions generated there
return SELF_HOSTED_RUNNERS_CPU_COUNT if self.runs_on == RUNS_ON_SELF_HOSTED_RUNNER else 1
+
+ @cached_property
+ def has_migrations(self) -> bool:
+ return any([file.startswith("airflow/migrations/") for file in self._files])
diff --git a/dev/breeze/src/airflow_breeze/utils/suspended_providers.py b/dev/breeze/src/airflow_breeze/utils/suspended_providers.py
index 19db6381d95f0..490393e86ff7f 100644
--- a/dev/breeze/src/airflow_breeze/utils/suspended_providers.py
+++ b/dev/breeze/src/airflow_breeze/utils/suspended_providers.py
@@ -16,8 +16,6 @@
# under the License.
from __future__ import annotations
-import yaml
-
from airflow_breeze.utils.path_utils import AIRFLOW_PROVIDERS_ROOT, AIRFLOW_SOURCES_ROOT
@@ -26,6 +24,8 @@ def get_suspended_providers_folders() -> list[str]:
Returns a list of suspended providers folders that should be
skipped when running tests (without any prefix - for example apache/beam, yandex, google etc.).
"""
+ import yaml
+
suspended_providers = []
for provider_path in AIRFLOW_PROVIDERS_ROOT.rglob("provider.yaml"):
provider_yaml = yaml.safe_load(provider_path.read_text())
@@ -42,6 +42,8 @@ def get_suspended_provider_ids() -> list[str]:
"""
Yields the ids of suspended providers.
"""
+ import yaml
+
suspended_provider_ids = []
for provider_path in AIRFLOW_PROVIDERS_ROOT.rglob("provider.yaml"):
provider_yaml = yaml.safe_load(provider_path.read_text())
diff --git a/dev/breeze/tests/conftest.py b/dev/breeze/tests/conftest.py
index 732638dda698d..17d7366abdc81 100644
--- a/dev/breeze/tests/conftest.py
+++ b/dev/breeze/tests/conftest.py
@@ -17,7 +17,6 @@
from __future__ import annotations
-# content of conftest.py
def pytest_configure(config):
import sys
diff --git a/dev/breeze/tests/test_cache.py b/dev/breeze/tests/test_cache.py
index 27d7eef487f96..95fae7696b587 100644
--- a/dev/breeze/tests/test_cache.py
+++ b/dev/breeze/tests/test_cache.py
@@ -34,8 +34,8 @@
@pytest.mark.parametrize(
"parameter, value, result, exception",
[
- ("backend", "mysql", (True, ["sqlite", "mysql", "postgres", "mssql"]), None),
- ("backend", "xxx", (False, ["sqlite", "mysql", "postgres", "mssql"]), None),
+ ("backend", "mysql", (True, ["sqlite", "mysql", "postgres", "mssql", "none"]), None),
+ ("backend", "xxx", (False, ["sqlite", "mysql", "postgres", "mssql", "none"]), None),
("python_major_minor_version", "3.8", (True, ["3.8", "3.9", "3.10", "3.11"]), None),
("python_major_minor_version", "3.7", (False, ["3.8", "3.9", "3.10", "3.11"]), None),
("missing", "value", None, AttributeError),
diff --git a/dev/breeze/tests/test_general_utils.py b/dev/breeze/tests/test_general_utils.py
new file mode 100644
index 0000000000000..b32b410d17bb6
--- /dev/null
+++ b/dev/breeze/tests/test_general_utils.py
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from airflow_breeze.utils.general_utils import get_docs_filter_name_from_short_hand
+
+
+@pytest.mark.parametrize(
+ "short_form_providers, expected",
+ [
+ pytest.param(
+ ("awesome", "foo.bar"),
+ ("apache-airflow-providers-awesome", "apache-airflow-providers-foo-bar"),
+ id="providers",
+ ),
+ pytest.param(
+ ("apache-airflow", "helm-chart", "docker-stack"),
+ ("apache-airflow", "helm-chart", "docker-stack"),
+ id="non-providers-docs",
+ ),
+ pytest.param(("providers-index",), ("apache-airflow-providers",), id="providers-index"),
+ pytest.param(
+ ("docker", "docker-stack", "providers-index"),
+ ("apache-airflow-providers-docker", "docker-stack", "apache-airflow-providers"),
+ id="mixin",
+ ),
+ ],
+)
+def test_get_provider_name_from_short_hand(short_form_providers, expected):
+ assert get_docs_filter_name_from_short_hand(short_form_providers) == expected
diff --git a/dev/breeze/tests/test_pr_info.py b/dev/breeze/tests/test_pr_info.py
index 7a8d910fe99db..3513cb56704c6 100644
--- a/dev/breeze/tests/test_pr_info.py
+++ b/dev/breeze/tests/test_pr_info.py
@@ -39,7 +39,7 @@ def test_pr_info():
assert wi.head_repo == "test/airflow"
assert wi.event_name == "pull_request"
assert wi.pr_number == 26004
- assert wi.get_runs_on() == "ubuntu-22.04"
+ assert wi.get_runs_on() == '["ubuntu-22.04"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -53,7 +53,7 @@ def test_push_info():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "push"
assert wi.pr_number is None
- assert wi.get_runs_on() == "ubuntu-22.04"
+ assert wi.get_runs_on() == '["ubuntu-22.04"]'
assert wi.is_canary_run() == "true"
assert wi.run_coverage() == "true"
@@ -67,7 +67,7 @@ def test_schedule():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "schedule"
assert wi.pr_number is None
- assert wi.get_runs_on() == "ubuntu-22.04"
+ assert wi.get_runs_on() == '["ubuntu-22.04"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -81,7 +81,7 @@ def test_runs_on_self_hosted():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "pull_request"
assert wi.pr_number == 1234
- assert wi.get_runs_on() == "self-hosted"
+ assert wi.get_runs_on() == '["self-hosted", "Linux", "X64"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -95,7 +95,7 @@ def test_runs_on_forced_public_runner():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "pull_request"
assert wi.pr_number == 1234
- assert wi.get_runs_on() == "ubuntu-22.04"
+ assert wi.get_runs_on() == '["ubuntu-22.04"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -109,7 +109,7 @@ def test_runs_on_simple_pr_other_repo():
assert wi.head_repo == "test/airflow"
assert wi.event_name == "pull_request"
assert wi.pr_number == 1234
- assert wi.get_runs_on() == "ubuntu-22.04"
+ assert wi.get_runs_on() == '["ubuntu-22.04"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -123,7 +123,7 @@ def test_runs_on_push_other_branch():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "push"
assert wi.pr_number is None
- assert wi.get_runs_on() == "self-hosted"
+ assert wi.get_runs_on() == '["self-hosted", "Linux", "X64"]'
assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -137,6 +137,6 @@ def test_runs_on_push_v_test_branch():
assert wi.head_repo == "apache/airflow"
assert wi.event_name == "push"
assert wi.pr_number is None
- assert wi.get_runs_on() == "self-hosted"
+ assert wi.get_runs_on() == '["self-hosted", "Linux", "X64"]'
assert wi.is_canary_run() == "true"
assert wi.run_coverage() == "false"
diff --git a/dev/breeze/tests/test_pytest_args_for_test_types.py b/dev/breeze/tests/test_pytest_args_for_test_types.py
new file mode 100644
index 0000000000000..7a13b82032114
--- /dev/null
+++ b/dev/breeze/tests/test_pytest_args_for_test_types.py
@@ -0,0 +1,315 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from airflow_breeze.utils.run_tests import convert_parallel_types_to_folders, convert_test_type_to_pytest_args
+
+
+@pytest.mark.parametrize(
+ "test_type, pytest_args, skip_provider_tests",
+ [
+ # Those list needs to be updated every time we add a new directory to tests/ folder
+ (
+ "Core",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ ],
+ False,
+ ),
+ (
+ "Integration",
+ ["tests/integration"],
+ False,
+ ),
+ (
+ "Integration",
+ [
+ "tests/integration/api_experimental",
+ "tests/integration/cli",
+ "tests/integration/executors",
+ "tests/integration/security",
+ ],
+ True,
+ ),
+ (
+ "API",
+ ["tests/api", "tests/api_experimental", "tests/api_connexion", "tests/api_internal"],
+ False,
+ ),
+ (
+ "Serialization",
+ ["tests/serialization"],
+ False,
+ ),
+ (
+ "System",
+ ["tests/system"],
+ False,
+ ),
+ (
+ "Operators",
+ ["tests/operators", "--exclude-virtualenv-operator", "--exclude-external-python-operator"],
+ False,
+ ),
+ (
+ "Providers",
+ ["tests/providers"],
+ False,
+ ),
+ (
+ "Providers",
+ [],
+ True,
+ ),
+ (
+ "Providers[amazon]",
+ ["tests/providers/amazon"],
+ False,
+ ),
+ (
+ "Providers[amazon,google,apache.hive]",
+ ["tests/providers/amazon", "tests/providers/google", "tests/providers/apache/hive"],
+ False,
+ ),
+ (
+ "Providers[-amazon,google,microsoft.azure]",
+ [
+ "tests/providers",
+ "--ignore=tests/providers/amazon",
+ "--ignore=tests/providers/google",
+ "--ignore=tests/providers/microsoft/azure",
+ ],
+ False,
+ ),
+ (
+ "PlainAsserts",
+ [
+ "tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context",
+ "--assert=plain",
+ ],
+ False,
+ ),
+ (
+ "All-Quarantined",
+ ["tests", "-m", "quarantined", "--include-quarantined"],
+ False,
+ ),
+ (
+ "PythonVenv",
+ [
+ "tests/operators/test_python.py::TestPythonVirtualenvOperator",
+ ],
+ False,
+ ),
+ (
+ "ExternalPython",
+ [
+ "tests/operators/test_python.py::TestExternalPythonOperator",
+ ],
+ False,
+ ),
+ (
+ "BranchExternalPython",
+ [
+ "tests/operators/test_python.py::TestBranchExternalPythonOperator",
+ ],
+ False,
+ ),
+ (
+ "Other",
+ [
+ "tests/auth",
+ "tests/callbacks",
+ "tests/charts",
+ "tests/cluster_policies",
+ "tests/config_templates",
+ "tests/dag_processing",
+ "tests/datasets",
+ "tests/decorators",
+ "tests/hooks",
+ "tests/lineage",
+ "tests/listeners",
+ "tests/macros",
+ "tests/notifications",
+ "tests/plugins",
+ "tests/secrets",
+ "tests/security",
+ "tests/sensors",
+ "tests/task",
+ "tests/template",
+ "tests/testconfig",
+ "tests/timetables",
+ "tests/triggers",
+ ],
+ False,
+ ),
+ ],
+)
+def test_pytest_args_for_regular_test_types(
+ test_type: str,
+ pytest_args: list[str],
+ skip_provider_tests: bool,
+):
+ assert (
+ convert_test_type_to_pytest_args(
+ test_type=test_type,
+ skip_provider_tests=skip_provider_tests,
+ )
+ == pytest_args
+ )
+
+
+@pytest.mark.parametrize(
+ "helm_test_package, pytest_args",
+ [
+ (
+ None,
+ ["helm_tests"],
+ ),
+ (
+ "airflow_aux",
+ ["helm_tests/airflow_aux"],
+ ),
+ (
+ "all",
+ ["helm_tests"],
+ ),
+ ],
+)
+def test_pytest_args_for_helm_test_types(helm_test_package: str, pytest_args: list[str]):
+ assert (
+ convert_test_type_to_pytest_args(
+ test_type="Helm", skip_provider_tests=False, helm_test_package=helm_test_package
+ )
+ == pytest_args
+ )
+
+
+@pytest.mark.parametrize(
+ "parallel_test_types, folders, skip_provider_tests",
+ [
+ (
+ "API",
+ ["tests/api", "tests/api_experimental", "tests/api_connexion", "tests/api_internal"],
+ False,
+ ),
+ (
+ "CLI",
+ [
+ "tests/cli",
+ ],
+ False,
+ ),
+ (
+ "API CLI",
+ [
+ "tests/api",
+ "tests/api_experimental",
+ "tests/api_connexion",
+ "tests/api_internal",
+ "tests/cli",
+ ],
+ False,
+ ),
+ (
+ "Core",
+ ["tests/core", "tests/executors", "tests/jobs", "tests/models", "tests/ti_deps", "tests/utils"],
+ False,
+ ),
+ (
+ "Core Providers",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ "tests/providers",
+ ],
+ False,
+ ),
+ (
+ "Core Providers[amazon]",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ "tests/providers/amazon",
+ ],
+ False,
+ ),
+ (
+ "Core Providers[amazon] Providers[google]",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ "tests/providers/amazon",
+ "tests/providers/google",
+ ],
+ False,
+ ),
+ (
+ "Core Providers[-amazon,google]",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ "tests/providers",
+ ],
+ False,
+ ),
+ (
+ "Core Providers[amazon] Providers[google]",
+ [
+ "tests/core",
+ "tests/executors",
+ "tests/jobs",
+ "tests/models",
+ "tests/ti_deps",
+ "tests/utils",
+ ],
+ True,
+ ),
+ ],
+)
+def test_folders_for_parallel_test_types(
+ parallel_test_types: str, folders: list[str], skip_provider_tests: bool
+):
+ assert (
+ convert_parallel_types_to_folders(
+ parallel_test_types_list=parallel_test_types.split(" "), skip_provider_tests=skip_provider_tests
+ )
+ == folders
+ )
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index df765cb344c4f..3b8ab08502cd1 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -17,12 +17,14 @@
from __future__ import annotations
import re
+from functools import lru_cache
from typing import Any
import pytest
+from rich.console import Console
from airflow_breeze.global_constants import COMMITTERS, GithubEvents
-from airflow_breeze.utils.selective_checks import SelectiveChecks
+from airflow_breeze.utils.selective_checks import ALL_CI_SELECTIVE_TEST_TYPES, SelectiveChecks
ANSI_COLORS_MATCHER = re.compile(r"(?:\x1B[@-_]|[\x80-\x9F])[0-?]*[ -/]*[@-~]")
@@ -35,17 +37,51 @@ def escape_ansi_colors(line):
return ANSI_COLORS_MATCHER.sub("", line)
+# Can be replaced with cache when we move to Python 3.9 (when 3.8 is EOL)
+@lru_cache(maxsize=None)
+def get_rich_console() -> Console:
+ return Console(color_system="truecolor", force_terminal=True)
+
+
+def print_in_color(s: Any = ""):
+ get_rich_console().print(s)
+
+
def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
escaped_stderr = escape_ansi_colors(stderr)
- for name, value in expected_outputs.items():
- if value is None:
- search_string = rf"^{re.escape(name)}="
- if re.search(search_string, escaped_stderr, re.MULTILINE):
- raise AssertionError(f"The {name} output should not be in {escaped_stderr}")
+ received_output_as_dict = dict(line.split("=", 1) for line in escaped_stderr.splitlines() if "=" in line)
+ for expected_key, expected_value in expected_outputs.items():
+ if expected_value is None:
+ if expected_key in received_output_as_dict:
+ print_in_color(f"\n[red]ERROR: The '{expected_key}' should not be present in:[/]")
+ print_in_color(received_output_as_dict)
+ print_in_color("\n")
+ assert expected_key is not None
+
else:
- search_string = rf"^{re.escape(name)}={re.escape(value)}$"
- if not re.search(search_string, escaped_stderr, re.MULTILINE):
- raise AssertionError(f"Expected {name}={value} not found in {escaped_stderr}")
+ received_value = received_output_as_dict.get(expected_key)
+ if received_value != expected_value:
+ if received_value is not None:
+ print_in_color(f"\n[red]ERROR: The key '{expected_key}' has unexpected value:")
+ print(received_value)
+ print_in_color("Expected value:\n")
+ print(expected_value)
+ print_in_color("\nOutput received:")
+ print_in_color(received_output_as_dict)
+ print_in_color()
+ assert expected_value == received_value
+ else:
+ print(
+ f"\n[red]ERROR: The key '{expected_key}' missing but "
+ f"it is expected. Expected value:"
+ )
+ print_in_color(expected_value)
+ print_in_color("\nOutput received:")
+ print(received_output_as_dict)
+ print_in_color()
+ print(received_output_as_dict)
+ print_in_color()
+ assert received_value is not None
@pytest.mark.parametrize(
@@ -91,6 +127,26 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
id="Only API tests and DOCS should run",
)
),
+ (
+ pytest.param(
+ ("airflow/operators/file.py",),
+ {
+ "affected-providers-list-as-string": None,
+ "all-python-versions": "['3.8']",
+ "all-python-versions-list-as-string": "3.8",
+ "python-versions": "['3.8']",
+ "python-versions-list-as-string": "3.8",
+ "image-build": "true",
+ "needs-helm-tests": "false",
+ "run-tests": "true",
+ "run-amazon-tests": "false",
+ "docs-build": "true",
+ "upgrade-to-newer-dependencies": "false",
+ "parallel-test-types-list-as-string": "Always Operators",
+ },
+ id="Only Operator tests and DOCS should run",
+ )
+ ),
(
pytest.param(
(
@@ -109,8 +165,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Providers[amazon] "
- "API Always Providers[common.sql,openlineage,postgres] Providers[google]",
+ "parallel-test-types-list-as-string": "API Always Providers[amazon] "
+ "Providers[common.sql,openlineage,postgres] Providers[google]",
},
id="API and providers tests and docs should run",
)
@@ -176,8 +232,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"docs-build": "true",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Providers[amazon] "
- "Always Providers[common.sql,openlineage,postgres] Providers[google]",
+ "parallel-test-types-list-as-string": "Always Providers[amazon] "
+ "Providers[common.sql,openlineage,postgres] Providers[google]",
},
id="Helm tests, providers (both upstream and downstream),"
"kubernetes tests and docs should run",
@@ -204,8 +260,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"docs-build": "true",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Providers[amazon] Always "
- "Providers[airbyte,apache.livy,dbt.cloud,dingding,discord,http]",
+ "parallel-test-types-list-as-string": "Always "
+ "Providers[airbyte,apache.livy,dbt.cloud,dingding,discord,http] Providers[amazon]",
},
id="Helm tests, http and all relevant providers, kubernetes tests and "
"docs should run even if unimportant files were added",
@@ -277,9 +333,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"upgrade-to-newer-dependencies": "true",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Everything should run - including all providers and upgrading to "
"newer requirements as setup.py changed and all Python versions",
@@ -300,9 +354,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"upgrade-to-newer-dependencies": "true",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Everything should run and upgrading to newer requirements as dependencies change",
)
@@ -310,9 +362,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
pytest.param(
("airflow/providers/amazon/__init__.py",),
{
- "affected-providers-list-as-string": "amazon apache.hive cncf.kubernetes "
- "common.sql exasol ftp google http imap microsoft.azure "
- "mongo mysql postgres salesforce ssh",
+ "affected-providers-list-as-string": "amazon apache.hive cncf.kubernetes common.sql exasol "
+ "ftp google http imap microsoft.azure mongo mysql postgres salesforce ssh",
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
"python-versions": "['3.8']",
@@ -324,9 +375,10 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"run-amazon-tests": "true",
- "parallel-test-types-list-as-string": "Providers[amazon] Always "
- "Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,http,imap,microsoft.azure,"
- "mongo,mysql,postgres,salesforce,ssh] Providers[google]",
+ "parallel-test-types-list-as-string": "Always Providers[amazon] "
+ "Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,http,"
+ "imap,"
+ "microsoft.azure,mongo,mysql,postgres,salesforce,ssh] Providers[google]",
},
id="Providers tests run including amazon tests if amazon provider files changed",
),
@@ -352,9 +404,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
pytest.param(
("airflow/providers/amazon/file.py",),
{
- "affected-providers-list-as-string": "amazon apache.hive cncf.kubernetes "
- "common.sql exasol ftp google http imap microsoft.azure "
- "mongo mysql postgres salesforce ssh",
+ "affected-providers-list-as-string": "amazon apache.hive cncf.kubernetes common.sql exasol "
+ "ftp google http imap microsoft.azure mongo mysql postgres salesforce ssh",
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
"python-versions": "['3.8']",
@@ -366,9 +417,9 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"docs-build": "true",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Providers[amazon] Always "
- "Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,"
- "http,imap,microsoft.azure,mongo,mysql,postgres,salesforce,ssh] Providers[google]",
+ "parallel-test-types-list-as-string": "Always Providers[amazon] "
+ "Providers[apache.hive,cncf.kubernetes,common.sql,exasol,ftp,http,"
+ "imap,microsoft.azure,mongo,mysql,postgres,salesforce,ssh] Providers[google]",
},
id="Providers tests run including amazon tests if amazon provider files changed",
),
@@ -405,12 +456,10 @@ def test_expected_output_pull_request_main(
"image-build": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Everything should run including all providers when full tests are needed",
)
@@ -432,12 +481,10 @@ def test_expected_output_pull_request_main(
"image-build": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Everything should run including full providers when full "
"tests are needed even with different label set as well",
@@ -457,12 +504,10 @@ def test_expected_output_pull_request_main(
"image-build": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Everything should run including full providers when"
"full tests are needed even if no files are changed",
@@ -482,12 +527,13 @@ def test_expected_output_pull_request_main(
"image-build": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow "
- "--package-filter docker-stack",
+ "docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "true",
"skip-provider-tests": "true",
"upgrade-to-newer-dependencies": "false",
- "parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
+ "parallel-test-types-list-as-string": "API Always BranchExternalPython "
+ "CLI Core ExternalPython Operators Other PlainAsserts "
+ "PythonVenv Serialization WWW",
},
id="Everything should run except Providers when full tests are needed for non-main branch",
)
@@ -523,7 +569,7 @@ def test_expected_output_full_tests_needed(
"needs-helm-tests": "false",
"run-tests": "false",
"docs-build": "false",
- "docs-filter-list-as-string": None,
+ "docs-list-as-string": None,
"full-tests-needed": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
@@ -538,15 +584,15 @@ def test_expected_output_full_tests_needed(
),
{
"affected-providers-list-as-string": "amazon apache.beam apache.cassandra cncf.kubernetes "
- "common.sql facebook google hashicorp microsoft.azure microsoft.mssql "
- "mysql openlineage oracle postgres presto salesforce sftp ssh trino",
+ "common.sql facebook google hashicorp microsoft.azure microsoft.mssql mysql "
+ "openlineage oracle postgres presto salesforce sftp ssh trino",
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
"needs-helm-tests": "false",
"image-build": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter docker-stack",
+ "docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "false",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
@@ -562,17 +608,16 @@ def test_expected_output_full_tests_needed(
"tests/providers/google/file.py",
),
{
- "affected-providers-list-as-string": "amazon apache.beam apache.cassandra "
- "cncf.kubernetes common.sql facebook google "
- "hashicorp microsoft.azure microsoft.mssql mysql openlineage oracle postgres "
- "presto salesforce sftp ssh trino",
+ "affected-providers-list-as-string": "amazon apache.beam apache.cassandra cncf.kubernetes "
+ "common.sql facebook google hashicorp microsoft.azure microsoft.mssql "
+ "mysql openlineage oracle postgres presto salesforce sftp ssh trino",
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
"image-build": "true",
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter docker-stack",
+ "docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "false",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
@@ -594,12 +639,13 @@ def test_expected_output_full_tests_needed(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter docker-stack",
+ "docs-list-as-string": "apache-airflow docker-stack",
"full-tests-needed": "false",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
- "parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
+ "parallel-test-types-list-as-string": "API Always BranchExternalPython "
+ "CLI Core ExternalPython Operators Other PlainAsserts PythonVenv Serialization WWW",
},
id="All tests except Providers should run if core file changed in non-main branch",
),
@@ -632,7 +678,7 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "false",
"run-tests": "false",
"docs-build": "false",
- "docs-filter-list-as-string": None,
+ "docs-list-as-string": None,
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": None,
@@ -649,7 +695,7 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "Always",
@@ -663,9 +709,8 @@ def test_expected_output_pull_request_v2_3(
"tests/providers/google/file.py",
),
{
- "affected-providers-list-as-string": "amazon apache.beam apache.cassandra "
- "cncf.kubernetes common.sql "
- "facebook google hashicorp microsoft.azure microsoft.mssql mysql "
+ "affected-providers-list-as-string": "amazon apache.beam apache.cassandra cncf.kubernetes "
+ "common.sql facebook google hashicorp microsoft.azure microsoft.mssql mysql "
"openlineage oracle postgres presto salesforce sftp ssh trino",
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
@@ -673,39 +718,23 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter helm-chart "
- "--package-filter apache-airflow-providers-amazon "
- "--package-filter apache-airflow-providers-apache-beam "
- "--package-filter apache-airflow-providers-apache-cassandra "
- "--package-filter apache-airflow-providers-cncf-kubernetes "
- "--package-filter apache-airflow-providers-common-sql "
- "--package-filter apache-airflow-providers-facebook "
- "--package-filter apache-airflow-providers-google "
- "--package-filter apache-airflow-providers-hashicorp "
- "--package-filter apache-airflow-providers-microsoft-azure "
- "--package-filter apache-airflow-providers-microsoft-mssql "
- "--package-filter apache-airflow-providers-mysql "
- "--package-filter apache-airflow-providers-openlineage "
- "--package-filter apache-airflow-providers-oracle "
- "--package-filter apache-airflow-providers-postgres "
- "--package-filter apache-airflow-providers-presto "
- "--package-filter apache-airflow-providers-salesforce "
- "--package-filter apache-airflow-providers-sftp "
- "--package-filter apache-airflow-providers-ssh "
- "--package-filter apache-airflow-providers-trino",
+ "docs-list-as-string": "apache-airflow helm-chart amazon apache.beam apache.cassandra "
+ "cncf.kubernetes common.sql facebook google hashicorp microsoft.azure microsoft.mssql mysql "
+ "openlineage oracle postgres presto salesforce sftp ssh trino",
"run-kubernetes-tests": "true",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
- "parallel-test-types-list-as-string": "Providers[amazon] Always CLI "
- "Providers[apache.beam,apache.cassandra,cncf.kubernetes,common.sql,facebook,"
- "hashicorp,microsoft.azure,microsoft.mssql,mysql,openlineage,oracle,postgres,presto,"
- "salesforce,sftp,ssh,trino] Providers[google]",
+ "parallel-test-types-list-as-string": "Always CLI Providers[amazon] "
+ "Providers[apache.beam,apache.cassandra,cncf.kubernetes,common.sql,facebook,hashicorp,"
+ "microsoft.azure,microsoft.mssql,mysql,openlineage,oracle,postgres,presto,salesforce,"
+ "sftp,ssh,trino] Providers[google]",
},
id="CLI tests and Google-related provider tests should run if cli/chart files changed",
),
pytest.param(
(
"airflow/cli/file.py",
+ "airflow/operators/file.py",
"airflow/www/file.py",
"airflow/api/file.py",
),
@@ -717,13 +746,13 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow",
+ "docs-list-as-string": "apache-airflow",
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
- "parallel-test-types-list-as-string": "WWW API Always CLI",
+ "parallel-test-types-list-as-string": "API Always CLI Operators WWW",
},
- id="No providers tests should run if only CLI/API/WWW file changed",
+ id="No providers tests should run if only CLI/API/Operators/WWW file changed",
),
pytest.param(
("airflow/models/test.py",),
@@ -735,13 +764,11 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
- "Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="Tests for all providers should run if model file changed",
),
@@ -755,15 +782,13 @@ def test_expected_output_pull_request_v2_3(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
- "Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
- id="Tests for all providers should run if any other than API/WWW/CLI file changed.",
+ id="Tests for all providers should run if any other than API/WWW/CLI/Operators file changed.",
),
],
)
@@ -796,11 +821,9 @@ def test_expected_output_pull_request_target(
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "true",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
- "Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="All tests run on push even if unimportant file changed",
),
@@ -816,9 +839,10 @@ def test_expected_output_pull_request_target(
"needs-helm-tests": "false",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter docker-stack",
+ "docs-list-as-string": "apache-airflow docker-stack",
"upgrade-to-newer-dependencies": "true",
- "parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
+ "parallel-test-types-list-as-string": "API Always BranchExternalPython "
+ "CLI Core ExternalPython Operators Other PlainAsserts PythonVenv Serialization WWW",
},
id="All tests except Providers and Helm run on push"
" even if unimportant file changed in non-main branch",
@@ -835,11 +859,9 @@ def test_expected_output_pull_request_target(
"needs-helm-tests": "true",
"run-tests": "true",
"docs-build": "true",
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "true",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
- "Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
id="All tests run on push if core file changed",
),
@@ -890,9 +912,7 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
"upgrade-to-newer-dependencies": "true"
if github_event in [GithubEvents.PUSH, GithubEvents.SCHEDULE]
else "false",
- "parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
- "Other Providers[amazon] WWW "
- "API Always CLI Providers[google]",
+ "parallel-test-types-list-as-string": ALL_CI_SELECTIVE_TEST_TYPES,
},
str(stderr),
)
@@ -970,73 +990,33 @@ def test_upgrade_to_newer_dependencies(
pytest.param(
("docs/apache-airflow-providers-google/docs.rst",),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow-providers-amazon "
- "--package-filter apache-airflow-providers-apache-beam "
- "--package-filter apache-airflow-providers-apache-cassandra "
- "--package-filter apache-airflow-providers-cncf-kubernetes "
- "--package-filter apache-airflow-providers-common-sql "
- "--package-filter apache-airflow-providers-facebook "
- "--package-filter apache-airflow-providers-google "
- "--package-filter apache-airflow-providers-hashicorp "
- "--package-filter apache-airflow-providers-microsoft-azure "
- "--package-filter apache-airflow-providers-microsoft-mssql "
- "--package-filter apache-airflow-providers-mysql "
- "--package-filter apache-airflow-providers-openlineage "
- "--package-filter apache-airflow-providers-oracle "
- "--package-filter apache-airflow-providers-postgres "
- "--package-filter apache-airflow-providers-presto "
- "--package-filter apache-airflow-providers-salesforce "
- "--package-filter apache-airflow-providers-sftp "
- "--package-filter apache-airflow-providers-ssh "
- "--package-filter apache-airflow-providers-trino",
+ "docs-list-as-string": "amazon apache.beam apache.cassandra cncf.kubernetes common.sql "
+ "facebook google hashicorp microsoft.azure microsoft.mssql mysql "
+ "openlineage oracle postgres presto salesforce sftp ssh trino",
},
id="Google provider docs changed",
),
pytest.param(
("airflow/providers/common/sql/common_sql_python.py",),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow "
- "--package-filter apache-airflow-providers-amazon "
- "--package-filter apache-airflow-providers-apache-drill "
- "--package-filter apache-airflow-providers-apache-druid "
- "--package-filter apache-airflow-providers-apache-hive "
- "--package-filter apache-airflow-providers-apache-impala "
- "--package-filter apache-airflow-providers-apache-pinot "
- "--package-filter apache-airflow-providers-common-sql "
- "--package-filter apache-airflow-providers-databricks "
- "--package-filter apache-airflow-providers-elasticsearch "
- "--package-filter apache-airflow-providers-exasol "
- "--package-filter apache-airflow-providers-google "
- "--package-filter apache-airflow-providers-jdbc "
- "--package-filter apache-airflow-providers-microsoft-mssql "
- "--package-filter apache-airflow-providers-mysql "
- "--package-filter apache-airflow-providers-odbc "
- "--package-filter apache-airflow-providers-openlineage "
- "--package-filter apache-airflow-providers-oracle "
- "--package-filter apache-airflow-providers-postgres "
- "--package-filter apache-airflow-providers-presto "
- "--package-filter apache-airflow-providers-slack "
- "--package-filter apache-airflow-providers-snowflake "
- "--package-filter apache-airflow-providers-sqlite "
- "--package-filter apache-airflow-providers-trino "
- "--package-filter apache-airflow-providers-vertica",
+ "docs-list-as-string": "apache-airflow amazon apache.drill apache.druid apache.hive "
+ "apache.impala apache.pinot common.sql databricks elasticsearch "
+ "exasol google jdbc microsoft.mssql mysql odbc openlineage "
+ "oracle postgres presto slack snowflake sqlite trino vertica",
},
id="Common SQL provider package python files changed",
),
pytest.param(
("docs/apache-airflow-providers-airbyte/docs.rst",),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow-providers-airbyte "
- "--package-filter apache-airflow-providers-http",
+ "docs-list-as-string": "airbyte http",
},
id="Airbyte provider docs changed",
),
pytest.param(
("docs/apache-airflow-providers-airbyte/docs.rst", "docs/apache-airflow/docs.rst"),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow "
- "--package-filter apache-airflow-providers-airbyte "
- "--package-filter apache-airflow-providers-http",
+ "docs-list-as-string": "apache-airflow airbyte http",
},
id="Airbyte provider and airflow core docs changed",
),
@@ -1047,66 +1027,59 @@ def test_upgrade_to_newer_dependencies(
"docs/apache-airflow-providers/docs.rst",
),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow "
- "--package-filter apache-airflow-providers "
- "--package-filter apache-airflow-providers-airbyte "
- "--package-filter apache-airflow-providers-http",
+ "docs-list-as-string": "apache-airflow providers-index airbyte http",
},
id="Airbyte provider and airflow core and common provider docs changed",
),
pytest.param(
("docs/apache-airflow/docs.rst",),
{
- "docs-filter-list-as-string": "--package-filter apache-airflow",
+ "docs-list-as-string": "apache-airflow",
},
id="Only Airflow docs changed",
),
pytest.param(
("airflow/providers/celery/file.py",),
- {
- "docs-filter-list-as-string": "--package-filter apache-airflow "
- "--package-filter apache-airflow-providers-celery "
- "--package-filter apache-airflow-providers-cncf-kubernetes"
- },
+ {"docs-list-as-string": "apache-airflow celery cncf.kubernetes"},
id="Celery python files changed",
),
pytest.param(
("docs/conf.py",),
{
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
},
id="Docs conf.py changed",
),
pytest.param(
("airflow/test.py",),
{
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
},
id="Core files changed. All provider docs should also be built",
),
pytest.param(
("docs/docker-stack/test.rst",),
- {"docs-filter-list-as-string": "--package-filter docker-stack"},
+ {"docs-list-as-string": "docker-stack"},
id="Docker stack files changed. No provider docs to build",
),
pytest.param(
("airflow/test.py", "chart/airflow/values.yaml"),
{
- "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
+ "docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
},
id="Core files and helm chart files changed. All provider docs should be built",
),
pytest.param(
("chart/airflow/values.yaml",),
{
- "docs-filter-list-as-string": "--package-filter helm-chart",
+ "docs-list-as-string": "helm-chart",
},
id="Helm chart files changed. No provider, airflow docs to build",
),
pytest.param(
("docs/helm-chart/airflow/values.yaml",),
{
- "docs-filter-list-as-string": "--package-filter helm-chart",
+ "docs-list-as-string": "helm-chart",
},
id="Docs helm chart files changed. No provider, airflow docs to build",
),
@@ -1148,67 +1121,54 @@ def test_helm_tests_trigger_ci_build(files: tuple[str, ...], expected_outputs: d
@pytest.mark.parametrize(
- "files, labels, expected_outputs, should_fail",
+ "github_event, github_actor, github_repository, pr_labels, github_context_dict, "
+ "runs_on, "
+ "is_self_hosted_runner, is_airflow_runner, is_amd_runner, is_arm_runner, is_vm_runner, is_k8s_runner",
[
pytest.param(
- ("airflow/providers/yandex/test.py",),
- (),
- None,
- True,
- id="Suspended provider changes should fail",
- ),
- pytest.param(
- ("airflow/providers/yandex/test.py",),
- ("allow suspended provider changes",),
- {"affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED},
- False,
- id="Suspended provider changes should not fail if appropriate label is set",
+ GithubEvents.PUSH,
+ "user",
+ "apache/airflow",
+ [],
+ dict(),
+ '["self-hosted", "Linux", "X64"]',
+ "true",
+ "true",
+ "true",
+ "false",
+ "true",
+ "false",
+ id="Push event",
),
- pytest.param(
- ("airflow/providers/yandex/test.py", "airflow/providers/airbyte/test.py"),
- ("allow suspended provider changes",),
- {"affected-providers-list-as-string": "airbyte http"},
- False,
- id="Only non-suspended provider changes should be listed",
- ),
- ],
-)
-def test_suspended_providers(
- files: tuple[str, ...], labels: tuple[str], expected_outputs: dict[str, str], should_fail: bool
-):
- failed = False
- try:
- stderr = str(
- SelectiveChecks(
- files=files,
- commit_ref="HEAD",
- github_event=GithubEvents.PULL_REQUEST,
- pr_labels=labels,
- default_branch="main",
- )
- )
- except SystemExit:
- failed = True
- assert failed == should_fail
- if not failed:
- assert_outputs_are_printed(expected_outputs, str(stderr))
-
-
-@pytest.mark.parametrize(
- "github_event, github_actor, github_repository, pr_labels, github_context_dict, runs_on",
- [
- pytest.param(GithubEvents.PUSH, "user", "apache/airflow", [], dict(), "self-hosted", id="Push event"),
pytest.param(
GithubEvents.PUSH,
"user",
"private/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Push event for private repo",
),
pytest.param(
- GithubEvents.PULL_REQUEST, "user", "apache/airflow", [], dict(), "ubuntu-22.04", id="Pull request"
+ GithubEvents.PULL_REQUEST,
+ "user",
+ "apache/airflow",
+ [],
+ dict(),
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
+ id="Pull request",
),
pytest.param(
GithubEvents.PULL_REQUEST,
@@ -1216,7 +1176,13 @@ def test_suspended_providers(
"private/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request private repo",
),
pytest.param(
@@ -1225,7 +1191,13 @@ def test_suspended_providers(
"apache/airflow",
[],
dict(),
- "self-hosted",
+ '["self-hosted", "Linux", "X64"]',
+ "true",
+ "true",
+ "true",
+ "false",
+ "true",
+ "false",
id="Pull request committer",
),
pytest.param(
@@ -1234,7 +1206,13 @@ def test_suspended_providers(
"apache/airflow",
[],
dict(event=dict(pull_request=dict(user=dict(login="user")))),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request committer pr non-committer",
),
pytest.param(
@@ -1243,7 +1221,13 @@ def test_suspended_providers(
"private/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request private repo committer",
),
pytest.param(
@@ -1252,7 +1236,13 @@ def test_suspended_providers(
"apache/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request target",
),
pytest.param(
@@ -1261,7 +1251,13 @@ def test_suspended_providers(
"private/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request target private repo",
),
pytest.param(
@@ -1270,7 +1266,13 @@ def test_suspended_providers(
"apache/airflow",
[],
dict(),
- "self-hosted",
+ '["self-hosted", "Linux", "X64"]',
+ "true",
+ "true",
+ "true",
+ "false",
+ "true",
+ "false",
id="Pull request target committer",
),
pytest.param(
@@ -1279,7 +1281,13 @@ def test_suspended_providers(
"apache/airflow",
[],
dict(event=dict(pull_request=dict(user=dict(login="user")))),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request target committer pr non-committer",
),
pytest.param(
@@ -1288,7 +1296,13 @@ def test_suspended_providers(
"private/airflow",
[],
dict(),
- "ubuntu-22.04",
+ '["ubuntu-22.04"]',
+ "false",
+ "false",
+ "true",
+ "false",
+ "false",
+ "false",
id="Pull request targe private repo committer",
),
],
@@ -1300,6 +1314,12 @@ def test_runs_on(
pr_labels: list[str],
github_context_dict: dict[str, Any],
runs_on: str,
+ is_self_hosted_runner: str,
+ is_airflow_runner: str,
+ is_amd_runner: str,
+ is_arm_runner: str,
+ is_vm_runner: str,
+ is_k8s_runner: str,
):
stderr = SelectiveChecks(
files=(),
@@ -1312,3 +1332,36 @@ def test_runs_on(
default_branch="main",
)
assert_outputs_are_printed({"runs-on": runs_on}, str(stderr))
+ assert_outputs_are_printed({"is-self-hosted-runner": is_self_hosted_runner}, str(stderr))
+ assert_outputs_are_printed({"is-airflow-runner": is_airflow_runner}, str(stderr))
+ assert_outputs_are_printed({"is-amd-runner": is_amd_runner}, str(stderr))
+ assert_outputs_are_printed({"is-arm-runner": is_arm_runner}, str(stderr))
+ assert_outputs_are_printed({"is-vm-runner": is_vm_runner}, str(stderr))
+ assert_outputs_are_printed({"is-k8s-runner": is_k8s_runner}, str(stderr))
+
+
+@pytest.mark.parametrize(
+ "files, has_migrations",
+ [
+ pytest.param(
+ ("airflow/test.py",),
+ False,
+ id="No migrations",
+ ),
+ pytest.param(
+ ("airflow/migrations/test_sql", "aiflow/test.py"),
+ True,
+ id="With migrations",
+ ),
+ ],
+)
+def test_has_migrations(files: tuple[str, ...], has_migrations: bool):
+ stderr = str(
+ SelectiveChecks(
+ files=files,
+ commit_ref="HEAD",
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ )
+ assert_outputs_are_printed({"has-migrations": str(has_migrations).lower()}, str(stderr))
diff --git a/dev/provider_packages/PROVIDER_COMMITS_TEMPLATE.rst.jinja2 b/dev/provider_packages/PROVIDER_COMMITS_TEMPLATE.rst.jinja2
index 8997248cbbdab..182e907a5709c 100644
--- a/dev/provider_packages/PROVIDER_COMMITS_TEMPLATE.rst.jinja2
+++ b/dev/provider_packages/PROVIDER_COMMITS_TEMPLATE.rst.jinja2
@@ -40,6 +40,7 @@
specific language governing permissions and limitations
under the License.
+.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
Package {{ PACKAGE_PIP_NAME }}
------------------------------------------------------
diff --git a/dev/provider_packages/tag_providers.sh b/dev/provider_packages/tag_providers.sh
index 9a115c3ea07c4..19a8d17f59e74 100755
--- a/dev/provider_packages/tag_providers.sh
+++ b/dev/provider_packages/tag_providers.sh
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
+ else
+ echo "Failed to push tags, probably a connectivity issue to Github"
+ CLEAN_LOCAL_TAGS="${CLEAN_LOCAL_TAGS:-true}"
+ if [[ "$CLEAN_LOCAL_TAGS" == "true" ]]; then
+ echo "Cleaning up local tags..."
+ git tag -d "${tags[@]}"
+ else
+ echo "Local tags are not cleaned up, unset CLEAN_LOCAL_TAGS or set to true"
+ fi
+ fi
fi
diff --git a/dev/refresh_images.sh b/dev/refresh_images.sh
index 594db2cc36266..638e86930cc1f 100755
--- a/dev/refresh_images.sh
+++ b/dev/refresh_images.sh
@@ -36,7 +36,7 @@ breeze ci-image build \
rm -fv ./dist/* ./docker-context-files/*
breeze release-management prepare-provider-packages \
- --package-list-file ./scripts/ci/installed_providers.txt \
+ --package-list-file ./airflow/providers/installed_providers.txt \
--package-format wheel \
--version-suffix-for-pypi dev0
diff --git a/docker_tests/test_prod_image.py b/docker_tests/test_prod_image.py
index 01e271a4071a9..5dd4a1ebecf9d 100644
--- a/docker_tests/test_prod_image.py
+++ b/docker_tests/test_prod_image.py
@@ -38,7 +38,7 @@
# isort:on (needed to workaround isort bug)
from setup import PREINSTALLED_PROVIDERS
-INSTALLED_PROVIDER_PATH = SOURCE_ROOT / "scripts" / "ci" / "installed_providers.txt"
+INSTALLED_PROVIDER_PATH = SOURCE_ROOT / "airflow" / "providers" / "installed_providers.txt"
class TestCommands:
diff --git a/docs/apache-airflow-providers-opensearch/changelog.rst b/docs/apache-airflow-providers-opensearch/changelog.rst
new file mode 100644
index 0000000000000..d1bb0e4735929
--- /dev/null
+++ b/docs/apache-airflow-providers-opensearch/changelog.rst
@@ -0,0 +1,19 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../airflow/providers/opensearch/CHANGELOG.rst
diff --git a/docs/apache-airflow-providers-opensearch/commits.rst b/docs/apache-airflow-providers-opensearch/commits.rst
new file mode 100644
index 0000000000000..d36d55467eec3
--- /dev/null
+++ b/docs/apache-airflow-providers-opensearch/commits.rst
@@ -0,0 +1,30 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Package apache-airflow-providers-opensearch
+------------------------------------------------------
+
+`OpenSearch `.
+
+1.0.0
+.....
diff --git a/docs/apache-airflow-providers-opensearch/connections/index.rst b/docs/apache-airflow-providers-opensearch/connections/index.rst
new file mode 100644
index 0000000000000..72d2846638b3a
--- /dev/null
+++ b/docs/apache-airflow-providers-opensearch/connections/index.rst
@@ -0,0 +1,29 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+OpenSearch Connections
+=======================
+
+
+.. toctree::
+ :maxdepth: 1
+ :glob:
+
+ *
diff --git a/docs/apache-airflow-providers-opensearch/connections/opensearch.rst b/docs/apache-airflow-providers-opensearch/connections/opensearch.rst
new file mode 100644
index 0000000000000..781f5da502cd4
--- /dev/null
+++ b/docs/apache-airflow-providers-opensearch/connections/opensearch.rst
@@ -0,0 +1,37 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+OpenSearch Connection
+=====================
+The OpenSearch connection provides credentials for an OpenSearch instance.
+
+Configuring the Connection
+--------------------------
+Host (required)
+ The host address of the Open Search instance.
+Login (required)
+ The login user.
+Password (required)
+ The password for the login user.
+Extra (optional)
+ Specifying the extra parameters as a (json dictionary) that can be used in the OpenSearch connection.
+ The following parameters are all optional:
+
+ * ``use_ssl``: Boolean on requiring an ssl connection. Default is false.
+ * ``verify_certs``: Boolean indicating to verify certs for ssl. Default is false.
diff --git a/docs/apache-airflow-providers-opensearch/index.rst b/docs/apache-airflow-providers-opensearch/index.rst
new file mode 100644
index 0000000000000..d122ba3831577
--- /dev/null
+++ b/docs/apache-airflow-providers-opensearch/index.rst
@@ -0,0 +1,102 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+``apache-airflow-providers-opensearch``
+=======================================
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Basics
+
+ Home
+ Changelog
+ Security
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Guides
+
+ Connection types
+ Operators
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: References
+
+ Python API <_api/airflow/providers/opensearch/index>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: System tests
+
+ System Tests <_api/tests/system/providers/opensearch/index>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Resources
+
+ PyPI Repository
+
+.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Commits
+
+ Detailed list of commits
+
+
+Package apache-airflow-providers-opensearch
+------------------------------------------------------
+`OpenSearch `_ versioning policy. Depending on
+the scope of the change, Provider will get ''MAJOR'' version upgrade when there are
+breaking changes, ``MINOR`` version upgrade when there are new features or ``PATCHLEVEL`` version upgrade
+when there are only bug fixes (including security bugfixes) - and this is the only version that receives
+security fixes by default, so you should upgrade to latest version of the provider if you want to receive
+all released security fixes.
+
+The only exception to that rule is when we have a critical security fix and good reason to provide an
+out-of-band release for the provider, in which case stakeholders in the provider might decide to cherry-pick
+and prepare a branch for an older version of the provider following the
+`mixed governance model `_
+and requires interested parties to cherry-pick and test the fixes.
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
index 21ae801b64ba5..925013af8beef 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
@@ -30,7 +30,7 @@ can also customize it and configure it as you want by overriding Python logger c
be configured by providing custom logging configuration object. Some configuration options require
that the logging config class be overwritten. You can do it by copying the default
configuration of Airflow and modifying it to suit your needs. The default configuration can be seen in the
-`airflow_local_settings.py template `_
+`airflow_local_settings.py template `_
and you can see the loggers and handlers used there. Except the custom loggers and handlers configurable there
via the ``airflow.cfg``, the logging methods in Airflow follow the usual Python logging convention,
that Python objects log to loggers that follow naming convention of ``.``.
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/errors.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/errors.rst
index 1a114d5fb711f..cb09843422321 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/errors.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/errors.rst
@@ -49,7 +49,14 @@ The ``before_send`` option can be used to modify or drop events before they are
[sentry]
before_send = path.to.my.sentry.before_send
-You can supply `additional configuration options `__ based on the Python platform via ``[sentry]`` section. Unsupported options: ``integrations``, ``in_app_include``, ``in_app_exclude``, ``ignore_errors``, ``before_breadcrumb``, ``transport``.
+The ``transport`` option can be used to change the transport used to send events to Sentry, and possibly other Systems. To set this option, provide a dotted path to a Transport class that the sentry SDK should be configured to use.
+
+.. code-block:: ini
+
+ [sentry]
+ transport = path.to.my.sentry.Transport
+
+You can supply `additional configuration options `__ based on the Python platform via ``[sentry]`` section. Unsupported options: ``integrations``, ``in_app_include``, ``in_app_exclude``, ``ignore_errors``, ``before_breadcrumb``.
Tags
-----
diff --git a/docs/apache-airflow/authoring-and-scheduling/datasets.rst b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
index 60268bedff4e6..2cc34d9c8870a 100644
--- a/docs/apache-airflow/authoring-and-scheduling/datasets.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
@@ -230,8 +230,8 @@ Example:
@task
def print_triggering_dataset_events(triggering_dataset_events=None):
for dataset, dataset_list in triggering_dataset_events.items():
- print(dataset, dataset_list, dataset_list[dataset])
- print(dataset_list[dataset][0].source_dag_run.dag_run_id)
+ print(dataset, dataset_list)
+ print(dataset_list[0].source_dag_run.dag_id)
print_triggering_dataset_events()
diff --git a/docs/apache-airflow/core-concepts/dags.rst b/docs/apache-airflow/core-concepts/dags.rst
index aa95699ab8cea..7a5a216db2583 100644
--- a/docs/apache-airflow/core-concepts/dags.rst
+++ b/docs/apache-airflow/core-concepts/dags.rst
@@ -105,7 +105,7 @@ There are two main ways to declare individual task dependencies. The recommended
Or, you can also use the more explicit ``set_upstream`` and ``set_downstream`` methods::
- first_task.set_downstream(second_task, third_task)
+ first_task.set_downstream([second_task, third_task])
third_task.set_upstream(fourth_task)
There are also shortcuts to declaring more complex dependencies. If you want to make two lists of tasks depend on all parts of each other, you can't use either of the approaches above, so you need to use ``cross_downstream``::
@@ -677,6 +677,11 @@ This is especially useful if your tasks are built dynamically from configuration
SubDAGs
-------
+.. note::
+
+ SubDAG is deprecated hence TaskGroup is always the preferred choice.
+
+
Sometimes, you will find that you are regularly adding exactly the same set of tasks to every DAG, or you want to group a lot of tasks into a single, logical unit. This is what SubDAGs are for.
For example, here's a DAG that has a lot of parallel tasks in two sections:
@@ -754,10 +759,6 @@ You can see the core differences between these two constructs.
| Simple construct declaration with context manager | Complex DAG factory with naming restrictions |
+--------------------------------------------------------+--------------------------------------------------------+
-.. note::
-
- SubDAG is deprecated hence TaskGroup is always the preferred choice.
-
Packaging DAGs
diff --git a/docs/apache-airflow/core-concepts/executor/debug.rst b/docs/apache-airflow/core-concepts/executor/debug.rst
index fb14c7a6f0ece..896f79ce1ca76 100644
--- a/docs/apache-airflow/core-concepts/executor/debug.rst
+++ b/docs/apache-airflow/core-concepts/executor/debug.rst
@@ -15,8 +15,47 @@
specific language governing permissions and limitations
under the License.
+.. _executor:DebugExecutor:
+
+Debug Executor (deprecated)
+===========================
+
+The :class:`~airflow.executors.debug_executor.DebugExecutor` is meant as
+a debug tool and can be used from IDE. It is a single process executor that
+queues :class:`~airflow.models.taskinstance.TaskInstance` and executes them by running
+``_run_raw_task`` method.
+
+Due to its nature the executor can be used with SQLite database. When used
+with sensors the executor will change sensor mode to ``reschedule`` to avoid
+blocking the execution of DAG.
+
+Additionally ``DebugExecutor`` can be used in a fail-fast mode that will make
+all other running or scheduled tasks fail immediately. To enable this option set
+``AIRFLOW__DEBUG__FAIL_FAST=True`` or adjust ``fail_fast`` option in your ``airflow.cfg``.
+For more information on setting the configuration, see :doc:`../../howto/set-config`.
+
+**IDE setup steps:**
+
+1. Add ``main`` block at the end of your DAG file to make it runnable.
+
+It will run a backfill job:
+
+.. code-block:: python
+
+ if __name__ == "__main__":
+ from airflow.utils.state import State
+
+ dag.clear()
+ dag.run()
+
+
+2. Setup ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` in run configuration of your IDE. In
+ this step you should also setup all environment variables required by your DAG.
+
+3. Run / debug the DAG file.
+
Testing DAGs with dag.test()
-=============================
+*****************************
To debug DAGs in an IDE, you can set up the ``dag.test`` command in your dag file and run through your DAG in a single
serialized python process.
@@ -35,7 +74,7 @@ and that's it! You can add argument such as ``execution_date`` if you want to te
you can run or debug DAGs as needed.
Comparison with DebugExecutor
-*****************************
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``dag.test`` command has the following benefits over the :class:`~airflow.executors.debug_executor.DebugExecutor`
class, which is now deprecated:
@@ -46,7 +85,7 @@ class, which is now deprecated:
Debugging Airflow DAGs on the command line
-==========================================
+******************************************
With the same two line addition as mentioned in the above section, you can now easily debug a DAG using pdb as well.
Run ``python -m pdb .py`` for an interactive debugging experience on the command line.
@@ -63,42 +102,3 @@ Run ``python -m pdb .py`` for an interactive debugging experie
-> bash_command='echo 1',
(Pdb) run_this_last
-
-.. _executor:DebugExecutor:
-
-Debug Executor (deprecated)
-===========================
-
-The :class:`~airflow.executors.debug_executor.DebugExecutor` is meant as
-a debug tool and can be used from IDE. It is a single process executor that
-queues :class:`~airflow.models.taskinstance.TaskInstance` and executes them by running
-``_run_raw_task`` method.
-
-Due to its nature the executor can be used with SQLite database. When used
-with sensors the executor will change sensor mode to ``reschedule`` to avoid
-blocking the execution of DAG.
-
-Additionally ``DebugExecutor`` can be used in a fail-fast mode that will make
-all other running or scheduled tasks fail immediately. To enable this option set
-``AIRFLOW__DEBUG__FAIL_FAST=True`` or adjust ``fail_fast`` option in your ``airflow.cfg``.
-For more information on setting the configuration, see :doc:`../../howto/set-config`.
-
-**IDE setup steps:**
-
-1. Add ``main`` block at the end of your DAG file to make it runnable.
-
-It will run a backfill job:
-
-.. code-block:: python
-
- if __name__ == "__main__":
- from airflow.utils.state import State
-
- dag.clear()
- dag.run()
-
-
-2. Setup ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` in run configuration of your IDE. In
- this step you should also setup all environment variables required by your DAG.
-
-3. Run / debug the DAG file.
diff --git a/docs/apache-airflow/core-concepts/executor/index.rst b/docs/apache-airflow/core-concepts/executor/index.rst
index 10d95355bfd01..4a2761a0cf0f0 100644
--- a/docs/apache-airflow/core-concepts/executor/index.rst
+++ b/docs/apache-airflow/core-concepts/executor/index.rst
@@ -30,13 +30,6 @@ Built-in executors are referred to by name, for example:
[core]
executor = KubernetesExecutor
-You can also write your own custom executors, and refer to them by their full path:
-
-.. code-block:: ini
-
- [core]
- executor = my_company.executors.MyCustomExecutor
-
.. note::
For more information on Airflow's configuration, see :doc:`/howto/set-config`.
@@ -51,7 +44,7 @@ If you want to check which executor is currently set, you can use the ``airflow
Executor Types
--------------
-There are two types of executor - those that run tasks *locally* (inside the ``scheduler`` process), and those that run their tasks *remotely* (usually via a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` by default, which is a local executor, and the safest option for execution, but we *strongly recommend* you change this to ``LocalExecutor`` for small, single-machine installations, or one of the remote executors for a multi-machine/cloud installation.
+There are two types of executors - those that run tasks *locally* (inside the ``scheduler`` process), and those that run their tasks *remotely* (usually via a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` by default, which is a local executor, and the simplest option for execution. However, the ``SequentialExecutor`` is not suitable for production since it does not allow for parallel task running and due to that, some Airflow features (e.g. running sensors) will not work properly. You should instead use the ``LocalExecutor`` for small, single-machine production installations, or one of the remote executors for a multi-machine/cloud installation.
**Local Executors**
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below.
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: Executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* ``end``: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
+* ``terminate``: More forcefully stop the executor, even killing/stopping in-flight tasks instead of synchronously waiting for completion.
+* ``cleanup_stuck_queued_tasks``: If tasks are stuck in the queued state for longer than ``task_queued_timeout`` then they are collected by the scheduler and provided to the executor to have an opportunity to handle them (perform any graceful cleanup/teardown) via this method and return the Task Instances for a warning message displayed to users.
+* ``try_adopt_task_instances``: Tasks that have been abandoned (e.g. from a scheduler job that died) are provided to the executor to adopt or otherwise handle them via this method. Any tasks that cannot be adopted (by default the BaseExector assumes all cannot be adopted) should be returned.
+* ``get_cli_commands``: Executors may vend CLI commands to users by implementing this method, see the `CLI`_ section below for more details.
+* ``get_task_log``: Executors may vend log messages to Airflow task logs by implementing this method, see the `Logging`_ section below for more details.
+
+Compatibility Attributes
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``BaseExecutor`` class interface contains a set of attributes that Airflow core code uses to check the features that your executor is compatible with. When writing your own Airflow executor be sure to set these correctly for your use case. Each attribute is simply a boolean to enable/disable a feature or indicate that a feature is supported/unsupported by the executor:
+
+* ``supports_pickling``: Whether or not the executor supports reading pickled DAGs from the Database before execution (rather than reading the DAG definition from the file system).
+* ``supports_sentry``: Whether or not the executor supports `Sentry `_.
+
+* ``is_local``: Whether or not the executor is remote or local. See the `Executor Types`_ section above.
+* ``is_single_threaded``: Whether or not the executor is single threaded. This is particularly relevant to what database backends are supported. Single threaded executors can run with any backend, including SQLite.
+* ``is_production``: Whether or not the executor should be used for production purposes. A UI message is displayed to users when they are using a non-production ready executor.
+
+* ``change_sensor_mode_to_reschedule``: Running Airflow sensors in poke mode can block the thread of executors and in some cases Airflow.
+* ``serve_logs``: Whether or not the executor supports serving logs, see :doc:`/administration-and-deployment/logging-monitoring/logging-tasks`.
+
+CLI
+^^^
+
+Executors may vend CLI commands which will be included in the ``airflow`` command line tool by implementing the ``get_cli_commands`` method. Executors such as ``CeleryExecutor`` and ``KubernetesExecutor`` for example, make use of this mechanism. The commands can be used to setup required workers, initialize environment or set other configuration. Commands are only vended for the currently configured executor. A pseudo-code example of implementing CLI command vending from an executor can be seen below:
+
+.. code-block:: python
+
+ @staticmethod
+ def get_cli_commands() -> list[GroupCommand]:
+ sub_commands = [
+ ActionCommand(
+ name="command_name",
+ help="Description of what this specific command does",
+ func=lazy_load_command("path.to.python.function.for.command"),
+ args=(),
+ ),
+ ]
+
+ return [
+ GroupCommand(
+ name="my_cool_executor",
+ help="Description of what this group of commands do",
+ subcommands=sub_commands,
+ ),
+ ]
+
+.. note::
+ Currently there are no strict rules in place for the Airflow command namespace. It is up to developers to use names for their CLI commands that are sufficiently unique so as to not cause conflicts with other Airflow executors or components.
+
+.. note::
+ When creating a new executor, or updating any existing executors, be sure to not import or execute any expensive operations/code at the module level. Executor classes are imported in several places and if they are slow to import this will negatively impact the performance of your Airflow environment, especially for CLI commands.
+
+Logging
+^^^^^^^
+
+Executors may vend log messages which will be included in the Airflow task logs by implementing the ``get_task_logs`` method. This can be helpful if the execution environment has extra context in the case of task failures, which may be due to the execution environment itself rather than the Airflow task code. It can also be helpful to include setup/teardown logging from the execution environment.
+The ``KubernetesExecutor`` leverages this this capability to include logs from the pod which ran a specific Airflow task and display them in the logs for that Airflow task. A pseudo-code example of implementing task log vending from an executor can be seen below:
+
+.. code-block:: python
+
+ def get_task_log(self, ti: TaskInstance, try_number: int) -> tuple[list[str], list[str]]:
+ messages = []
+ log = []
+ try:
+ res = helper_function_to_fetch_logs_from_execution_env(ti, try_number)
+ for line in res:
+ log.append(remove_escape_codes(line.decode()))
+ if log:
+ messages.append("Found logs from execution environment!")
+ except Exception as e: # No exception should cause task logs to fail
+ messages.append(f"Failed to find logs from execution environment: {e}")
+ return messages, ["\n".join(log)]
+
+Next Steps
+^^^^^^^^^^
+
+Once you have created a new executor class implementing the ``BaseExecutor`` interface, you can configure Airflow to use it by setting the ``core.executor`` configuration value to the module path of your executor:
+
+.. code-block:: ini
+
+ [core]
+ executor = my_company.executors.MyCustomExecutor
+
+.. note::
+ For more information on Airflow's configuration, see :doc:`/howto/set-config` and for more information on managing Python modules in Airflow see :doc:`/administration-and-deployment/modules_management`.
diff --git a/docs/apache-airflow/core-concepts/params.rst b/docs/apache-airflow/core-concepts/params.rst
index e5bf04ae62343..42fd523688068 100644
--- a/docs/apache-airflow/core-concepts/params.rst
+++ b/docs/apache-airflow/core-concepts/params.rst
@@ -120,12 +120,13 @@ Another way to access your param is via a task's ``context`` kwarg.
.. code-block::
:emphasize-lines: 1,2
- def print_x(**context):
+ def print_my_int_param(**context):
print(context["params"]["my_int_param"])
PythonOperator(
task_id="print_my_int_param",
python_callable=print_my_int_param,
+ params={"my_int_param": 12345},
)
JSON Schema Validation
diff --git a/docs/apache-airflow/core-concepts/taskflow.rst b/docs/apache-airflow/core-concepts/taskflow.rst
index 8dcd52acd1916..0794fa8533c5e 100644
--- a/docs/apache-airflow/core-concepts/taskflow.rst
+++ b/docs/apache-airflow/core-concepts/taskflow.rst
@@ -31,7 +31,7 @@ TaskFlow takes care of moving inputs and outputs between your Tasks using XComs
def get_ip():
return my_ip_service.get_main_ip()
- @task
+ @task(multiple_outputs=True)
def compose_email(external_ip):
return {
'subject':f'Server connected from {external_ip}',
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index 7b091d8854aa8..d9e7d1eb0d32a 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -299,6 +299,8 @@ These are extras that provide support for integration with external systems via
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| openlineage | ``pip install 'apache-airflow[openlineage]'`` | Sending OpenLineage events | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
+| opensearch | ``pip install 'apache-airflow[opensearch]'`` | Opensearch hooks and operators | |
++---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| papermill | ``pip install 'apache-airflow[papermill]'`` | Papermill hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| sftp | ``pip install 'apache-airflow[sftp]'`` | SFTP hooks, operators and sensors | |
diff --git a/docs/apache-airflow/howto/connection.rst b/docs/apache-airflow/howto/connection.rst
index 0db100fd2974f..2c07b03b6d396 100644
--- a/docs/apache-airflow/howto/connection.rst
+++ b/docs/apache-airflow/howto/connection.rst
@@ -190,8 +190,26 @@ Passwords cannot be manipulated or read without the key. For information on conf
Testing Connections
^^^^^^^^^^^^^^^^^^^
+For security reasons, the test connection functionality is disabled by default across Airflow UI, API and CLI.
-Airflow Web UI, REST API, and CLI allow you to test connections. The test connection feature can be used from
+For more information on capabilities of users, see the documentation:
+https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users.
+It is strongly advised to not enable the feature until you make sure that only
+highly trusted UI/API users have "edit connection" permissions.
+
+The availability of the
+functionality can be controlled by the test_connection flag in
+the core section of the Airflow configuration (airflow.cfg).
+It can also be controlled by the environment variable
+``AIRFLOW__CORE__TEST_CONNECTION``.
+
+The following values are accepted for this config param:
+
+* Disabled: Disables the test connection functionality and disables the Test Connection button in the UI. This is also the default value set in the Airflow configuration.
+* Enabled: Enables the test connection functionality and activates the Test Connection button in the UI.
+* Hidden: Disables the test connection functionality and hides the Test Connection button in UI.
+
+After enabling Test Connection, it can be used from the
:ref:`create ` or :ref:`edit ` connection page in the UI, through calling
:doc:`Connections REST API `, or running the ``airflow connections test`` :ref:`CLI command `.
diff --git a/docs/apache-airflow/howto/define-extra-link.rst b/docs/apache-airflow/howto/define-extra-link.rst
index 8f2f918d45511..733a965bc7d26 100644
--- a/docs/apache-airflow/howto/define-extra-link.rst
+++ b/docs/apache-airflow/howto/define-extra-link.rst
@@ -28,6 +28,7 @@ The following code shows how to add extra links to an operator via Plugins:
.. code-block:: python
from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.plugins_manager import AirflowPlugin
@@ -80,8 +81,9 @@ tasks using :class:`~airflow.providers.amazon.aws.transfers.gcs_to_s3.GCSToS3Ope
.. code-block:: python
+ from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
+ from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.plugins_manager import AirflowPlugin
- from airflow.models.baseoperator import BaseOperatorLink
from airflow.providers.amazon.aws.transfers.gcs_to_s3 import GCSToS3Operator
@@ -117,9 +119,10 @@ Console, but if we wanted to change that link we could:
.. code-block:: python
- from airflow.plugins_manager import AirflowPlugin
- from airflow.models.baseoperator import BaseOperatorLink
+ from airflow.models.baseoperator import BaseOperatorLink, BaseOperator
+ from airflow.models.taskinstancekey import TaskInstanceKey
from airflow.models.xcom import XCom
+ from airflow.plugins_manager import AirflowPlugin
from airflow.providers.google.cloud.operators.bigquery import BigQueryOperator
# Change from https to http just to display the override
diff --git a/docs/apache-airflow/howto/set-config.rst b/docs/apache-airflow/howto/set-config.rst
index 1e0679c886584..bf1cae0d37cd3 100644
--- a/docs/apache-airflow/howto/set-config.rst
+++ b/docs/apache-airflow/howto/set-config.rst
@@ -66,7 +66,7 @@ For example consider the pretend section ``providers.some_provider``:
.. code-block:: ini
- [providers.some_provider>]
+ [providers.some_provider]
this_param = true
.. code-block:: bash
diff --git a/docs/apache-airflow/howto/set-up-database.rst b/docs/apache-airflow/howto/set-up-database.rst
index c5cab0a2e5e34..3c8f7a9bb113b 100644
--- a/docs/apache-airflow/howto/set-up-database.rst
+++ b/docs/apache-airflow/howto/set-up-database.rst
@@ -298,18 +298,9 @@ We recommend using the ``mysqlclient`` driver and specifying it in your SqlAlche
mysql+mysqldb://:@[:]/
-We also support the ``mysql-connector-python`` driver, which lets you connect through SSL
-without any cert options provided. If you wish to use ``mysql-connector-python`` driver, please install it with extras.
-
-.. code-block:: text
-
- $ pip install mysql-connector-python
-
-The connection string in this case should look like:
-
-.. code-block:: text
-
- mysql+mysqlconnector://:@[:]/
+.. important::
+ The integration of MySQL backend has only been validated using the ``mysqlclient`` driver
+ during Apache Airflow's continuous integration (CI) process.
If you want to use other drivers visit the `MySQL Dialect `__ in SQLAlchemy documentation for more information regarding download
and setup of the SqlAlchemy connection.
diff --git a/docs/apache-airflow/installation/supported-versions.rst b/docs/apache-airflow/installation/supported-versions.rst
index 236d46ecb1651..1364f98e9307b 100644
--- a/docs/apache-airflow/installation/supported-versions.rst
+++ b/docs/apache-airflow/installation/supported-versions.rst
@@ -29,7 +29,7 @@ Apache Airflow™ version life cycle:
========= ===================== ========= =============== ================= ================
Version Current Patch/Minor State First Release Limited Support EOL/Terminated
========= ===================== ========= =============== ================= ================
-2 2.7.2 Supported Dec 17, 2020 TBD TBD
+2 2.7.3 Supported Dec 17, 2020 TBD TBD
1.10 1.10.15 EOL Aug 27, 2018 Dec 17, 2020 June 17, 2021
1.9 1.9.0 EOL Jan 03, 2018 Aug 27, 2018 Aug 27, 2018
1.8 1.8.2 EOL Mar 19, 2017 Jan 03, 2018 Jan 03, 2018
diff --git a/docs/apache-airflow/public-airflow-interface.rst b/docs/apache-airflow/public-airflow-interface.rst
index 534cc0d0e63c3..eeeaa618e7b42 100644
--- a/docs/apache-airflow/public-airflow-interface.rst
+++ b/docs/apache-airflow/public-airflow-interface.rst
@@ -318,11 +318,11 @@ Executors
Executors are the mechanism by which task instances get run. All executors are
derived from :class:`~airflow.executors.base_executor.BaseExecutor`. There are several
-executor implementations built-in Airflow, each with its own unique characteristics and capabilities.
+executor implementations built-in Airflow, each with their own unique characteristics and capabilities.
-The executor interface itself (the BaseExecutor class) is public, but the built-in executors are not (i.e. KubernetesExecutor, LocalExecutor, etc). This means that, to use KubernetesExecutor as an example, we may make changes to KubernetesExecutor in minor or patch Airflow releases which could break an executor that subclasses KubernetesExecutor. This is necessary to allow Airflow developers sufficient freedom to continue to improve the executors we offer. Accordingly if you want to modify or extend a built-in executor, you should incorporate the full executor code into your project so that such changes will not break your derivative executor.
+The executor interface itself (the BaseExecutor class) is public, but the built-in executors are not (i.e. KubernetesExecutor, LocalExecutor, etc). This means that, to use KubernetesExecutor as an example, we may make changes to KubernetesExecutor in minor or patch Airflow releases which could break an executor that subclasses KubernetesExecutor. This is necessary to allow Airflow developers sufficient freedom to continue to improve the executors we offer. Accordingly, if you want to modify or extend a built-in executor, you should incorporate the full executor code into your project so that such changes will not break your derivative executor.
-You can read more about executors in :doc:`core-concepts/executor/index`.
+You can read more about executors and how to write your own in :doc:`core-concepts/executor/index`.
.. versionadded:: 2.6
diff --git a/docs/apache-airflow/security/secrets/secrets-backend/index.rst b/docs/apache-airflow/security/secrets/secrets-backend/index.rst
index b91bef347dff9..514f874d07850 100644
--- a/docs/apache-airflow/security/secrets/secrets-backend/index.rst
+++ b/docs/apache-airflow/security/secrets/secrets-backend/index.rst
@@ -101,7 +101,7 @@ Roll your own secrets backend
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A secrets backend is a subclass of :py:class:`airflow.secrets.base_secrets.BaseSecretsBackend` and must implement either
-:py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_connection` or :py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_conn_value`.
+:py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_connection` or :py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_conn_value` for retrieving connections, :py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_variable` for retrieving variables and :py:meth:`~airflow.secrets.base_secrets.BaseSecretsBackend.get_config` for retrieving Airflow configurations.
After writing your backend class, provide the fully qualified class name in the ``backend`` key in the ``[secrets]``
section of ``airflow.cfg``.
diff --git a/docs/apache-airflow/security/security_model.rst b/docs/apache-airflow/security/security_model.rst
index 6c9e3da15102b..4b426ea041b14 100644
--- a/docs/apache-airflow/security/security_model.rst
+++ b/docs/apache-airflow/security/security_model.rst
@@ -110,6 +110,7 @@ Airflow is deployed. This includes but is not limited to:
* applying authentication and authorization to the web application so that only known and authorized
users can have access to Airflow
* any kind of detection of unusual activity and protection against it
+* choosing the right session backend and configuring it properly including timeouts for the session
Airflow does not implement any of those feature natively, and delegates it to the deployment managers
to deploy all the necessary infrastructure to protect the deployment - as external infrastructure components.
diff --git a/docs/apache-airflow/templates-ref.rst b/docs/apache-airflow/templates-ref.rst
index 763d3550fdc8e..215e35f589792 100644
--- a/docs/apache-airflow/templates-ref.rst
+++ b/docs/apache-airflow/templates-ref.rst
@@ -136,6 +136,7 @@ Just like with ``var`` it's possible to fetch a connection by string (e.g. ``{{
Additionally, the ``extras`` field of a connection can be fetched as a Python Dictionary with the ``extra_dejson`` field, e.g.
``conn.my_aws_conn_id.extra_dejson.region_name`` would fetch ``region_name`` out of ``extras``.
+This way, defaults in ``extras`` can be provided as well (e.g. ``{{ conn.my_aws_conn_id.extra_dejson.get('region_name', 'Europe (Frankfurt)') }}``).
Filters
-------
diff --git a/docs/conf.py b/docs/conf.py
index d26edb6c82788..8aa9bb0f28496 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -102,6 +102,12 @@
# behavior of the utils.apply_default that was hiding function headers
os.environ["BUILDING_AIRFLOW_DOCS"] = "TRUE"
+# Use for generate rst_epilog and other post-generation substitutions
+global_substitutions = {
+ "version": PACKAGE_VERSION,
+ "airflow-version": airflow.__version__,
+}
+
# == Sphinx configuration ======================================================
# -- Project information -------------------------------------------------------
@@ -114,18 +120,19 @@
# The full version, including alpha/beta/rc tags.
release = PACKAGE_VERSION
-rst_epilog = f"""
-.. |version| replace:: {version}
-.. |airflow-version| replace:: {airflow.__version__}
-.. |experimental| replace:: This is an :ref:`experimental feature `.
-"""
-
-smartquotes_excludes = {"builders": ["man", "text", "spelling"]}
-
-
# -- General configuration -----------------------------------------------------
# See: https://www.sphinx-doc.org/en/master/usage/configuration.html
+rst_epilog = "\n".join(
+ f".. |{key}| replace:: {replace}"
+ for key, replace in {
+ **global_substitutions,
+ "experimental": "This is an :ref:`experimental feature `.",
+ }.items()
+)
+
+smartquotes_excludes = {"builders": ["man", "text", "spelling"]}
+
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
@@ -171,8 +178,7 @@
elif PACKAGE_NAME == "helm-chart":
extensions.append("sphinx_jinja")
elif PACKAGE_NAME == "docker-stack":
- # No extra extensions
- pass
+ extensions.append("extra_files_with_substitutions")
elif PACKAGE_NAME.startswith("apache-airflow-providers-"):
extensions.extend(
[
@@ -323,17 +329,17 @@ def _get_rst_filepath_from_path(filepath: pathlib.Path):
]
html_extra_with_substitutions = [
f"{ROOT_DIR}/docs/apache-airflow/howto/docker-compose/docker-compose.yaml",
- f"{ROOT_DIR}/docs/docker-stack/build.rst",
]
- # Replace "|version|" in links
+ # Substitute in links
manual_substitutions_in_generated_html = [
"installation/installing-from-pypi.html",
"installation/installing-from-sources.html",
+ "administration-and-deployment/logging-monitoring/advanced-logging-configuration.html",
]
if PACKAGE_NAME.startswith("apache-airflow-providers"):
manual_substitutions_in_generated_html = ["example-dags.html", "operators.html", "index.html"]
if PACKAGE_NAME == "docker-stack":
- # Replace "|version|" inside ```` quotes
+ # Substitute in links
manual_substitutions_in_generated_html = ["build.html"]
# -- Theme configuration -------------------------------------------------------
diff --git a/docs/docker-stack/README.md b/docs/docker-stack/README.md
index 10129d80feea5..c7e8dde81181e 100644
--- a/docs/docker-stack/README.md
+++ b/docs/docker-stack/README.md
@@ -31,12 +31,12 @@ Every time a new version of Airflow is released, the images are prepared in the
[apache/airflow DockerHub](https://hub.docker.com/r/apache/airflow)
for all the supported Python versions.
-You can find the following images there (Assuming Airflow version `2.7.2`):
+You can find the following images there (Assuming Airflow version `2.7.3`):
* `apache/airflow:latest` - the latest released Airflow image with default Python version (3.8 currently)
* `apache/airflow:latest-pythonX.Y` - the latest released Airflow image with specific Python version
-* `apache/airflow:2.7.2` - the versioned Airflow image with default Python version (3.8 currently)
-* `apache/airflow:2.7.2-pythonX.Y` - the versioned Airflow image with specific Python version
+* `apache/airflow:2.7.3` - the versioned Airflow image with default Python version (3.8 currently)
+* `apache/airflow:2.7.3-pythonX.Y` - the versioned Airflow image with specific Python version
Those are "reference" regular images. They contain the most common set of extras, dependencies and providers that are
often used by the users and they are good to "try-things-out" when you want to just take Airflow for a spin,
@@ -47,8 +47,8 @@ via [Building the image](https://airflow.apache.org/docs/docker-stack/build.html
* `apache/airflow:slim-latest` - the latest released Airflow image with default Python version (3.8 currently)
* `apache/airflow:slim-latest-pythonX.Y` - the latest released Airflow image with specific Python version
-* `apache/airflow:slim-2.7.2` - the versioned Airflow image with default Python version (3.8 currently)
-* `apache/airflow:slim-2.7.2-pythonX.Y` - the versioned Airflow image with specific Python version
+* `apache/airflow:slim-2.7.3` - the versioned Airflow image with default Python version (3.8 currently)
+* `apache/airflow:slim-2.7.3-pythonX.Y` - the versioned Airflow image with specific Python version
The Apache Airflow image provided as convenience package is optimized for size, and
it provides just a bare minimal set of the extras and dependencies installed and in most cases
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index 9b2cd6f209799..aa3315473c2ec 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you want to build a custom
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.2.1`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.3.1`` | PIP version used. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_PIP_INSTALL_FLAGS`` | | additional ``pip`` flags passed to the |
| | | installation commands (except when |
@@ -171,6 +171,11 @@ for examples of using those arguments.
| | | The mysql extra is removed from extras |
| | | if the client is not installed. |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``INSTALL_MYSQL_CLIENT_TYPE`` | ``mysql`` | (*Experimental*) Type of MySQL client |
+| | | library. This can be ``mysql`` or |
+| | | ``mariadb``. Regardless of the parameter |
+| | | will always be used ``mariadb`` on ARM. |
++------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_MSSQL_CLIENT`` | ``true`` | Whether MsSQL client should be installed |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_POSTGRES_CLIENT`` | ``true`` | Whether Postgres client should be |
@@ -273,3 +278,6 @@ Docker context files.
| | | This allows to optimize iterations for |
| | | Image builds and speeds up CI builds. |
+------------------------------------------+------------------------------------------+------------------------------------------+
+| ``PIP_CACHE_EPOCH`` | ``"0"`` | Allow to invalidate cache by passing a |
+| | | new argument. |
++------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
index 41a52b0c07a42..5bad72c0031e8 100644
--- a/docs/docker-stack/build.rst
+++ b/docs/docker-stack/build.rst
@@ -533,7 +533,7 @@ Before attempting to customize the image, you need to download flexible and cust
You can extract the officially released version of the Dockerfile from the
`released sources `_.
You can also conveniently download the latest released version
-`from GitHub `_. You can save it
+`from GitHub `_. You can save it
in any directory - there is no need for any other files to be present there. If you wish to use your own
files (for example custom configuration of ``pip`` or your own ``requirements`` or custom dependencies,
you need to use ``DOCKER_CONTEXT_FILES`` build arg and place the files in the directory pointed at by
@@ -972,3 +972,15 @@ The architecture of the images
You can read more details about the images - the context, their parameters and internal structure in the
`IMAGES.rst `_ document.
+
+
+Pip packages caching
+....................
+
+To enable faster iteration when building the image locally (especially if you are testing different combination of
+python packages), pip caching has been enabled. The caching id is based on four different parameters:
+
+1. ``PYTHON_BASE_IMAGE``: Avoid sharing same cache based on python version and target os
+2. ``AIRFLOW_PIP_VERSION``
+3. ``TARGETARCH``: Avoid sharing architecture specific cached package
+4. ``PIP_CACHE_EPOCH``: Enable changing cache id by passing ``PIP_CACHE_EPOCH`` as ``--build-arg``
diff --git a/docs/docker-stack/changelog.rst b/docs/docker-stack/changelog.rst
index 26256eac4332b..3ec48be4348b6 100644
--- a/docs/docker-stack/changelog.rst
+++ b/docs/docker-stack/changelog.rst
@@ -58,6 +58,18 @@ here so that users affected can find the reason for the changes.
Airflow 2.7
~~~~~~~~~~~
+* 2.7.3
+
+ * Add experimental feature for select type of MySQL Client libraries during the build custom image via ``INSTALL_MYSQL_CLIENT_TYPE``
+ build arg. ``mysql`` for install MySQL client libraries from `Oracle APT repository `_.
+ The selection of MySQL Client libraries only available on AMD64 (x86_64) for ARM docker image it will always install
+ MariaDB client.
+
+ * Docker CLI version in the image is bumped to 24.0.6 version.
+
+ * PIP caching for local builds has been enabled to speed up local custom image building
+
* 2.7.0
* As of now, Python 3.7 is no longer supported by the Python community. Therefore, to use Airflow 2.7.0 and above, you must ensure your Python version is
diff --git a/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile b/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
index 6888723306d78..8d6a4b63251d3 100644
--- a/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-airflow-configuration/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
index 31c4d91d540eb..1eb04526d4446 100644
--- a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
index 2baf12487ff39..aeae3ff84f214 100644
--- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
index 34490290d90af..8d567fa8cce45 100644
--- a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
index baf4c1f30a4f7..6524fc2fe6d78 100644
--- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
@@ -15,6 +15,6 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
index 1704f64173d65..cd957d12a6c9b 100644
--- a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
index 73c62e6a9a1a2..d3aed5f3efe4e 100644
--- a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile
@@ -15,6 +15,6 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
index b82de6e70d433..4ec92a973a7ae 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
COPY --chown=airflow:root test_dag.py /opt/airflow/dags
diff --git a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
index 516628daaba0e..5fbf07f2d5729 100644
--- a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow:2.7.2
+FROM apache/airflow:2.7.3
RUN umask 0002; \
mkdir -p ~/writeable-directory
# [END Dockerfile]
diff --git a/docs/docker-stack/entrypoint.rst b/docs/docker-stack/entrypoint.rst
index 25fb2ec62680d..c07de558716f8 100644
--- a/docs/docker-stack/entrypoint.rst
+++ b/docs/docker-stack/entrypoint.rst
@@ -132,7 +132,7 @@ if you specify extra arguments. For example:
.. code-block:: bash
- docker run -it apache/airflow:2.7.2-python3.8 bash -c "ls -la"
+ docker run -it apache/airflow:2.7.3-python3.8 bash -c "ls -la"
total 16
drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 .
drwxr-xr-x 1 root root 4096 Jun 5 18:12 ..
@@ -144,7 +144,7 @@ you pass extra parameters. For example:
.. code-block:: bash
- > docker run -it apache/airflow:2.7.2-python3.8 python -c "print('test')"
+ > docker run -it apache/airflow:2.7.3-python3.8 python -c "print('test')"
test
If first argument equals to "airflow" - the rest of the arguments is treated as an airflow command
@@ -152,13 +152,13 @@ to execute. Example:
.. code-block:: bash
- docker run -it apache/airflow:2.7.2-python3.8 airflow webserver
+ docker run -it apache/airflow:2.7.3-python3.8 airflow webserver
If there are any other arguments - they are simply passed to the "airflow" command
.. code-block:: bash
- > docker run -it apache/airflow:2.7.2-python3.8 help
+ > docker run -it apache/airflow:2.7.3-python3.8 help
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
@@ -363,7 +363,7 @@ database and creating an ``admin/admin`` Admin user with the following command:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD=admin" \
- apache/airflow:2.7.2-python3.8 webserver
+ apache/airflow:2.7.3-python3.8 webserver
.. code-block:: bash
@@ -372,7 +372,7 @@ database and creating an ``admin/admin`` Admin user with the following command:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \
- apache/airflow:2.7.2-python3.8 webserver
+ apache/airflow:2.7.3-python3.8 webserver
The commands above perform initialization of the SQLite database, create admin user with admin password
and Admin role. They also forward local port ``8080`` to the webserver port and finally start the webserver.
@@ -412,6 +412,6 @@ Example:
--env "_AIRFLOW_DB_MIGRATE=true" \
--env "_AIRFLOW_WWW_USER_CREATE=true" \
--env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \
- apache/airflow:2.7.2-python3.8 webserver
+ apache/airflow:2.7.3-python3.8 webserver
This method is only available starting from Docker image of Airflow 2.1.1 and above.
diff --git a/docs/exts/extra_files_with_substitutions.py b/docs/exts/extra_files_with_substitutions.py
index cb91b163f045a..e0e0f20f05db4 100644
--- a/docs/exts/extra_files_with_substitutions.py
+++ b/docs/exts/extra_files_with_substitutions.py
@@ -19,19 +19,27 @@
import os
-def copy_docker_compose(app, exception):
+def _manual_substitution(line: str, replacements: dict[str, str]) -> str:
+ for value, repl in replacements.items():
+ line = line.replace(f"|{value}|", repl)
+ return line
+
+
+def build_postprocess(app, exception):
"""Sphinx "build-finished" event handler."""
from sphinx.builders import html as builders
if exception or not isinstance(app.builder, builders.StandaloneHTMLBuilder):
return
+ global_substitutions = app.config.global_substitutions
+
# Replace `|version|` in the docker-compose.yaml that requires manual substitutions
for path in app.config.html_extra_with_substitutions:
with open(path) as file:
with open(os.path.join(app.outdir, os.path.basename(path)), "w") as output_file:
for line in file:
- output_file.write(line.replace("|version|", app.config.version))
+ output_file.write(_manual_substitution(line, global_substitutions))
# Replace `|version|` in the installation files that requires manual substitutions (in links)
for path in app.config.manual_substitutions_in_generated_html:
@@ -41,15 +49,16 @@ def copy_docker_compose(app, exception):
os.path.join(app.outdir, os.path.dirname(path), os.path.basename(path)), "w"
) as output_file:
for line in content:
- output_file.write(line.replace("|version|", app.config.version))
+ output_file.write(_manual_substitution(line, global_substitutions))
def setup(app):
"""Setup plugin"""
- app.connect("build-finished", copy_docker_compose)
+ app.connect("build-finished", build_postprocess)
- app.add_config_value("html_extra_with_substitutions", [], "[str]")
- app.add_config_value("manual_substitutions_in_generated_html", [], "[str]")
+ app.add_config_value("html_extra_with_substitutions", [], "html", [str])
+ app.add_config_value("manual_substitutions_in_generated_html", [], "html", [str])
+ app.add_config_value("global_substitutions", {}, "html", [dict])
return {
"parallel_write_safe": True,
diff --git a/docs/exts/operators_and_hooks_ref.py b/docs/exts/operators_and_hooks_ref.py
index 06c37cf0b4275..e324053c15b6d 100644
--- a/docs/exts/operators_and_hooks_ref.py
+++ b/docs/exts/operators_and_hooks_ref.py
@@ -246,7 +246,7 @@ def run(self):
def render_content(self, *, tags: set[str] | None, header_separator: str = DEFAULT_HEADER_SEPARATOR):
"""Return content in RST format"""
- raise NotImplementedError("Tou need to override render_content method.")
+ raise NotImplementedError("You need to override render_content method.")
def _common_render_list_content(*, header_separator: str, resource_type: str, template: str):
diff --git a/docs/integration-logos/opensearch/opensearch.png b/docs/integration-logos/opensearch/opensearch.png
new file mode 100644
index 0000000000000..a51d300b3bb83
Binary files /dev/null and b/docs/integration-logos/opensearch/opensearch.png differ
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index f8f613dfc3c87..661c3cf7a98af 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1046,6 +1046,8 @@ Oozie
openapi
openfaas
openlineage
+OpenSearch
+opensearch
oper
Opsgenie
opsgenie
diff --git a/generated/PYPI_README.md b/generated/PYPI_README.md
index f0ecd7cb5fdab..d6a9a97383e2a 100644
--- a/generated/PYPI_README.md
+++ b/generated/PYPI_README.md
@@ -47,7 +47,7 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.7.2) |
+| | Main version (dev) | Stable version (2.7.3) |
|-------------|------------------------------|------------------------|
| Python | 3.8, 3.9, 3.10, 3.11 | 3.8, 3.9, 3.10, 3.11 |
| Platform | AMD64/ARM64(\*) | AMD64/ARM64(\*) |
@@ -55,7 +55,7 @@ Apache Airflow is tested with:
| PostgreSQL | 11, 12, 13, 14, 15 | 11, 12, 13, 14, 15 |
| MySQL | 5.7, 8.0, 8.1 | 5.7, 8.0 |
| SQLite | 3.15.0+ | 3.15.0+ |
-| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*), 2019(\*) |
+| MSSQL | 2017(\*\*), 2019(\*\*) | 2017(\*\*), 2019(\*\*) |
\* Experimental
@@ -126,15 +126,15 @@ them to the appropriate format and workflow that your tool requires.
```bash
-pip install 'apache-airflow==2.7.2' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt"
+pip install 'apache-airflow==2.7.3' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
```
2. Installing with extras (i.e., postgres, google)
```bash
-pip install 'apache-airflow[postgres,google]==2.7.2' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt"
+pip install 'apache-airflow[postgres,google]==2.7.3' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.8.txt"
```
For information on installing provider packages, check
diff --git a/generated/provider_dependencies.json b/generated/provider_dependencies.json
index 616827e66144b..16eb2cb65d256 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -549,9 +549,10 @@
"azure-datalake-store>=0.0.45",
"azure-identity>=1.3.1",
"azure-keyvault-secrets>=4.1.0",
- "azure-kusto-data>=0.0.43,<0.1",
- "azure-mgmt-containerinstance>=1.5.0,<2.0",
- "azure-mgmt-datafactory>=1.0.0,<2.0",
+ "azure-kusto-data>=4.1.0",
+ "azure-mgmt-containerinstance>=9.0.0",
+ "azure-mgmt-containerregistry>=8.0.0",
+ "azure-mgmt-datafactory>=2.0.0",
"azure-mgmt-datalake-store>=0.5.0",
"azure-mgmt-resource>=2.2.0",
"azure-servicebus>=7.6.1",
@@ -659,6 +660,14 @@
],
"excluded-python-versions": []
},
+ "opensearch": {
+ "deps": [
+ "apache-airflow>=2.5.0",
+ "opensearch-py>=2.2.0"
+ ],
+ "cross-providers-deps": [],
+ "excluded-python-versions": []
+ },
"opsgenie": {
"deps": [
"apache-airflow>=2.4.0",
@@ -721,7 +730,7 @@
"apache-airflow-providers-common-sql>=1.3.1",
"apache-airflow>=2.4.0",
"pandas>=0.17.1",
- "presto-python-client>=0.8.2"
+ "presto-python-client>=0.8.4"
],
"cross-providers-deps": [
"common.sql",
diff --git a/generated/provider_metadata.json b/generated/provider_metadata.json
index bcf3dcf113cf8..390b76420e2b4 100644
--- a/generated/provider_metadata.json
+++ b/generated/provider_metadata.json
@@ -49,8 +49,16 @@
"date_released": "2023-05-23T14:20:24Z"
},
"3.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"alibaba": {
@@ -95,8 +103,28 @@
"date_released": "2023-05-23T14:20:24Z"
},
"2.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "2.5.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "2.5.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:46Z"
+ },
+ "2.5.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "2.5.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "2.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"amazon": {
@@ -233,8 +261,48 @@
"date_released": "2023-05-23T14:20:24Z"
},
"8.2.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "8.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-10T17:51:39Z"
+ },
+ "8.3.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "8.4.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "8.5.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "8.5.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "8.6.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "8.7.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "8.7.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-18T01:35:53Z"
+ },
+ "8.8.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ },
+ "8.9.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-21T20:02:16Z"
}
},
"apache.beam": {
@@ -307,8 +375,28 @@
"date_released": "2023-05-23T14:20:24Z"
},
"5.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "5.2.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "5.2.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "5.2.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.2.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "5.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.cassandra": {
@@ -361,8 +449,12 @@
"date_released": "2023-05-23T14:20:24Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.drill": {
@@ -419,8 +511,24 @@
"date_released": "2023-05-23T14:20:24Z"
},
"2.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "2.4.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "2.4.3": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "2.4.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "2.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.druid": {
@@ -501,8 +609,20 @@
"date_released": "2023-05-23T14:20:24Z"
},
"3.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.4.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.flink": {
@@ -519,8 +639,20 @@
"date_released": "2023-05-23T14:20:24Z"
},
"1.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "1.1.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "1.1.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "1.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.hdfs": {
@@ -585,8 +717,16 @@
"date_released": "2023-05-23T14:20:24Z"
},
"4.1.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.1.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "4.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.hive": {
@@ -699,8 +839,32 @@
"date_released": "2023-05-23T14:20:24Z"
},
"6.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "6.1.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "6.1.3": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "6.1.4": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "6.1.5": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "6.1.6": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "6.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.impala": {
@@ -713,8 +877,20 @@
"date_released": "2023-05-23T14:20:24Z"
},
"1.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "1.1.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "1.1.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:50Z"
+ },
+ "1.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.kafka": {
@@ -729,6 +905,14 @@
"1.1.1": {
"associated_airflow_version": "2.0.0",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "1.1.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "1.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.kylin": {
@@ -773,8 +957,12 @@
"date_released": "2023-05-23T14:20:24Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.livy": {
@@ -839,8 +1027,24 @@
"date_released": "2023-05-23T14:20:24Z"
},
"3.5.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.5.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "3.5.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.5.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.pig": {
@@ -885,8 +1089,16 @@
"date_released": "2023-05-23T14:20:24Z"
},
"4.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.1.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.pinot": {
@@ -947,8 +1159,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.1.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.1.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.1.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.spark": {
@@ -1017,8 +1245,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.1.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.1.3": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "4.1.4": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.1.5": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"apache.sqoop": {
@@ -1075,8 +1323,34 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.0.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "4.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ }
+ },
+ "apprise": {
+ "1.0.0": {
+ "associated_airflow_version": "2.0.0",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "1.0.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "1.0.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "1.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"arangodb": {
@@ -1101,8 +1375,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "2.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "2.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"asana": {
@@ -1143,8 +1425,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "2.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "2.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"atlassian.jira": {
@@ -1169,8 +1459,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "2.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"celery": {
@@ -1219,8 +1513,36 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.3.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "3.3.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ },
+ "3.4.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-21T20:02:16Z"
}
},
"cloudant": {
@@ -1265,8 +1587,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"cncf.kubernetes": {
@@ -1407,8 +1733,44 @@
"date_released": "2023-05-23T14:20:25Z"
},
"7.1.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "7.2.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "7.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "7.4.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "7.4.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "7.4.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "7.5.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-09-01T12:37:25Z"
+ },
+ "7.5.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "7.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-18T01:35:53Z"
+ },
+ "7.7.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"common.sql": {
@@ -1457,8 +1819,50 @@
"date_released": "2023-05-27T16:42:58Z"
},
"1.5.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "1.6.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "1.6.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "1.6.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "1.7.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "1.7.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "1.7.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "1.8.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ }
+ },
+ "daskexecutor": {
+ "1.0.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "1.0.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "1.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"databricks": {
@@ -1547,8 +1951,32 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.3.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "4.3.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "4.3.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.3.3": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "4.4.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"datadog": {
@@ -1597,8 +2025,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"dbt.cloud": {
@@ -1651,8 +2087,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "3.2.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-14T06:02:26Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"dingding": {
@@ -1701,8 +2153,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"discord": {
@@ -1751,8 +2207,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.3.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.3.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"docker": {
@@ -1861,8 +2325,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.7.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:45Z"
+ },
+ "3.7.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "3.7.3": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "3.7.4": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.7.5": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.8.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"elasticsearch": {
@@ -1963,8 +2447,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.5.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.0.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-21T07:16:52Z"
+ },
+ "5.0.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.0.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"exasol": {
@@ -2041,16 +2541,36 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
- }
- },
- "facebook": {
- "1.0.0": {
- "associated_airflow_version": "2.0.0",
- "date_released": "2020-12-09T21:48:29Z"
},
- "1.0.1": {
+ "4.2.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.2.3": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "4.2.4": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.2.5": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ }
+ },
+ "facebook": {
+ "1.0.0": {
+ "associated_airflow_version": "2.0.0",
+ "date_released": "2020-12-09T21:48:29Z"
+ },
+ "1.0.1": {
"associated_airflow_version": "2.0.1",
"date_released": "2021-02-04T09:11:00Z"
},
@@ -2103,8 +2623,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"ftp": {
@@ -2169,8 +2697,24 @@
"date_released": "2023-05-27T16:42:58Z"
},
"3.4.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "3.5.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.5.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"github": {
@@ -2211,8 +2755,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "2.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "2.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"google": {
@@ -2232,10 +2784,46 @@
"associated_airflow_version": "2.6.2",
"date_released": "2023-05-27T16:42:58Z"
},
+ "10.10.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
+ },
+ "10.10.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-21T20:02:16Z"
+ },
"10.2.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
},
+ "10.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "10.4.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "10.5.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "10.6.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-10T13:03:11Z"
+ },
+ "10.7.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "10.8.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "10.9.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-18T01:35:53Z"
+ },
"2.0.0": {
"associated_airflow_version": "2.0.1",
"date_released": "2021-02-04T09:11:00Z"
@@ -2407,8 +2995,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"hashicorp": {
@@ -2481,8 +3077,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.4.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "3.4.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"http": {
@@ -2559,8 +3167,24 @@
"date_released": "2023-05-27T16:42:58Z"
},
"4.4.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.5.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "4.5.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.5.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"imap": {
@@ -2621,8 +3245,24 @@
"date_released": "2023-05-27T16:42:58Z"
},
"3.2.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "3.3.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"influxdb": {
@@ -2659,8 +3299,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "2.2.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "2.2.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "2.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"jdbc": {
@@ -2721,8 +3373,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.0.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.0.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.0.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"jenkins": {
@@ -2795,8 +3459,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"microsoft.azure": {
@@ -2933,8 +3605,40 @@
"date_released": "2023-05-27T16:42:58Z"
},
"6.1.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "6.2.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "6.2.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "6.2.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "6.2.3": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "6.2.4": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "6.3.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-09-01T12:37:25Z"
+ },
+ "7.0.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "8.0.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"microsoft.mssql": {
@@ -3007,8 +3711,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.4.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"microsoft.psrp": {
@@ -3057,8 +3769,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"2.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "2.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "2.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"microsoft.winrm": {
@@ -3119,8 +3839,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"mongo": {
@@ -3177,8 +3905,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"mysql": {
@@ -3271,8 +4007,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.2.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "5.2.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "5.3.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.3.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:16Z"
}
},
"neo4j": {
@@ -3333,8 +4089,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.3.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"odbc": {
@@ -3395,8 +4163,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.0.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"openfaas": {
@@ -3441,8 +4213,40 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
+ }
+ },
+ "openlineage": {
+ "1.0.0": {
+ "associated_airflow_version": "2.0.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "1.0.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "1.0.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "1.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "1.1.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
+ }
+ },
+ "opensearch": {
+ "1.0.0": {
+ "associated_airflow_version": "2.0.0",
+ "date_released": "2023-10-21T20:02:16Z"
}
},
"opsgenie": {
@@ -3499,8 +4303,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"oracle": {
@@ -3581,8 +4389,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.7.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.7.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.7.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.7.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.8.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"pagerduty": {
@@ -3631,8 +4455,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.3.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"papermill": {
@@ -3693,8 +4525,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"plexus": {
@@ -3739,8 +4575,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"postgres": {
@@ -3829,8 +4673,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.5.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.5.2": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "5.6.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "5.6.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.7.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
+ },
+ "5.7.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-21T20:02:16Z"
}
},
"presto": {
@@ -3915,78 +4779,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
- }
- },
- "qubole": {
- "1.0.0": {
- "associated_airflow_version": "2.0.0",
- "date_released": "2020-12-09T21:48:29Z"
- },
- "1.0.1": {
- "associated_airflow_version": "2.0.1",
- "date_released": "2021-02-04T09:11:00Z"
- },
- "1.0.2": {
- "associated_airflow_version": "2.0.2",
- "date_released": "2021-02-27T14:47:19Z"
- },
- "2.0.0": {
- "associated_airflow_version": "2.1.2",
- "date_released": "2021-06-23T12:50:28Z"
- },
- "2.0.1": {
- "associated_airflow_version": "2.2.1",
- "date_released": "2021-08-30T21:27:32Z"
- },
- "2.1.0": {
- "associated_airflow_version": "2.2.4",
- "date_released": "2022-02-08T21:57:49Z"
},
- "2.1.1": {
- "associated_airflow_version": "2.2.4",
- "date_released": "2022-03-07T20:27:47Z"
- },
- "2.1.2": {
- "associated_airflow_version": "2.2.4",
- "date_released": "2022-03-14T22:17:57Z"
- },
- "2.1.3": {
- "associated_airflow_version": "2.3.2",
- "date_released": "2022-03-22T22:09:01Z"
- },
- "3.0.0": {
- "associated_airflow_version": "2.3.3",
- "date_released": "2022-06-08T22:56:47Z"
- },
- "3.1.0": {
- "associated_airflow_version": "2.3.3",
- "date_released": "2022-07-13T20:26:57Z"
- },
- "3.2.0": {
- "associated_airflow_version": "2.3.4",
- "date_released": "2022-08-10T11:26:48Z"
- },
- "3.2.1": {
- "associated_airflow_version": "2.4.1",
- "date_released": "2022-09-28T12:31:00Z"
- },
- "3.3.0": {
- "associated_airflow_version": "2.4.1",
- "date_released": "2022-11-18T10:44:03Z"
+ "5.1.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
},
- "3.3.1": {
- "associated_airflow_version": "2.6.0",
- "date_released": "2022-11-29T14:19:45Z"
+ "5.1.3": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
},
- "3.4.0": {
- "associated_airflow_version": "2.6.2",
- "date_released": "2023-05-23T14:20:25Z"
+ "5.1.4": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
},
- "3.4.1": {
- "associated_airflow_version": "2.6.2",
- "date_released": "2023-06-23T15:38:46Z"
+ "5.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"redis": {
@@ -4031,8 +4841,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.3.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-09T10:34:19Z"
+ },
+ "3.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"salesforce": {
@@ -4109,8 +4935,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.4.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.4.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"samba": {
@@ -4159,8 +4997,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"segment": {
@@ -4205,8 +5051,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"sendgrid": {
@@ -4255,8 +5105,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"sftp": {
@@ -4357,8 +5215,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.4.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "4.5.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.6.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "4.6.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.7.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"singularity": {
@@ -4407,8 +5285,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"slack": {
@@ -4485,8 +5371,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"7.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "7.3.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "8.0.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "8.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "8.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"smtp": {
@@ -4503,8 +5405,24 @@
"date_released": "2023-05-23T14:20:25Z"
},
"1.2.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "1.3.0": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "1.3.1": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "1.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "1.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"snowflake": {
@@ -4621,8 +5539,40 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.2.0": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.3.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "4.3.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-16T05:37:52Z"
+ },
+ "4.4.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "4.4.1": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-08-08T06:19:11Z"
+ },
+ "4.4.2": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-14T06:04:47Z"
+ },
+ "5.0.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.0.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.1.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"sqlite": {
@@ -4699,8 +5649,16 @@
"date_released": "2023-05-27T16:42:58Z"
},
"3.4.2": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.4.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"ssh": {
@@ -4793,8 +5751,20 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.7.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.7.2": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.7.3": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.8.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"tableau": {
@@ -4863,8 +5833,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.2.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"tabular": {
@@ -4885,8 +5863,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"1.2.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "1.3.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"telegram": {
@@ -4943,8 +5925,12 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.7.2",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.2.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"trino": {
@@ -5029,8 +6015,28 @@
"date_released": "2023-05-23T14:20:25Z"
},
"5.1.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "5.2.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "5.2.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "5.3.0": {
+ "associated_airflow_version": "2.7.1",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "5.3.1": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "5.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"vertica": {
@@ -5095,8 +6101,82 @@
"date_released": "2023-05-23T14:20:25Z"
},
"3.4.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.6.3",
+ "date_released": "2023-07-09T14:32:07Z"
+ },
+ "3.5.1": {
+ "associated_airflow_version": "2.7.0",
+ "date_released": "2023-08-01T17:21:45Z"
+ },
+ "3.5.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "3.6.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
+ }
+ },
+ "yandex": {
+ "1.0.0": {
+ "associated_airflow_version": "2.0.0",
+ "date_released": "2020-12-09T21:48:29Z"
+ },
+ "1.0.1": {
+ "associated_airflow_version": "2.0.1",
+ "date_released": "2021-02-04T09:11:00Z"
+ },
+ "2.0.0": {
+ "associated_airflow_version": "2.1.2",
+ "date_released": "2021-06-23T12:50:28Z"
+ },
+ "2.1.0": {
+ "associated_airflow_version": "2.2.1",
+ "date_released": "2021-08-30T21:27:32Z"
+ },
+ "2.2.0": {
+ "associated_airflow_version": "2.2.4",
+ "date_released": "2021-12-31T20:54:29Z"
+ },
+ "2.2.1": {
+ "associated_airflow_version": "2.2.4",
+ "date_released": "2022-03-07T20:27:47Z"
+ },
+ "2.2.2": {
+ "associated_airflow_version": "2.2.4",
+ "date_released": "2022-03-14T22:17:57Z"
+ },
+ "2.2.3": {
+ "associated_airflow_version": "2.3.2",
+ "date_released": "2022-03-22T22:09:01Z"
+ },
+ "3.0.0": {
+ "associated_airflow_version": "2.3.3",
+ "date_released": "2022-06-08T22:56:47Z"
+ },
+ "3.1.0": {
+ "associated_airflow_version": "2.3.4",
+ "date_released": "2022-08-10T11:26:48Z"
+ },
+ "3.2.0": {
+ "associated_airflow_version": "2.5.0",
+ "date_released": "2022-11-29T14:19:45Z"
+ },
+ "3.3.0": {
+ "associated_airflow_version": "2.6.0",
+ "date_released": "2023-03-06T20:29:01Z"
+ },
+ "3.4.0": {
+ "associated_airflow_version": "2.6.0",
+ "date_released": "2023-08-29T06:43:23Z"
+ },
+ "3.5.0": {
+ "associated_airflow_version": "2.6.0",
+ "date_released": "2023-10-17T07:49:17Z"
}
},
"zendesk": {
@@ -5149,8 +6229,16 @@
"date_released": "2023-05-23T14:20:25Z"
},
"4.3.1": {
- "associated_airflow_version": "2.6.2",
+ "associated_airflow_version": "2.6.3",
"date_released": "2023-06-23T15:38:46Z"
+ },
+ "4.3.2": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-09-12T06:46:51Z"
+ },
+ "4.4.0": {
+ "associated_airflow_version": "2.7.2",
+ "date_released": "2023-10-17T07:49:17Z"
}
}
}
diff --git a/helm_tests/airflow_aux/test_cleanup_pods.py b/helm_tests/airflow_aux/test_cleanup_pods.py
index 33e475d59f168..79563aee4a654 100644
--- a/helm_tests/airflow_aux/test_cleanup_pods.py
+++ b/helm_tests/airflow_aux/test_cleanup_pods.py
@@ -373,7 +373,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"cleanup": {"enabled": True, "serviceAccount": {"automountServiceAccountToken": False}},
diff --git a/helm_tests/airflow_aux/test_create_user_job.py b/helm_tests/airflow_aux/test_create_user_job.py
index 3ed0cf04af4eb..bb85a7d3cf315 100644
--- a/helm_tests/airflow_aux/test_create_user_job.py
+++ b/helm_tests/airflow_aux/test_create_user_job.py
@@ -408,7 +408,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"createUserJob": {
diff --git a/helm_tests/airflow_aux/test_migrate_database_job.py b/helm_tests/airflow_aux/test_migrate_database_job.py
index 6bd52a110012f..27f86c8dbb77f 100644
--- a/helm_tests/airflow_aux/test_migrate_database_job.py
+++ b/helm_tests/airflow_aux/test_migrate_database_job.py
@@ -339,7 +339,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"migrateDatabaseJob": {
diff --git a/helm_tests/airflow_core/test_dag_processor.py b/helm_tests/airflow_core/test_dag_processor.py
index 74035222e7a35..4ba7ebf28a1b8 100644
--- a/helm_tests/airflow_core/test_dag_processor.py
+++ b/helm_tests/airflow_core/test_dag_processor.py
@@ -641,7 +641,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"dagProcessor": {
diff --git a/helm_tests/airflow_core/test_scheduler.py b/helm_tests/airflow_core/test_scheduler.py
index 57ab3bbe5bcb8..08971dea69520 100644
--- a/helm_tests/airflow_core/test_scheduler.py
+++ b/helm_tests/airflow_core/test_scheduler.py
@@ -815,7 +815,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"scheduler": {
diff --git a/helm_tests/airflow_core/test_triggerer.py b/helm_tests/airflow_core/test_triggerer.py
index 64f1a3419b9f0..10366a219bed9 100644
--- a/helm_tests/airflow_core/test_triggerer.py
+++ b/helm_tests/airflow_core/test_triggerer.py
@@ -602,7 +602,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"triggerer": {
diff --git a/helm_tests/airflow_core/test_worker.py b/helm_tests/airflow_core/test_worker.py
index 81432ee7ac6f3..a93f450dc00e5 100644
--- a/helm_tests/airflow_core/test_worker.py
+++ b/helm_tests/airflow_core/test_worker.py
@@ -814,7 +814,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"workers": {
diff --git a/helm_tests/other/test_flower.py b/helm_tests/other/test_flower.py
index e195d146e737f..f48ac1501d351 100644
--- a/helm_tests/other/test_flower.py
+++ b/helm_tests/other/test_flower.py
@@ -592,7 +592,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"flower": {
diff --git a/helm_tests/other/test_pgbouncer.py b/helm_tests/other/test_pgbouncer.py
index 454912b4fb6b3..122057762b76e 100644
--- a/helm_tests/other/test_pgbouncer.py
+++ b/helm_tests/other/test_pgbouncer.py
@@ -584,7 +584,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"pgbouncer": {
diff --git a/helm_tests/other/test_redis.py b/helm_tests/other/test_redis.py
index c99503a4b7711..51a84dc532cd5 100644
--- a/helm_tests/other/test_redis.py
+++ b/helm_tests/other/test_redis.py
@@ -344,7 +344,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"redis": {
diff --git a/helm_tests/other/test_statsd.py b/helm_tests/other/test_statsd.py
index c383cd99d3629..bf3737e6bf886 100644
--- a/helm_tests/other/test_statsd.py
+++ b/helm_tests/other/test_statsd.py
@@ -311,7 +311,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"statsd": {
diff --git a/helm_tests/webserver/test_webserver.py b/helm_tests/webserver/test_webserver.py
index d76cdb2eaa522..8b9a58058e1c7 100644
--- a/helm_tests/webserver/test_webserver.py
+++ b/helm_tests/webserver/test_webserver.py
@@ -1026,7 +1026,7 @@ def test_default_automount_service_account_token(self):
)
assert jmespath.search("automountServiceAccountToken", docs[0]) is True
- def test_overriden_automount_service_account_token(self):
+ def test_overridden_automount_service_account_token(self):
docs = render_chart(
values={
"webserver": {
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index 09b6278bf9419..0cfd8e390e08c 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -1,8 +1,8 @@
# This file is automatically generated by pre-commit. If you have a conflict with this file
# Please do not solve it but run `breeze setup regenerate-command-images`.
# This command should fix the conflict and regenerate help images that you have conflict with.
-main:344261ca3aa7ff31e098b1d88280566a
-build-docs:9b540ef8f8c2bd942647595d1e250cae
+main:62ab7b1184c5986a3672268f07c8b959
+build-docs:31467c83a91ec364c026ae3874f13e96
ci:find-backtracking-candidates:17fe56b867a745e5032a08dfcd3f73ee
ci:fix-ownership:3e5a73533cc96045e72cb258783cfc96
ci:free-space:49af17b032039c05c41a7a8283f365cc
@@ -10,63 +10,66 @@ ci:get-workflow-info:8246038093359b9c3c110043419473e2
ci:resource-check:bfcca92f18a403ca630955074eb5e9ad
ci:selective-check:6657ed5d42affb7264b5efcc86f17a2a
ci:5315c29bd9f68725ef92e4db8aff5cda
-ci-image:build:96ed67d4e1008b3860e20faccfad1069
-ci-image:pull:7f14482a588f018f76df84719e77723f
+ci-image:build:c6d03ef18fbb392568b6a73a8bd4cef9
+ci-image:pull:f9248c6026da61fe0acdb5d8f37b20da
ci-image:verify:c90dc7e20fce2351eb89d8d1ebbd35e7
-ci-image:ecf4756b5c00d0574b660d3b27658c2a
+ci-image:7b7aac96dfcb7d690898e4f14c843683
cleanup:8d92d453a6700f6d8cb11fb6a8b50461
compile-www-assets:0963f1409f0aa1e3b137cddd4cc52e87
down:4580f5b3b178ea00182694f134a751f3
exec:9d0fb86607526afb6b161115ae7bf9cc
-k8s:build-k8s-image:4c299179e91066626e5b4a61b06fd6c6
-k8s:configure-cluster:3bd5d4614ffcd02952fadd0f61bc0cdf
-k8s:create-cluster:cef6b3203035af1741d6acc87a26fe12
+k8s:build-k8s-image:b625255c3e8f3f794ee404f9a4476836
+k8s:configure-cluster:5f9f8d52adc36ca38a894a9c519330f3
+k8s:create-cluster:45cedc9a839d7cf1e968fdc2c278808e
k8s:delete-cluster:76a40b5a9001d8a933672ae3175ad59f
-k8s:deploy-airflow:bb5f9212ad00de2f4131df2bb2beebf3
+k8s:deploy-airflow:a616b16546476e6ee624112c74cf51e6
k8s:k9s:f25f0df6b0fdb2604822e0c8d5b64d06
k8s:logs:547c07e859eafcf67591952b785d381d
-k8s:run-complete-tests:0cc62d0fbea8ae35849dc84fa786c960
+k8s:run-complete-tests:72140e054826d3823dd4fb6d4b99523f
k8s:setup-env:a34e94744ca4e0592371fe55478c3d54
k8s:shell:ced753917db3d1bd89e7ea3314ac0d69
k8s:status:1529ccd444b41c4b0b5f943289957100
-k8s:tests:2a1e2928faea2eddafaff94176a46690
-k8s:upload-k8s-image:6b3a20cdeb692f3c3d727f6b9e68c901
-k8s:8c1e4287deb0533a74f3b302f9c574be
-prod-image:build:789f32a07099033b49936d4cfbeb5322
-prod-image:pull:76f1f27e6119928412abecf153fce4bb
+k8s:tests:737ce3fc8d7906eff9932d3c7c3130f2
+k8s:upload-k8s-image:30c898ce07d7c655d1b01b0256dd3b0d
+k8s:50226c047e5ce6b717ded38de22f3d31
+prod-image:build:2777de42de868b1fbae4222f321072a1
+prod-image:pull:3817ef211b023b76df84ee1110ef64dd
prod-image:verify:bd2b78738a7c388dbad6076c41a9f906
-prod-image:2946a9cee30c8ad7bd738ea3c77bff55
-release-management:add-back-references:9c565c08edd57ab2b484efc2be53672e
+prod-image:2a4a95019e577a46ccf729959526c80f
+release-management:add-back-references:e733b42fcf2d3d3868ba2622feb788a6
release-management:create-minor-branch:a3834afc4aa5d1e98002c9e9e7a9931d
-release-management:generate-constraints:b8fcaf8f0acd35ed5dbd48659bdb6485
-release-management:generate-issue-content-providers:71fccecb5187faf66aaada89f3db9091
+release-management:generate-constraints:01aef235b11e59ed7f10c970a5cdaba7
+release-management:generate-issue-content-providers:f760bc29c06a3ff447188fb421984b58
release-management:generate-providers-metadata:d4e8e5cfaa024e3963af02d7a873048d
-release-management:install-provider-packages:7a5aa75736387c442c05163581543cbc
+release-management:install-provider-packages:1a1f03a5086edb1ecd578d86347ce113
release-management:prepare-airflow-package:85d01c57e5b5ee0fb9e5f9d9706ed3b5
-release-management:prepare-provider-documentation:a87bae187db8bb988f4cace124b23171
-release-management:prepare-provider-packages:cd39b58b8c1b46d6ce26ad53e3577aa4
-release-management:publish-docs:1eb6f4721cd52be497d1623d37170175
+release-management:prepare-provider-documentation:dca9e1e3eec36789d3504a6df1586747
+release-management:prepare-provider-packages:a5dd282567a8e8d3061c92ef3ae33674
+release-management:publish-docs:e784a3aa3f45da93e6340770b84d21bf
release-management:release-prod-images:cfbfe8b19fee91fd90718f98ef2fd078
release-management:start-rc-process:b27bd524dd3c89f50a747b60a7e892c1
release-management:start-release:419f48f6a4ff4457cb9de7ff496aebbe
-release-management:update-constraints:596d970210f9a71cf6b688995797c0e2
+release-management:update-constraints:02ec4b119150e3fdbac52026e94820ef
release-management:verify-provider-packages:2bfa1015b2a4682f7b098587e09026d4
-release-management:394f619fa3803bf148f200c9f0213c5d
-sbom:generate-provider-requirements:230cea8fb70cb4fddd73583d82ca4ede
-sbom:update-sbom-information:0ce56884e5f842e3e80d6619df1ccc64
-sbom:531007d8e9f8415e6c17b7b04dd89b9d
+release-management:7ce4fd8ea3baf261cb6d251b5d1118bb
+sbom:build-all-airflow-images:32f8acade299c2b112e986bae99846db
+sbom:generate-providers-requirements:d0677219f2c42fe8e454f0850b62e002
+sbom:update-sbom-information:653be48be70b4b7ff5172d491aadc694
+sbom:2d1e9e5569157acb68af5196c415a651
setup:autocomplete:fffcd49e102e09ccd69b3841a9e3ea8e
-setup:check-all-params-in-groups:639dfb061b8da02dcaa4a57b567ff42a
-setup:config:38ebaaf93ed42bc7b2a3000eeea2631d
-setup:regenerate-command-images:d5e29ec6acb1a6af7d83772c2962f89d
+setup:check-all-params-in-groups:5c5e3c382fc8ce84899d224448b3f48a
+setup:config:3928be029b3272805915aebc9ae55189
+setup:regenerate-command-images:3c3023afd1761030c64bd6f1d6ef4793
setup:self-upgrade:4af905a147fcd6670a0e33d3d369a94b
setup:version:be116d90a21c2afe01087f7609774e1e
-setup:8de3ed2645928e8f9f89c58f0ccd2a60
-shell:1e901a677a6df6ba1d64d3fe79b42587
-start-airflow:ff0f63e20b9ff454e5d3c7b9ba9080d7
-static-checks:d319b1c7972a6623bc8ee1b174cacb48
-testing:docker-compose-tests:0c810047fc66a0cfe91119e2d08b3507
-testing:helm-tests:8e491da2e01ebd815322c37562059d77
-testing:integration-tests:486e4d91449ecdb7630ef2a470d705a3
-testing:tests:b3b921fd5a7d3435a0ad34e90b75cb2f
-testing:13325e047fc32d9e40b51bfd15212a91
+setup:cf6e4e5f42f09dd34682c86b69b49983
+shell:7c98cbb7bc893bbc80765af3e56f6017
+start-airflow:b9e24f09365a8e3c19371f28f3e738ff
+static-checks:81a162939cfbce8c8e0f1082b6761090
+testing:db-tests:195ac3879e402df1e510df72336a543e
+testing:docker-compose-tests:a4dfe7dadbe3e95fdf2b8d2107f7e208
+testing:helm-tests:98a9ba6631249762b1633b76a29f4461
+testing:integration-tests:7007231e960b91cfda126da374aee3a4
+testing:non-db-tests:7d48842aca56b3757238a48e813f1887
+testing:tests:7c6dc0d43c2b63476e103b8eeb3bd8bb
+testing:20d5728130253c010974a0fa2d43bafa
diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg
index 65ed6cdbb0c21..4ad484e5ce951 100644
--- a/images/breeze/output-commands.svg
+++ b/images/breeze/output-commands.svg
@@ -1,4 +1,4 @@
-