diff --git a/IMAGES.rst b/IMAGES.rst index ae47255e6dac1..bd7dcba626255 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -22,8 +22,8 @@ Airflow docker images Airflow has two images (build from Dockerfiles): +* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation * CI image (Dockerfile.ci) - used for running tests and local development -* Production image (Dockerfile) - used to run production-ready Airflow installations Image naming conventions ======================== @@ -34,7 +34,7 @@ The images are named as follows: where: -* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.10`` +* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.12`` The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. The ``1.10.*`` and in the future ``2.*`` labels are build from git tags and they are "fixed" once built. * ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.5``, ``3.7`` @@ -67,8 +67,8 @@ By adding ``--python `` parameter you can build the image version for the chosen python version. The images are build with default extras - different extras for CI and production image and you -can change the extras via the ``--extras`` parameters. You can see default extras used via -``./breeze flags``. +can change the extras via the ``--extras`` parameters and add new ones with ``--additional-extras``. +You can see default extras used via ``./breeze flags``. For example if you want to build python 3.7 version of production image with "all" extras installed you should run this command: @@ -77,6 +77,12 @@ For example if you want to build python 3.7 version of production image with ./breeze build-image --python 3.7 --extras "all" --production-image +If you just want to add new extras you can add them like that: + +.. code-block:: bash + + ./breeze build-image --python 3.7 --additional-extras "all" --production-image + The command that builds the CI image is optimized to minimize the time needed to rebuild the image when the source code of Airflow evolves. This means that if you already have the image locally downloaded and built, the scripts will determine whether the rebuild is needed in the first place. Then the scripts will @@ -92,13 +98,15 @@ parameter to Breeze: .. code-block:: bash - ./breeze build-image --python 3.7 --extras=gcp --production-image --install-airflow-version=1.10.12 + ./breeze build-image --python 3.7 --additional-extras=presto \ + --production-image --install-airflow-version=1.10.12 This will build the image using command similar to: .. code-block:: bash - pip install apache-airflow[sendgrid]==1.10.12 \ + pip install \ + apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.12 \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt" You can also build production images from specific Git version via providing ``--install-airflow-reference`` @@ -186,8 +194,8 @@ For example: apache/airflow:master-python3.6 - production "latest" image from current master apache/airflow:master-python3.6-ci - CI "latest" image from current master apache/airflow:v1-10-test-python2.7-ci - CI "latest" image from current v1-10-test branch - apache/airflow:1.10.10-python3.6 - production image for 1.10.10 release - apache/airflow:1.10.10-1-python3.6 - production image for 1.10.10 with some patches applied + apache/airflow:1.10.12-python3.6 - production image for 1.10.12 release + apache/airflow:1.10.12-1-python3.6 - production image for 1.10.12 with some patches applied You can see DockerHub images at ``_ @@ -225,7 +233,6 @@ in ``_. Note that you need to be committer and have the right to push to DockerHub and GitHub and you need to be logged in. Only committers can push images directly. - Technical details of Airflow images =================================== @@ -360,199 +367,12 @@ This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headl docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.6 --build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" - - Production images -................. - -The following build arguments (``--build-arg`` in docker build command) can be used for production images: +----------------- -+------------------------------------------+------------------------------------------+------------------------------------------+ -| Build argument | Default value | Description | -+==========================================+==========================================+==========================================+ -| ``PYTHON_BASE_IMAGE`` | ``python:3.6-slim-buster`` | Base python image | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``PYTHON_MAJOR_MINOR_VERSION`` | ``3.6`` | major/minor version of Python (should | -| | | match base image) | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_VERSION`` | ``2.0.0.dev0`` | version of Airflow | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_REPO`` | ``apache/airflow`` | the repository from which PIP | -| | | dependencies are pre-installed | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_BRANCH`` | ``master`` | the branch from which PIP dependencies | -| | | are pre-installed initially | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | ``constraints-master`` | reference (branch or tag) from GitHub | -| | | repository from which constraints are | -| | | used. By default it is set to | -| | | ``constraints-master`` but can be | -| | | ``constraints-1-10`` for 1.10.* versions | -| | | or it could point to specific version | -| | | for example ``constraints-1.10.12`` | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is | -| | | installed | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_AIRFLOW_EXTRAS`` | | Optional additional extras with which | -| | | airflow is installed | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend | -| | | the image with some extra dependencies | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_DEV_DEPS`` | | additional apt dev dependencies to | -| | | install | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``ADDITIONAL_RUNTIME_DEPS`` | | additional apt runtime dependencies to | -| | | install | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_HOME`` | ``/opt/airflow`` | Airflow’s HOME (that’s where logs and | -| | | sqlite databases are stored) | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_UID`` | ``50000`` | Airflow user UID | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_GID`` | ``50000`` | Airflow group GID. Note that most files | -| | | created on behalf of airflow user belong | -| | | to the ``root`` group (0) to keep | -| | | OpenShift Guidelines compatibility | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user | -+------------------------------------------+------------------------------------------+------------------------------------------+ -| ``CASS_DRIVER_BUILD_CONCURRENCY`` | ``8`` | Number of processors to use for | -| | | cassandra PIP install (speeds up | -| | | installing in case cassandra extra is | -| | | used). | -+------------------------------------------+------------------------------------------+------------------------------------------+ +You can find details about using, building, extending and customising the production images in the +`Latest documentation `_ -There are build arguments that determine the installation mechanism of Apache Airflow for the -production image. There are three types of build: - -* From local sources (by default for example when you use ``docker build .``) -* You can build the image from released PyPi airflow package (used to build the official Docker image) -* You can build the image from any version in GitHub repository(this is used mostly for system testing). - -+-----------------------------------+-----------------------------------+ -| Build argument | What to specify | -+===================================+===================================+ -| ``AIRFLOW_INSTALL_SOURCES`` | Should point to the sources of | -| | of Apache Airflow. It can be | -| | either "." for installation from | -| | local sources, "apache-airflow" | -| | for installation from packages | -| | and URL to installation from | -| | GitHub repository (see below) | -| | to install from any GitHub | -| | version | -+-----------------------------------+-----------------------------------+ -| ``AIRFLOW_INSTALL_VERSION`` | Optional - might be used for | -| | package installation case to | -| | set Airflow version for example | -| | "==1.10.10" | -+-----------------------------------+-----------------------------------+ -| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | reference (branch or tag) from | -| | GitHub where constraints file | -| | is taken from. By default it is | -| | ``constraints-master`` but can be | -| | ``constraints-1-10`` for 1.10.* | -| | constraint or if you want to | -| | point to specific varsion | -| | ``constraints-1.10.12 | -+-----------------------------------+-----------------------------------+ -| ``SLUGIFY_USES_TEXT_UNIDECODE`` | In case of of installing airflow | -| | 1.10.2 or 1.10.1 you need to | -| | set this arg to ``yes``. | -+-----------------------------------+-----------------------------------+ -| ``AIRFLOW_WWW`` | In case of Airflow 2.0 it should | -| | be "www", in case of Airflow 1.10 | -| | series it should be "www_rbac". | -| | See examples below | -+-----------------------------------+-----------------------------------+ -| ``AIRFLOW_SOURCES_FROM`` | Sources of Airflow. Set it to | -| | "empty" to avoid costly | -| | Docker context copying | -| | in case of installation from | -| | the package or from GitHub URL. | -| | See examples below | -+-----------------------------------+-----------------------------------+ -| ``AIRFLOW_SOURCES_TO`` | Target for Airflow sources. Set | -| | to "/empty" to avoid costly | -| | Docker context copying | -| | in case of installation from | -| | the package or from GitHub URL. | -| | See examples below | -+-----------------------------------+-----------------------------------+ - - -This builds production image in version 3.6 with default extras from the local sources (master version -of 2.0 currently): - -.. code-block:: bash - - docker build . - -This builds the production image in version 3.7 with default extras from 1.10.12 tag and -constraints taken from constraints-1-10-12 branch in GitHub. - -.. code-block:: bash - - docker build . \ - --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ - --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ - --build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.12.tar.gz#egg=apache-airflow" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ - --build-arg AIRFLOW_BRANCH="v1-10-test" \ - --build-arg AIRFLOW_SOURCES_FROM="empty" \ - --build-arg AIRFLOW_SOURCES_TO="/empty" - -This builds the production image in version 3.7 with default extras from 1.10.12 Pypi package and -constraints taken from 1.10.12 tag in GitHub and pre-installed pip dependencies from the top -of v1-10-test branch. - -.. code-block:: bash - - docker build . \ - --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ - --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ - --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ - --build-arg AIRFLOW_BRANCH="v1-10-test" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.12" \ - --build-arg AIRFLOW_SOURCES_FROM="empty" \ - --build-arg AIRFLOW_SOURCES_TO="/empty" - -This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and -additional python dependencies and pre-installed pip dependencies from 1.10.10 tagged constraints. - -.. code-block:: bash - - docker build . \ - --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ - --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ - --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \ - --build-arg AIRFLOW_BRANCH="v1-10-test" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.10" \ - --build-arg AIRFLOW_SOURCES_FROM="empty" \ - --build-arg AIRFLOW_SOURCES_TO="/empty" \ - --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" - --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client" - -This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and -additional apt dev and runtime dependencies. - -.. code-block:: - - docker build . \ - --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ - --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ - --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ - --build-arg AIRFLOW_SOURCES_FROM="empty" \ - --build-arg AIRFLOW_SOURCES_TO="/empty" \ - --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" - --build-arg ADDITIONAL_DEV_DEPS="gcc g++" - --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" Image manifests --------------- @@ -590,7 +410,6 @@ however uou can also force it with the same flag. ./breeze build-image --force-pull-images - Embedded image scripts ====================== @@ -598,7 +417,6 @@ Both images have a set of scripts that can be used in the image. Those are: * /entrypoint - entrypoint script used when entering the image * /clean-logs - script for periodic log cleaning - Running the CI image ==================== @@ -635,85 +453,13 @@ The entrypoint performs those operations: * Runs system tests if RUN_SYSTEM_TESTS flag is specified, otherwise runs regular unit and integration tests -Using the PROD image -==================== - -The entrypoint in the PROD image contains all the initialisation needed for tests to be immediately executed. -It is copied from ``scripts/in_container/entrypoint_prod.sh``. - -The PROD image entrypoint works as follows: - -* In case the user is not "airflow" (with undefined user id) and the group id of the user is set to 0 (root), - then the user is dynamically added to /etc/passwd at entry using USER_NAME variable to define the user name. - This is in order to accommodate the - `OpenShift Guidelines `_ - -* If ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is passed to the container and it is either mysql or postgres - SQL alchemy connection, then the connection is checked and the script waits until the database is reachable. - -* If no ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is set or if it is set to sqlite SQL alchemy connection - then db reset is executed. - -* If ``AIRFLOW__CELERY__BROKER_URL`` variable is passed and scheduler, worker of flower command is used then - the connection is checked and the script waits until the Celery broker database is reachable. - -* If first argument equals to "bash" - it dropped in bash shell or executes bash command if you specify - extra arguments. For example: - -.. code-block:: bash - - docker run -it apache/airflow:master-python3.6 bash -c "ls -la" - total 16 - drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . - drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. - drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 dags - drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 logs - -* If first argument is equal to "python" - you are dropped in python shell or python commands are executed if - you pass extra parameters. For example: - -.. code-block:: bash - - > docker run -it apache/airflow:master-python3.6 python -c "print('test')" - test - -* If there are any other arguments - they are passed to "airflow" command - -.. code-block:: bash +Using, customising, and extending the production image +====================================================== - > docker run -it apache/airflow:master-python3.6 --help - - usage: airflow [-h] - {celery,config,connections,dags,db,info,kerberos,plugins,pools,roles,rotate_fernet_key,scheduler,sync_perm,tasks,users,variables,version,webserver} - ... - - positional arguments: - - Groups: - celery Start celery components - connections List/Add/Delete connections - dags List and manage DAGs - db Database operations - pools CRUD operations on pools - roles Create/List roles - tasks List and manage tasks - users CRUD operations on users - variables CRUD operations on variables - - Commands: - config Show current application configuration - info Show information about current Airflow and environment - kerberos Start a kerberos ticket renewer - plugins Dump information about loaded plugins - rotate-fernet-key Rotate encrypted connection credentials and variables - scheduler Start a scheduler instance - sync-perm Update permissions for existing roles and DAGs - version Show the version - webserver Start a Airflow webserver instance - - optional arguments: - -h, --help show this help message and exit +You can read more about using, customising, and extending the production image in the documentation: +* [Stable docs](https://airflow.apache.org/docs/stable/production-deployment.html) +* [Latest docs from master branch](https://airflow.readthedocs.io/en/latest/production-deployment.html Alpha versions of 1.10.10 production-ready images ================================================= diff --git a/README.md b/README.md index e434158ab98f5..3486767aa942a 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The - [Getting started](#getting-started) - [Installing from PyPI](#installing-from-pypi) - [Official source code](#official-source-code) +- [Convenience packages](#convenience-packages) - [Project Focus](#project-focus) - [Principles](#principles) - [User Interface](#user-interface) @@ -87,7 +88,7 @@ Official Docker (container) images for Apache Airflow are described in [IMAGES.r ## Installing from PyPI -Airflow is published as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky +We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open and applications usually pin them, but we should do neither and both at the same time. We decided to keep our dependencies as open as possible (in `setup.py`) so users can install different versions of libraries @@ -116,31 +117,42 @@ pip install apache-airflow[postgres,google]==1.10.12 \ ## Official source code -Apache Airflow is an [Apache Software Foundation](http://www.apache.org) (ASF) project, and our official source code releases: +Apache Airflow is an [Apache Software Foundation](http://www.apache.org) (ASF) project, +and our official source code releases: - Follow the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) - Can be downloaded from [the ASF Distribution Directory](https://downloads.apache.org) - Are cryptographically signed by the release manager -- Are officially voted on by the PMC members during the [Release Approval Process](http://www.apache.org/legal/release-policy.html#release-approval) +- Are officially voted on by the PMC members during the + [Release Approval Process](http://www.apache.org/legal/release-policy.html#release-approval) -Following the ASF rules, the source packages released must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. +Following the ASF rules, the source packages released must be sufficient for a user to build and test the +release provided they have access to the appropriate platform and tools. -Other ways of retrieving source code are "convenience" methods. For example: +## Convenience packages -- Tagging in GitHub to mark the git project sources that were used to generate official source packages +There are other ways of installing and using Airflow. Those are "convenience" methods - they are +not "official releases" as stated by the `ASF Release Policy`, but they can be used by the users +who do not want to build the software themselves. -We also have binary "convenience" packages: +Those are - in the order of most common ways people install Airflow: -- PyPI releases to install Airflow using standard python tools -- Docker Images published in the [Apache Airflow DockerHub](https://hub.docker.com/repository/docker/apache/airflow). +- [PyPI releases](https://pypi.org/project/apache-airflow/) to install Airflow using standard `pip` tool +- [Docker Images](https://hub.docker.com/repository/docker/apache/airflow) to install airflow via + `docker` tool, use them in Kubernetes, Helm Charts, `docker-compose`, `docker swarm` etc. You can + read more about using, customising, and extending the images in the + [Latest docs](https://airflow.readthedocs.io/en/latest/production-deployment.html), and + learn details on the internals in the [IMAGES.rst](IMAGES.rst) document. +- [Tags in GitHub](https://github.com/apache/airflow/tags) to retrieve the git project sources that + were used to generate official source packages via git -These artifacts are not official releases, but they are built using officially released sources. - -> Note: Airflow Summit 2020's ["Production Docker Image talk"](https://youtu.be/wDr3Y7q2XoI) explains context, architecture and customization/extension methods. +All those artifacts are not official releases, but they are prepared using officially released sources. +Some of those artifacts are "development" or "pre-release" ones, and they are clearly marked as such +following the ASF Policy. ## Project Focus -Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). +Airflow works best with workflows that are mostly static and slowly changing. When the structure is similarfrom one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. diff --git a/docs/index.rst b/docs/index.rst index 45ea799b0ed8b..f21041fb97152 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -98,6 +98,7 @@ Content smart-sensor changelog best-practices + production-deployment faq privacy_notice diff --git a/docs/production-deployment.rst b/docs/production-deployment.rst new file mode 100644 index 0000000000000..9f33e5cdc5c1b --- /dev/null +++ b/docs/production-deployment.rst @@ -0,0 +1,457 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Production Deployment +^^^^^^^^^^^^^^^^^^^^^ + +This document describes various aspects of the production deployments of Apache Airflow. + +Production Container Images +=========================== + +Customizing or extending the Production Image +--------------------------------------------- + +Before you dive-deeply in the way how the Airflow Image is build, named and why we are doing it the +way we do, you might want to know very quickly how you can extend or customize the existing image +for Apache Airflow. This chapter gives you a short answer to those questions. + +The docker image provided (as convenience binary package) in the +`Apache Airflow DockerHub `_ is a bare image +that has not many external dependencies and extras installed. Apache Airflow has many extras +that can be installed alongside the "core" airflow image and they often require some additional +dependencies. The Apache Airflow image provided as convenience package is optimized for size, so +it provides just a bare minimal set of the extras and dependencies installed and in most cases +you want to either extend or customize the image. + +Airflow Summit 2020's `Production Docker Image `_ talk provides more +details about the context, architecture and customization/extension methods for the Production Image. + +Extending the image +................... + +Extending the image is easiest if you just need to add some dependencies that do not require +compiling. The compilation framework of Linux (so called ``build-essential``) is pretty big, and +for the production images, size is really important factor to optimize for, so our Production Image +does not contain ``build-essential``. If you need compiler like gcc or g++ or make/cmake etc. - those +are not found in the image and it is recommended that you follow the "customize" route instead. + +How to extend the image - it is something you are most likely familiar with - simply +build a new image using Dockerfile's ``FROM:`` directive and add whatever you need. Then you can add your +Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or any other stuff you need. + +You should be aware, about a few things: + +* The production image of airflow uses "airflow" user, so if you want to add some of the tools + as ``root`` user, you need to switch to it with ``USER`` directive of the Dockerfile. Also you + should remember about following the + `best practises of Dockerfiles `_ + to make sure your image is lean and small. + +.. code-block:: dockerfile + + FROM: apache/airflow:1.10.12 + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + my-awesome-apt-dependency-to-add \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + USER airflow + + +* PyPI dependencies in Apache Airflow are installed in the user library, of the "airflow" user, so + you need to install them with the ``--user`` flag and WITHOUT switching to airflow user. Note also + that using --no-cache-dir is a good idea that can help to make your image smaller. + +.. code-block:: dockerfile + + FROM: apache/airflow:1.10.12 + RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add + + +* If your apt, or PyPI dependencies require some of the build-essentials, then your best choice is + to follow the "Customize the image" route. However it requires to checkout sources of Apache Airflow, + so you might still want to choose to add build essentials to your image, even if your image will + be significantly bigger. + +.. code-block:: dockerfile + + FROM: apache/airflow:1.10.12 + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential my-awesome-apt-dependency-to-add \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + USER airflow + RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add + + +* You can also embed your dags in the image by simply adding them with COPY directive of Airflow. + The DAGs in production image are in /opt/airflow/dags folder. + +Customizing the image +..................... + +Customizing the image is an alternative way of adding your own dependencies to the image - better +suited to prepare optimized production images. + +The advantage of this method is that it produces optimized image even if you need some compile-time +dependencies that are not needed in the final image. You need to use Airflow Sources to build such images +from the `official distribution folder of Apache Airflow `_ for the +released versions, or checked out from the Github project if you happen to do it from git sources. + +The easiest way to build the image image is to use ``breeze`` script, but you can also build such customized +image by running appropriately crafted docker build in which you specify all the ``build-args`` +that you need to add to customize it. You can read about all the args and ways you can build the image +in the `<#production-image-build-arguments>`_ chapter below. + +Here just a few examples are presented which should give you general understanding of what you can customize. + +This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and +additional apt dev and runtime dependencies. + +.. code-block:: bash + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" \ + --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" + --build-arg ADDITIONAL_PYTHON_DEPS="pandas" + --build-arg ADDITIONAL_DEV_DEPS="gcc g++" + --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" + --tag my-image + + +the same image can be built using ``breeze`` (it supports auto-completion of the options): + +.. code-block:: bash + + ./breeze build-image \ + --production-image --python 3.7 --install-airflow-version=1.10.12 \ + --additional-extras=jdbc --additional-python-deps="pandas" \ + --additional-dev-deps="gcc g++" --additional-runtime-deps="default-jre-headless" + +Customizing & extending the image together +.......................................... + +You can combine both - customizing & extending the image. You can build the image first using +``customize`` method (either with docker command or with ``breeze`` and then you can ``extend`` +the resulting image using ``FROM:`` any dependencies you want. + +External sources for dependencies +--------------------------------- + +In corporate environments, there is often the need to build your Container images using +other than default sources of dependencies. The docker file uses standard sources (such as +Debian apt repositories or PyPI repository. However, in corporate environments, the dependencies +are often only possible to be installed from internal, vetted repositories that are reviewed and +approved by the internal security teams. In those cases, you might need to use those different +sources. + +This is rather easy if you extend the image - you simply write your extension commands +using the right sources - either by adding/replacing the sources in apt configuration or +specifying the source repository in pip install command. + +It's a bit more involved in the case of customizing the image. We do not have yet (but we are working +on it) a capability of changing the sources via build args. However, since the builds use +Dockerfile that is a source file, you can rather easily simply modify the file manually and +specify different sources to be used by either of the commands. + + +Comparing extending and customizing the image +--------------------------------------------- + +Here is the comparison of the two types of building images. + ++----------------------------------------------------+---------------------+-----------------------+ +| | Extending the image | Customizing the image | ++====================================================+=====================+=======================+ +| Produces optimized image | No | Yes | ++----------------------------------------------------+---------------------+-----------------------+ +| Use Airflow Dockerfile sources to build the image | No | Yes | ++----------------------------------------------------+---------------------+-----------------------+ +| Requires Airflow sources | No | Yes | ++----------------------------------------------------+---------------------+-----------------------+ +| You can build it with Breeze | No | Yes | ++----------------------------------------------------+---------------------+-----------------------+ +| Allows to use non-default sources for dependencies | Yes | No [1] | ++----------------------------------------------------+---------------------+-----------------------+ + +[1] When you combine customizing and extending the image, you can use external sources + in the "extend" part. There are plans to add functionality to add external sources + option to image customization. You can also modify Dockerfile manually if you want to + use non-default sources for dependencies. + +Using the production image +-------------------------- + +The PROD image entrypoint works as follows: + +* In case the user is not "airflow" (with undefined user id) and the group id of the user is set to 0 (root), + then the user is dynamically added to /etc/passwd at entry using USER_NAME variable to define the user name. + This is in order to accommodate the + `OpenShift Guidelines `_ + +* If ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is passed to the container and it is either mysql or postgres + SQL alchemy connection, then the connection is checked and the script waits until the database is reachable. + +* If no ``AIRFLOW__CORE__SQL_ALCHEMY_CONN`` variable is set or if it is set to sqlite SQL alchemy connection + then db reset is executed. + +* If ``AIRFLOW__CELERY__BROKER_URL`` variable is passed and scheduler, worker of flower command is used then + the connection is checked and the script waits until the Celery broker database is reachable. + +* The ``AIRFLOW_HOME`` is set by default to ``/opt/airflow/`` - this means that DAGs + are in default in the ``/opt/airflow/dags`` folder and logs are in the ``/opt/airflow/logs`` + +* The working directory is ``/opt/airflow`` by default. + +* If first argument equals to "bash" - you are dropped to a bash shell or you can executes bash command + if you specify extra arguments. For example: + +.. code-block:: bash + + docker run -it apache/airflow:master-python3.6 bash -c "ls -la" + total 16 + drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . + drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 dags + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 logs + + +* If first argument is equal to "python" - you are dropped in python shell or python commands are executed if + you pass extra parameters. For example: + +.. code-block:: bash + + > docker run -it apache/airflow:master-python3.6 python -c "print('test')" + test + +* If there are any other arguments - they are passed to "airflow" command + +.. code-block:: bash + + > docker run -it apache/airflow:master-python3.6 + 2.0.0.dev0 + +Production image build arguments +-------------------------------- + +The following build arguments (``--build-arg`` in docker build command) can be used for production images: + ++------------------------------------------+------------------------------------------+------------------------------------------+ +| Build argument | Default value | Description | ++==========================================+==========================================+==========================================+ +| ``PYTHON_BASE_IMAGE`` | ``python:3.6-slim-buster`` | Base python image | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``PYTHON_MAJOR_MINOR_VERSION`` | ``3.6`` | major/minor version of Python (should | +| | | match base image) | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_VERSION`` | ``2.0.0.dev0`` | version of Airflow | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_REPO`` | ``apache/airflow`` | the repository from which PIP | +| | | dependencies are pre-installed | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_BRANCH`` | ``master`` | the branch from which PIP dependencies | +| | | are pre-installed initially | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | ``constraints-master`` | reference (branch or tag) from GitHub | +| | | repository from which constraints are | +| | | used. By default it is set to | +| | | ``constraints-master`` but can be | +| | | ``constraints-1-10`` for 1.10.* versions | +| | | or it could point to specific version | +| | | for example ``constraints-1.10.12`` | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is | +| | | installed | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_AIRFLOW_EXTRAS`` | | Optional additional extras with which | +| | | airflow is installed | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend | +| | | the image with some extra dependencies | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_DEV_DEPS`` | | additional apt dev dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``ADDITIONAL_RUNTIME_DEPS`` | | additional apt runtime dependencies to | +| | | install | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_HOME`` | ``/opt/airflow`` | Airflow’s HOME (that’s where logs and | +| | | sqlite databases are stored) | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_UID`` | ``50000`` | Airflow user UID | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_GID`` | ``50000`` | Airflow group GID. Note that most files | +| | | created on behalf of airflow user belong | +| | | to the ``root`` group (0) to keep | +| | | OpenShift Guidelines compatibility | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``CASS_DRIVER_BUILD_CONCURRENCY`` | ``8`` | Number of processors to use for | +| | | cassandra PIP install (speeds up | +| | | installing in case cassandra extra is | +| | | used). | ++------------------------------------------+------------------------------------------+------------------------------------------+ + +There are build arguments that determine the installation mechanism of Apache Airflow for the +production image. There are three types of build: + +* From local sources (by default for example when you use ``docker build .``) +* You can build the image from released PyPi airflow package (used to build the official Docker image) +* You can build the image from any version in GitHub repository(this is used mostly for system testing). + ++-----------------------------------+-----------------------------------+ +| Build argument | What to specify | ++===================================+===================================+ +| ``AIRFLOW_INSTALL_SOURCES`` | Should point to the sources of | +| | of Apache Airflow. It can be | +| | either "." for installation from | +| | local sources, "apache-airflow" | +| | for installation from packages | +| | and URL to installation from | +| | GitHub repository (see below) | +| | to install from any GitHub | +| | version | ++-----------------------------------+-----------------------------------+ +| ``AIRFLOW_INSTALL_VERSION`` | Optional - might be used for | +| | package installation case to | +| | set Airflow version for example | +| | "==1.10.12" | ++-----------------------------------+-----------------------------------+ +| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | reference (branch or tag) from | +| | GitHub where constraints file | +| | is taken from. By default it is | +| | ``constraints-master`` but can be | +| | ``constraints-1-10`` for 1.10.* | +| | constraint or if you want to | +| | point to specific version | +| | might be ``constraints-1.10.12`` | ++-----------------------------------+-----------------------------------+ +| ``SLUGIFY_USES_TEXT_UNIDECODE`` | In case of of installing airflow | +| | 1.10.2 or 1.10.1 you need to | +| | set this arg to ``yes``. | ++-----------------------------------+-----------------------------------+ +| ``AIRFLOW_WWW`` | In case of Airflow 2.0 it should | +| | be "www", in case of Airflow 1.10 | +| | series it should be "www_rbac". | +| | See examples below | ++-----------------------------------+-----------------------------------+ +| ``AIRFLOW_SOURCES_FROM`` | Sources of Airflow. Set it to | +| | "empty" to avoid costly | +| | Docker context copying | +| | in case of installation from | +| | the package or from GitHub URL. | +| | See examples below | ++-----------------------------------+-----------------------------------+ +| ``AIRFLOW_SOURCES_TO`` | Target for Airflow sources. Set | +| | to "/empty" to avoid costly | +| | Docker context copying | +| | in case of installation from | +| | the package or from GitHub URL. | +| | See examples below | ++-----------------------------------+-----------------------------------+ + + +This builds production image in version 3.6 with default extras from the local sources (master version +of 2.0 currently): + +.. code-block:: bash + + docker build . + +This builds the production image in version 3.7 with default extras from 1.10.12 tag and +constraints taken from constraints-1-10-12 branch in GitHub. + +.. code-block:: bash + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.12.tar.gz#egg=apache-airflow" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ + --build-arg AIRFLOW_BRANCH="v1-10-test" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" + +This builds the production image in version 3.7 with default extras from 1.10.12 Pypi package and +constraints taken from 1.10.12 tag in GitHub and pre-installed pip dependencies from the top +of v1-10-test branch. + +.. code-block:: bash + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_BRANCH="v1-10-test" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.12" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" + +This builds the production image in version 3.7 with additional airflow extras from 1.10.12 Pypi package and +additional python dependencies and pre-installed pip dependencies from 1.10.12 tagged constraints. + +.. code-block:: bash + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_BRANCH="v1-10-test" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.12" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" \ + --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" + --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client" + +This builds the production image in version 3.7 with additional airflow extras from 1.10.12 Pypi package and +additional apt dev and runtime dependencies. + +.. code-block:: bash + + docker build . \ + --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ + --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ + --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ + --build-arg AIRFLOW_SOURCES_FROM="empty" \ + --build-arg AIRFLOW_SOURCES_TO="/empty" \ + --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" + --build-arg ADDITIONAL_DEV_DEPS="gcc g++" + --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" + + +More details about the images +----------------------------- + +You can read more details about the images - the context, their parameters and internal structure in the +`IMAGES.rst `_ document. diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index 3853b97af413c..dd9d517b6abda 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -532,6 +532,7 @@ cloudant cloudml cloudsqldatabehook cloudwatch +cmake cmd cmdline cmds @@ -735,6 +736,7 @@ gRPC ga gantt gbq +gcc gcloud gcp gcpcloudsql