Airflow cli is very slow

[Wats0ns]

Apache Airflow version

2.2.2

What happened

Typing any command using airflow, like airflow, airflow version is very slow in our production environment (~20-30s for an airflow version), which is not the cas in our dev env. Airflow is run in a docker image based on the apache/airflow:latest-python3.8 image in both env. We use the helm package airflow-stable https://airflow-helm.github.io/charts to run airflow in prod, bu as stated previously it is very slow when executing commands in a pod.

We believe that this may be the one of the cause of another problem we have, which is task unable to start because of max task nb reached.

What you expected to happen

airflow version should be immediate

How to reproduce

No response

Operating System

apache/airflow:latest-python3.8, Debian GNU/Linux 10 (buster)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jedcunningham]

Would it be possible to to wrap it with py-spy, so we can determine what's being slow?

e.g.

py-spy record -o airflow_version.svg -- airflow version

[hterik]

Python is inherently a bit slow to start up so don't expect any magic. Airflow though is a bit of an extra bad case though since it imports so many other modules.

If you use the Docker image it is even slower (assume you do since using K8), because

1. The entrypoint performs some airflow db check before starting any tasks. Not sure why. This takes 5 seconds some times.
2. You loose the .pyc-caching since it starts a fresh container each time. I did some test long ago by pre-baking the .pyc files by simply ending the Dockerfile with RUN airflow --help and it shaved off almost a complete second on subsequent docker runs. Maybe i should upstream this fix to the official image?

[potiuk]

Python is inherently a bit slow to start up so don't expect any magic. Airflow though is a bit of an extra bad case though since it imports so many other modules.

If you use the Docker image it is even slower (assume you do since using K8), because

1. The entrypoint performs some airflow db check before starting any tasks. Not sure why. This takes 5 seconds some times.

This is explained in the docs:
https://airflow.apache.org/docs/docker-stack/entrypoint.html#waits-for-airflow-db-connection

The entrypoint is waiting for a connection to the database independent of the database engine. This allows us to increase the stability of the environment.

Alongside the documentation how to disable this check:

CONNECTION_CHECK_MAX_COUNT=0

But you gave me thought that we can only run it for specific commands - so if you run airflow commands as "separate container" commands, this might help a bit.

2. You loose the .pyc-caching since it starts a fresh container each time. I did some test long ago by pre-baking the .pyc files by simply ending the Dockerfile with RUN airflow --help and it shaved off almost a complete second on subsequent docker runs. Maybe i should upstream this fix to the official image?

This is deliberate decision and baking in .pyc files is bad idea as it increases the size of the image significantly (you are basically trading of the size of the image, network, and storage with first time start for some commands.

If you want to run airflow commands repetitively, rather than running new container every time, run a single comtainer and exec command in the running container.

@Wats0ns : Airlfow version SHOULD be fast (als in terms of .pyc) because it imports very littler, so I also second @jedcunningham here - py-spy would be useful. I just checked my "airflow version" and I looked where the slowness might come from.

Almost for sure those are your local_settings or your log configuration. Parsing settings.py (and local settings) and establshing logging configuration is the one thing that happens in airflow version.

So please - run tha py-spy and let us know here by posting it (or mostl likely you will find where it comes from in your configuration).

I am converting this one into a discussion, until we hear more about the py-spy results as this is likely not an airflow issue/