Improve speed to run airflow by 6x#21438
Conversation
By delaying expensive/slow imports to where they are needed, this gets `airflow` printing it's usage information in under 0.8s, down from almost 3s which makes it feel much much snappier.
| status_code = 400 | ||
|
|
||
|
|
||
| class AirflowNotFoundException(AirflowException, ApiConnexionNotFound): |
There was a problem hiding this comment.
I have a feeling that this might break something. Lets see.
|
This was profiled using |
ashb
changed the title
airflow by 3xairflow by 6x
| log = logging.getLogger(__name__) | ||
|
|
||
| if TYPE_CHECKING: | ||
| from airflow.executors.base_executor import BaseExecutor |
There was a problem hiding this comment.
By moving this here it prevents loading all the models and saves a goodly chunk of time.
|
I love the "perceived speed" factor. You can't imagine how many tricks mobile apps do to load "percivably faster". And to be perfectly hones this is one of the most important factors that improve "quality" of the apps. |
|
(This was totally not what I "meant" to be work on tonight. But sometimes you have to roll with it) |
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
Let's see - but it looks good.
BTW. I have a vague feeling this is really a band-aid and we need to refactor the modules to avoid local imports altogether. For me the need to add "TYPE_CHECKING" and local imports is a sign we have the structure of "airflow" package wrong.
However it is likely unsolvable until airfliow as it might lead to breaking compatibility, so for now it is "good enoug".
github-actions
Bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Yeah no disagreement from me -- and primarirly one of the thngs I have always not liked about the code is the number of side-effects we get from just |
|
@ashb - seems like you've been ahead of our users reporting this: #21802 (reply in thread) (connected to the fact that someone's filesystem is SLOW) |
3 participants
By delaying expensive/slow imports to where they are needed, this gets
airflowprinting it's usage information in under 0.5s, down from almost 3s which makes it feel much much snappier.I can't really think of any way of adding tests to prevent regressions here. Ideas welcome.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.