No scheduling when max_active_runs is 1

[alexakra]

Apache Airflow version

2.2.2 (latest released)

Operating System

Debian

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

Since version 2.2 a field DAG.next_dagrun_create_after is not calculated when DAG.max_active_runs is 1.

What you expected to happen

airflow/airflow/models/dag.py

Line 2466 in fca2b19

if num_active_runs.get(dag.dag_id, 0) >= orm_dag.max_active_runs:

If this condition is evaluated when a state is "running" then it is incorrect.

How to reproduce

Create a DAG with a schedule_interval and max_active_runs=1.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[easontm]

Just out of curiosity, does the scheduler create another run if you modify the file?

[ephraimbuddy]

@alexakra This issue need more information. What do you mean by no scheduling? Is it that dagruns are not created at all? Please explain with more information

[alexakra]

Just out of curiosity, does the scheduler create another run if you modify the file?

yes, just another single run

[alexakra]

What do you mean by no scheduling? Is it that dagruns are not created at all?

There is a record in the "dag" table in which next_dagrun_create_after remains null.
In the description above, I pointed to the exact line in the code. It was changed in 2.2.
And "If this condition is evaluated when a state is "running" then it is incorrect.", i.e. condition should be ">" instead of ">=".
When next_dagrun_create_after remains null, the dag doesn't run/scheduled anymore.

[ephraimbuddy]

What do you mean by no scheduling? Is it that dagruns are not created at all?

There is a record in the "dag" table in which next_dagrun_create_after remains null. In the description above, I pointed to the exact line in the code. It was changed in 2.2. And "If this condition is evaluated when a state is "running" then it is incorrect.", i.e. condition should be ">" instead of ">=". When next_dagrun_create_after remains null, the dag doesn't run/scheduled anymore.

max_active_runs is for dag_runs in running state. Do you mean that when you set max_active_runs=1 in your DAG that you don't see any dagruns in running state? I'm trying to understand your issue. What is the issue? Not asking for the code explanation but the issue you face when you use max_active_runs in your dag

[alexakra]

Yes, when I set max_active_runs=1 in a DAG, I don't see any dagruns in running state. It is not triggered anymore after it ran only once.

[ephraimbuddy]

Yes, when I set max_active_runs=1 in a DAG, I don't see any dagruns in running state. It is not triggered anymore after it ran only once.

This looks like an issue with your schedule interval and not max_active_runs because I have a dag with max_active_runs=1 that is running correctly with airflow 2.2.2.
Can you share your DAG file

[alexakra]

test_dag.py.txt

[ephraimbuddy]

I ran your dag and it ran perfectly well

[alexakra]

Could it be related to an upgraded from 2.1.4?

[ephraimbuddy]

Could it be related to an upgraded from 2.1.4?

Not sure but check your logs if there's anything you can see

[easontm]

yes, just another single run

I think I'm experiencing a similar circumstance (on 2.2.2), however I usually get more than one run after causing a re-process.

It's not specific to any particular DAG, but there is some kind of condition (that I can't replicate on-demand) which results in the scheduler no longer creating new DAGruns. It always happens with DAGs whose max_active_runs is easy to hit, in the 1-3 range. IIRC every time I've seen this, the column DAG.next_dagrun_create_after is empty for that DAG -- but not every DAG that has this field empty has this issue. Another correlated symptom is that affected DAGs no longer get any scheduler/processing logs (in $AIRFLOW_HOME/logs/scheduler/<DATE>/...).

This happens to me maybe 2-3 times a month and I have all my (INFO) scheduler logs, although I haven't seen any anomalies myself. If anyone with better insight on the scheduler's state machine can tell me what to look for in the logs I can search deeper. Or I can even add some log points on my installation to collect until the anomaly occurs again.

@alexakra do your blocked DAGs have processing logs in the above path? And just wondering, what's your setting for schedule_after_task_execution?