Add DagProcessor status to health endpoint.

[tirkarthi]

Description

Add DagProcessor status including latest heartbeat to health endpoint similar to Triggerer status added recently. Related PRs.

#31529
#27755

Use case/motivation

It helps in dag processor monitoring

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[utkarsharma2]

@tirkarthi I don't see any downside to adding this information in Healthcheck. However, DagProcessor is not a separate component as opposed to the trigger. If DagProcessor is having some issues it won't affect airflow as a whole. It's probably an issue with a dag and if an instance of airflow is shared with multiple teams, they won't be interested to know if some dag that they don't own has some issues. WDYT?

[tirkarthi]

It's an issue from the perspective that updates to dag or new dags won't be picked up by UI and scheduler till the dag processor is healthy.

[utkarsharma2]

On further looking into this I think you can run DagProcessor as a separate process, having this feature would certainly help.
Ref - https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dagfile-processing.html

[eladkal]

We also have a task for health endpoints for workers #17191