Surface YARN diagnostics for spark on yarn clusters

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Continuation of the exception-enrichment work for spark like: Better log when spark k8s driver remains in Unknown Phase and Include spark submit canonical logs in failure exceptions. Retry policies (LLM-based or deterministic) can only reason from what in the exception message, and Spark's YARN tracking
path was the thinnest of the three deploy modes -- _start_yarn_application_status_tracking
raised only "ended with state: FAILED, final status: FAILED", with no indication of why.

Solution

YARN has a "diagnostics" field to provide detailed diag bundle for failures: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Rather than assume the RM API's diagnostics field is populated with anything useful, I triggered real jobs against a live cluster and captured the actual RM responses.

Job tested	Diagnostics returned by RM	Message before	Message after
MapReduce job whose AM container fails to launch (missing HADOOP_MAPRED_HOME env)	Application application_...0002 failed 2 times due to AM Container for appattempt_...000002 exited with exitCode: 1 ... Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster ...	YARN application application_...0002 ended with state: FAILED, final status: FAILED	YARN application application_...0002 ended with state: FAILED, final status: FAILED Diagnostics: Application application_...0002 failed 2 times due to AM Container for appattempt_...000002 exited with exitCode: 1 ... Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster ...
Long-running pi job killed mid-flight via yarn application -kill	Application application_...0005 was killed by user root at 192.168.117.5	YARN application application_...0005 ended with state: KILLED, final status: KILLED	YARN application application_...0005 ended with state: KILLED, final status: KILLED Diagnostics: Application application_...0005 was killed by user root at 192.168.117.5

Both captured payloads were fed into LLMRetryPolicy across runbook instruction variants and all consistently classified FAIL, including on bare defaults (none), once the message carried the real diagnostics text. The killed job case also resolved an open question: "was killed by user root at <ip>" is unambiguous, distinct from how YARN phrases
scheduler-preemption terminations -- so diagnostics alone is enough to tell an explicit kill from an infra-driven one.

Proposed Change

• _query_yarn_application_status now returns tuple[str, str, str] (state, finalStatus, diagnostics), reading app.get("diagnostics", "").
• Both failure raises in _start_yarn_application_status_tracking append \nDiagnostics: {diagnostics} when present; unchanged when absent.
• query_yarn_application_status (the normalized-status public method) updated to unpack the 3-tuple.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[amoghrajesh]

cc @nailo2c fyi

[amoghrajesh]

Unrelated failures, merging.