AIP-97 (draft): failure context propagation and transparent infra retry

[1fanwang]

Concrete draft backing AIP-97 (in DRAFT on the cwiki), opened so the listener-side API shape can be discussed against working code. Marked draft for that reason. This is the consolidated view; the Merge plan below decomposes it into three reviewable PRs.

Why

on_task_instance_failed sees only the worker-side error. When a task is killed from outside the worker — a pod eviction, an OOM kill, a force-deleted pod, a lost heartbeat — the worker never raises, so the listener gets nothing that says why. A pod evicted mid-run and a real ValueError look identical to alerting and lineage. The same blind spot burns retries: is_eligible_to_retry() is try_number <= max_tries with max_tries = task.retries, so an eviction spends a retry the user meant for their own code.

What this adds

Pillar 1 — failure context. on_task_instance_failed gains an optional failure_kind argument, a TaskFailureKind string enum (infra / application / timeout / manual). pluggy dispatches by parameter name, so a listener that does not declare it keeps working. The executor's reason token (Evicted, PodDeleted, ...) persists on the task instance as a new nullable infra_reason column. The Kubernetes executor's classify_pod_failure() maps a node-level disruption (Evicted, Preempting, DisruptionTarget, node shutdown) or a pod removed while running (PodDeleted — drain, preemption, spot reclaim, force-delete) to infra; a container that ended on its own (an app crash, an OOM against its own limit) to application. An unclassified death stays None.

Pillar 2 — transparent infra retry (opt-in). When failure_kind == infra, [core] infra_failure_refund_retries refunds the attempt (max_tries += 1) instead of drawing on the user's retries, bounded by max_infra_refunds. Off by default; application, timeout, and manual failures never refund. This extends the executor's existing pre-execution behavior — a pod that dies before the task starts is already requeued without consuming a retry — to the mid-run case.

Merge plan

Lands as three incremental PRs, each reviewable on its own:

1. failure_kind on the listener + infra_reason column (the type/hookspec foundation).
2. The Kubernetes executor bridge — classify_pod_failure() surfaces infra to the scheduler.
3. The opt-in transparent infra retry (refund max_tries, gated and capped).

Testing

Full-stack on a real KubernetesExecutor cluster (kind): a running task pod force-deleted, classify_pod_failure reads PodDeleted, the scheduler refunds while the TI is still RUNNING:

PRE : running try=1 max_tries=1 infra_reason=-
kubectl delete pod <task-pod> --force --grace-period=0
POST: running try=2 max_tries=2 infra_reason=PodDeleted   # refunded, retrying
infra failure (PodDeleted) refunded attempt ...; max_tries now 2 (refund 1/3)
on_task_instance_failed ... failure_kind=infra infra_reason='PodDeleted'

The non-infra paths spend the retry, verified live on LocalExecutor + Postgres: a task mark-failed is manual, a DAG-run mark-failed sets the terminal state first so it is never misclassified, an execution_timeout is timeout, a dagrun_timeout skips the task. None refunds. 34 unit tests plus scheduler-wiring, ruff clean.

Open questions (for the AIP discussion)

• The infra budget's shape: a server-side default plus a per-DAG or per-task override, versus the single max_infra_refunds cap here.
• The refund mechanism: max_tries += 1 (one write, no migration, but shares the counter with application retries) versus a dedicated infra_retry_count column.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: GitHub Copilot CLI following the guidelines