Skip to content

Better log when spark k8s driver remains in Unknown Phase - #70176

Merged
shahar1 merged 2 commits into
apache:mainfrom
astronomer:betterlog-spark-k8s
Jul 22, 2026
Merged

Better log when spark k8s driver remains in Unknown Phase#70176
shahar1 merged 2 commits into
apache:mainfrom
astronomer:betterlog-spark-k8s

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

The retry loop that monitored spark driver pod status gave up after repeated "Unknown" pod phases with no explanation of what that phase means, leaving retry policies (and humans) with no way to tell if it was worth retrying. Kubernetes documents Unknown phase as an error communicating with the node the pod should be running on: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase which is a credible transient condition -- so
we should state that explicitly instead of leaving it opaque.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, no harm to have more explicit error message, thanks.

@shahar1
shahar1 merged commit 523a803 into apache:main Jul 22, 2026
79 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in Durable / Crash-Safe Execution Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

4 participants