Skip to content

Remove leftovers from #45184 clear_not_launched_queued_tasks removal#68698

Merged
jason810496 merged 1 commit into
apache:mainfrom
FrankYang0529:remove-dead-code-in-KubernetesExecutor
Jun 23, 2026
Merged

Remove leftovers from #45184 clear_not_launched_queued_tasks removal#68698
jason810496 merged 1 commit into
apache:mainfrom
FrankYang0529:remove-dead-code-in-KubernetesExecutor

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Jun 18, 2026

Copy link
Copy Markdown
Member

#45184 removed clear_not_launched_queued_tasks() (switching from the proactive stuck-QUEUED sweep to internal retries on pod deletion) but left two pieces of its supporting machinery behind:

  • KubernetesExecutor.last_handled — written on every task launch (execute_async) but never read or cleared anywhere, since both its only reader and its only del lived inside the removed method. It grows by one entry per launched task for the scheduler's lifetime — an unbounded leak that serves no purpose.
  • The kubernetes_executor.clear_not_launched_queued_tasks.duration metric in the registry YAML — its Stats.timer(...) was removed with the method, so the catalog now lists a metric that is never emitted.

related: #45184

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jun 18, 2026
…moval

Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529 FrankYang0529 force-pushed the remove-dead-code-in-KubernetesExecutor branch from 30a1b56 to 09aa636 Compare June 18, 2026 05:53
@FrankYang0529 FrankYang0529 changed the title KubernetesExecutor: remove orphaned last_handled dict leaked since #45184 Remove leftovers from #45184 clear_not_launched_queued_tasks removal Jun 18, 2026
@FrankYang0529 FrankYang0529 marked this pull request as ready for review June 18, 2026 06:54
@arkadiuszbach

Copy link
Copy Markdown
Contributor

Just spent few hours analyzing this memory leak in Scheduler on Airflow 3.2.2

@FrankYang0529

Copy link
Copy Markdown
Member Author

Thanks for the approval, @Lee-W 🙏

Since this touches the KubernetesExecutor scheduler path and fixes an unbounded memory leak, would another committer be willing to give it a second look? Thank you.

@Lee-W Lee-W requested a review from jason810496 June 20, 2026 02:29

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@jason810496 jason810496 merged commit 7ce8504 into apache:main Jun 23, 2026
198 checks passed
@FrankYang0529 FrankYang0529 deleted the remove-dead-code-in-KubernetesExecutor branch June 23, 2026 05:39
cetingokhan pushed a commit to cetingokhan/airflow that referenced this pull request Jun 24, 2026
…moval (apache#68698)

Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants