fix: clear next_method and next_kwargs on task retry

[YoannAbriel]

fix: clear next_method and next_kwargs on task retry

Problem

When a deferrable operator fails during trigger resumption and enters retry, next_method and next_kwargs are not cleared in Airflow 3.x. This means the retry attempt skips execute() entirely and jumps straight to the stale next_method(**next_kwargs) callback from the previous attempt.

This causes two failure modes:

1. The task processes "zombie" trigger events from the previous attempt
2. The task fails because initial setup logic in execute() was bypassed

This was a known issue in Airflow 2.x, fixed in #18146 / #18210 by nullifying next_method and next_kwargs on retry. With the transition to the Task SDK and Internal API in 3.x, the reset logic was not carried over to the new prepare_db_for_next_try() method.

Root Cause

TaskInstance.prepare_db_for_next_try() is the central method in 3.x for preparing a TI for its next attempt (generating new UUID, incrementing try_number, resetting state). It was missing a call to clear_next_method_args(), so the deferred execution state persisted across retries.

Fix

Added self.clear_next_method_args() in prepare_db_for_next_try(), right before the new attempt UUID is generated. This ensures the retry always starts fresh from execute().

One line change in airflow-core/src/airflow/models/taskinstance.py, plus two tests:

• test_prepare_db_for_next_try_clears_next_method — verifies prepare_db_for_next_try() resets deferred fields
• test_handle_failure_clears_next_method_on_retry — verifies the full handle_failure() path clears deferred state when retries remain

All 216 model tests + 127 execution API tests pass.

Closes: #62845

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code

Generated-by: Claude Code following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[tirkarthi]

As confirmed by the user this is resolved as of Airflow 3.1.7. I am closing this PR. Please feel free to reopen if this can be reproduced.

#62845 (comment)