Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs

[korex-f]

Three related bugs all produce WatchtowerWarning: "Received message after logging system shutdown", causing CloudWatch log streams to be truncated or never created.

Bug 1 (task-sdk/src/airflow/sdk/log.py):
In configure_logging(), remote.processors was accessed before the inner configure_logging() call ran dictConfig(). dictConfig() calls _clearExistingHandlers() → logging.shutdown() on all existing handlers, killing the just-created watchtower handler (shutting_down=True) before any task log is emitted. Fix: move getattr(remote, "processors") to after dictConfig runs, injecting via a second structlog.configure() call.

Bug 2 (providers/amazon/.../cloudwatch_task_handler.py):
CloudWatchRemoteLogIO.handler was a cached_property — once killed by logging.shutdown() the dead instance was never replaced. The processors cached_property also captured _handler in a closure, pinning the dead instance. Fix: convert handler to a regular property that recreates when shutting_down=True; access self.handler dynamically in the processors closure.

Bug 3 (task-sdk/src/airflow/sdk/execution_time/supervisor.py):
_configure_logging() returned a plain tuple with no cleanup path. The only code that ever closed the remote handler was logging.shutdown() at process exit, which fired after supervise_task() returned while final task messages were still in flight. Fix: convert _configure_logging() to a contextmanager that explicitly flushes and closes the handler after process.wait() drains all task log messages.

closes: #66475

---

Important

🛠️ Maintainer triage note for @korex-f · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @jason810496 is waiting on you:

• 6 unresolved review thread(s) from @jason810496 need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[chris-stetter]

I can confirm that this works in our ECS environment. Thank you!

[korex-f]

You're very much welcome.

[o-nikolas]

I would have expected some new or more updated test cases to ensure this doesn't regress again? Do you not think that's required here @korex-f ?

[korex-f]

I would have expected some new or more updated test cases to ensure this doesn't regress again? Do you not think that's required here @korex-f ?

Good point, I have added tests to test_cloudwatch_task_handler.py covering the self-healing handler property (verifies a new handler is created when shutting_down=True) and the dynamic self.handler access in the processors closure.

[o-nikolas]

Looks reasonable to me now, but I still think @ashb or @amoghrajesh should have a look

[potiuk]

@korex-f — There are 2 unresolved review threads on this PR from @o-nikolas. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[seanghaeli]

Verified end-to-end this fixes remote logging. Thanks @korex-f this issue has been floating around for a while.

[1fanwang]

Thanks for addressing the comments, lgtm!

[potiuk]

@korex-f — A reviewer (@ashb) has requested changes on this PR, so I've removed the ready for maintainer review label — the next step is on your side. Could you address the review comments (push a fix, or reply in-thread explaining why the feedback doesn't apply)? Once addressed, re-request review from @ashb or re-mark the PR ready and it returns to the maintainer queue. Thank you.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[korex-f]

Thanks for the review, @ashb. I have addressed the requested changes, pushed the fixes, and re-requested review. Please do take another look when you have the chance.

[sarvesh371]

Hey, running into this exact issue on Airflow 3.2.2 with KubernetesExecutor — task pods emit WatchtowerWarning: Received message after logging system shutdown and logs never make it to CloudWatch.

Any idea on ETA for merge and which provider version will ship the fix? Also is there any interim workaround while we wait?

Happy to test if there's a pre-release available.

[jason810496]

Thanks for fixing this and I verify the root case as well. The overall direction LGTM.
However, this PR touches both Task-SDK and provider.

IIRC, the current convention is separating PRs of Task-SDK and provider. So we need to make this PR Task-SDK only and raise another one for provider or vice versa.

Any idea on ETA for merge and which provider version will ship the fix? Also is there any interim workaround while we wait?

If this PR doesn't make it to the 3.3 release (since it change the Task-SDK). Then I will merge the #68779 which solve the exact issue in different direction with provider-only changes (and I verify on k8s executor setup) -> the next AWS provider version should solve this problem.

If this PR catches 3.3 release -> using Airflow 3.3 plus the next AWS provider version could solve.

[korex-f]

Thanks for sharing #68779 @jason810496, having read through it, I agree it's the cleaner approach to the provider side. The _closed flag correctly distinguishes dictConfig mid-task closures from genuine teardown, which is something my approach didn't handle.

Given this, I'd like to scope this PR down to task-sdk only:

• Issue 1 (log.py): getattr(remote, "processors") called before dictConfig runs — Fix CloudWatch remote logging for ephemeral lifecycle executor #68779 doesn't fix this and it's the root cause
  that makes Issue 2 worse
• Issue 3 (supervisor.py): _close_remote_log_handler context manager to ensure the handler is flushed after process.wait() returns

That means reverting all provider changes from this PR and letting #68779 carry those.

Does that scope make sense @ashb? Happy to also drop the close() protocol addition from RemoteLogIO per your comment, since #68779 handles the provider side without needing it.

[jason810496]

Happy to also drop the close() protocol addition from RemoteLogIO per your comment

Yes, I think we should make this PR targeting task-sdk changes only. (more specific, only change task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py)

[korex-f]

Addressed all outstanding review comments:

• Reverted unrelated changes in _handle_request (GetXComCount ordering and send_msg parameter style)
• Restored the dropped TODO comment in _configure_logging
• Used walrus operator for load_remote_log_handler() check
• Added hasattr(handler, "close") guard in _close_remote_log_handler to avoid mypy error after reverting the RemoteLogIO protocol change
• Reverted all provider and shared/logging changes per @jason810496 guidance, this PR now touches only supervisor.py and log.py

[eladkal]

Can the provider related changes be split into a dedicated PR?

[korex-f]

Can the provider related changes be split into a dedicated PR?

Done!
All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.

[o-nikolas]

@jason810496 and @ferruzzi, does this look good to approve/merge for y'all?

@ashb it looks like your feedback was addressed, but you "requested changes" which makes me think you saw something very serious, are you happy with this PR now?