Fix batchoperator deferrable xcom links by kakatur · Pull Request #64745 · apache/airflow

kakatur · 2026-04-05T17:22:27Z

Problem

Fixes missing CloudWatch links when BatchOperator runs in deferrable mode. Previously, CloudWatch log links were not persisted to the Airflow UI when tasks deferred to the triggerer.

When deferrable=True, the operator would defer to the triggerer after submitting the Batch job, but the CloudWatch link persistence logic only ran in the non-deferrable path. This meant users couldn't easily access CloudWatch logs from the Airflow UI for deferred tasks, even though the logs existed in AWS.

Additionally, the fast-path completion scenario (when a job completes before deferring) and failure cases also lacked proper CloudWatch link persistence.

How to Reproduce

Create a DAG with BatchOperator using deferrable=True
Run the task and let it defer to the triggerer
Check the task instance UI - the CloudWatch logs link is missing from the operator links section
The logs exist in CloudWatch but users have no easy way to access them from Airflow UI

Solution

Core fix: Added CloudWatch link persistence to deferrable execution paths by calling link persistence logic in execute_complete() after the triggerer signals job completion. (commits dca1ffa, 42bdfd9, 0364247)

Comprehensive coverage: Extended fix to handle all execution scenarios:

Fast-path completion: When job finishes before the operator defers, persist CloudWatch links immediately in execute()
Failure cases: Persist CloudWatch links even when jobs fail, so users can debug via CloudWatch logs
Success cases: Ensure links persist in normal deferrable completion flow (commits 3f17702, 3b78fc9)

Code improvements: Extracted _persist_cloudwatch_link() helper method to eliminate code duplication and ensure consistent link persistence across all code paths. (commits 0a6cbad, 3f17702)

Additional Changes

Optimizations based on Copilot Comments:

Reduced API calls - 67% reduction (3→1) before deferring by removing redundant wait_for_job() call in execute_complete() (commit 0a6cbad)
Context validation - Added check for empty context (if not context or "ti" not in context) to skip CloudWatch API calls when context lacks task instance (commit 0058f4d)
Removed XCom placeholder - Added if not self.do_xcom_push: return to respect do_xcom_push setting instead of always attempting to push CloudWatch links (commit 3b78fc9)
Bug fix - Reordered code in execute_complete() to set job_id before persisting CloudWatch link, ensuring links work on both success and failure paths (commit 3b78fc9)
Conditional logic based on awslogs_enabled - Initially gated CloudWatch links with if self.awslogs_enabled, then reverted to preserve UI links in all cases (commits 0058f4d, 4dc6b9d)

Test and Type Fixes:

test_execute_without_failures - Updated to mock get_job_all_awslogs_info() instead of expecting multiple get_job_description() calls, enabled xcom_push, added proper dict context (commit e9b8d80)
test_execute_complete_success_with_logs - Added proper mocking of hook methods (get_job_all_awslogs_info, check_job_success) and enabled xcom_push with proper context (commit e9b8d80)
test_execute_complete_success_without_logs - Same mocking updates to handle CloudWatch link persistence in all scenarios (commit e9b8d80)
mypy type check / prek hook - Replaced AirflowException with ValueError for missing job_id, added type guard for job_id in _persist_cloudwatch_link() (commit 9b210ef)

How to Verify the Fix

Manual Testing:

Create a DAG with BatchOperator using deferrable=True and awslogs_enabled=True
Run the task and verify it defers to the triggerer
After task completion, check the task instance UI → "Operator Extra Links" section
The "CloudWatch Events" link should now appear and take you to the job's CloudWatch logs

Automated Testing:

Run uv run --project providers/amazon pytest providers/amazon/tests/unit/amazon/aws/operators/test_batch.py::TestBatchOperator::test_execute_without_failures -xvs
Run uv run --project providers/amazon pytest providers/amazon/tests/unit/amazon/aws/operators/test_batch.py::TestBatchOperator::test_execute_complete_success_with_logs -xvs
All tests should pass with the new mocking expectations

Expected Behavior:

CloudWatch links persist in all execution paths: normal deferrable completion, fast-path completion (job finishes before defer), and failure scenarios
Links appear regardless of awslogs_enabled setting (flag only controls log streaming during wait, not link persistence)
AWS Batch API calls reduced from 3 to 1 in execute_complete() for better performance

Was generative AI tooling used to co-author this PR?

Yes — Claude Code

Generated-by: Claude Code following the guidelines

When BatchOperator runs with deferrable=True, operator links (job definition, job queue, CloudWatch logs) were not being persisted as XCom values, causing ERROR-level logs when the UI tried to render them. This change: - Extracts link persistence logic into _persist_links() method - Calls _persist_links() from execute_complete() for deferrable tasks - Refactors monitor_job() to use the same method (DRY principle) Fixes XCom not found errors for: - batch_job_definition - batch_job_queue - cloudwatch_events

Added _persist_links(context) call in execute() method before deferring. This ensures operator links (batch_job_definition, batch_job_queue, cloudwatch_events) are persisted as XCom values immediately after job submission, making them available in the UI while the task is deferred.

- Persist placeholder None value for cloudwatch_events XCom when logs aren't available at submission time - Change log level from WARNING to INFO for missing CloudWatch logs - Prevents 'XCom not found' warning in UI while task is deferred - CloudWatch link will be populated when job completes and logs exist

boring-cyborg · 2026-04-05T17:22:30Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Copilot

Pull request overview

This PR aims to ensure the AWS Batch operator’s extra UI links (job definition/queue/logs) are available for deferrable runs by persisting the related XComs before/around deferral and completion.

Changes:

Persist operator extra-link XComs before deferring in execute() and again in execute_complete().
Refactor link persistence logic into a new _persist_links() helper.
Adjust CloudWatch log-link persistence and logging behavior.

o-nikolas · 2026-04-08T00:59:22Z

Converting this to draft since there is unaddressed copilot feedback and failing tests. change this to ready for review once all feedback is addressed and the tests are green 🙂

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

- Reduce AWS API calls by 67% to avoid throttling - Fix missing job success check when awslogs disabled - Improve CloudWatch log error visibility - Handle edge cases in unit tests This addresses code review feedback on PR apache#64745.

kakatur · 2026-04-11T03:22:35Z

I've implemented the feedback from Guthub copilot.

Reduced API calls - 67% reduction (3→1) before deferring
Context validation - Handles empty context in unit tests
Removed XCom placeholder - Respects do_xcom_push setting
Eliminated duplication in monitor_job - CloudWatch fetched once
Better error logging - WARNING level for CloudWatch failures
No duplication in execute_complete - Conditional logic based on awslogs_enabled
Bug fix - Added missing check_job_success() when awslogs_enabled=False

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

Address GitHub Copilot code review feedback: - Reduce AWS Batch API calls by 67% to prevent throttling - Remove dead code and redundant operations - Fix missing job success check when awslogs disabled - Improve CloudWatch error logging visibility - Use keyword arguments for better test compatibility This optimizes the deferrable execution path by reusing job descriptions and eliminating duplicate link persistence and CloudWatch log fetching. Addresses code review feedback on PR apache#64745. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…hub.com/kakatur/airflow into fix-batchoperator-deferrable-xcom-links

ferruzzi

Left two nipicks, otherwise LGTM

Changed AirflowException to ValueError for job_id checks in execute() and _persist_links() to be consistent with monitor_job() and follow the pattern that ValueError should be used for invalid state/parameters rather than runtime errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

potiuk · 2026-05-18T10:52:06Z

@kakatur — There are 1 unresolved review thread on this PR from @ferruzzi. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

The test test_defer_but_failed_due_to_job_id_not_found expects an AirflowException to be raised when job_id is not found before deferral. Changed the validation to raise AirflowException instead of ValueError to match the test expectation and the method's documented behavior. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

The BatchOperator now has 6 raise AirflowException statements (down from 7 on main): - execute() method: raises AirflowException for missing job_id (1) - _persist_links() and monitor_job(): now raise ValueError instead (2 replacements) - Other existing raises remain unchanged (5 unchanged) Total: 7 - 2 + 1 = 6 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

…chOperator Changed job_id validation errors from AirflowException to ValueError since these represent invalid state rather than runtime failures. This follows the coding standard of using specific exceptions instead of the broad AirflowException. Updated test to expect ValueError and use pytest.raises match parameter for more specific error matching. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…tions - Remove do_xcom_push guard from _persist_cloudwatch_link() to decouple UI link persistence from XCom return value behavior. CloudWatch links now appear in UI regardless of do_xcom_push setting, consistent with other operator links (BatchJobDetailsLink, BatchJobDefinitionLink, etc.) - Update _persist_links type annotations from bare dict to dict[str, Any] for better static type checking and codebase consistency - Remove test workaround that set do_xcom_push=True just to exercise link persistence, making test intent clearer Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

When using BatchOperator in deferrable mode with awslogs_enabled=True, logs were not being fetched from CloudWatch. Only the CloudWatch link was persisted in the UI, but the actual log messages were not printed to the task log. This created an inconsistency with non-deferrable mode where logs are streamed during execution. This commit adds log fetching functionality to execute_complete() that: - Fetches up to 10,000 completed log messages from CloudWatch - Prints each message to the task log for visibility - Works for both success and failure cases - Respects the awslogs_enabled flag - Gracefully handles missing log streams Test coverage expanded to cover all scenarios: - Success with logs enabled - Success with logs disabled - Failure with logs enabled - Failure with logs disabled (new test) All tests ensure CloudWatch links are persisted regardless of awslogs_enabled setting, maintaining UI link access in all cases. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Add robust error handling and performance optimizations based on Copilot review feedback: 1. Exception handling for CloudWatch API failures - Wrap get_last_log_messages() in try/except - Log warning with traceback instead of failing task - Ensures task continues even if log retrieval fails - Matches background thread error handling pattern 2. Performance optimization for XCom disabled case - Add early return when do_xcom_push=False in _persist_cloudwatch_link() - Prevents unnecessary get_job_all_awslogs_info() API call - BaseAwsLink.persist() already checks this, now we skip earlier - Saves 1 API call when XCom push disabled 3. Documentation accuracy - Update execute() docstring to document both exception types - ValueError: raised when job_id is not found - AirflowException: raised for job submission/execution failures - Matches actual code behavior throughout operator All changes validated: - 23/23 unit tests pass - All static checks pass (ruff, prek, mypy) - Custom exception handling test verifies graceful degradation - No breaking changes, fully backward compatible Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

boring-cyborg · 2026-06-24T22:57:13Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

When BatchOperator runs with deferrable=True, operator links (job definition, job queue, CloudWatch logs) were not being persisted as XCom values, causing ERROR-level logs when the UI tried to render them. This change: - Extracts link persistence logic into _persist_links() method - Calls _persist_links() from execute_complete() for deferrable tasks - Refactors monitor_job() to use the same method (DRY principle)

kakatur added 3 commits April 4, 2026 13:46

kakatur requested a review from o-nikolas as a code owner April 5, 2026 17:22

boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Apr 5, 2026

o-nikolas requested a review from Copilot April 6, 2026 19:49

Copilot started reviewing on behalf of o-nikolas April 6, 2026 19:50 View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

eladkal requested review from ferruzzi and vincbeck April 7, 2026 05:43

o-nikolas marked this pull request as draft April 8, 2026 00:58

kaxil requested a review from Copilot April 10, 2026 19:55

Copilot started reviewing on behalf of kaxil April 10, 2026 19:56 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated

kakatur and others added 3 commits April 10, 2026 18:18

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

e9c3ee4

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

ea5178b

kakatur marked this pull request as ready for review April 11, 2026 03:20

kakatur requested a review from Copilot April 11, 2026 04:04

Copilot started reviewing on behalf of kakatur April 11, 2026 04:05 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

kakatur and others added 3 commits April 11, 2026 07:58

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

23cffcf

Merge branch 'fix-batchoperator-deferrable-xcom-links' of https://git…

f61f1d5

…hub.com/kakatur/airflow into fix-batchoperator-deferrable-xcom-links

kakatur requested a review from Copilot April 11, 2026 15:05

Copilot started reviewing on behalf of kakatur April 11, 2026 15:06 View session

ferruzzi approved these changes May 12, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated

kakatur and others added 2 commits May 22, 2026 08:21

kakatur requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, jscheffl and potiuk as code owners May 22, 2026 15:27

kakatur requested a review from Copilot May 22, 2026 15:28

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Updated ValueError to AirflowException

95bb2a1

kakatur requested a review from Copilot May 25, 2026 23:55

Copilot AI reviewed May 25, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated

Comment thread providers/amazon/tests/unit/amazon/aws/operators/test_batch.py Outdated

kakatur and others added 2 commits May 25, 2026 16:59

o-nikolas reviewed Jun 22, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Comment thread providers/amazon/tests/unit/amazon/aws/operators/test_batch.py

kakatur requested a review from Copilot June 23, 2026 23:01

Copilot started reviewing on behalf of kakatur June 23, 2026 23:01 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

o-nikolas approved these changes Jun 24, 2026

View reviewed changes

o-nikolas merged commit 37d8acb into apache:main Jun 24, 2026
145 checks passed

vincbeck mentioned this pull request Jul 7, 2026

Status of testing Providers that were prepared on July 07, 2026 #69562

Closed

75 tasks

Uh oh!

Conversation

kakatur commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

How to Reproduce

Solution

Additional Changes

How to Verify the Fix

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg Bot commented Apr 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

o-nikolas commented Apr 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

kakatur commented Apr 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ferruzzi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

potiuk commented May 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

boring-cyborg Bot commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kakatur commented Apr 5, 2026 •

edited

Loading