Add a task instance dependency for mapped dependencies#37498
Merged
eladkal merged 17 commits intoApr 2, 2024
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
uranusjr
reviewed
Mar 12, 2024
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
from
March 13, 2024 15:29
4bd1303 to
c2e94c2
Compare
uranusjr
reviewed
Mar 13, 2024
uranusjr
reviewed
Mar 13, 2024
uranusjr
reviewed
Mar 13, 2024
Contributor
|
@stevenschaerer can you address the comments and rebase ? |
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
2 times, most recently
from
March 25, 2024 21:07
adbc13a to
325eace
Compare
Contributor
Author
|
@uranusjr Could you please review my latest changes and responses to your comments? Thanks! |
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
from
March 28, 2024 22:16
a6b93e7 to
dc90eb3
Compare
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
from
March 31, 2024 09:50
dc90eb3 to
ab8da53
Compare
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
from
March 31, 2024 17:51
42a694d to
2f69980
Compare
uranusjr
reviewed
Apr 2, 2024
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
…om the base operator deps
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
stevenschaerer
force-pushed
the
feature/mapped-task-upstream-issues
branch
from
April 2, 2024 10:28
86dbbe7 to
e4031b3
Compare
uranusjr
approved these changes
Apr 2, 2024
eladkal
approved these changes
Apr 2, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes #37091.
In particular, dynamically mapped tasks currently show as
failed_upstreaminstead ofskippedif some of the tasks that generate the mapping are skipped and the trigger rule isnone_failed,all_doneor similar.This is because when trying to expand the mapped task, a failed and skipped mapped dependency behave identically and both result in a NotFullyPopulated being raised.
This PR introduces a new task instance dependency that sets a dynamically mapped task to
skippedif any of the mapped dependencies are skipped (but not failed) and toupstream_failedif any of the mapped dependencies are failed. As such, an expansion of the mapped task is never attempted.Before (from #37091):

After:
