fix(providers/databricks): DatabricksWorkflowTaskGroup launch task now waits for upstream tasks

[shubhgurav0590]

Closes #51598

Problem

When using DatabricksWorkflowTaskGroup, the internal launch task
ignored upstream task dependencies and executed immediately at DAG
start, regardless of trigger_rule configuration.

Fix

Added create_databricks_workflow_task.set_upstream(root_task) in
the __exit__ method so the launch task correctly waits for all
upstream dependencies before executing.

Changes

• One-line fix in databricks_workflow.py
• Added unit test test_launch_task_waits_for_upstream_tasks
  validating that launch task correctly registers upstream task IDs

Testing

Unit test added in test_databricks_workflow.py covering the
external upstream dependency scenario from issue #51598.

Note on DAG Graph Screenshots

Previous PR #53559 was asked for before/after DAG graph screenshots.
Running into Windows/WSL encoding issues with Breeze setup.
Can provide screenshots if required during review — happy to
set up on a Linux environment if needed.

---

Note

🛠️ Maintainer triage note for @shubhgurav0590 · by @potiuk · 2026-06-18 21:56 UTC

Closing to keep the review queue clean — this draft was triaged ~9 days ago and there's been no update since:

• No worries and nothing lost — reopen it or open a fresh PR whenever you're ready to pick it back up.
• See the Pull Request quality criteria.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

@shubhgurav0590 A few things need addressing before review — see our Pull Request quality criteria.

• ❌ CI fails: CI image checks / Static checks (and possibly other checks — see the Checks tab for the full list).

No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

---

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

[shubhgurav0590]

"After running CI I found my fix was incorrect — it created a DAG cycle (launch → inner_task → launch). I've reverted the change.
After studying the full exit method I need clarification before implementing the correct fix:
The roots variable contains INTERNAL workflow tasks. The current code correctly makes them wait for launch (root_task.set_upstream(launch)). My one-line addition reversed this causing a cycle.
My question: when external_task >> task_group is called after the with block, does Airflow not correctly connect external_task to the launch task as the group root? Or is the bug triggered by a different pattern — such as setting dependencies before the with block? A minimal reproduction example would help me implement the correct fix.
Happy to implement once the correct approach is confirmed."

[jroachgolf84]

Hmmmm, I don't think that there are any changes here. Am I missing something?

:)

[shubhgurav0590]

"Hi @jroachgolf84 — you're correct, I reverted my initial fix after CI showed it created a DAG cycle.
My original fix added create_databricks_workflow_task.set_upstream(root_task) inside the for root_task in roots loop. But roots contains INTERNAL workflow tasks, so this created a circular dependency: launch → inner_task → launch.
The actual bug from #51598 is that when external_task >> task_group is set OUTSIDE the with block, the launch task ignores that external upstream dependency and runs immediately.
My proposed correct fix would be to transfer the task group's upstream dependencies to the launch task inside exit:
for upstream_task_id in self.upstream_task_ids:
upstream_task = self.dag.get_task(upstream_task_id)
create_databricks_workflow_task.set_upstream(upstream_task)
Could you confirm if this is the right approach before I implement it? I want to make sure I'm solving the actual bug correctly."

[potiuk]

@shubhgurav0590 A few things need addressing before review — see our Pull Request quality criteria.

• ❌ Pre-commit / static checks. See docs.

No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

---

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

[potiuk]

@shubhgurav0590 Converting to draft — this PR does not yet meet our Pull Request quality criteria.

• ❌ Merge conflicts with main. See docs.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Closing as a stale triaged draft — no update in ~9 days since the triage note. No worries: reopen this or open a fresh PR whenever you're ready to continue. Thanks for the contribution!