Fix coroutine serialization error in PowerBIDatasetRefreshOperator by henry3260 · Pull Request #63829 · apache/airflow

henry3260 · 2026-03-17T14:37:31Z

Why

When the PowerBIDatasetRefreshOperator is executed with wait_for_completion=False, it takes the synchronous "fire-and-forget" path. However, the underlying hook method trigger_dataset_refresh is an async function.

What

Forced Synchronous Execution: Wrapped the hook.trigger_dataset_refresh(...) call with asyncio.run(...) inside the wait_for_completion=False block. This ensures the async hook method executes fully within the current synchronous thread and returns the actual string ID.

closes: #63811

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

henry3260 · 2026-03-17T18:26:44Z

cc @morelgeorge

SameerMesiah97

I think you found the cause but I feel the fix might be at the wrong layer. Also, did you manage to reproduce the bug locally?

henry3260 · 2026-03-18T01:30:03Z

I think you found the cause but I feel the fix might be at the wrong layer. Also, did you manage to reproduce the bug locally?

Thanks for reviewing! Yes, I reproduced the error locally with the unit test.

morelgeorge · 2026-03-18T13:04:38Z

Hello @henry3260,
thank you for your effort! I tried to run the updated provider in our test instance. Unfortunately, I still have an issue with wait_for_completion=False.
Currently I am receiving an AirflowNotFoundException connection error. See :

PowerBIDatasetRefreshException: Failed to trigger dataset refresh.
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1068 in run
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1477 in _execute_task
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417 in wrapper
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/operators/powerbi.py", line 126 in execute
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 201 in trigger_dataset_refresh_sync
File "/appl/airflow/lib/lib/python3.12/asyncio/runners.py", line 195 in run
File "/appl/airflow/lib/lib/python3.12/asyncio/runners.py", line 118 in run
File "/appl/airflow/lib/lib/python3.12/asyncio/base_events.py", line 691 in run_until_complete
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 236 in trigger_dataset_refresh
AirflowNotFoundException: The conn_id `PowerBI_Con8` isn't defined
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 222 in trigger_dataset_refresh
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 511 in run
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 583 in send_request
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 404 in get_async_conn
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/common/compat/connection/__init__.py", line 43 in get_async_connection
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/bases/hook.py", line 75 in aget_connection
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/definitions/connection.py", line 234 in async_get
File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/context.py", line 220 in _async_get_connection

The connection is defined directly in UI. In case the wait_for_completion is set to True, DAG works fine. So, it seems that once you run it synchronously through asyncio.run() then get_async_conn does not work properly.

I did a quick test to test this behaviour. I changed line 583 in msgraph hook in method send_request. Instead of conn = await self.get_async_conn() I tried to use deprecated conn = self.get_conn().

Then wait_for_completion works fine - even for True or False

henry3260 · 2026-03-20T11:18:58Z

Hello @henry3260,

thank you for your effort! I tried to run the updated provider in our test instance. Unfortunately, I still have an issue with wait_for_completion=False.

Currently I am receiving an AirflowNotFoundException connection error. See :


PowerBIDatasetRefreshException: Failed to trigger dataset refresh.

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1068 in run

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1477 in _execute_task

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417 in wrapper

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/operators/powerbi.py", line 126 in execute

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 201 in trigger_dataset_refresh_sync

File "/appl/airflow/lib/lib/python3.12/asyncio/runners.py", line 195 in run

File "/appl/airflow/lib/lib/python3.12/asyncio/runners.py", line 118 in run

File "/appl/airflow/lib/lib/python3.12/asyncio/base_events.py", line 691 in run_until_complete

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 236 in trigger_dataset_refresh

AirflowNotFoundException: The conn_id `PowerBI_Con8` isn't defined

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/powerbi.py", line 222 in trigger_dataset_refresh

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 511 in run

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 583 in send_request

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/msgraph.py", line 404 in get_async_conn

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/providers/common/compat/connection/__init__.py", line 43 in get_async_connection

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/bases/hook.py", line 75 in aget_connection

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/definitions/connection.py", line 234 in async_get

File "/appl/airflow/lib/lib/python3.12/site-packages/airflow/sdk/execution_time/context.py", line 220 in _async_get_connection

The connection is defined directly in UI. In case the wait_for_completion is set to True, DAG works fine. So, it seems that once you run it synchronously through asyncio.run() then get_async_conn does not work properly.

I did a quick test to test this behaviour. I changed line 583 in msgraph hook in method send_request. Instead of conn = await self.get_async_conn() I tried to use deprecated conn = self.get_conn().

Then wait_for_completion works fine - even for True or False

Hi @morelgeorge! Thanks for the information! I think this version is better and will avoid the above error.

henry3260 · 2026-04-01T16:59:00Z

Hi @dabla, just checking in to see if we can move forward with this PR. Please let me know if there's anything else you need from my end—I'd be happy to help!

dabla · 2026-04-01T18:44:04Z

Hi @dabla, just checking in to see if we can move forward with this PR. Please let me know if there's anything else you need from my end—I'd be happy to help!

Looking good, I just would be nice if you could test the double deferral using the execute_operator test helper method, check in test_msgraph of the operators, this helper methods tests the full operator lifcyle with deferrals. That way we are certain the full operator lifecycle works. So an additional test for that would be nice, then I think we are ready to merge.

Thx for the effort!

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

henry3260 · 2026-04-02T17:34:12Z

Hi @dabla, just checking in to see if we can move forward with this PR. Please let me know if there's anything else you need from my end—I'd be happy to help!

Looking good, I just would be nice if you could test the double deferral using the execute_operator test helper method, check in test_msgraph of the operators, this helper methods tests the full operator lifcyle with deferrals. That way we are certain the full operator lifecycle works. So an additional test for that would be nice, then I think we are ready to merge.

Thx for the effort!

Add it, thanks dabla!

henry3260 · 2026-04-15T16:19:41Z

Hi @dabla, is there anything else I can do to help move this PR forward?

henry3260 · 2026-05-02T06:52:48Z

Hi @dabla Could I get your final review on this when you get a chance?

vincbeck

I am trying to understand (sorry for the late review). As far as I understand you're waiting regardless of wait_for_completion value right? What's wait_for_completion for now?

henry3260 · 2026-05-04T18:58:41Z

I am trying to understand (sorry for the late review). As far as I understand you're waiting regardless of wait_for_completion value right? What's wait_for_completion for now?

first really appreciate your review and your time!

wait_for_completion=True

execute() first defers to the trigger.
The trigger sends the refresh request and gets a dataset_refresh_id.
Control returns to handle_refresh().
Because wait_for_completion=True, it defers a second time.
The second trigger keeps polling until the refresh completes or fails.
Finally, execution resumes in execute_complete().

wait_for_completion=False

1. execute() still first defers to the trigger.
2. The trigger sends the refresh request and gets a dataset_refresh_id.
3. Control returns to handle_refresh().
4. Because wait_for_completion=False, it returns immediately there and does not defer a second time.

vincbeck · 2026-05-04T19:04:24Z

I am trying to understand (sorry for the late review). As far as I understand you're waiting regardless of wait_for_completion value right? What's wait_for_completion for now?

first really appreciate your review and your time!

wait_for_completion=True

execute() first defers to the trigger.

The trigger sends the refresh request and gets a dataset_refresh_id.

Control returns to handle_refresh().

Because wait_for_completion=True, it defers a second time.

The second trigger keeps polling until the refresh completes or fails.

Finally, execution resumes in execute_complete().

wait_for_completion=False

1. execute() still first defers to the trigger. 2. The trigger sends the refresh request and gets a dataset_refresh_id. 3. Control returns to handle_refresh(). 4. Because wait_for_completion=False, it returns immediately there and does not defer a second time.

Oh I understand now, thank you

vincbeck · 2026-05-04T19:05:27Z

@dabla Can you review it by any chance?

henry3260 · 2026-05-24T14:29:42Z

@dabla Could I get your review when you get a chance?

dabla · 2026-05-24T17:28:56Z

Overal looking good to me.

henry3260 · 2026-05-25T15:24:45Z

@vincbeck I think this one is good to go!

simonp92 · 2026-06-09T11:16:57Z

Hi

I see that the provider package was updated 2 days ago (13.4.0):
https://pypi.org/project/apache-airflow-providers-microsoft-azure/#history

Was this fix included in the release?

henry3260 requested a review from dabla as a code owner March 17, 2026 14:37

boring-cyborg Bot added area:providers provider:microsoft-azure Azure-related issues labels Mar 17, 2026

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Mar 17, 2026

SameerMesiah97 reviewed Mar 18, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

henry3260 force-pushed the fix-wait-for-completion branch 2 times, most recently from 8b7e0c8 to 0593eae Compare March 18, 2026 04:34

dabla reviewed Mar 18, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py Outdated

henry3260 force-pushed the fix-wait-for-completion branch from 0593eae to 395b3a3 Compare March 20, 2026 10:17

henry3260 requested a review from dabla March 20, 2026 15:02

dabla requested changes Mar 23, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

henry3260 force-pushed the fix-wait-for-completion branch from 395b3a3 to 9dacf17 Compare March 24, 2026 09:39

henry3260 requested a review from dabla March 27, 2026 12:09

kaxil requested a review from Copilot April 2, 2026 00:45

Copilot AI reviewed Apr 2, 2026

henry3260 force-pushed the fix-wait-for-completion branch from b4d7aba to 27e20e5 Compare April 2, 2026 20:26

henry3260 force-pushed the fix-wait-for-completion branch from 27e20e5 to 341cd00 Compare April 23, 2026 16:45

henry3260 added 2 commits May 2, 2026 14:51

Fix coroutine serialization error in PowerBIDatasetRefreshOperator

359f487

add unit test

e499fb1

henry3260 force-pushed the fix-wait-for-completion branch from 341cd00 to e499fb1 Compare May 2, 2026 06:51

vincbeck reviewed May 4, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py

vincbeck reviewed May 4, 2026

View reviewed changes

vincbeck approved these changes May 4, 2026

View reviewed changes

potiuk removed the ready for maintainer review Set after triaging when all criteria pass. label May 18, 2026

dabla reviewed May 24, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py

dabla requested changes May 24, 2026

View reviewed changes

dabla approved these changes May 24, 2026

View reviewed changes

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label May 28, 2026

potiuk approved these changes Jun 5, 2026

View reviewed changes

potiuk merged commit 47faf3c into apache:main Jun 5, 2026
94 checks passed

henry3260 deleted the fix-wait-for-completion branch June 5, 2026 10:17

shahar1 mentioned this pull request Jun 19, 2026

Status of testing Providers that were prepared on June 19, 2026 #68751

Open

75 tasks

Conversation

henry3260 commented Mar 17, 2026

Why

What

Was generative AI tooling used to co-author this PR?

Uh oh!

henry3260 commented Mar 17, 2026

Uh oh!

SameerMesiah97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henry3260 commented Mar 18, 2026

Uh oh!

morelgeorge commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

henry3260 commented Mar 20, 2026

Uh oh!

Uh oh!

henry3260 commented Apr 1, 2026

Uh oh!

dabla commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

henry3260 commented Apr 2, 2026

Uh oh!

henry3260 commented Apr 15, 2026

Uh oh!

henry3260 commented May 2, 2026

Uh oh!

Uh oh!

vincbeck left a comment

Choose a reason for hiding this comment

Uh oh!

henry3260 commented May 4, 2026

Uh oh!

vincbeck commented May 4, 2026

Uh oh!

vincbeck commented May 4, 2026

Uh oh!

henry3260 commented May 24, 2026

Uh oh!

Uh oh!

dabla commented May 24, 2026

Uh oh!

henry3260 commented May 25, 2026

Uh oh!

Uh oh!

simonp92 commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

morelgeorge commented Mar 18, 2026 •

edited

Loading