TaskFlow unexpectedly unwinds 'dict'

[bolkedebruin]

Apache Airflow version

main (development)

What happened

if using the TaskFlow API and one does the following

@task()
def MyFunc(myarg: int) -> dict:
  return dict({x: "y", z: "x"})

Airflow automatically unwinds dict automatically into separate variables x and z and if you do not want that kind of bahavior you will need to set multiple_outputs=False. While this is in the documentation, it is rather non intuitive requiring someone to be aware of the documentation that the default of multiple_outputs changes depending the return type of your function. Moreover, it is unexpected behavior for people coming from plain python (e.g. someone converting their functions to Airflow tasks) and wanting to do:

@task()
def f1(myarg: int) -> dict:
  return dict({x: myarg, z: "x"})

@task
def f2(in: dict):
  print(in)

a = f1(10)
f2(a)

Which would suddenly have issues in unittesting.

cc @josh-fell @BasPH @uranusjr

What you think should happen instead

Airflow should only unwind dicts if explicitly asked to do so.

How to reproduce

No response

Operating System

Not relevant

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

This is multiple output inference: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#multiple-outputs-inference

Personally I’ve always found this feature confusing. You could argue it’s not a good design and I would not object, but it is not a bug.

[bolkedebruin]

Agreed it is not a bug, but an awkward design choice. It is indeed confusing as @BasPH also encountered it and didnt know what was causing it.

[uranusjr]

So it seems this is introduced in 2.0.0 via #8996, so I’m going to cc @turbaszek for some context. Type annotations have come a long way since 2020 so maybe the feature does not blend well with modern best practice anymore?

[bolkedebruin]

I don't think it was introduced there (i.e. the automatic inference)? multiple_outputs in itself is fine of course, the change default based on return type is the part that is confusing.

[uranusjr]

It was added there, and you can see the documentation addition in the pull request.