Apache Airflow version
main (development)
What happened
>>> from airflow.utils.json import XComEncoder
>>> XComEncoder().encode(set())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/airflow/airflow/utils/json.py", line 91, in encode
return super().encode(o)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/opt/airflow/airflow/utils/json.py", line 82, in default
return serialize(o)
File "/opt/airflow/airflow/serialization/serde.py", line 97, in serialize
if isinstance(o, _primitives):
RecursionError: maximum recursion depth exceeded in __instancecheck__What you think should happen instead
The encoder should correctly encode a value, or raise TypeError if it is not encodable.
How to reproduce
Simply use XComEncoder to encode anything that’s not encodable by the built-in JSONEncoder (i.e. something that’d go through the custom default hook).
Operating System
Any
Versions of Apache Airflow Providers
Irrelevant
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
The problem here is because serialize does not guarantee to return a JSON-serializable object. So when default receives an set, it returns a set, and the encoder would try to serialize that set again, calling default, and eventually the stack explodes.
The simplest solution would be to add an additional flag to serialize so it can return a JSON-compatible type if requested.
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
main (development)
What happened
What you think should happen instead
The encoder should correctly encode a value, or raise TypeError if it is not encodable.
How to reproduce
Simply use
XComEncoderto encode anything that’s not encodable by the built-in JSONEncoder (i.e. something that’d go through the customdefaulthook).Operating System
Any
Versions of Apache Airflow Providers
Irrelevant
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
The problem here is because
serializedoes not guarantee to return a JSON-serializable object. So whendefaultreceives an set, it returns a set, and the encoder would try to serialize that set again, calling default, and eventually the stack explodes.The simplest solution would be to add an additional flag to
serializeso it can return a JSON-compatible type if requested.Are you willing to submit PR?
Code of Conduct