'dict' object has no attribute 'dump'

[11010cy]

Apache Airflow version

2.2.2

What happened

get 500 while request 'api/v1/dags/{dag_id}/tasks'. It happens randomly.

AttributeError: 'dict' object has no attribute 'dump'
  File "flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 39, in reraise
    raise value
  File "flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "airflow/_vendor/connexion/decorators/decorator.py", line 48, in wrapper
    response = function(request)
  File "airflow/_vendor/connexion/decorators/uri_parsing.py", line 144, in wrapper
    response = function(request)
  File "airflow/_vendor/connexion/decorators/validation.py", line 384, in wrapper
    return function(request)
  File "airflow/_vendor/connexion/decorators/response.py", line 103, in wrapper
    response = function(request)
  File "airflow/_vendor/connexion/decorators/parameter.py", line 121, in wrapper
    return function(**kwargs)
  File "airflow/api_connexion/security.py", line 47, in decorated
    return func(*args, **kwargs)
  File "airflow/api_connexion/endpoints/task_endpoint.py", line 71, in get_tasks
    return task_collection_schema.dump(task_collection)
  File "marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "marshmallow/fields.py", line 765, in _serialize
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "marshmallow/fields.py", line 765, in <listcomp>
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "marshmallow/fields.py", line 634, in _serialize
    return schema.dump(nested_obj, many=many)
  File "marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "marshmallow/fields.py", line 1870, in _serialize
    return self._serialize_method(obj)
  File "airflow/api_connexion/schemas/task_schema.py", line 70, in get_params
    return {k: v.dump() for k, v in params.items()}
  File "airflow/api_connexion/schemas/task_schema.py", line 70, in <dictcomp>
    return {k: v.dump() for k, v in params.items()}

After some inspection, I have located the cause of task.params.

In some cases, current_app.dag_bag.get_dag(dag_id) will return a dag with one of task.params that is type dict instead of type ParamsDict. It happens in the same gunicorn worker process 15994, while the current_app.dag_bag.dags supposed to be the same instance. Before the error happened, there was nothing wrong. But 2 minutes later, the error happend.

I have print the pid and the id of task.params before and when the above error happend.

BEFORE

[params] check_updated sd_last_updated_datetime=2022-03-30 04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at 0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at 0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at 0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at 0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at 0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at 0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at 0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at 0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at 0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at 0x7f4761704850>}

[params] get_tasks 139949716343280 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d970>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d3a0>, 'business': <airflow.models.param.Param object at 0x7f476175dac0>}

[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at 0x7f476174cd60>}
127.0.0.1 - airflow [30/Mar/2022:14:04:38 +0800] "GET /api/v1/dags/xxx/tasks HTTP/1.0" 200 64063 "-" "OpenAPI-Generator/2.2.0/python"

WHEN

[params] check_updated sd_last_updated_datetime=2022-03-30 04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at 0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at 0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at 0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at 0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at 0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at 0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at 0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at 0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at 0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji': <airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at 0x7f4761704850>}


[params] get_tasks 139944557266624 <class 'dict'> {xxx}

[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'> {'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji': <airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at 0x7f476174cd60>}

[^[[34m2022-03-30 14:06:30,706^[[0m] {^[[34mapp.py:^[[0m1891} ERROR^[[0m - Exception on /api/v1/dags/troubleshooter_15/tasks [GET]^[[0m

We could see that task 139949716343280 with type ParamsDict changed to 139944557266624 with type dict, while the other task object hasn't changed.

Does anyone knows: when and where could the task.params in a dag of current_app.dag_bag.dags be changed from ParamsDict to dict

What you think should happen instead

There's maybe a bug related to current_app.dag_bag.dags.

How to reproduce

request log_url to make current_app.dag_bag.dags.get_dag(dag_id) task.params become dict，then request task_url to get tasks, then we will get 500 error.

log_url = "xxx/dags/{}/dagRuns/{}/taskInstances/{}/logs/1"
task_url = "xxx/dags/{}/dagRuns/{}/taskInstances?limit=30&offset=0"

# request log_url for 8 times
import requests, threading

def req_url(url):
    req = requests.get(url)
    print(f"[req] status_code={req.status_code}")

for i in range(8):
    th = threading.Thread(target=req_url, args=(log_url, ))
    th.start()
    th.join()

for i in range(5):
    th = threading.Thread(target=req_url, args=(task_url, ))
    th.start()
    th.join()

Operating System

Linux Server 7.2 (Paladin)

Versions of Apache Airflow Providers

apache-airflow-providers-alibaba==1.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-sqlite==2.0.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[11010cy]

I found that it caused by requesting /dags/{}/dagRuns/{dag_run_id}/taskInstances/{task_instance_id}/logs/1.

// utils/log/file_task_handler.py _render_filename
context = ti.get_template_context()

but get_template_context has the side effect that it changed task.params to dict. (params.validate() return a dict)

//  models/taskintance.py get_template_context
validated_params = task.params = params.validate()

The related commits are:
2258e13
23a68fa

The problem was introduced by 23a68fa, fixed at 2258e13 (not released). And exists in apache-airflow >= 2.1.4.

[potiuk]

So seems that one is fixed in #21495. Closing it then. Will be released as part of 2.3.0 I believe.