Skip to content

Flaky OTEL connection in integration tests #62769

Closed
Closed
Flaky OTEL connection in integration tests#62769
Labels
area:dev-envCI, pre-commit, pylint and other changes that do not change the behavior of the final code

Description

@jason810496
jason810496
opened on Mar 3, 2026

Body

What

The test sometimes fails.
For example: https://github.com/apache/airflow/actions/runs/22609497337/job/65509753048?pr=62758

Traceback Logs

summary:

======================================================================================== short test summary info =========================================================================================
FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes - AssertionError: Span name 'task2' wasn't found in children span names. It's not a child of span 'otel_test_dag_with_pause_between_tasks'.
FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_exits_gracefully_in_the_middle_of_the_first_task - AssertionError: Span name 'task1_sub_span3' wasn't found in children span names. It's not a child of span 'task1_sub_span2'.
===================================================================== 2 failed, 7 passed, 10 skipped, 1 warning in 580.16s (0:09:40) =====================================================================
full traceback: 
Error:    ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
    self._exporter.export(
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
    resp = self._export(serialized_data, deadline_sec - time())
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Warning:  ] Invalid type UUID for attribute 'airflow.task.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types [opentelemetry.attributes] loc=__init__.py:111
Error:    ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
    self._exporter.export(
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
    resp = self._export(serialized_data, deadline_sec - time())
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Error:    ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
    self._exporter.export(
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
    resp = self._export(serialized_data, deadline_sec - time())
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Error:    ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
    sock = connection.create_connection(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
    self.endheaders()
  File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
    self.sock = self._new_conn()
  File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
    self._exporter.export(
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
    resp = self._export(serialized_data, deadline_sec - time())
  File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
    resp = self._session.post(
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))

-- err-end
ERROR    integration.otel.otel_utils:otel_utils.py:146 Failed to parse JSON: {
    "resource_metrics": [
        {
            "resource": {
                "attributes": {
                    "telemetry.sdk.language": "python",
                    "telemetry.sdk.name": "opentelemetry",
                    "telemetry.sdk.version": "1.39.1",
                    "service.name": "airflow"
                },
 /root/airflow/logs/dag_id=otel_test_dag_with_pause_in_task/run_id=manual__2026-03-03T05:53:30.782643+00:00/task_id=task1/attempt=1.log
  File "/usr/python/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/python/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 1154, in format
    ed = cast(EventDict, proc(logger, meth_name, ed))
  File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 764, in __call__
    if len(args) == 1 and isinstance(args[0], dict) and args[0]:
  File "/usr/python/bin/celery", line 10, in <module>
    sys.exit(main())
  File "/usr/python/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/usr/python/lib/python3.10/site-packages/celery/bin/celery.py", line 227, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
  File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/python/lib/python3.10/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/celery/bin/worker.py", line 367, in worker
    worker.start()
  File "/usr/python/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/python/lib/python3.10/site-packages/celery/concurrency/base.py", line 130, in start
    self.on_start()
  File "/usr/python/lib/python3.10/site-packages/celery/concurrency/prefork.py", line 112, in on_start
    P = self._pool = Pool(processes=self.limit,
  File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 475, in __init__
    super().__init__(processes, *args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1046, in __init__
    self._create_worker_process(i)
  File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 493, in _create_worker_process
    return super()._create_worker_process(i)
  File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1158, in _create_worker_process
    w.start()
  File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 120, in start
    self._popen = self._Popen(self)
  File "/usr/python/lib/python3.10/site-packages/billiard/context.py", line 331, in _Popen
    return Popen(process_obj)
  File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 22, in __init__
    self._launch(process_obj)
  File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 77, in _launch
    code = process_obj._bootstrap()
  File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 323, in _bootstrap
    self.run()
  File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 110, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 292, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 362, in workloop
    result = (True, prepare_result(fun(*args, **kwargs)))
  File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 694, in fast_trace_task
    R, I, T, Rstr = tasks[task].__trace__(
  File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 479, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 779, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/airflow/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
    supervise(
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 2090, in supervise
    process = ActivitySubprocess.start(
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 1004, in start
    proc: Self = super().start(id=what.id, client=client, target=target, logger=logger, **kwargs)
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 514, in start
    _fork_main(child_requests, child_stdout, child_stderr, child_logs.fileno(), target)
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 413, in _fork_main
    target()
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main
    main()
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1825, in main
    state, _, error = run(ti, context, log)
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1236, in run
    result = _execute_task(context=context, ti=ti, log=log)
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1651, in _execute_task
    result = ctx.run(execute, context=context)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/decorator.py", line 360, in execute
    return_value = super().execute(context)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 232, in execute
    return_value = self.execute_callable()
  File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 255, in execute_callable
    return runner.run(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/callback_runner.py", line 97, in run
    return func(*args, **kwargs)
  File "/opt/airflow/airflow-core/tests/integration/otel/dags/otel_test_dag_with_pause_in_task.py", line 76, in task1
    logger.info("Found ti.context_carrier: %s.", context_carrier)
{
    "name": "task1_sub_span3",
    "context": {
        "trace_id": "0x186713f4ffcd23351586d08d9fd5ed34",
        "span_id": "0x57ffb0578ea05d12",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x70dad54d80c553f6",
    "start_time": "2026-03-03T05:53:42.271939Z",
    "end_time": "2026-03-03T05:53:42.272071Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "attr3": "val3"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.39.1",
            "service.name": "airflow"
        },
        "schema_url": ""
    }
}

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dev-envCI, pre-commit, pylint and other changes that do not change the behavior of the final code

    Type

    No type

    Projects

    CI / DEV ENV planned work
    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions