Skip to content

Fix socket/fd leak in streaming endpoints (#2766)#3412

Open
ykstorm wants to merge 2 commits into
docker:mainfrom
ykstorm:fix-streaming-socket-leak-2766
Open

Fix socket/fd leak in streaming endpoints (#2766)#3412
ykstorm wants to merge 2 commits into
docker:mainfrom
ykstorm:fix-streaming-socket-leak-2766

Conversation

@ykstorm

@ykstorm ykstorm commented Jun 23, 2026

Copy link
Copy Markdown

Closes #2766.

What's wrong

container.logs(stream=True), events(), attach(), stats() and the raw
exec streams all return a generator that reads from a socket to the daemon. If
you stop reading before the stream ends - you break, an exception is raised,
or the generator just goes out of scope - the response never gets closed, so the
socket and its fd stay open. _get_raw_response_socket() also stores a reference
to the response on the socket, so the garbage collector can't always clean it up
on its own. _read_from_socket(stream=True) even said so in a comment: "the
caller is responsible for closing the response."

If you have a long-running process that tails logs and bails out early, those
fds add up until you run out. That's the "ResourceWarning: unclosed
<socket.socket>" people kept hitting on the issue.

The change

Wrap the read loop in each streaming generator in try/finally: response.close(). When you stop iterating, Python throws GeneratorExit into
the generator (on .close() and on garbage collection), so finally is where
the socket gets released. Socket setup still happens on the first read like
before, so the calls behave exactly the same otherwise.

for line in container.logs(stream=True):
    if got_what_i_need(line):
        break          # socket/fd gets closed now

Only three helpers in docker/api/client.py changed: _stream_raw_result,
_multiplexed_response_stream_helper, and _read_from_socket.

Proof it leaked

benchmarks/stream_leak.py reproduces it without a daemon. It serves an endless
chunked response locally, opens 200 streams, reads one chunk from each, stops,
and counts how many sockets are still open (and double-checks with psutil):

opening 200 streams, reading one chunk, then stopping each early

impl       streams  sockets leaked   ESTABLISHED conns
------------------------------------------------------
old            200             200                 200
fixed          200               0                   0

Tests and docs

  • tests/unit/stream_leak_test.py covers early break, an exception in the
    consumer, and an explicit .close(). The tests use fakes, so they don't lean
    on any requests/urllib3 internals.
  • ruff is clean and the unit tests pass locally. I don't have a daemon set up
    here, so the integration tests are up to CI.
  • Added a short "Streaming endpoints" docs page and a changelog entry.

No new dependencies, and nothing changes for existing callers except that
streams now get closed when you stop reading them.

@ykstorm ykstorm changed the title Fix socket/fd leak in streaming endpoints (#2766) + opt-in stream collector Fix socket/fd leak in streaming endpoints (#2766) Jun 24, 2026
@ykstorm

ykstorm commented Jun 24, 2026

Copy link
Copy Markdown
Author

Pushed an update: added benchmarks/stream_leak.py, a daemon-free reproduction of the leak (opens 200 streams, reads one chunk, stops early — old code leaks all 200 sockets, the fix closes every one), and referenced it from the streaming docs. Also reworded the PR description. No code changes to the fix itself.

@ykstorm ykstorm force-pushed the fix-streaming-socket-leak-2766 branch from 7533234 to 3e5cfca Compare June 24, 2026 01:22
@ykstorm

ykstorm commented Jun 24, 2026

Copy link
Copy Markdown
Author

Scoped this PR down to just the leak fix. I removed the optional stream-collector that was bundled in earlier — it added a background thread and a new public API that turn a surgical fix into a design discussion, and the try/finally change already closes the leak for normal consumers. The PR is now: the fix in docker/api/client.py, unit tests, a daemon-free benchmark, and docs. Happy to discuss the collector separately if there's interest, but it doesn't belong in this change.

ykstorm added 2 commits June 25, 2026 23:17
The streaming helpers (logs, events, attach, stats and raw exec output)
hand back a generator that reads from the daemon socket. If you stop
reading before the end - break out of the loop, hit an exception, or just
drop the generator - the response was never closed, so the socket and its
fd leaked. _get_raw_response_socket also keeps a reference to the response
on the socket, so the garbage collector doesn't reliably clean it up.

Wrap the read loops in try/finally so the response is closed when the
generator is closed or garbage collected. Socket setup still happens on
the first read, so nothing else about these calls changes.

Closes docker#2766

Signed-off-by: lakshyaraj <raolakshyaraj@gmail.com>
Document that streams now close themselves when you stop reading, and add
benchmarks/stream_leak.py - a small reproduction that needs no daemon: it
opens a few hundred streams, reads one line from each, walks away, and
counts the sockets left open. Before the fix every one leaks; after it,
none do.

Signed-off-by: lakshyaraj <raolakshyaraj@gmail.com>
@ykstorm ykstorm force-pushed the fix-streaming-socket-leak-2766 branch from 3e5cfca to c2fe936 Compare June 25, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Leaking file descriptors

1 participant