server + ui: SSE Replay Buffer by ServeurpersoCom · Pull Request #23226 · ggml-org/llama.cpp

ServeurpersoCom · 2026-05-17T18:28:38Z

Overview

Resumable SSE streaming, keyed on conversation id

Today the generation stops the moment the HTTP socket drops. F5, tab close, iOS Safari going to background, transient network: the generation aborts, the WebUI loses the live stream.

This PR keeps the generation running server side as a background task and lets the client reattach to the live SSE stream. Opt in via X-Conversation-Id on POST /v1/chat/completions. The OAI strict path is unchanged when that header is absent.

Lifecycle

The producer keeps running server side after the client is gone, SSE bytes land in a bounded ring buffer. Clients drain via:

GET /v1/stream/<conv_id>?from=N to replay from offset N
GET /v1/streams[?conversation_id=X] to list live sessions
DELETE /v1/stream/<conv_id> for Stop, idempotent

The conv id is the only identity end to end: server map, localStorage key, URL path. No extra opaque token.

Router mode

The parent enforces one session per conv across all children: fans out DELETE on POST to evict any prior owner, probes children on GET to route the replay, aggregates on list, fans out on DELETE.

WebUI

The layout snapshots /v1/streams at mount and on visibilitychange, so the sidebar reflects every live inference across convs. The chat page reattaches on mount, detects append vs fresh generation from existing message content so a mid stream continue keeps its prefix.

Additional information

SSE-Replay-Buffer.mp4

Closes #23136

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

coder543 · 2026-05-17T18:35:28Z

Opt in via X-Stream-Resume: 1 and X-Conversation-Id

Is there any reason to require both? What happens if the client only passes the first one? If the second one is always required, then maybe it should be the only header required for this?

Just a thought, but this looks good!

ServeurpersoCom · 2026-05-17T18:58:05Z

That's a good idea. Because the "X-Stream-Resume: 1" option was intended to allow the system to be completely disabled from the web interface (via developer options) while maintaining its current functionality in case of a bug.
However, it is preferable for Llama's user interface to function this way permanently; the PR is still in draft form, and many edge cases remain to be tested.

ngxson

IMO this feature should be confined inside server-http component instead of spreading across 3 different parts: http/context/models. In other words, I feel like the current direction won't be easy to maintain in the long term.

Instead, I propose:

Implement a stream callback hook in server-http.cpp, something like set_stream_hook(...) and whenever a SSE chunk is sent in the http layer, the callback hook will be invoked
Implement the "stream tracking" system as a dedicated component, ideally no changes should be made to server-context
/v1/stream routes will be registered in server.cpp, directly interface with http component (bypassing server-context)

ServeurpersoCom · 2026-05-18T08:28:50Z

Rebased and refactoring+testing in progress

ServeurpersoCom · 2026-05-18T14:40:14Z

Rebase

ServeurpersoCom · 2026-05-18T14:40:33Z

Rerebase

coder543 · 2026-05-18T20:10:26Z

+    // stop_producer) to server-stream. when the header is absent this is a no op, when set
+    // the session is created or replaced and the response carries the tee that mirrors every
+    // SSE chunk into the ring buffer, plus the cancel hook for DELETE /v1/stream/<conv_id>
+    stream_session_attach_hooks(*res, res->rd, req.headers);


If I'm understanding correctly, then I believe this happens only after prompt processing has begun/completed, based on the rd.next() call on line 3519. If the connection fails before that finishes, then the stream may not initialize at all, leaving the client unable to reattach. Fixing it so we can attach the stream session before that first blocking call without breaking the contract of not sending a 200 until things are known to be working might require some refactoring.

But, maybe this is better left as a future improvement. Just pointing out something I noticed.

Good catch. Real edge case but the fix touches server-context and server-http again, I'd rather keep this PR scoped and tackle it in a follow-up.

ServeurpersoCom · 2026-05-18T21:05:43Z

A few stress tests at this point. (White flash -> I press F5)

Tests.mp4

ServeurpersoCom · 2026-05-18T21:14:52Z

Thanks a lot for the reviews, this kind of feedback is invaluable for getting the architecture right.

coder543 · 2026-05-18T23:03:07Z

GET /v1/streams[?conversation_id=X] to list live sessions

I feel like this could be a significant privacy issue. Part of the idea of having a random UUID as the conversation ID was that it is literally impossible for anyone to guess which UUIDs exist unless the client is using broken RNG or intentionally selecting obvious ones.

With the ability to list sessions, anyone could attach to any session.

Other than that issue, I think this PR seems good to me. Very big improvement!

ServeurpersoCom · 2026-05-19T00:21:22Z

Excellent catch, I ended up exposing the list of identifiers just to drive the spinning loading indicators on the left.

I could add a client-side random UUID stored in localStorage, sent as X-Client-Id on every stream route, and scope listings and lookups to that id server-side.

Or even simpler (secure by design): drop the listing route altogether and replace it with a POST that takes the conv_ids the client already owns and returns their status. No client_id scoping needed, and the body stays small since the client only sends the conv_ids it actually displays in the sidebar.

Going with the second approach unless you see a reason to prefer the first.

ServeurpersoCom · 2026-05-19T02:13:31Z

Open question: the buffer constants in server-stream.cpp (TTL = 300 s, MAX_BYTES = 4 MiB, GC interval = 60 s) are hardcoded. TTL is the one most likely to deserve a CLI flag for workflows that want longer reattach windows. Expose via CLI now, or keep hardcoded for the initial landing?

@ngxson

Address @ngxson review: one class covering both ends was messy. stream_pipe is now a base holding the session and is_cancelled, with stream_pipe_producer (write, mark_producer_done, finish_producer, cleanup, finalizes on destruct) and stream_pipe_consumer (read only, no finalize) deriving from it. Drops the is_producer_ discriminator and its runtime guards, the type now encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer> since it is only ever a producer. No behavior change.

@ngxson

Address @ngxson review: mark_producer_done becomes done(), finish_producer becomes close(), matching a unix pipe write end. The producer_done_ member follows as done_. write() is unchanged. No behavior change.

…entity Address ngxson review: drop the polling probe, proxy_post records a conv_id -> model map and the stream routes resolve the owning child with one lookup. The map is the single source of truth, the ::model suffix stays for child session uniqueness but the router never parses it. UI: the server keys a session by the POST time identity (conv::model), but reload probed with the bare conv id and missed model tagged sessions, so F5 stopped the stream and sidebar spinners stayed off. Persist the model and rebuild the exact identity on resume, single conv and bulk sidebar both send it. Add unit coverage for the identity round trip.

… switch

Address review from ngxson: server_models held two mutexes side by side, the global one and a bare conv_model_mu guarding a loose map, which made the locking hard to follow. Wrap the map and its lock in a small conv_model_tracker struct that owns its mutex, one mutex per struct. The remember, lookup and forget methods move inline into the tracker, server_models exposes a single conv_models member and the routes call models.conv_models.lookup and friends. No behavior change, the map stays the single source of truth for routing resumable streams to a child.

Address review from allozaur: lift the inline literals around the resumable stream code into named symbols so the intent is explicit and reusable.

Address review from allozaur: drop the two standalone stream-*.service files. They were used only by the chat service and store, carried no shared state, and did not follow the static class pattern the other services use, so a separate abstraction was not warranted. Move the helpers onto ChatService as static methods. No behavior change, tests now exercise them through ChatService.

Add the resumable streaming section, list stream_session_manager in the backend component inventory, and link PR 23226 in the related PRs.

…StreamResponse

allozaur · 2026-06-25T12:48:10Z

+			if (document.visibilityState !== 'visible') return;
+			void chatStore.syncRemoteRunningStreams();
+		};
+		document.addEventListener('visibilitychange', onVisibility);


let's use <svelte:document>¹

Footnotes

https://svelte.dev/docs/svelte/svelte-document ↩

address review from allozaur: replace the manual document.addEventListener in onMount with a declarative <svelte:document onvisibilitychange>. svelte handles attach, detach and SSR, so the typeof document guard and the onMount cleanup go away. onMount keeps only the first load snapshot.

Address review from ngxson

remove redundant comments and tighten the verbose ones across the resumable stream code, keeping the concurrency and lifetime rationale that is not obvious from the code. also fix two stale comments in server.cpp and server-models.h that still described the old ::model suffix probe and fan out routing, now replaced by the conv_id -> model map Address review from ngxson

dedup repeated rationale (frozen conv::model identity, the lookup privacy note, the abort patterns) down to one canonical spot, tighten the verbose blocks, and keep the concurrency and resume-offset reasoning. fix stale comments in stream-identity.ts and chat.service.ts that still described the old loopback probe and fan out routing, now the conv_id -> model map.

ggerganov · 2026-06-26T08:37:38Z

+#pragma once
+
+#include "server-http.h"
+
+#include <atomic>
+#include <condition_variable>
+#include <cstddef>
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <string>
+#include <thread>
+#include <unordered_map>
+#include <vector>
+
+enum class stream_read_status {
+    OK,
+    OFFSET_LOST,
+};


There is a lot of implementation stuff exposed in this header. Consider moving it to the implementation. Some directions: 5004859

cherry-picked: I'm going to open a follow-up PR based on your reviews

Thanks, moved the implementation into the .cpp on top of your 5004859.

ggerganov · 2026-06-26T08:38:27Z

+// process wide manager, linked by both llama-server and llama-cli. llama-server main() drives
+// start_gc/stop_gc, llama-cli leaves it idle. the dtor calls stop_gc() unconditionally so exit
+// is safe whether or not the GC thread ran
+extern stream_session_manager g_stream_sessions;


When we have to, it's better to use the singleton pattern instead of extern. Although in this case it's better to hide the entire session menager implementation in the .cpp file.

Hid the whole session manager in the .cpp, it's a file-static singleton now, no extern.

ggerganov · 2026-06-26T08:39:28Z

+// route handler factories operating on g_stream_sessions, wired under /v1/stream/* by server.cpp.
+// keeps the resumable stream surface confined to server-stream
+server_http_context::handler_t make_stream_get_handler();
+server_http_context::handler_t make_streams_lookup_handler();
+server_http_context::handler_t make_stream_delete_handler();
+
+// extract the X-Conversation-Id header value (case-insensitive), empty when absent. exposed so
+// the router can track which child serves a forwarded POST
+std::string stream_conv_id_from_headers(const std::map<std::string, std::string> & headers);
+
+// on an X-Conversation-Id header, create or replace the session and attach a producer pipe to
+// res. no-op when absent, called from the server_res_generator constructor
+void stream_session_attach_pipe(server_http_res & res, const std::map<std::string, std::string> & headers);
+
+// should_stop closure that ignores peer disconnect when a pipe is attached, so only an explicit
+// DELETE stops the producer and generation keeps flowing into the ring buffer. without a pipe it
+// delegates to fallback, the legacy non-resumable flow
+std::function<bool()> stream_aware_should_stop(server_http_res * res, std::function<bool()> fallback);


Generally, we should prefix names with server_stream_ for better scoping.

Prefixed all the public stream functions with server_stream_.

ggerganov · 2026-06-26T08:40:36Z

+    mutable std::mutex      mu;
+    std::condition_variable cv;
+    std::vector<char>       buffer;
+    size_t                  prefix_dropped;
+    size_t                  cap_bytes;
+    std::atomic<bool>       done;
+    std::atomic<bool>       cancelled;
+    std::atomic<int64_t>    completed_ts;
+    std::function<void()>   stop_producer; // protected by mu


Haven't looked at the implementation, but mixing atomic with mutex usually should not be necessary.

Made done, completed_ts and the GC running flag plain members under their mutex, kept only cancelled atomic for the lock-free should_stop poll.

ggerganov · 2026-06-26T08:41:59Z

+    // start the stream session manager GC right after common init, before any HTTP route can
+    // touch it. lifecycle is symmetric, stop_gc() runs in clean_up() before backend free
+    g_stream_sessions.start_gc();


Generally, less comments is better. The logic should be easy to understand by reading the code.

Trimmed the comments back to just the concurrency, lifetime and ordering rationale.

@ngxson

* server: SSE replay buffer, survives client disconnect Opt in on POST /v1/chat/completions when the client sends X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is the session identity end to end, no extra opaque token. The drain runs detached server side and buffers SSE bytes, the generation survives HTTP disconnect, F5, or lets users switch from iOS Safari to another app without losing the actively generated response. Routes: GET /v1/stream/<conv_id>?from=N replay GET /v1/streams[?conversation_id=X] list, drives sidebar spinners DELETE /v1/stream/<conv_id> Stop, idempotent Router parent fans out to children for list and delete, probes on GET to route to the owner, fans out DELETE on POST so "one session per conv" holds across model swaps. WebUI: the layout snapshots /v1/streams at mount and on visibilitychange, the sidebar reflects live inferences across all convs. The chat page reattaches on mount, append vs fresh is detected from existing content so continue mid stream keeps its prefix. update_slots: on llama_memory_seq_rm refusal at a deep position, full clear of the seq and reprefill from zero instead of GGML_ABORT. OAI strict path unchanged when the opt in headers are absent. * server: create stream session only after post_tasks succeeds * server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer * server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time * refactor: address review feedback from ngxson * server-context: cleaning * server-stream: fix use-after-free on rd Guard stop_producer with a shared alive flag, flipped by on_stream_end before rd dies. Prevents a late cancel (session eviction by a later POST on the same conv_id, or a DELETE arriving after the producer ended) from touching a destroyed rd. * ui: fix cross-conversation contamination Scope streaming flags per conv so one finishing does not unflag the others, guard discoverActiveStream against concurrent runs to avoid duplicate attaches, and stop racing syncRemoteRunningStreams for the sidebar set. * server-http: keep request alive in detached SSE drain The response next() lambda may reach into *request via &req long after on_complete reset the request shared_ptr. Capture request in the detached thread so it outlives the drain. * ui: address review feedback from coder543 Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes must obey --api-key like the rest of the API. Wrap reader.read() in a try/catch, the underlying connection drop rejects with TypeError instead of resolving done=true, treat it as a premature end of stream so the existing resume loop kicks in. Freeze the model at session start in chatStreamingStates.model and thread it through cancel and resume, the dropdown selection may have changed since the POST and the server side identity is fixed at that time. * format * ui: remove unused selectedModelName * server-stream: poll session->is_cancelled() in stream_aware_should_stop Address review feedback from coder543. The cancel propagation through rd.stop() relies on the slot eventually processing the cancel task and posting a result that notifies the recv condvar, remove_waiting_task_ids does not notify directly. Add a defensive poll on session->is_cancelled() so the producer-side next() loop exits on its next iteration after cancel() without waiting for the cancel task to round trip through a slot. * server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup Address review feedback from coder543. Listing live sessions leaks the conversation_id of every concurrent user, which defeats the random UUID unguessability. The new route takes {conversation_ids: [...]} in the body and returns matches only for the ids the caller already owns, so foreign UUIDs stay private. The router fans out the same POST to every child and aggregates, the WebUI passes the convs visible in its sidebar. * ui: read conv ids from IndexedDB in syncRemoteRunningStreams The conversations store is not hydrated yet at +layout onMount, so the sidebar spinners stayed off for background convs until the user clicked on them. Read straight from the DB to dodge the init race. * server-models: deduplicate stream lookup timeouts behind one constant * ui: extract visibility kick grace into a stream constant, bump to 1000 ms * make it safer & more simple * server-stream: survive client disconnect via stream_pipe::finish_producer After the RAII rewrite the generation stopped the moment the client disconnected. httplib bails its content provider on the is_peer_alive check at the top of write_content_chunked, so returning true from the provider never keeps it producing: the response resets, rd is destroyed and its task gets cancelled. Reinstate the disconnect survival inside the pipe. stream_pipe gains finish_producer, which pumps the response next() into the ring buffer until the generation ends, and mark_producer_done for the clean wire end. server-http only triggers them: mark before sink.done on a clean close, finish in on_complete when the peer left early. No detach, no stream logic in server-http beyond the trigger, and the strict OAI path is untouched when no pipe is attached. Known limitation: finish_producer pumps synchronously on the http worker, so a disconnected stream keeps its worker busy until the generation ends. A follow-up will move the drain off the http worker so no worker is held. * server-stream: drain disconnected streams on a manager owned thread The previous commit pumped the post disconnect drain synchronously in on_complete, on the http worker, so a disconnected stream kept its worker busy until the generation ended. Under a wave of reloads or tab closes that pins workers from the pool. Move the drain off the http worker. on_complete now hands the response to stream_session_manager::adopt_orphan, which pumps it to completion on a manager owned thread and releases the worker at once. One thread per disconnected stream still generating, stored in a list, joined and reaped on the next adopt, by the GC, and at shutdown. No detach, the thread lifecycle is fully owned by the manager. needs_drain gates the handoff so a cleanly finished stream never spawns a thread, and the strict OAI path stays untouched when no pipe is attached. stop_gc now cancels sessions before finalizing them, so an in flight drain sees is_cancelled and exits instead of blocking the shutdown join until the generation ends naturally. * ui: add missing JSDoc * server-stream: drain on the http worker, drop the manager thread Address @ngxson review: httplib runs a large dynamic pool and a worker blocked in next() sits on a condvar instead of burning cpu, so draining the rest of the generation on that worker is fine and much simpler than a dedicated thread. on_complete calls finish_producer directly again. Removes adopt_orphan, the orphan thread list and its reaping, the stop_gc session cancel that only existed to unblock those threads, and the now dead drain_shutdown flag. * server-stream: split stream_pipe into producer and consumer classes Address @ngxson review: one class covering both ends was messy. stream_pipe is now a base holding the session and is_cancelled, with stream_pipe_producer (write, mark_producer_done, finish_producer, cleanup, finalizes on destruct) and stream_pipe_consumer (read only, no finalize) deriving from it. Drops the is_producer_ discriminator and its runtime guards, the type now encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer> since it is only ever a producer. No behavior change. * server-stream: rename producer methods to unix pipe semantics Address @ngxson review: mark_producer_done becomes done(), finish_producer becomes close(), matching a unix pipe write end. The producer_done_ member follows as done_. write() is unchanged. No behavior change. * server, ui: route resumable streams via a conv map, persist resume identity Address ngxson review: drop the polling probe, proxy_post records a conv_id -> model map and the stream routes resolve the owning child with one lookup. The map is the single source of truth, the ::model suffix stays for child session uniqueness but the router never parses it. UI: the server keys a session by the POST time identity (conv::model), but reload probed with the bare conv id and missed model tagged sessions, so F5 stopped the stream and sidebar spinners stayed off. Persist the model and rebuild the exact identity on resume, single conv and bulk sidebar both send it. Add unit coverage for the identity round trip. * ui: resolve continue target by id to stop cross-conversation flash on switch * ui: skip stream resume when the abort is intentional * server: move the conv id to model map into a self contained tracker Address review from ngxson: server_models held two mutexes side by side, the global one and a bare conv_model_mu guarding a loose map, which made the locking hard to follow. Wrap the map and its lock in a small conv_model_tracker struct that owns its mutex, one mutex per struct. The remember, lookup and forget methods move inline into the tracker, server_models exposes a single conv_models member and the routes call models.conv_models.lookup and friends. No behavior change, the map stays the single source of truth for routing resumable streams to a child. * ui: replace stream magic values with enums and shared constants Address review from allozaur: lift the inline literals around the resumable stream code into named symbols so the intent is explicit and reusable. * ui: fold the stream resume and discovery helpers into ChatService Address review from allozaur: drop the two standalone stream-*.service files. They were used only by the chat service and store, carried no shared state, and did not follow the static class pattern the other services use, so a separate abstraction was not warranted. Move the helpers onto ChatService as static methods. No behavior change, tests now exercise them through ChatService. * docs: document the SSE replay buffer in server README-dev Add the resumable streaming section, list stream_session_manager in the backend component inventory, and link PR 23226 in the related PRs. * ui: align attachServerStream call with onCompletionId param in handleStreamResponse * server-http: rename del_ to del to match get and post * ui: address review feedback from allozaur * ui: drop duplicate SSE constants, keep sse.ts canonical * ui: use svelte:document for the visibilitychange listener address review from allozaur: replace the manual document.addEventListener in onMount with a declarative <svelte:document onvisibilitychange>. svelte handles attach, detach and SSR, so the typeof document guard and the onMount cleanup go away. onMount keeps only the first load snapshot. * server: trim redundant stream drain comments Address review from ngxson * server: balance and clean up stream comments remove redundant comments and tighten the verbose ones across the resumable stream code, keeping the concurrency and lifetime rationale that is not obvious from the code. also fix two stale comments in server.cpp and server-models.h that still described the old ::model suffix probe and fan out routing, now replaced by the conv_id -> model map Address review from ngxson * ui: balance and clean up stream comments dedup repeated rationale (frozen conv::model identity, the lookup privacy note, the abort patterns) down to one canonical spot, tighten the verbose blocks, and keep the concurrency and resume-offset reasoning. fix stale comments in stream-identity.ts and chat.service.ts that still described the old loopback probe and fan out routing, now the conv_id -> model map. --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

@ngxson

* server: SSE replay buffer, survives client disconnect Opt in on POST /v1/chat/completions when the client sends X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is the session identity end to end, no extra opaque token. The drain runs detached server side and buffers SSE bytes, the generation survives HTTP disconnect, F5, or lets users switch from iOS Safari to another app without losing the actively generated response. Routes: GET /v1/stream/<conv_id>?from=N replay GET /v1/streams[?conversation_id=X] list, drives sidebar spinners DELETE /v1/stream/<conv_id> Stop, idempotent Router parent fans out to children for list and delete, probes on GET to route to the owner, fans out DELETE on POST so "one session per conv" holds across model swaps. WebUI: the layout snapshots /v1/streams at mount and on visibilitychange, the sidebar reflects live inferences across all convs. The chat page reattaches on mount, append vs fresh is detected from existing content so continue mid stream keeps its prefix. update_slots: on llama_memory_seq_rm refusal at a deep position, full clear of the seq and reprefill from zero instead of GGML_ABORT. OAI strict path unchanged when the opt in headers are absent. * server: create stream session only after post_tasks succeeds * server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer * server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time * refactor: address review feedback from ngxson * server-context: cleaning * server-stream: fix use-after-free on rd Guard stop_producer with a shared alive flag, flipped by on_stream_end before rd dies. Prevents a late cancel (session eviction by a later POST on the same conv_id, or a DELETE arriving after the producer ended) from touching a destroyed rd. * ui: fix cross-conversation contamination Scope streaming flags per conv so one finishing does not unflag the others, guard discoverActiveStream against concurrent runs to avoid duplicate attaches, and stop racing syncRemoteRunningStreams for the sidebar set. * server-http: keep request alive in detached SSE drain The response next() lambda may reach into *request via &req long after on_complete reset the request shared_ptr. Capture request in the detached thread so it outlives the drain. * ui: address review feedback from coder543 Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes must obey --api-key like the rest of the API. Wrap reader.read() in a try/catch, the underlying connection drop rejects with TypeError instead of resolving done=true, treat it as a premature end of stream so the existing resume loop kicks in. Freeze the model at session start in chatStreamingStates.model and thread it through cancel and resume, the dropdown selection may have changed since the POST and the server side identity is fixed at that time. * format * ui: remove unused selectedModelName * server-stream: poll session->is_cancelled() in stream_aware_should_stop Address review feedback from coder543. The cancel propagation through rd.stop() relies on the slot eventually processing the cancel task and posting a result that notifies the recv condvar, remove_waiting_task_ids does not notify directly. Add a defensive poll on session->is_cancelled() so the producer-side next() loop exits on its next iteration after cancel() without waiting for the cancel task to round trip through a slot. * server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup Address review feedback from coder543. Listing live sessions leaks the conversation_id of every concurrent user, which defeats the random UUID unguessability. The new route takes {conversation_ids: [...]} in the body and returns matches only for the ids the caller already owns, so foreign UUIDs stay private. The router fans out the same POST to every child and aggregates, the WebUI passes the convs visible in its sidebar. * ui: read conv ids from IndexedDB in syncRemoteRunningStreams The conversations store is not hydrated yet at +layout onMount, so the sidebar spinners stayed off for background convs until the user clicked on them. Read straight from the DB to dodge the init race. * server-models: deduplicate stream lookup timeouts behind one constant * ui: extract visibility kick grace into a stream constant, bump to 1000 ms * make it safer & more simple * server-stream: survive client disconnect via stream_pipe::finish_producer After the RAII rewrite the generation stopped the moment the client disconnected. httplib bails its content provider on the is_peer_alive check at the top of write_content_chunked, so returning true from the provider never keeps it producing: the response resets, rd is destroyed and its task gets cancelled. Reinstate the disconnect survival inside the pipe. stream_pipe gains finish_producer, which pumps the response next() into the ring buffer until the generation ends, and mark_producer_done for the clean wire end. server-http only triggers them: mark before sink.done on a clean close, finish in on_complete when the peer left early. No detach, no stream logic in server-http beyond the trigger, and the strict OAI path is untouched when no pipe is attached. Known limitation: finish_producer pumps synchronously on the http worker, so a disconnected stream keeps its worker busy until the generation ends. A follow-up will move the drain off the http worker so no worker is held. * server-stream: drain disconnected streams on a manager owned thread The previous commit pumped the post disconnect drain synchronously in on_complete, on the http worker, so a disconnected stream kept its worker busy until the generation ended. Under a wave of reloads or tab closes that pins workers from the pool. Move the drain off the http worker. on_complete now hands the response to stream_session_manager::adopt_orphan, which pumps it to completion on a manager owned thread and releases the worker at once. One thread per disconnected stream still generating, stored in a list, joined and reaped on the next adopt, by the GC, and at shutdown. No detach, the thread lifecycle is fully owned by the manager. needs_drain gates the handoff so a cleanly finished stream never spawns a thread, and the strict OAI path stays untouched when no pipe is attached. stop_gc now cancels sessions before finalizing them, so an in flight drain sees is_cancelled and exits instead of blocking the shutdown join until the generation ends naturally. * ui: add missing JSDoc * server-stream: drain on the http worker, drop the manager thread Address @ngxson review: httplib runs a large dynamic pool and a worker blocked in next() sits on a condvar instead of burning cpu, so draining the rest of the generation on that worker is fine and much simpler than a dedicated thread. on_complete calls finish_producer directly again. Removes adopt_orphan, the orphan thread list and its reaping, the stop_gc session cancel that only existed to unblock those threads, and the now dead drain_shutdown flag. * server-stream: split stream_pipe into producer and consumer classes Address @ngxson review: one class covering both ends was messy. stream_pipe is now a base holding the session and is_cancelled, with stream_pipe_producer (write, mark_producer_done, finish_producer, cleanup, finalizes on destruct) and stream_pipe_consumer (read only, no finalize) deriving from it. Drops the is_producer_ discriminator and its runtime guards, the type now encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer> since it is only ever a producer. No behavior change. * server-stream: rename producer methods to unix pipe semantics Address @ngxson review: mark_producer_done becomes done(), finish_producer becomes close(), matching a unix pipe write end. The producer_done_ member follows as done_. write() is unchanged. No behavior change. * server, ui: route resumable streams via a conv map, persist resume identity Address ngxson review: drop the polling probe, proxy_post records a conv_id -> model map and the stream routes resolve the owning child with one lookup. The map is the single source of truth, the ::model suffix stays for child session uniqueness but the router never parses it. UI: the server keys a session by the POST time identity (conv::model), but reload probed with the bare conv id and missed model tagged sessions, so F5 stopped the stream and sidebar spinners stayed off. Persist the model and rebuild the exact identity on resume, single conv and bulk sidebar both send it. Add unit coverage for the identity round trip. * ui: resolve continue target by id to stop cross-conversation flash on switch * ui: skip stream resume when the abort is intentional * server: move the conv id to model map into a self contained tracker Address review from ngxson: server_models held two mutexes side by side, the global one and a bare conv_model_mu guarding a loose map, which made the locking hard to follow. Wrap the map and its lock in a small conv_model_tracker struct that owns its mutex, one mutex per struct. The remember, lookup and forget methods move inline into the tracker, server_models exposes a single conv_models member and the routes call models.conv_models.lookup and friends. No behavior change, the map stays the single source of truth for routing resumable streams to a child. * ui: replace stream magic values with enums and shared constants Address review from allozaur: lift the inline literals around the resumable stream code into named symbols so the intent is explicit and reusable. * ui: fold the stream resume and discovery helpers into ChatService Address review from allozaur: drop the two standalone stream-*.service files. They were used only by the chat service and store, carried no shared state, and did not follow the static class pattern the other services use, so a separate abstraction was not warranted. Move the helpers onto ChatService as static methods. No behavior change, tests now exercise them through ChatService. * docs: document the SSE replay buffer in server README-dev Add the resumable streaming section, list stream_session_manager in the backend component inventory, and link PR 23226 in the related PRs. * ui: align attachServerStream call with onCompletionId param in handleStreamResponse * server-http: rename del_ to del to match get and post * ui: address review feedback from allozaur * ui: drop duplicate SSE constants, keep sse.ts canonical * ui: use svelte:document for the visibilitychange listener address review from allozaur: replace the manual document.addEventListener in onMount with a declarative <svelte:document onvisibilitychange>. svelte handles attach, detach and SSR, so the typeof document guard and the onMount cleanup go away. onMount keeps only the first load snapshot. * server: trim redundant stream drain comments Address review from ngxson * server: balance and clean up stream comments remove redundant comments and tighten the verbose ones across the resumable stream code, keeping the concurrency and lifetime rationale that is not obvious from the code. also fix two stale comments in server.cpp and server-models.h that still described the old ::model suffix probe and fan out routing, now replaced by the conv_id -> model map Address review from ngxson * ui: balance and clean up stream comments dedup repeated rationale (frozen conv::model identity, the lookup privacy note, the abort patterns) down to one canonical spot, tighten the verbose blocks, and keep the concurrency and resume-offset reasoning. fix stale comments in stream-identity.ts and chat.service.ts that still described the old loopback probe and fan out routing, now the conv_id -> model map. --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

* server-stream : pimpl * server-stream: prefix free functions with server_stream_ address review from ggerganov: scope the public stream functions under the server_stream_ prefix, matching server_stream_session_manager_start/stop. * server-stream: guard session and manager state with the mutex address review from ggerganov: make done, completed_ts and the GC running flag plain members under their mutex and set the condvar predicates under the lock. keep cancelled atomic for the lock-free should_stop poll. * server-stream: trim comments to the non-obvious address review from ggerganov: drop comments that restate the code, keep the concurrency, lifetime and ordering rationale. de-stale a few comments left by the pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone. * server-stream: update dev docs for the pimpl and prefix reflect server_stream_session_manager_start/stop and the server_stream_ prefix, note the manager is now a file-static singleton hidden in the .cpp * server-stream: move stream traces to debug level keep the bring-up traces for diagnostics but off the default log: skip drain, draining, drain ended, DELETE evict, attach_pipe, and the router stream resume proxy. * server-stream: align router stream resume proxy trace with upstream the child-side bring-up traces are already SRV_TRC on master, move the router stream resume proxy trace to the same level. * server-stream: move stream_read_status enum to the cpp it is only used by the hidden session and consumer types, so it belongs with them behind the pimpl boundary, not on the public header surface. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

ServeurpersoCom requested review from a team as code owners May 17, 2026 18:28

ServeurpersoCom marked this pull request as draft May 17, 2026 18:28

allozaur changed the title ~~Sse replay buffer~~ server + ui: SSE Replay Buffer May 17, 2026

ngxson reviewed May 17, 2026

View reviewed changes

Comment thread tools/server/server-context.cpp Outdated

Comment thread tools/server/server-models.cpp Outdated

github-actions Bot added examples server server/ui labels May 17, 2026

ServeurpersoCom force-pushed the sse-replay-buffer branch from 9c981bc to 59340bc Compare May 18, 2026 08:21

ServeurpersoCom force-pushed the sse-replay-buffer branch from 7cac13d to 1dd03e4 Compare May 18, 2026 14:33

ServeurpersoCom force-pushed the sse-replay-buffer branch from 1dd03e4 to ac4ef37 Compare May 18, 2026 14:40

coder543 reviewed May 18, 2026

View reviewed changes

Comment thread tools/ui/src/lib/services/stream-resume.service.ts Outdated

Comment thread tools/server/server-stream.cpp Outdated

Comment thread tools/ui/src/lib/services/chat.service.ts Outdated

Comment thread tools/ui/src/lib/services/chat.service.ts Outdated

coder543 reviewed May 18, 2026

View reviewed changes

ServeurpersoCom marked this pull request as ready for review May 18, 2026 21:12

ServeurpersoCom force-pushed the sse-replay-buffer branch from 6b3720b to 3aa3752 Compare May 19, 2026 12:03

ServeurpersoCom mentioned this pull request May 19, 2026

Feature Request: WebUI response streaming is fragile #21754

Closed

4 tasks

ngxson reviewed May 19, 2026

View reviewed changes

Comment thread tools/server/server-http.h Outdated

Comment thread tools/server/server-stream.cpp Outdated

Comment thread tools/server/server-stream.cpp Outdated

Comment thread tools/server/server-stream.h

ngxson reviewed May 19, 2026

View reviewed changes

Comment thread tools/server/server-http.cpp Outdated

ngxson reviewed May 19, 2026

View reviewed changes

Comment thread tools/server/server-http.cpp

ngxson mentioned this pull request May 19, 2026

SSE replay cleanup ServeurpersoCom/llama.cpp#4

Merged

ServeurpersoCom added 12 commits June 25, 2026 12:00

server-stream: rename producer methods to unix pipe semantics

4524ea7

Address @ngxson review: mark_producer_done becomes done(), finish_producer becomes close(), matching a unix pipe write end. The producer_done_ member follows as done_. write() is unchanged. No behavior change.

ui: resolve continue target by id to stop cross-conversation flash on…

eeb62dd

… switch

ui: skip stream resume when the abort is intentional

c572968

ui: replace stream magic values with enums and shared constants

8667f14

Address review from allozaur: lift the inline literals around the resumable stream code into named symbols so the intent is explicit and reusable.

docs: document the SSE replay buffer in server README-dev

7962c30

Add the resumable streaming section, list stream_session_manager in the backend component inventory, and link PR 23226 in the related PRs.

ui: align attachServerStream call with onCompletionId param in handle…

99f8ae0

…StreamResponse

server-http: rename del_ to del to match get and post

2f4be74

ui: address review feedback from allozaur

5a3a981

ServeurpersoCom force-pushed the sse-replay-buffer branch from 295a3ca to 5a3a981 Compare June 25, 2026 12:01

ui: drop duplicate SSE constants, keep sse.ts canonical

9ddbaf8

allozaur requested changes Jun 25, 2026

View reviewed changes

ngxson reviewed Jun 25, 2026

View reviewed changes

Comment thread tools/server/server-http.cpp Outdated

ServeurpersoCom added 3 commits June 25, 2026 15:59

server: trim redundant stream drain comments

ac02404

Address review from ngxson

allozaur approved these changes Jun 25, 2026

View reviewed changes

ServeurpersoCom merged commit 1a87dcd into ggml-org:master Jun 26, 2026
25 of 31 checks passed

ggerganov reviewed Jun 26, 2026

View reviewed changes

ServeurpersoCom mentioned this pull request Jun 26, 2026

server-stream: follow-up on SSE Replay Buffer (#23226) #25047

Merged

atmosfar mentioned this pull request Jun 27, 2026

Misc. bug: pressing stop while model is reasoning in webui breaks send button and doesn't stop generation #25055

Closed

sdroege mentioned this pull request Jun 30, 2026

Misc. bug: ui: "Stream connection lost and could not be resumed" with slow models after 1a87dcdc #25158

Closed

ServeurpersoCom mentioned this pull request Jul 2, 2026

server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections #25241

Merged

Uh oh!

Conversation

ServeurpersoCom commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Lifecycle

Router mode

WebUI

Additional information

Requirements

Uh oh!

coder543 commented May 17, 2026

Uh oh!

ServeurpersoCom commented May 17, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ServeurpersoCom commented May 18, 2026

Uh oh!

ServeurpersoCom commented May 18, 2026

Uh oh!

ServeurpersoCom commented May 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented May 18, 2026

Uh oh!

coder543 commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented May 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented May 17, 2026 •

edited

Loading

ServeurpersoCom commented May 18, 2026 •

edited

Loading

coder543 commented May 18, 2026 •

edited

Loading

ServeurpersoCom commented May 19, 2026 •

edited

Loading

ServeurpersoCom Jun 26, 2026 •

edited

Loading