feat: live 2-process disaggregated serving handoff (B3b2a)#192
Merged
Conversation
Wire the disaggregated serving role end to end so two real mlxcel-server
processes hand a sequence off over TCP: a --node-role prefill node prefills
and ships the KV to a --node-role decode node, which continues decoding. The
worker now flips from the standard scheduler loop to a networked role loop
when a non-hybrid role is configured.
- serving_protocol.rs: PrefillRequestFrame / DecodeMetaFrame / ResultFrame
control frames (serde_json over TransportMessage::Control) plus the
SamplingConfig <-> SerializableSamplingState converters.
- coordinator.rs: run_{prefill,decode}_role_networked read frames off the
transport, drive the B2b scheduler entries, and return results to each
request's reply_to; serve_{prefill,decode}_role_networked_blocking drivers.
- config chain: new --serving-bind flag; thread serving_bind + decode_peers
from ServerStartupInput through ServerConfig and the provider ladder to
WorkerSchedulerConfig. prefill_peers stays in ServerConfig for the future
dedicated router.
- model_worker.rs: a non-hybrid worker binds its role transport and runs the
role loop; a misconfigured role falls back to the single-node loop.
- tests/disaggregated_serving_e2e.rs: spawns two real servers and verifies the
merged stream is byte-identical to the single-node greedy reference.
Result split (prefill first token + decode continuation) matches the
StreamBridge contract the dedicated router will merge in B3b2b. Additive only;
the in-crate channel-driven role loops and single-node path are unchanged.
Part of #126
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
B3b2a of the disaggregated serving productionization (epic #116, capstone #126). Wires the serving role end to end so two real
mlxcel-serverprocesses hand a sequence off over TCP and reproduce the single-node output byte-for-byte. The dedicated router HTTP front and 3-process client E2E land next in B3b2b.What this does
serving_protocol.rs):PrefillRequestFrame(client to prefill),DecodeMetaFrame(prefill to decode, precedes the KV frame),ResultFramewith FirstToken/Continuation phase (nodes to client). Carried asTransportMessage::Control+ serde_json.coordinator.rs):run_{prefill,decode}_role_networkedread frames off the transport, drive the existing B2b scheduler entries, and return results to each request'sreply_to;serve_{prefill,decode}_role_networked_blockingdrivers build the runtime and bind the transport.--serving-bind <addr>flag on both binaries;serving_bind+decode_peersthreaded fromServerStartupInputthroughServerConfigand the provider ladder toWorkerSchedulerConfig.prefill_peersstays inServerConfig(pub) for the future dedicated router.model_worker.rs): a non-hybrid worker binds its role transport and runs the role loop instead ofscheduler.run(); a misconfigured role (no--serving-bind, or prefill with no--decode-peers) logs and falls back to the single-node loop rather than hanging.Design notes
StreamBridge.submit_first_token/submit_decode_tokencontract the B3b2b router will merge.--serving-bind: no existing flag gives a node its own role-transport listener (HTTP host:port feeds the cluster-control address).Verification (orchestrator-local, the real metal gate)
-D warningsboth feature sets (metal,accelerateand+test-utils)--all-targets: cleancargo fmt --check: cleantests/disaggregated_serving_e2e.rs): two realmlxcel-serverprocesses hand off over TCP, merged output" \n\n###Answer: 4\n\n###Step 1: 2 + "is byte-identical to the single-node greedy referencePart of #126