Skip to content

feat: live 2-process disaggregated serving handoff (B3b2a)#192

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-126-b3b2a-live-2process-handoff
Jun 9, 2026
Merged

feat: live 2-process disaggregated serving handoff (B3b2a)#192
inureyes merged 1 commit into
mainfrom
feature/issue-126-b3b2a-live-2process-handoff

Conversation

@inureyes

@inureyes inureyes commented Jun 9, 2026

Copy link
Copy Markdown
Member

Summary

B3b2a of the disaggregated serving productionization (epic #116, capstone #126). Wires the serving role end to end so two real mlxcel-server processes hand a sequence off over TCP and reproduce the single-node output byte-for-byte. The dedicated router HTTP front and 3-process client E2E land next in B3b2b.

What this does

  • Wire protocol (serving_protocol.rs): PrefillRequestFrame (client to prefill), DecodeMetaFrame (prefill to decode, precedes the KV frame), ResultFrame with FirstToken/Continuation phase (nodes to client). Carried as TransportMessage::Control + serde_json.
  • Networked role loops (coordinator.rs): run_{prefill,decode}_role_networked read frames off the transport, drive the existing B2b scheduler entries, and return results to each request's reply_to; serve_{prefill,decode}_role_networked_blocking drivers build the runtime and bind the transport.
  • Config chain: new --serving-bind <addr> flag on both binaries; serving_bind + decode_peers threaded from ServerStartupInput through ServerConfig and the provider ladder to WorkerSchedulerConfig. prefill_peers stays in ServerConfig (pub) for the future dedicated router.
  • Worker LIVE flip (model_worker.rs): a non-hybrid worker binds its role transport and runs the role loop instead of scheduler.run(); a misconfigured role (no --serving-bind, or prefill with no --decode-peers) logs and falls back to the single-node loop rather than hanging.

Design notes

  • Result stays split (prefill first token + decode continuation to the client), matching the StreamBridge.submit_first_token / submit_decode_token contract the B3b2b router will merge.
  • New --serving-bind: no existing flag gives a node its own role-transport listener (HTTP host:port feeds the cluster-control address).
  • Decode intake is two frames (decode-meta control frame then KV frame), keeping the KV serde focused on KV.
  • Additive only: the in-crate channel-driven role loops (B3a) and the single-node path are unchanged.

Verification (orchestrator-local, the real metal gate)

  • Cold clippy -D warnings both feature sets (metal,accelerate and +test-utils) --all-targets: clean
  • cargo fmt --check: clean
  • Protocol unit tests 5/5; in-crate B3a/B2c/B3b1 serving parity 4/4 unchanged
  • 2-process spawn E2E (tests/disaggregated_serving_e2e.rs): two real mlxcel-server processes hand off over TCP, merged output " \n\n###Answer: 4\n\n###Step 1: 2 + " is byte-identical to the single-node greedy reference
  • Full lib suite: 3086 passed, 0 failed

Part of #126

Wire the disaggregated serving role end to end so two real mlxcel-server
processes hand a sequence off over TCP: a --node-role prefill node prefills
and ships the KV to a --node-role decode node, which continues decoding. The
worker now flips from the standard scheduler loop to a networked role loop
when a non-hybrid role is configured.

- serving_protocol.rs: PrefillRequestFrame / DecodeMetaFrame / ResultFrame
  control frames (serde_json over TransportMessage::Control) plus the
  SamplingConfig <-> SerializableSamplingState converters.
- coordinator.rs: run_{prefill,decode}_role_networked read frames off the
  transport, drive the B2b scheduler entries, and return results to each
  request's reply_to; serve_{prefill,decode}_role_networked_blocking drivers.
- config chain: new --serving-bind flag; thread serving_bind + decode_peers
  from ServerStartupInput through ServerConfig and the provider ladder to
  WorkerSchedulerConfig. prefill_peers stays in ServerConfig for the future
  dedicated router.
- model_worker.rs: a non-hybrid worker binds its role transport and runs the
  role loop; a misconfigured role falls back to the single-node loop.
- tests/disaggregated_serving_e2e.rs: spawns two real servers and verifies the
  merged stream is byte-identical to the single-node greedy reference.

Result split (prefill first token + decode continuation) matches the
StreamBridge contract the dedicated router will merge in B3b2b. Additive only;
the in-crate channel-driven role loops and single-node path are unchanged.

Part of #126
@inureyes
inureyes merged commit f117990 into main Jun 9, 2026
5 checks passed
@inureyes
inureyes deleted the feature/issue-126-b3b2a-live-2process-handoff branch June 9, 2026 13:39
@inureyes inureyes self-assigned this Jun 10, 2026
@inureyes inureyes added status:done Completed type:enhancement New features, capabilities, or significant additions priority:medium Medium priority labels Jun 10, 2026
@inureyes inureyes added this to the 0.2 milestone Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:medium Medium priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant