MCP client does not auto-reconnect after disconnect

[lihah111222333-cloud]

Summary

MCP server connections in Codex currently do not auto-recover after a disconnect, while model SSE streams already have retry/backoff logic.

This causes MCP tools/resources to remain unavailable until users manually reload MCP servers or restart the CLI.

Observed behavior

1. AsyncManagedClient uses a cached Shared<Future> startup result, so initialization is effectively one-shot.
2. RmcpClient has Connecting -> Ready but no explicit Disconnected/Reconnecting lifecycle.
3. There is no heartbeat/health monitor to proactively detect dropped MCP transport.
4. Model API SSE flow has reconnect (retry + backoff), but MCP path has no equivalent resilience.
5. Practical recovery is manual (RefreshMcpServers / config/mcpServer/reload) or full process restart.

Why this matters

• Long-running Codex sessions with stdio or streamable-http MCP servers are fragile.
• Any transient server restart/network hiccup can permanently break MCP calls in the current session.
• Users experience random "tool call failed" errors without automatic self-healing.

Minimal repro

• Configure any MCP server (stdio is enough).
• Start a session and successfully call an MCP tool once.
• Restart/kill the MCP server process.
• Call MCP tool again.
• Expected: Codex reconnects and retries automatically.
• Actual: MCP call fails until manual reload/restart.

Suggested direction

• Add reconnect support to MCP connection management:
  • Detect connection-class errors on MCP operations.
  • Recreate client transport and re-run initialize/list-tools.
  • Retry failed request once after successful reconnect.
  • Re-send relevant state (e.g. sandbox-state capability update) after reconnect.
• Optionally add periodic health check/heartbeat for proactive reconnection.

If helpful, I can open a follow-up PR with a concrete implementation and tests.

[HamburgJ]

Adding a startup-time variant of this that the "reconnect after disconnect" framing doesn't cover, plus a small complementary fix that already half-exists in the tree. (All against current main.)

A server that never connects in the first place. The repro here is "connect successfully, then kill the server." But the same one-shot Shared startup future (observed-behavior #1) also makes the initial handshake terminal: if a non-required server's initialize + tools/list doesn't finish within startup_timeout_sec (default DEFAULT_STARTUP_TIMEOUT = 30s) — e.g. a backend that's briefly slow on its first tools/list — the future resolves to Err and is cached for the life of the McpConnectionManager. In a long-lived / warm process the server then stays toolless for the rest of the session with no retry, even though a handshake a few seconds later would have succeeded. A reconnect-on-disconnect path wouldn't help, because the connection never reached Ready.

There's already a survivorship mechanism — but only for one server. AsyncManagedClient::listed_tools() serves cached_tool_info_snapshot when the live startup result isn't usable (still initializing, or Err). That snapshot is only ever populated for the built-in codex_apps server — load_startup_cached_codex_apps_tools_snapshot returns None for any server_name != CODEX_APPS_MCP_SERVER_NAME. So codex_apps already survives a transient startup failure; every user-configured server does not. Generalizing that snapshot to all servers would be a smaller, complementary fix to the full reconnect lifecycle proposed above — it specifically covers the cold-start / first-handshake case the reconnect path can't.

Interaction with #28407. The in-flight "[codex-mcp] Avoid blocking on optional MCP startup" makes list_all_tools concurrent and skips pending optional servers that have no snapshot (early return None). That fixes the first-turn latency in #19556 / #21318, but for the startup-failure case it turns "block" into "silently skipped, still toolless, still no retry" — so it doesn't address the durability concern here for non-codex_apps servers (and it removes the stall that previously made the problem visible).

Failing-test outline (mirrors list_all_tools_uses_cached_tool_info_snapshot_when_client_startup_fails, but for a generic server): build an AsyncManagedClient with a non-codex_apps server_name, cached_tool_info_snapshot: None, startup_complete = true, and a client future resolving to Err(StartupOutcomeError::Failed { .. }), then assert its tools are recoverable. Today that path yields None and the server contributes no tools, so it fails — demonstrating the gap.

Happy to put up a PR for the snapshot-generalization piece if that's a direction you'd take.

[NarryG]

We really need something like this -- Claude has "reconnect" under /mcp. opencode has a solution. Currently, Codex has no great solution for MCP's flakiness

[junovhs]

Please prioritize this, this issue is extremely disruptive and comes up frequently.