Share compute: route to external OpenAI-compatible endpoints — local servers (Ollama / LM Studio / MLX / vLLM) and, opt-in, cloud/enterprise gateways

[bradhallett]

Type: proposal / dependency-tracking (issue-first per CONTRIBUTING, no PR yet)

Problem

Share compute can only serve a model the mesh node loads and runs itself — a GGUF / hf:// / catalog ref resolved to local weights. A member who already runs a capable model in Ollama, LM Studio, oMLX (Apple MLX), vLLM, TGI, or a llama.cpp server cannot contribute it to the community pool; they must let Buzz re-download and re-load a GGUF twin (if one even exists). And a business relay that wants every employee agent to draw from one vetted compute source has no mesh-native way to provide it.

This blocks two constituencies:

• Apple-Silicon users running MLX. Share compute is llama.cpp/Metal (GGUF) only; MLX is the dominant native local-LLM runtime on M-series Macs, and there's no native MLX path in mesh-llm (Mesh-LLM/mesh-llm#813 is closed/unmerged). Without external-endpoint support, MLX models are unreachable from the mesh entirely.
• Ollama / LM Studio / vLLM users who already have a model loaded and don't want a second copy through Buzz.
• Business / enterprise relays. An org that wants agents to use a single, reliable, governed compute source — a company OpenAI/Azure account, a self-hosted vLLM cluster, or a compliance/rate-limit gateway — currently has to have each employee paste their own key into openai-compat. The mesh could centralize that.

What the upstream already provides

mesh-llm ships an openai-endpoint plugin (catalog, v0.1.2) — "Routes mesh-llm inference to an external OpenAI-compatible endpoint such as llama.cpp, vLLM, TGI, Ollama, or Lemonade Server." It probes GET /v1/models, health-checks, and routes /v1/chat/completions. Local and cloud endpoints are the same mechanism to it — the base URL differs, nothing else.

What I verified (standalone, on 0.74.0)

Ran mesh-llm standalone with the plugin pointed at a local oMLX (MLX) server:

• ✅ Discovery works. A local gate model + the plugin → GET /v1/models surfaces 11 models — the local GGUF plus 8 oMLX MLX models (e.g. Qwen3.6-35B-A3B-6bit). Plugin process runs and advertises the backend's catalog.
• ❌ Routing does not. POST /v1/chat/completions { model: "Qwen3.6-35B-A3B-6bit" } → 404 model not found. Node logs startup plan ready: 1 process(es), 1 endpoint(s), 1 model(s) — the plugin registers as an endpoint but its models aren't in the request router.
• ❌ No zero-model daemon. serve exits ("needs at least one startup model"); client starts but doesn't load plugins; [runtime] mode = "on_demand" is ignored by serve in this version.

Routing to plugin/external models + the zero-model daemon are Mesh-LLM/mesh-llm#910 ("Runtime daemon and model lifecycle"), closed-as-completed 2026-07-27 — but not in the 0.74.0 release (cut the same day). Code on main only.

Dependency chain for Buzz

Buzz vendors mesh-llm at 0.73.1 (desktop/src-tauri/Cargo.toml:93 rev f455d49…; Cargo.lock), already behind 0.74.0 — and Mesh-LLM/mesh-llm#910 isn't in 0.74.0 yet. Path:

1. mesh-llm releases a version including Mesh-LLM/mesh-llm#910 (pending release)
2. Buzz bumps the vendored mesh-llm 0.73.1 → that release
3. Share-compute wiring (below)

Proposed wiring (once the upstream fix is vendored)

• classifyModelRef (desktop/src/.../classifyModelRef.ts): add an http(s):// → external-endpoint kind alongside GGUF / hf:// / catalog.
• StartMeshNodeRequest (desktop/src/shared/api/tauriMesh.ts): carry a base URL (+ optional key) alongside modelId, so a serving node can register an external backend instead of loading weights.
• Discovery: advertise external-backed models in the existing mesh membership events so peers see them and "auto" can route to them.
• buzz-agent relay-mesh provider: no change expected — it already consumes the mesh's OpenAI-compatible API; it just gains more models in the catalog.

Two trust postures, one mechanism

The capability is one piece of plumbing; local vs. cloud is a policy/governance distinction, not extra engineering:

• Local member-owned backends (Ollama / LM Studio / MLX / vLLM on a member's machine) — a faithful extension of VISION_MESH: still "hardware owned by people you already chose to work with," still no cloud/API keys. Fits the sovereign default.
• Cloud / enterprise gateways (a vendor endpoint, or an org's self-hosted vLLM/compliance gateway) — genuinely diverges from the "no vendor, no cloud bill" default. Should be a distinct, explicitly-enabled, per-community mode, never the default. It does buy something today's per-employee-key flow can't: the vendor credential lives only on the serving node/gateway; member agents never see it (consistent with the security: keep BUZZ_PRIVATE_KEY out of agent-controlled processes #2883 principle of keeping secrets out of member-controlled processes). The consent panel must say plainly that these prompts reach a vendor.

Non-goals

• Native MLX runtime in mesh-llm (Mesh-LLM/mesh-llm#813) — separate; this proposal makes external MLX servers (oMLX) work without it.
• Silently routing to vendor endpoints — cloud/enterprise must be an explicit, per-community opt-in with honest consent, never a hidden default.

Cross-links

• Depends on: Mesh-LLM/mesh-llm#910 (closed-completed, pending release) + a Buzz vendor bump from 0.73.1.
• Coordinate with: Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal) #2062 (in-flight desktop/src-tauri/src/commands/mesh_llm.rs ingress-liveness / evict-rebuild — this work shares that code path).
• Related: Track Compute Share managed-agent startup, MeshLLM diagnostics, and relay-mesh text-only media handling #3300 (Compute Share / MeshLLM tracking); security principle from security: keep BUZZ_PRIVATE_KEY out of agent-controlled processes #2883 (keep signing/vendor secrets out of member-controlled processes).

Happy to turn this into a PR once Mesh-LLM/mesh-llm#910 is in a vendored release.