[RFC] CD deployment agent/runner — lightweight, forge-agnostic, idle-and-wait

[ChrisonSimtian]

Problem

CD needs to run deployment steps on machines the forge runner can't reach. Forge runners live in the public cloud; targets often sit behind a firewall (Proxmox host, on-prem VMware, air-gapped subnet, homelab box).

Self-hosted runners are the current workaround, but they're forge-coupled, heavy, and tie deployment topology to one forge. We want something lighter and portable.

Proposal

A small Fallout deployment agent: one binary that runs on a target host, authenticates to a coordinator, and idles for instructions. On an instruction it runs a script (Bash / PowerShell / C# / dotnet-script), streams logs back, and reports status.

It's a script executor with auth and logging — not a workflow engine. Orchestration stays in the CD pipeline (C#, in the Fallout repo); the agent runs the leaves.

Shape

• Distribution: single self-contained fallout-agent per OS/arch, and/or a global dotnet tool.
• Hosting: everything defining the agent's behaviour (allowed scripts, valid targets, auth secrets) lives in the consumer's git repo. Agent pulls on startup. No external control plane.
• Idle: long-poll / WebSocket / SSE to the coordinator. No inbound port on the agent host.
• Execution: spawn script with clean env, capture stdout/stderr line-by-line, stream upstream, exit code → status.
• Auth: mutual. Agent proves identity (pinned token/cert); coordinator proves the command is signed by an authorised source (forge OIDC, signed payload).
• Logging: structured per-run logs, kept on the agent host and streamed to the coordinator. Local retention survives a coordinator outage.

Forge-agnostic

GitHub is the reference (OIDC from actions/ runs is easy to verify). Must also work with Gitea, Forgejo, GitLab, Azure DevOps — anything that can sign a JWT or call a webhook. The agent knows nothing about "GitHub", only an abstract "coordinator I trust to sign commands".

Open questions

• Wire protocol: HTTP long-poll (simplest), persistent WebSocket (bidirectional, more state), git-ref pull (everything-in-repo, but git-as-queue has sharp edges), or hybrid (forge issues signed dispatch, agent polls a small endpoint).
• Coordinator topology: separate service, or the forge itself (agent talks to GitHub/GitLab API directly)?
• Auth model: forge OIDC, pre-shared keys, certs, SPIFFE? Prefer OIDC.
• Concurrency: instinct is single-tenant, single-command by default; run multiple agents for parallelism.
• Script runtime: pure exec, or a richer C# API (FalloutAgent.Log(), .Artifact())? Start with exec/capture/report; add on demand.
• Updates: pinned version, auto-update from a release feed, or manual?
• vs v12 plugin SDK: separate component or a hosted plugin? Probably separate (different deployment + threat model) — confirm.

Why v13

The agent is the substrate the rest of CD sits on — environment promotion, approval gates, release tracking all need a way to run commands on real machines. Filed as RFC because the shape is open. Most useful feedback: deployment topologies you'd use Fallout for, and where self-hosted-runner/agent solutions fall short.

Related: parent CD RFC #106.

[ChrisonSimtian]

Design pass — 2026-05-28

Working through this with a consumer hat on (currently running build-as-deployment-agent-over-SSH in another project — brittle for the obvious reasons: outbound SSH from CI per target, key sprawl, host fingerprint drift, opaque failure modes). This captures concrete positions on several of the open questions above and pushes others further.

Poll, not push — closes Command structure

Default: HTTPS long-poll, agent → coordinator. Outbound 443 only; works through any corporate proxy; no inbound firewall holes. The mental model has to be clean: deploy the agent once, it listens to the coordinator, grabs a job when one's there, runs it, reports.

WebSocket / SSE / push remain as a pluggable transport for shops that already have inbound holes and need sub-second dispatch — but they ship as a plugin, not as the default. Git-as-queue is out — too many sharp edges, audit story is worse than a real coordinator.

Deployment scripts = published Fallout build artifact — closes Script runtime contract

Don't invent a script host. Ship a published Fallout build artifact and exec it.

The deployment is already a Fallout target (Deploy) in someone's _build project. The coordinator runs dotnet publish once, signs the output, uploads to blob storage. Job descriptor is roughly:

{ jobId, target: "Deploy", artifactUrl, artifactSha256, sealedSecrets, parameters }

Agent flow per job:

1. Download artifact → verify hash + signature
2. Decrypt sealed secrets in-memory
3. Spawn dotnet fallout Deploy --param ... with secrets in env
4. Stream stdout/stderr back over OTLP (see Logs)
5. Report exit code → wipe secrets → loop

Killer property: agent is target-agnostic. It doesn't know what "deploy" means. Same agent runs DB migrations, k8s rollouts, IIS site swaps, ECS task updates — anything that fits in a Fallout target. C# in a real OS process, real exit codes, full isolation, attach a debugger if you need to. Cold start is one dotnet invocation (~300ms); artifacts cached by hash so repeats are instant.

Implication: no separate "script runtime" — no Roslyn-on-agent, no AssemblyLoadContext games, no Bash/PowerShell/dotnet-script branching. The runtime contract is the Fallout target contract, which we already own. This narrows the option space the original RFC left open ("Bash / PowerShell / C# / dotnet-script / something else with a registered runner") — and the narrowing is the point.

Artifact format: self-contained single-file binary in v1 (one hash, one blob, no agent↔build runtime coupling). Framework-dependent is a size optimisation we can add later.

Secrets — sealed envelope per job, in-memory only

• Agent generates an X25519 keypair at startup; registers the public key on enrollment.
• Each job descriptor carries secrets sealed to that public key (libsodium sealed boxes — one primitive, no PKI).
• Agent decrypts in-memory into byte[] (never string — string interning + LOH = forever-reachable), injects to child via env, then CryptographicOperations.ZeroMemorys the buffer after spawn. The OS owns the child's env block from that point and zeros it at child exit.
• Agent restart = new keypair = coordinator re-seals next job. Rotation for free.

Coordinator-side secret storage: ship a simple Day-1 implementation, but be explicit it's a sample. Encrypted blob, key from env / file, no HSM, no rotation, no audit log. Label accordingly (FalloutVault.Simple or similar). Production operators who want hardening wire one of the third-party adapters we already track on the roadmap:

• 1Password (already-tracked feature)
• Bitwarden (already-tracked feature)
• Azure KeyVault / AWS Secrets Manager / HashiCorp Vault (future adapters)

The Day-1 impl exists so first-run works without standing up a vault — not as the answer for steel-hardened production. The README has to say so plainly.

Logs — OTLP, agent as collector — refines Logging

Agent emits OTLP; coordinator is (or fronts) an OTLP collector.

The deployment is naturally a distributed trace: coordinator → agent → child process. Each Fallout target maps to a span. Stdout/stderr → log records on the span. This buys us:

• Standard tooling (otelcol) — operators fan out to their existing observability stack with config, not code.
• Resumable streaming is part of the spec — agent checkpoints and replays on poll reconnect.
• Secret redaction is a stock OTLP processor.
• Coordinator may not even need to store logs long-term — forward to whatever the operator already runs (Grafana Loki, Datadog, Honeycomb, …).

Tradeoffs to keep honest:

• OTLP batches. Fine for durable record, laggy for live-tail UX. We'll likely want a small WebSocket/SSE tail channel on top of OTLP — same source, different consumer — for the "watching deploys go" experience. Not duplicate work; optimising for different consumption patterns.
• Dep weight on the agent. OTel SDK is non-trivial. Should be fine (it's AOT-friendly now), but worth measuring against hand-rolled HTTP+JSON line streaming before locking in — the agent is meant to be a small single binary.
• "Coordinator doesn't need to store logs" only holds if operators run a backend (Loki/etc.) to view them. For first-run, no-infra UX the coordinator probably still stashes a copy (SQLite) for its own UI. OTLP buys us fan-out, not coordinator-replacement.

Local retention on the agent host stays for audit (coordinator-outage durability), as the RFC already proposed.

Concurrency, scheduling, rerun — TickerQ-shaped coordinator — closes Concurrency, refines Coordinator topology

If we model the coordinator's job state on TickerQ, the open questions collapse:

• Concurrent jobs: native — multiple agents claim independent jobs from the queue.
• Scheduled: cron-style scheduling is TickerQ's main feature.
• Rerun: re-trigger a completed job with the same descriptor.
• Persistence: survives coordinator restart.

The TickerQ job isn't "run the deployment in-process" — it's "dispatch this to a remote agent and await completion." Concurrency, retries, and schedules become coordinator-side concerns; the agent stays dumb.

Per-agent concurrency stays single-job-at-a-time by default ("deployments shouldn't race against themselves on the same host"). Operators run multiple agents for parallelism, which the topology naturally supports.

Signing — pinned key for v1

Coordinator-pinned public key in agent config at enrollment time. Sign the job descriptor; agent verifies before doing anything. Sigstore / cosign for public verifiability is a follow-up if there's demand — not v1.

Status against the open questions in the issue body

Question	Position
Command structure	HTTPS long-poll default; push as plugin
Coordinator topology	Separate coordinator (TickerQ-shaped); forges are auth sources, not the dispatcher
Auth model	mTLS or signed JWT at the transport; sealed envelope for secrets; coordinator-pinned key for signing job descriptors
Concurrency	Single-job-per-agent; multi-agent for parallelism; concurrency lives in coordinator / TickerQ
Script runtime contract	Published Fallout build artifact, exec out-of-process — no separate runtime
Update story	Still open — pinned version + operator-driven upgrade is the boring-correct v1; auto-update needs a real threat model
Relationship to v12 plugin SDK	Separate component. Agent depends on the SDK only insofar as it spawns published Fallout targets — no in-proc plugin loading on the agent

Still open

• Update story. Pinned version + operator-driven upgrade is my default; anything beyond that needs a threat model first.
• Enrollment flow ergonomics. One-time token printed by the coordinator, pasted into the agent install, exchanged for the long-lived identity. Worth a sub-issue when sized.
• Artifact storage location. Coordinator-fronted blob vs forge-native artifact storage vs both. Probably both, with a default.
• Per-target host requirements. At minimum, a .NET runtime matching the artifact's TargetFramework or a self-contained artifact. Self-contained removes a config knob; size is the trade.