feat(audit): run the audit on the hosted agent platform behind a flag by daniloc · Pull Request #915 · PostHog/wizard

daniloc · 2026-07-16T19:31:23Z

Problem (by Danilo)

We need to try agent platform with real people and real projects. The audit program is a great candidate: enough volume to be real, not an emergency if it breaks. this uses the new machinery, along with a feature flag, to route half of audits to the Agent Platform.

we shouldn't merge until platform-side changes land

PostHog/posthog#71696

Changes (by the robot)

wizard audit can now run server-side on the PostHog agent platform instead of a local subprocess, behind the boolean wizard-cloud-audit flag.

The inversion is the whole point. In the local arm the model runs here and reaches out for a skill; in the hosted arm the model runs off a frozen agent bundle server-side and reaches back for this machine's filesystem through read-only client tools. The agent's resolve_checks results stream back over SSE and mirror onto the same local ledger the existing audit TUI file-watches — so the screens, live checklist, and on-disk report are unchanged. The audit's checklist and report format now change by re-freezing the bundle, with no wizard release.

Built on the switchboard architecture. This branch was rebased onto main's runner refactor (switchboard → sequence + harness) per reviewer guidance. The hosted audit is not a branch inside a program — it's a first-class remote sequence on an agents-platform harness, routed through the switchboard:

remote sequence (runner/sequence/remote.ts) — remote bundle execution: no local skill install or prompt assembly. It resolves the program's hosted run, drives it through the harness, and owns the shared pipeline around that call (seed, failover, outro), mirroring how linear.ts wraps anthropic/pi.
agents-platform harness (runner/harness/agents-platform/) — the SDK adapter for the platform: opens a session, streams the turn, fulfils client tool calls, mirrors the checklist. The transport, client tools, and ledger bridge live here (moved out of the program, mirroring what refactor(agent): split runner into agent/runner, move orchestrator machinery out of programs #688 did for the orchestrator).
Switchboard wiring — Sequence.remote + Harness.agentsPlatform registered; a wizard-cloud-audit flag middleware on both axes binds audit to {remote, agents-platform}; the explicit wizard audit cloud command gets a default cloud-audit binding. The orchestrator run-task clamp now only affects orchestrator.
ProgramConfig.remoteRun — how a program declares its hosted arm. run stays the local default and the remote sequence's fallback.

Other pieces:

Silent local fallback. On any hosted failure (error, unexpected stream end, or a completed run that wrote no report), the remote sequence degrades to the local linear pipeline on the same bootstrap — no re-auth.
read_ledger client tool + re-grounding. The client hosts the run's authoritative ledger; the agent reads it back via read_ledger and builds the report's per-check results from it (not from its own context) — protecting the result against context drift over a long run. If it's missing a check the agent recorded, the agent re-records and re-reads (reconciliation), with a fall-through to its recorded state when the client keeps no ledger.
Path containment on the client tools: a model-supplied path is untrusted input, resolved strictly inside the project root (lexical check; symlink caveat documented — strictly more contained than the local arm, which runs with unrestricted filesystem access).
Staff-gated. The platform's posthog auth 403s anyone outside the agent's org, so the flag is targeted to PostHog staff; an external user who got it flipped would only reach a 403, surfaced as a readable message.
Shared ledger helpers. The read-modify-write logic moved from wizard-tools.ts to a shared audit/ledger.ts so the local MCP tools and the cloud bridge share one implementation.

Depends on the platform primitives in PostHog/posthog#71696 (the authenticated-class audience story and the agent's client-tool session protocol).

Known follow-up: the agents-platform harness is still audit-coupled (imports buildAuditTask + the ledger bridge directly). Fine while audit is the only remote program; a second one would inject those from the program via remoteRun.

Test plan

Switchboard (runner/__tests__/switchboard.test.ts) — added cases pinning the routing: cloud-audit binds to {remote, agents-platform}, and audit + wizard-cloud-audit flag resolves to {remote, agents-platform} while staying {linear, anthropic} off the flag.
Failover (failover.test.ts, rewritten for the sequence) — hosted success stops there; hosted error degrades to the local linear pipeline with the classic run; no remote arm → linear.
Ledger bridge + read_ledger (cloud-audit-ledger-bridge.test.ts) — catalog/resolved folding, idempotent replay, stale-ledger reset, drift-drop coercion; read_ledger returns the accumulated ledger and empty before the first resolution.
59 tests green; tsdown builds; prettier + eslint clean. (The remaining strict-tsc errors are in main's own untouched files — pi harness, orchestrator, ci-region.)
Manual: POSTHOG_WIZARD_AGENT_URL points the wizard at a local ingress for end-to-end runs against the seeded wizard-audit agent.

LLM context

Co-authored in a Claude Code session. Rebased onto main's switchboard/sequence/harness architecture and re-homed the hosted audit as a remote sequence + agents-platform harness per reviewer (gewenyu99) guidance; the wizard-development skill's architecture doc was updated to match. The platform-side counterpart (auth audience + cross-agent memory) is PostHog/posthog#71696.

`wizard audit` can now run server-side on the PostHog agent platform instead of a local Claude Agent SDK subprocess. The model runs off a frozen agent bundle and reaches back for this machine's filesystem via read-only client tools (list/grep/read, plus a single write for the report); its `resolve_checks` results mirror onto the same local ledger the existing audit TUI renders, so the screens, checklist, and report are unchanged. Keeping the audit's behaviour in a re-freezable bundle is the point — the checklist and report format change without a wizard release. Gated by the boolean `wizard-cloud-audit` flag (staff-targeted; the platform's posthog auth 403s anyone outside the agent's org). Plain `wizard audit` forks here when the flag is on and silently falls back to the classic local run on any cloud failure — same bootstrap, no re-auth; `wizard audit cloud` reaches the hosted arm directly. The ledger read-modify-write helpers move to a shared `audit/ledger.ts` so both the local MCP tools and the cloud bridge share one implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKGXDLPogR2b41oGaJfyk5

gewenyu99 · 2026-07-16T19:42:22Z

-import { registerCleanup } from '../../../utils/wizard-abort';
+import { runLinearProgram } from './linear';
+import { runCloudProgram } from './cloud';
+import { WizardError, wizardAbort } from '../../../utils/wizard-abort';


Suggested change

import { WizardError, wizardAbort } from '../../../utils/wizard-abort';

import { WizardError, wizardAbort } from ''@utils/wizard-abort';

ditto other similar imports. We have these aliases

gewenyu99

Fly by,

What I'll ask you('re agent) to consider is that flags for switching between harness (which is what this is), sequence, model, and effort are all patched through the switch board for maintainability.

The cloud runner itself looks clean, but I believe we could achieve a better shape that way.

Issue 2 is that I see tools and ledger bridgers and cloud runner specific tools in the cloud-audit program. Might I suggest that they go in the runner?

for your consideration

daniloc · 2026-07-16T19:56:43Z

@gewenyu99 indeed; I'm untangling its confused first cut a few different ways rn, thanks so much for this context

The hosted audit agent accumulates its checklist in the CLI-hosted ledger (.posthog-audit-checks.json) but had no way to read it back — so its final report was built from its own context, which drifts over a long run. Add a read_ledger client tool that returns the wizard's accumulated ledger, so the agent re-grounds from the authoritative on-disk state instead of memory. The agent-side instruction (agent.md) and spec live in the wizard-audit bundle and re-deploy separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKGXDLPogR2b41oGaJfyk5

daniloc · 2026-07-16T20:51:47Z

lmao the problem was that it built this on an inexplicably stale checkout without my catching it. reworking

Merge main and rebuild the hosted audit onto the runner architecture that landed since this branch forked (switchboard -> sequence + harness). The config.executor === 'cloud' fork and the machinery under programs/cloud-audit/ were the pre-switchboard shape; this replaces them with: - a remote sequence (sequence/remote.ts): remote bundle execution, no local skill install or prompt, drives the harness, owns seed/failover/outro - an agents-platform harness (harness/agents-platform/): the platform session transport, client tools, and ledger bridge, moved out of the program - switchboard wiring: Sequence.remote + Harness.agentsPlatform, a wizard-cloud-audit flag middleware on both axes, and a cloud-audit binding; the run-task clamp now only affects orchestrator - ProgramConfig.remoteRun: how a program declares its hosted arm; run stays the local default and the remote sequence's fallback Migrated to boot.credentials, dropped the obsolete executor/fallback fields, and rewrote the failover test for the new sequence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKGXDLPogR2b41oGaJfyk5

audit/index.ts and cloud-audit/index.ts import each other's run-builders at module-init time. buildCloudAuditRun is a hoisted `function`, so it survives the cycle, but buildClassicAuditRun was a non-hoisted `const` arrow. When the bundle emits the cloud-audit region first, cloudAuditConfig's `run: buildClassicAuditRun` reads the const before the audit region initializes it, throwing `ReferenceError: Cannot access 'buildClassicAuditRun' before initialization` at the top of dist/bin.js — a module-init crash that takes down the whole CLI. Convert buildClassicAuditRun to a hoisted `function` declaration, symmetric with buildCloudAuditRun, so both survive the cycle regardless of bundler ordering. Reproduced via the postbuild smoke test (bin.js crashed on load); fixed after. Generated-By: PostHog Code Task-Id: d5a911ec-e7d6-4b49-a651-9f56cb33ace7

gewenyu99 · 2026-07-16T22:08:49Z

lmao the problem was that it built this on an inexplicably stale checkout without my catching it. reworking

Classic.

gewenyu99 · 2026-07-16T22:09:49Z

There we go. A glance and it looks miles and miles cleaner to read.

daniloc · 2026-07-16T22:17:59Z

and I think I can make the diff even smaller 😎

The wizard's client-tool hands (read/grep/list/write) hand-rolled ~100 lines of node:fs logic. Route them through pi's built-in tool implementations (createReadTool/createGrepTool/createLsTool/createWriteTool) instead — the same engine the platform runs — so we reuse pi's read/grep/list/write logic, arg schemas, and truncation rather than maintaining our own. We keep only the path jail (the remote model is untrusted) and the audit-specific read_ledger. Follow-ups (need the platform stack): align the wizard-audit bundle's client tool descriptions to pi's result shape, and validate end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKGXDLPogR2b41oGaJfyk5

gewenyu99 · 2026-07-16T23:10:19Z

This should be an ultra neat shape. If we can prove this to be a good 1:1 drop over like we did pi vs anthropic agent SDK, we can just flag this for testing with any program

daniloc requested a review from a team as a code owner July 16, 2026 19:31

gewenyu99 reviewed Jul 16, 2026

View reviewed changes

daniloc force-pushed the agent-platform/cloud-audit branch from b75bb2c to 0619572 Compare July 16, 2026 21:45

daniloc force-pushed the agent-platform/cloud-audit branch from 0619572 to a77f887 Compare July 16, 2026 21:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(audit): run the audit on the hosted agent platform behind a flag - #915

feat(audit): run the audit on the hosted agent platform behind a flag#915
daniloc wants to merge 5 commits into
mainfrom
agent-platform/cloud-audit

daniloc commented Jul 16, 2026 •

edited

Loading

Uh oh!

gewenyu99 Jul 16, 2026

Uh oh!

gewenyu99 left a comment

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

gewenyu99 commented Jul 16, 2026 •

edited

Loading

Uh oh!

gewenyu99 commented Jul 16, 2026

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

gewenyu99 commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	import { WizardError, wizardAbort } from '../../../utils/wizard-abort';
	import { WizardError, wizardAbort } from ''@utils/wizard-abort';

Uh oh!

Conversation

daniloc commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem (by Danilo)

Changes (by the robot)

Test plan

LLM context

Uh oh!

gewenyu99 Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

gewenyu99 left a comment

Choose a reason for hiding this comment

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

gewenyu99 commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gewenyu99 commented Jul 16, 2026

Uh oh!

daniloc commented Jul 16, 2026

Uh oh!

gewenyu99 commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniloc commented Jul 16, 2026 •

edited

Loading

gewenyu99 commented Jul 16, 2026 •

edited

Loading