fix(yara): triage skill installs on every harness - #997
Open
gewenyu99 wants to merge 4 commits into
Open
Conversation
Skill-install scans built their triage provider from a default that only read the ANTHROPIC_* env vars, which pi and the orchestrator never set — so 5 of 6 install sites ran untriaged, fail-closed, and terminated on first-party skills (~15 users/day since 07-24). #977 fixed only pi's `install_skill` tool; the orchestrator pre-installs skills itself and never calls it. Resolve the provider once in bootstrapProgram, carry it on BootstrapResult, and pick its model from the harness — haiku on anthropic, luna on pi. Dispatch through the pi SDK's completeSimple instead of a hand-rolled axios POST, so transport, reasoning effort and the gateway trace headers come from buildGatewayModel rather than being restated. temperature 0 is anthropic-only: the gpt-5 line 400s on it, which warlock's fail-safe turned into `true_positive` — a failed triage call now logs instead of passing silently. Generated-By: PostHog Code Task-Id: a3f36102-7dc4-47ef-b49d-a05e0e12146b
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
Collaborator
Author
|
/wizard-ci basic-integration |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
`isTerminalMatch` ended the session on `action: 'block'` at any severity, so the medium `prompt_injection_posthog_integration_attack` was fatal wherever it fired. Block is about refusing the operation — PreToolUse still does — not about ending the run. Skill installs bypassed the shared verdict entirely and terminated on ANY post-triage match; both install scanners now resolve `scanVerdict` like every other surface and keep a non-terminal match's skill instead of deleting it. Also folds `skillsRoot`/`triage` into a named `SkillInstallOptions` so no call site passes a bare positional `undefined`. Generated-By: PostHog Code Task-Id: a3f36102-7dc4-47ef-b49d-a05e0e12146b
… change Critical-only termination is only safe if the payloads that matter are actually critical, and nothing checked that. The smoke test now resolves real scans through the real policy: six injection classes (instruction override, role hijack, DAN persona, chat markup, base64-in-comment, poisoned SKILL.md) must stay terminal, and the three first-party doc strings that blocked ~200 runs must not. Four whole-skill fixtures run the real install scanner with triage unavailable, so a poisoned skill is still deleted on severity alone and first-party prose still installs. Lives in the smoke script rather than the unit suite because vitest aliases @posthog/warlock to a manual mock (WASM can't load in the runner), so only this script sees the real rules — and it gates every build and publish. Generated-By: PostHog Code Task-Id: a3f36102-7dc4-47ef-b49d-a05e0e12146b
Triage is a boolean classifier, so it should follow the current 4.5 release rather than pin a dated snapshot the way dispatchable agent models do. Kept out of MODEL_CAPABILITIES deliberately: it is not a model the switchboard may route agent work to, and defaultCaps already resolves it correctly. Generated-By: PostHog Code Task-Id: a3f36102-7dc4-47ef-b49d-a05e0e12146b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skill-install scans ran untriaged on 5 of 6 install sites, so YARA terminated on first-party skills — ~15 users/day since 07-24.
scanInstalledSkilldefaulted its triage provider to one built from theANTHROPIC_*env vars, which pi and the orchestrator never set (they auth the gateway programmatically), so the default resolved toundefinedand every flagged match was acted on. #977 wired auth into pi'sinstall_skilltool, but the orchestrator pre-installs skills itself atorchestrator-runner.ts:292/:478and never calls that tool. Now the provider resolves once inbootstrapProgram, rides onBootstrapResult, and takes its model from the harness (haiku on anthropic, luna on pi), dispatched through the pi SDK'scompleteSimpleso transport/effort/trace-headers come frombuildGatewayModelinstead of a hand-rolled axios POST.temperature: 0is anthropic-only — the gpt-5 line 400s on it, and warlock's fail-safe turned that failure intotrue_positive, which is why a failed triage call now logs instead of passing silently.Verified against
wizard-workbench/apps/basic-integration/react-native/react-native-saasover wizard-ci: on stock main, 5 skills blocked (triage skipped (no provider)); with this branch, both paths install all 5 (overruled: 6 · poisoned: 0 · install-failed: 0) — orchestrator onopenai/gpt-5.6-lunaand linear onclaude-haiku-4-5.Not addressed here: the warlock rules still match PostHog's own doc prose ("disable PostHog"), so every install pays a triage call to overrule them, and skill-install still terminates on any severity while Read/Grep only terminate on critical.
Created with PostHog Code