Skip to content

chore(ci): Skeptic Gate + Evidence Gate (portable install) #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jleechan2015 merged 4 commits into from
Jun 5, 2026
Merged

chore(ci): Skeptic Gate + Evidence Gate (portable install) #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fc5d656
chore(ci): add portable Skeptic Gate and Evidence Gate workflows
Apr 1, 2026
a8f41a8
fix(ci): skeptic gate — use latest completed check; optional thread g…
Apr 1, 2026
63305bd
fix(ci): skeptic gate — drop check-runs --paginate (use per_page=100)
Apr 1, 2026
5ae8cda
fix(ci): skeptic gate — pipe check-runs JSON through jq (gh --jq --ar…
Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
322 changes: 322 additions & 0 deletions .github/workflows/evidence-gate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,322 @@
name: Evidence Gate

# Portable install (scripts/gates/install-gates.sh): validates PR Evidence bundles.
# Upstream: jleechanorg/agent-orchestrator .github/workflows/evidence-gate.yml

on:
pull_request:
types: [opened, synchronize, edited, reopened]

concurrency:
group: evidence-gate-${{ github.event.pull_request.number }}
cancel-in-progress: true

permissions:
contents: read
pull-requests: read
issues: read

jobs:
evidence-gate:
name: Evidence Gate
runs-on: ubuntu-latest
# bd-fisn: Skip entirely when PR is merged or closed — a merged PR stops receiving
# pull_request events so a stale failed check run cannot be overwritten. Instead of
# leaving a permanent failure that blocks the PR from showing green, we exit 0.
# Evidence gate is a pre-merge gate; post-merge it has no function.
if: github.event.pull_request.merged == false && github.event.action != 'closed'
steps:
- uses: actions/checkout@v4.1.1

- name: Write PR body to temp file
env:
PR_BODY: ${{ github.event.pull_request.body }}
run: |
# Use the PR body from the workflow context — available without auth scope
# Skip gracefully if PR body is empty (valid GitHub state — no evidence bundle)
printf '%s' "$PR_BODY" > "$RUNNER_TEMP/pr_body.txt"
echo "Body fetched: ${#PR_BODY} chars"
if [ ${#PR_BODY} -eq 0 ]; then
echo "PR body is empty — treating as no evidence bundle (workflow skips)"
echo "found=false" >> "$GITHUB_OUTPUT"
echo "skip=true" >> "$GITHUB_OUTPUT"
fi

- name: Check for evidence bundle in PR body
id: check
run: |
# If body was empty, skip was already set
if [ "${{ steps.check.outputs.skip }}" = "true" ]; then
echo "Skipping — empty PR body"
exit 0
fi
Comment on lines +45 to +52

Copilot AI Apr 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The step output reference is self-referential and will never be set: within the step with id: check, it reads steps.check.outputs.skip, but no prior step with id check has produced that output. As written, the “empty PR body” skip path will not work (and this line will always evaluate to empty/false). Give the PR-body step an id and read steps.<id>.outputs.skip, or remove the skip mechanism entirely and use a single found/missing path.

Copilot uses AI. Check for mistakes.
Comment on lines +45 to +52

@coderabbitai coderabbitai Bot Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Bug: Self-referencing step output that doesn't exist.

Line 49 references steps.check.outputs.skip, but this step (id: check) is the current step — it cannot read its own outputs. The skip output was set in the previous step (lines 41-42), which has no id:. This condition will always evaluate to false, bypassing the intended skip logic.

The static analysis tool correctly flagged this: property "check" is not defined in object type {}.

🐛 Proposed fix: Add id to the previous step and reference it correctly 
       - name: Write PR body to temp file
+        id: write_body
         env:
           PR_BODY: ${{ github.event.pull_request.body }}
         run: |
           # Use the PR body from the workflow context — available without auth scope
           # Skip gracefully if PR body is empty (valid GitHub state — no evidence bundle)
           printf '%s' "$PR_BODY" > "$RUNNER_TEMP/pr_body.txt"
           echo "Body fetched: ${`#PR_BODY`} chars"
           if [ ${`#PR_BODY`} -eq 0 ]; then
             echo "PR body is empty — treating as no evidence bundle (workflow skips)"
             echo "found=false" >> "$GITHUB_OUTPUT"
             echo "skip=true" >> "$GITHUB_OUTPUT"
           fi

       - name: Check for evidence bundle in PR body
         id: check
         run: |
           # If body was empty, skip was already set
-          if [ "${{ steps.check.outputs.skip }}" = "true" ]; then
+          if [ "${{ steps.write_body.outputs.skip }}" = "true" ]; then
             echo "Skipping — empty PR body"
             exit 0
           fi
🧰 Tools
🪛 actionlint (1.7.11)

[error] 47-47: property "check" is not defined in object type {}

(expression)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/evidence-gate.yml around lines 45 - 52, The previous step
that sets the "skip" output is missing an id, and the current step incorrectly
references its own id; give the earlier step a stable id (e.g., id: detect_skip
or id: check_pre) and change the conditional to reference
steps.<that_id>.outputs.skip (e.g., steps.detect_skip.outputs.skip) instead of
steps.check.outputs.skip; ensure the earlier step emits the output under that id
(using the workflow output mechanism your runner expects) so the conditional can
read it correctly.

@cursor cursor Bot Apr 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip outputs reference wrong step missing id

Low Severity

The "Write PR body to temp file" step (line 31) writes skip=true and found=false to $GITHUB_OUTPUT but has no id, making those outputs inaccessible. The next step (id: check, line 46) references steps.check.outputs.skip — its own not-yet-set output — so the skip guard on line 49 never triggers. This is dead code; the empty-body case still correctly falls through to found=false via the grep path.

Fix in Cursor Fix in Web

@cursor cursor Bot Apr 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-referencing step output makes skip logic unreachable

Medium Severity

The "Write PR body to temp file" step (no id) writes skip=true to $GITHUB_OUTPUT, but those outputs are inaccessible without a step id. The next step (id: check) then checks steps.check.outputs.skip — a self-reference to its own not-yet-set output, which is always empty. This makes the skip guard dead code. Critically, if someone "fixes" this by wiring the reference correctly and adding an id, the exit 0 on line 51 would leave found unset, causing all downstream if: steps.check.outputs.found == 'true' and == 'false' conditions to be false — silently passing the evidence gate with no validation at all.

Additional Locations (1)
Fix in Cursor Fix in Web


Comment on lines +36 to +53

Copilot AI Apr 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow currently treats an empty PR body as a reason to "skip" the evidence gate (writing skip=true and exiting 0). That contradicts the later enforcement step (“MUST fail when Evidence section is absent”) and creates a bypass where a PR can pass the required check by leaving the body empty. Remove the skip-on-empty behavior and instead fail when the Evidence section (and required fields) are missing, even if the body is empty.

Suggested change
# Skip gracefully if PR body is empty (valid GitHub state — no evidence bundle)
printf '%s' "$PR_BODY" > "$RUNNER_TEMP/pr_body.txt"
echo "Body fetched: ${#PR_BODY} chars"
if [ ${#PR_BODY} -eq 0 ]; then
echo "PR body is empty — treating as no evidence bundle (workflow skips)"
echo "found=false" >> "$GITHUB_OUTPUT"
echo "skip=true" >> "$GITHUB_OUTPUT"
fi
- name: Check for evidence bundle in PR body
id: check
run: |
# If body was empty, skip was already set
if [ "${{ steps.check.outputs.skip }}" = "true" ]; then
echo "Skipping — empty PR body"
exit 0
fi
printf '%s' "$PR_BODY" > "$RUNNER_TEMP/pr_body.txt"
echo "Body fetched: ${#PR_BODY} chars"
- name: Check for evidence bundle in PR body
id: check
run: |

Copilot uses AI. Check for mistakes.
BODY=$(cat "$RUNNER_TEMP/pr_body.txt")

if printf '%s' "$BODY" | grep -qi '^[[:space:]]*## Evidence'; then
echo "found=true" >> "$GITHUB_OUTPUT"
else
echo "found=false" >> "$GITHUB_OUTPUT"
fi

- name: Fail when no Evidence section found
# This step MUST fail when no Evidence section is present. Exiting 0 (skipping)
# enables a silent bypass of the merge gate: workers omit the ## Evidence section
# from the PR body, the workflow passes, and gh pr merge --auto merges without
# CodeRabbit approval or any human review. Branch protection has no required checks,
# so the evidence gate is the only enforcement. Failing hard here is the correct
# behavior — it prevents unevidenced PRs from sneaking through the merge gate.
if: steps.check.outputs.found == 'false'
run: |
echo "ERROR: No ## Evidence section found in PR body."
echo "The evidence gate MUST fail when Evidence section is absent —"
echo "skipping allows workers to bypass the merge gate by omitting Evidence."
echo "Fix: add a ## Evidence section with a Claim class and verdict to the PR body."
exit 1

- name: Validate evidence bundle format
if: steps.check.outputs.found == 'true'
run: |
BODY=$(cat "$RUNNER_TEMP/pr_body.txt")

# Extract claim class — normalize to lowercase
# Support two common markdown formats:
# - **Claim class**: pipeline-e2e (bold inline — colon INSIDE bold)
# - **Claim class**: PR-lifecycle E2E (with parenthetical — colon INSIDE bold)
# grep -i '\*\*Claim class' matches bold-markdown lines
# | grep -v '^- ' filters list bullets (extracted separately below)
# First sed: strip markdown and extract text after "**Claim class**: " (space after colon)
# Supports: **Claim class**: value (colon inside bold, space after colon)
# Second sed: strip any trailing parenthetical or extra text
# tr -d '*' removes markdown bold markers; tr '[:upper:]' '[:lower:]' lowercases
# tr ' ' '-' converts remaining spaces to hyphens so "Pipeline E2E" → "pipeline-e2e"
# tr -d '[:space:]' removes ALL whitespace (not just leading/trailing) — collapses
# multi-word names cleanly; sed then strips leading/trailing characters
# sed 'I' flag makes the pattern case-insensitive so "**Claim Class**:" (title case)
# is also correctly stripped even though the pattern uses lowercase 'c'
CLAIM=$(printf '%s' "$BODY" \
| grep -i '\*\*Claim class' | grep -v '^- ' | head -1 \
| sed 's/.*\*\*Claim class\*\*: *//I' \
| sed 's/(.*//' \
| tr -d '*' | tr '[:upper:]' '[:lower:]' \
| tr ' ' '-' | tr -d '\t' \
| sed 's/^[ \t-]*//;s/[ \t-]*$//')

# Fallback: also try extracting from list-bullet format "- **Claim class**: ..."
if [ -z "$CLAIM" ]; then
CLAIM=$(printf '%s' "$BODY" \
| grep -i '^-.*\*\*Claim class' | head -1 \
| sed 's/.*\*\*Claim class\*\*: *//I' \
| sed 's/(.*//' \
| tr -d '*' | tr '[:upper:]' '[:lower:]' \
| tr ' ' '-' | tr -d '\t' \
| sed 's/^[ \t-]*//;s/[ \t-]*$//')
fi

echo "Claim class detected: $CLAIM"

# Validate claim class is recognized
# Accept both short forms (unit, integration, merge-gate) used internally
# and the longer forms documented in CLAUDE.md (unit-test-coverage,
# integration-test, merge-gate-green) which normalize to hyphenated strings
# after tr ' ' '-' transforms "Unit test coverage" → "unit-test-coverage"
case "$CLAIM" in
unit|unit-test-coverage|unit-test)
CLAIM="unit"
echo "Validated claim class: unit"
;;
integration|integration-test)
CLAIM="integration"
echo "Validated claim class: integration"
;;
pipeline-e2e)
echo "Validated claim class: $CLAIM"
;;
pr-lifecycle-e2e|pr-lifecycle)
CLAIM="pr-lifecycle-e2e"
echo "Validated claim class: pr-lifecycle-e2e"
;;
merge-gate|merge-gate-green)
CLAIM="merge-gate"
echo "Validated claim class: merge-gate"
;;
*)
echo "ERROR: Unrecognized claim class: '$CLAIM'"
echo "Valid classes: unit, integration, pipeline-e2e, pr-lifecycle-e2e, merge-gate"
echo "(Also accepted: unit-test-coverage, integration-test, merge-gate-green — CLAUDE.md forms)"
exit 1
Comment on lines +118 to +147

Copilot AI Apr 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow comments and error output reference “CLAUDE.md forms”, but this repository does not contain a CLAUDE.md. This is misleading for contributors trying to fix a failing evidence gate. Either add the referenced documentation file to the repo or update the wording to point at an existing doc (or just list the accepted claim classes without referencing a non-existent file).

Copilot uses AI. Check for mistakes.
;;
esac

- name: Validate PR-lifecycle-e2e required proofs
if: steps.check.outputs.found == 'true'
run: |
BODY=$(cat "$RUNNER_TEMP/pr_body.txt")

CLAIM=$(printf '%s' "$BODY" \
| grep -i '\*\*Claim class' | grep -v '^- ' | head -1 \
| sed 's/.*\*\*Claim class\*\*: *//I' \
| sed 's/(.*//' \
| tr -d '*' | tr '[:upper:]' '[:lower:]' \
| tr ' ' '-' | tr -d '\t' \
| sed 's/^[ \t-]*//;s/[ \t-]*$//')

# Fallback: also try list-bullet format
if [ -z "$CLAIM" ]; then
CLAIM=$(printf '%s' "$BODY" \
| grep -i '^-.*\*\*Claim class' | head -1 \
| sed 's/.*\*\*Claim class\*\*: *//I' \
| sed 's/(.*//' \
| tr -d '*' | tr '[:upper:]' '[:lower:]' \
| tr ' ' '-' | tr -d '\t' \
| sed 's/^[ \t-]*//;s/[ \t-]*$//')
fi

# Normalize CLAUDE.md long forms to canonical short forms
case "$CLAIM" in
unit-test-coverage|unit-test) CLAIM="unit" ;;
integration-test) CLAIM="integration" ;;
pr-lifecycle) CLAIM="pr-lifecycle-e2e" ;;
merge-gate-green) CLAIM="merge-gate" ;;
esac

if [ "$CLAIM" = "pr-lifecycle-e2e" ]; then
echo "Checking required proofs for: $CLAIM"

# Proof 1: PR creation — must have GitHub PR URL
if ! printf '%s' "$BODY" | grep -qiE 'https://github\.com/[^/]+/[^/]+/pull/[0-9]+'; then
echo "ERROR: Missing PR creation URL in evidence bundle"
exit 1
fi
echo "PR creation proof: PASS"

# Proof 2: Transition — require structured evidence of CI/review activity
if ! printf '%s' "$BODY" | grep -qiE '(transition proof|CI timeline|review timeline|check run|workflow run|codecov|status check)'; then
echo "ERROR: Missing transition proof (CI/review timeline) in evidence bundle"
exit 1
fi
echo "Transition proof: PASS"

# Proof 3: Merge outcome — structured phrase required
if ! printf '%s' "$BODY" | grep -qiE '(commit SHA|mergeable|merged|merge commit)'; then
echo "ERROR: Missing merge outcome proof in evidence bundle"
exit 1
fi
echo "Merge outcome proof: PASS"

# Proof 4: Cleanup — structured phrase required
if ! printf '%s' "$BODY" | grep -qiE '(cleanup proof|branch.*deleted|session.*kill|worktree.*remov)'; then
echo "ERROR: Missing cleanup proof in evidence bundle"
exit 1
fi
echo "Cleanup proof: PASS"

echo "All 4 required proofs present for $CLAIM"
elif [ "$CLAIM" = "merge-gate" ]; then
echo "Checking required proofs for: $CLAIM (merge-gate = all 7 green conditions)"
# merge-gate requires evidence of all 7 merge-gate conditions per CLAUDE.md:
# 1. CI green
if ! printf '%s' "$BODY" | grep -qiE '(CI (passing|success|green)|checks (passing|success|green))'; then
echo "ERROR: Missing CI-green proof in merge-gate evidence"
exit 1
fi
echo "Condition 1/7 (CI green): PASS"

# 2. No merge conflicts
if ! printf '%s' "$BODY" | grep -qiE '(mergeable|MERGEABLE|conflict-free|no conflict)'; then
echo "ERROR: Missing mergeable-state proof in merge-gate evidence"
exit 1
fi
echo "Condition 2/7 (no merge conflicts): PASS"

# 3. CR approved
if ! printf '%s' "$BODY" | grep -qiE '(CodeRabbit APPROVED|CR APPROVED|crab.*approved)'; then
echo "ERROR: Missing CR-approved proof in merge-gate evidence"
exit 1
fi
echo "Condition 3/7 (CR approved): PASS"

# 4. Bugbot finished
if ! printf '%s' "$BODY" | grep -qiE '(Bugbot finished|Cursor Bugbot|bugbot (finished|neutral|success))'; then
echo "ERROR: Missing Bugbot-finished proof in merge-gate evidence"
exit 1
fi
echo "Condition 4/7 (Bugbot finished): PASS"

# 5. All threads resolved
if ! printf '%s' "$BODY" | grep -qiE '(all threads resolved|threads resolved|no unresolved threads)'; then
echo "ERROR: Missing all-threads-resolved proof in merge-gate evidence"
exit 1
fi
echo "Condition 5/7 (all threads resolved): PASS"

# 6. Evidence review passed
if ! printf '%s' "$BODY" | grep -qiE '(evidence review passed|/er PASS|evidence-gate PASS)'; then
echo "ERROR: Missing evidence-review-passed proof in merge-gate evidence"
exit 1
fi
echo "Condition 6/7 (evidence review passed): PASS"

# Condition 7/7: Skeptic gate CI passed
# Check both PR body (evidence bundle) AND skeptic-agent verdict comments.
# The verdict may be in the evidence bundle text OR posted by skeptic-gate
# as a PR comment (<!-- skeptic-agent-verdict -->). Either source is valid.
SKEPTIC_PASSED=false
if printf '%s' "$BODY" | grep -qiE 'skeptic.*(verdict|gate).*pass'; then
SKEPTIC_PASSED=true
else
# Fallback: check skeptic-agent verdict comments via GitHub API
# Filter by author (app/skeptic-agent) and HTML marker to prevent spoofing
SKEPTIC_COMMENT=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
--jq '[.[] | select(
(.user.login == "app/skeptic-agent" and (.body | test("<!--\\s*skeptic-agent-verdict\\s*-->"; "i")))
or (.user.login == "github-actions[bot]" and (.body | test("skeptic-gate-result"; "i")))
)] | sort_by(.created_at) | reverse | .[0].body // ""' 2>/dev/null)
if printf '%s' "$SKEPTIC_COMMENT" | grep -qiE 'VERDICT:\\s*PASS|\\*\\*Result:\\s*PASS\\*\\*'; then
Comment on lines +264 to +275

Copilot AI Apr 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gh api is used to fetch PR comments when validating merge-gate evidence, but this step never sets GITHUB_TOKEN/GH_TOKEN in the environment. On GitHub-hosted runners, gh will fail to authenticate unless a token env var is provided. Add env: { GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} } (or GH_TOKEN: ${{ github.token }}) to this step (and any other steps invoking gh).

Copilot uses AI. Check for mistakes.

@cursor cursor Bot Apr 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grep regex double-backslash won't match verdict text

High Severity

The grep -qiE pattern 'VERDICT:\\s*PASS|\\*\\*Result:\\s*PASS\\*\\*' uses doubled backslashes inside single quotes. In GNU ERE, \\ matches a literal backslash, so \\s* matches a backslash followed by zero or more s characters — not whitespace. The skeptic-gate comment outputs VERDICT: PASS (with a space), which this pattern will never match. Similarly, \\*\\* matches zero or more backslashes, not literal ** in markdown. The fallback check for merge-gate condition 7/7 is effectively dead code, causing false failures when the PR body doesn't contain the primary skeptic.*(verdict|gate).*pass pattern.

Fix in Cursor Fix in Web

SKEPTIC_PASSED=true
fi
fi
Comment on lines +268 to +278

@coderabbitai coderabbitai Bot Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for other references to skeptic-agent login format in the codebase
rg -n -C2 'skeptic-agent' --type yaml

Repository: jleechanorg/mctrl_test

Length of output: 214

🌐 Web query:

What is the login format for GitHub App bots in API responses?

💡 Result:

The login format for GitHub App bots (app installations) in API responses, such as in issue authors or actors, is the app's slug (the URL-friendly name of the GitHub App). This is indicated in official docs where API schemas for fields like performed_via_github_app include the app object with a slug field, and login fields in user-like objects for apps reference the app identifier. When an app installation acts, the actor is attributed to the app itself using its slug as the login in responses like issues or events. For example, in REST API issue responses, the performed_via_github_app shows the app details, and the effective login for attribution is the app slug.

Citations:

🌐 Web query:

GitHub App bot login format issue comments "app/skeptic-agent" vs "[bot]" suffix

💡 Result:

GitHub Apps and bot accounts display differently in issue comments. GitHub Apps (installed on repositories) typically appear as "[bot]" (e.g., "github-actions[bot]") when posting comments via installation access tokens. User accounts running GitHub Actions bots also use this [bot] suffix. The "app/skeptic-agent" format likely refers to the internal API representation in the performed_via_github_app field (where slug is "skeptic-agent"), not the visible author login. No sources confirm "app/" as a display format for comment authors; official docs show user.login as the app's slug with [bot] suffix. "Skeptic-agent" appears in AI agent/skill contexts, possibly a custom app name without public issues showing the format.

Citations:

🌐 Web query:

GitHub Actions bot login format in API responses github-actions[bot]

💡 Result:

The GitHub Actions bot has the login "github-actions[bot]" in GitHub API responses. This is the username format used for commits and actions performed by GitHub Actions workflows. To access its details via the GitHub REST API, use the endpoint GET /users/github-actions%5Bbot%5D (URL-encoded [ as %5B and ] as %5D), which returns user information including id: 41898282, display name: "github-actions[bot]", and email format: "41898282+github-actions[bot]@users.noreply.github.com". This format ensures proper attribution with the correct avatar and bot indicator in GitHub's UI, such as commit histories and pull requests.

Citations:

Fix the app login format for skeptic-agent on line 272.

The check for .user.login == "app/skeptic-agent" is incorrect. GitHub App bot logins use the format <app-slug>[bot] (e.g., skeptic-agent[bot]), not app/<app-slug>. This condition will never match when the skeptic-agent bot posts comments. Change it to .user.login == "skeptic-agent[bot]" to align with the correct format used by GitHub Apps in API responses, consistent with how github-actions[bot] is already handled on line 273.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/evidence-gate.yml around lines 268 - 278, The check that
builds SKEPTIC_COMMENT filters comments by `.user.login == "app/skeptic-agent"`,
which never matches GitHub App bot usernames; update that predicate to
`.user.login == "skeptic-agent[bot]"` so it matches the app comment author
format used by the API (keep the rest of the jq filter, tests for the HTML
marker, and the github-actions[bot] branch unchanged).

if [ "$SKEPTIC_PASSED" = "false" ]; then
echo "ERROR: Missing Condition 7/7 (skeptic gate PASS) in evidence bundle or skeptic-agent comment"
exit 1
fi
echo "Condition 7/7 (skeptic gate PASS): PASS"
echo "All 7 merge-gate conditions verified for $CLAIM"
else
echo "No additional validation required for claim class: $CLAIM"
fi

- name: Validate verdict is present and consistent
if: steps.check.outputs.found == 'true'
run: |
BODY=$(cat "$RUNNER_TEMP/pr_body.txt")

# Extract from "## Evidence" to EOF to avoid false matches from text that
# appears before the Evidence section (e.g. "Previous PR's verdict: PASS"
# in the Background or Goals sections). Extracting to EOF ensures that a
# "## Verdict:" section header following ## Evidence is also captured.
EVIDENCE_SECTION=$(printf '%s' "$BODY" \
| sed -n '/^[[:space:]]*## [Ee]vidence/,$p')

# Fallback to full body if Evidence section could not be isolated
if [ -z "$EVIDENCE_SECTION" ]; then
EVIDENCE_SECTION="$BODY"
fi

# Check for PASS verdict — scoped to Evidence section
if printf '%s' "$EVIDENCE_SECTION" | grep -qi '[Vv]erdict.*:.*[Pp][Aa][Ss][Ss]'; then
echo "Verdict: PASS — evidence gate passes"

# Check for INSUFFICIENT verdict — gate passes; missing proof correctly triggers this
elif printf '%s' "$EVIDENCE_SECTION" | grep -qi '[Vv]erdict.*:.*[Ii][Nn][Ss][Uu][Ff][Ff][Ii][Cc][Ii][Ee][Nn][Tt]'; then
echo "Verdict: INSUFFICIENT — gate passes (INSUFFICIENT is a valid fail-closed verdict)"

# Check for FAIL verdict — gate passes; claiming FAIL with a present bundle is contradictory
elif printf '%s' "$EVIDENCE_SECTION" | grep -qi '[Vv]erdict.*:.*[Ff][Aa][Ii][Ll]'; then
echo "Verdict: FAIL with present bundle — contradiction; gate passes but this bundle should be re-examined"

# No verdict found — hard requirement, gate fails
else
echo "ERROR: No verdict found in evidence bundle. Verdict is a mandatory field — missing verdict fails the evidence gate."
exit 1
fi
Loading
Loading