Skip to content

feat(fleet-monitor): track high-failure workflows as Issues with dev-lead label#231

Merged
don-petry merged 7 commits into
mainfrom
feat/fleet-monitor-high-failure-issues
May 18, 2026
Merged

feat(fleet-monitor): track high-failure workflows as Issues with dev-lead label#231
don-petry merged 7 commits into
mainfrom
feat/fleet-monitor-high-failure-issues

Conversation

@don-petry

@don-petry don-petry commented May 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Enhances the Actions Fleet Monitor to find/update or create a dedicated GitHub Issue for every workflow whose failure rate exceeds 10%, and applies the dev-lead label automatically.

Changes

scripts/fleet_monitor.sh

  • New section 3b runs after metrics collection (before temp-file cleanup).
  • Uses jq to parse the raw metrics TSV and emit fleet_high_failure.json — an array of all workflows with rate_int > 10 (covers WARNING ≥10%, DEGRADED ≥20%, CRITICAL ≥50%, ERROR).
  • Exports HIGH_FAILURE_COUNT to GITHUB_ENV for downstream logging.

.github/workflows/actions-fleet-monitor.yml

  • New step: Track high-failure workflows as Issues (runs after Run fleet monitor, before Annotate clean run).
  • Reads fleet_high_failure.json; no-ops if the file is absent or empty.
  • Creates dev-lead and fleet-tracker labels on first run (idempotent — 422 ignored).
  • For each high-failure item:
    • Searches for an existing open Issue titled [Fleet Monitor] repo — workflow.yml
    • Updates body + ensures dev-lead / fleet-tracker labels if found
    • Creates a new Issue with those labels if not found
  • Emits core.notice annotations for every created/updated Issue URL.

Testing

Triggered a workflow_dispatch run with lookback_days=30 to surface real CRITICAL items. See linked Issue for results.

Labels created

Label Color Purpose
dev-lead 🔴 #d93f0b Requires dev-lead attention
fleet-tracker 🔵 #0075ca Tracked by the Actions Fleet Monitor

Summary by CodeRabbit

  • New Features
    • Workflows exceeding failure thresholds are now automatically tracked as repository issues with deterministic titles and applied labels for easier triage.
    • Issues are created or updated with per-workflow metrics (failure counts/rates, duration percentiles) and links to runs for investigation.
    • A high-failure JSON report is produced after fleet metrics run and the high-failure count is exported for use in subsequent steps.

Review Change Stack

don-petry added 2 commits May 18, 2026 09:22
…issue tracking

Adds a section (3b) that writes fleet_high_failure.json after metrics collection.
The file contains every workflow with failure rate > 10%, ready for the
downstream GitHub Actions step to find/update or create tracked Issues.
…>10% failure rate

For each workflow whose failure rate exceeds 10%, the new step:
- Searches for an existing open Issue titled '[Fleet Monitor] repo — workflow.yml'
- Updates the body and ensures dev-lead + fleet-tracker labels if found
- Creates a new Issue with those labels if not found

Labels dev-lead and fleet-tracker are created on first run (idempotent).
Copilot AI review requested due to automatic review settings May 18, 2026 14:22
@coderabbitai

coderabbitai Bot commented May 18, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@don-petry has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 12 minutes and 26 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 419b4812-5b59-4e05-af7c-b35939ac3cbe

📥 Commits

Reviewing files that changed from the base of the PR and between 87237ff and 8c00a45.

📒 Files selected for processing (2)
  • .github/workflows/actions-fleet-monitor.yml
  • scripts/fleet_monitor.sh
📝 Walkthrough

Walkthrough

This PR extends the fleet monitor workflow to automatically track high-failure workflows as GitHub issues. The script filters workflow metrics by failure threshold and severity labels, outputting a curated list; the workflow step then reads that list and creates or updates corresponding tracking issues with relevant metrics and labels.

Changes

High-Failure Workflow Issue Tracking

Layer / File(s) Summary
High-failure workflow data generation
scripts/fleet_monitor.sh
Adds a post-processing stage that parses the metrics TSV, filters workflows with failure rate >10% and severity labels (WARNING/DEGRADED/CRITICAL/ERROR), sorts by failure rate, writes fleet_high_failure.json, and exports HIGH_FAILURE_COUNT to GITHUB_ENV.
Workflow issue creation and updates
.github/workflows/actions-fleet-monitor.yml
Changes checkout ref to github.ref_name and adds a GitHub Script step that reads fleet_high_failure.json, ensures dev-lead and fleet-tracker labels exist, and upserts deterministic “[Fleet Monitor] …” issues per entry (update body/labels for existing open issues or create new issues with dev-lead, fleet-tracker, health-check).

Sequence Diagram

sequenceDiagram
  participant Script as fleet_monitor.sh
  participant Artifact as fleet_high_failure.json
  participant Workflow as actions-fleet-monitor
  participant GH as GitHub API
  Script->>Script: Parse metrics TSV, filter by rate>10% & severity
  Script->>Artifact: Write high-failure workflows array
  Workflow->>Artifact: Read fleet_high_failure.json
  Workflow->>GH: Ensure labels exist (dev-lead, fleet-tracker)
  loop For each high-failure workflow
    Workflow->>GH: Search for existing open issue by title
    alt Issue found
      Workflow->>GH: Update issue body and labels
    else Issue not found
      Workflow->>GH: Create new issue with title, body, labels
    end
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

Possibly related PRs

  • petry-projects/.github-private#197: Prior PR that modified the fleet monitor baseline; this PR extends that work to emit fleet_high_failure.json and add issue upsert logic.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: enhancing fleet-monitor to automatically track and create/update GitHub Issues for high-failure workflows with dev-lead labeling.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/fleet-monitor-high-failure-issues

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — human-pr (no-changes)

No changes were needed for this PR.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Engine ran but made no changes.

1 similar comment
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Engine ran but made no changes.

coderabbitai[bot]
coderabbitai Bot previously requested changes May 18, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/actions-fleet-monitor.yml:
- Around line 144-152: The search query for
github.rest.search.issuesAndPullRequests is being built by joining terms with
'+' which gets URL-encoded and breaks the search; change the join to use a space
' ' (or simply pass a single string with spaces) so the q parameter is a proper
space-separated GitHub search query. Update the construction around const {
data: results } and the array that includes
`repo:${context.repo.owner}/${context.repo.repo}`, 'is:issue', 'is:open',
`in:title`, and `"${title}"` to join with ' ' instead of '+' (or interpolate
them into one space-separated string) and ensure the resulting q value is passed
to github.rest.search.issuesAndPullRequests.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3365d2e2-cf65-4f26-98c5-3e881521da56

📥 Commits

Reviewing files that changed from the base of the PR and between 89b7695 and 5ed089c.

📒 Files selected for processing (2)
  • .github/workflows/actions-fleet-monitor.yml
  • scripts/fleet_monitor.sh

Comment thread .github/workflows/actions-fleet-monitor.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds per-workflow Issue tracking to the Actions Fleet Monitor: any workflow with a failure rate above 10% gets a dedicated GitHub Issue labeled dev-lead / fleet-tracker, created on first detection and updated on subsequent runs.

Changes:

  • New section in scripts/fleet_monitor.sh that parses the metrics TSV with jq and emits fleet_high_failure.json containing each WARNING/DEGRADED/CRITICAL/ERROR workflow with rate_int > 10.
  • New Track high-failure workflows as Issues step in actions-fleet-monitor.yml that ensures the dev-lead and fleet-tracker labels exist, searches for an existing open Issue per repo — workflow.yml, and either updates it or creates a new one with a metrics table and run link.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
scripts/fleet_monitor.sh Adds section 3b that exports high-failure metrics rows to fleet_high_failure.json and writes HIGH_FAILURE_COUNT to GITHUB_ENV.
.github/workflows/actions-fleet-monitor.yml Adds a github-script step that reads the JSON and creates/updates per-workflow tracking issues with dev-lead/fleet-tracker labels.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

'is:open',
`in:title`,
`"${title}"`,
].join('+'),
Comment on lines +161 to +173
// Ensure dev-lead label is present (addLabels is idempotent)
await github.rest.issues.addLabels({
owner: context.repo.owner, repo: context.repo.repo,
issue_number: existing.number,
labels: ['dev-lead', 'fleet-tracker'],
});
core.notice(`Updated issue #${existing.number}: ${existing.html_url}`);
} else {
const created = await github.rest.issues.create({
owner: context.repo.owner, repo: context.repo.repo,
title,
body,
labels: ['dev-lead', 'fleet-tracker', 'health-check'],
Comment thread scripts/fleet_monitor.sh Outdated
Comment on lines +226 to +232
.rate_int > 10 and
(.label | IN("WARNING","DEGRADED","CRITICAL","ERROR"))
)) |
sort_by(.rate_int) | reverse
' < "$metrics_file" > "$HIGH_FAILURE_FILE"
hf_count=$(jq 'length' "$HIGH_FAILURE_FILE")
echo "High-failure items (>10%): ${hf_count}"
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (no-changes)

No changes were needed for the open review threads.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a new step to the fleet monitor script to export workflows with failure rates exceeding 10% into a JSON file for automated issue tracking. A logic bug was identified where workflows labeled as 'ERROR' would be excluded from the report because their failure rate is hardcoded to zero, failing the current filter threshold. Additionally, a more idiomatic and memory-efficient jq implementation was suggested for processing the metrics file.

Comment thread scripts/fleet_monitor.sh Outdated
Comment on lines +208 to +230
jq -Rsn '
[inputs | split("\n")[] | select(length > 0) | split("\t") |
select(length >= 12) |
{
sort_key: .[0],
repo: .[1],
workflow: .[2],
total: (.[3] | tonumber? // 0),
success: (.[4] | tonumber? // 0),
failed: (.[5] | tonumber? // 0),
cancelled:(.[6] | tonumber? // 0),
rate: .[7],
p50: (.[8] | tonumber? // 0),
p95: (.[9] | tonumber? // 0),
label: .[10],
rate_int: (.[11] | tonumber? // 0)
}] |
map(select(
.rate_int > 10 and
(.label | IN("WARNING","DEGRADED","CRITICAL","ERROR"))
)) |
sort_by(.rate_int) | reverse
' < "$metrics_file" > "$HIGH_FAILURE_FILE"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The jq filter logic currently excludes workflows with the ERROR label because their rate_int is hardcoded to 0 (see line 114), but the filter requires .rate_int > 10. This prevents monitor failures from being tracked as issues, which contradicts the PR description and the comments on lines 203 and 227.

Additionally, the jq command can be simplified by removing the -s (slurp) flag and using inputs to process lines directly. This is more idiomatic and avoids unnecessary memory overhead for large metrics files.

  jq -Rn '
    [inputs | select(length > 0) | split("\t") |
     select(length >= 12) |
     {
       sort_key:  .[0],
       repo:      .[1],
       workflow:  .[2],
       total:    (.[3]  | tonumber? // 0),
       success:  (.[4]  | tonumber? // 0),
       failed:   (.[5]  | tonumber? // 0),
       cancelled:(.[6]  | tonumber? // 0),
       rate:      .[7],
       p50:      (.[8]  | tonumber? // 0),
       p95:      (.[9]  | tonumber? // 0),
       label:     .[10],
       rate_int: (.[11] | tonumber? // 0)
     }] |
    map(select(
      .rate_int > 10 or .label == "ERROR"
    )) |
    sort_by(.rate_int) | reverse
  ' < "$metrics_file" > "$HIGH_FAILURE_FILE"

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (no-changes)

No changes were needed for the open review threads.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5ed089c139

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/fleet_monitor.sh Outdated
rate_int: (.[11] | tonumber? // 0)
}] |
map(select(
.rate_int > 10 and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compute the threshold from the exact failure rate

For workflows whose true failure rate is just over 10% but below 11% (for example 21 failures out of 200 runs = 10.5%), rate_int has already been truncated to 10, so this filter drops them even though the new tracker says it files issues for failure rates > 10%. Use the exact failed / total ratio, or preserve a non-truncated percentage, for this cutoff so these high-failure workflows are not silently skipped.

Useful? React with 👍 / 👎.

Comment thread scripts/fleet_monitor.sh Outdated
Comment on lines +207 to +208
if [ -s "$metrics_file" ]; then
jq -Rsn '

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the existing confidence filter before filing issues

When the lookback window has fewer than 5 runs, the report path intentionally re-labels CRITICAL rows as LOW-CONF, but this new export reads the raw metrics before that filter runs. A workflow with 1 failed run out of 1 will therefore open a dev-lead tracking issue even though the generated fleet report suppresses it as low-confidence, creating noisy issues for sparse run history; build this JSON from the filtered metrics or repeat the same < 5 run guard here.

Useful? React with 👍 / 👎.

Comment thread scripts/fleet_monitor.sh Outdated
Comment on lines +226 to +227
.rate_int > 10 and
(.label | IN("WARNING","DEGRADED","CRITICAL","ERROR"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Let ERROR sentinel rows reach the tracker

When reading runs fails, the script records an ERROR row with rate_int set to 0, but this new filter also requires .rate_int > 10, so those monitor failures are never exported even though ERROR is included in the allowed labels. In any repo/workflow where the Actions runs API is inaccessible, the fleet report still marks the monitor failure, but the new per-workflow dev-lead tracking issue is silently skipped; special-case ERROR rows or give them a rate that passes the filter.

Useful? React with 👍 / 👎.

'is:open',
`in:title`,
`"${title}"`,
].join('+'),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use spaces, not literal plus signs, in the search query

This q value is passed as an Octokit parameter, so the literal + characters from join('+') are sent as part of the search string rather than acting as URL-level spaces; GitHub search syntax expects qualifiers to be separated by whitespace. Once a matching fleet issue already exists, this can miss the existing title search and create another issue on each run, so build the query with spaces (for example join(' ')) and let Octokit encode it.

Useful? React with 👍 / 👎.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (no-changes)

No changes were needed for the open review threads.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — human-pr (no-changes)

No changes were needed for this PR.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b0d1bf613

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

repository: petry-projects/.github-private
ref: main # Always checkout .github-private@main; workflow_call callers target @main per docs
# TEST: use calling branch so feature-branch scripts are available; revert to 'main' before merge
ref: ${{ github.ref_name }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore the shared repo checkout to main

When this reusable workflow is called from another repository, github.ref_name is the caller repository's branch or tag, not a ref in petry-projects/.github-private; I checked docs/actions-fleet-monitor.md, which documents callers using petry-projects/.github-private/.github/workflows/actions-fleet-monitor.yml@main. For any caller running from a branch that does not also exist in .github-private (for example a PR branch or a repo whose default branch is not main), this checkout of repository: petry-projects/.github-private will fail before the monitor can run. Keep this pinned to main for reusable callers and use a separate testing override if needed.

Useful? React with 👍 / 👎.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Engine ran but made no changes.

…sues:write)

The DON_PETRY_BOT_GH_PAT lacks label-assignment permission on issue creation.
Switching to github.token (which has the explicit issues:write grant from the
workflow permissions block) fixes labels being silently dropped on new issues.
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — human-pr (no-changes)

No changes were needed for this PR.

coderabbitai[bot]
coderabbitai Bot previously requested changes May 18, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/actions-fleet-monitor.yml:
- Around line 53-54: The workflow is using the dynamic ref value github.ref_name
which allows scheduled or manually triggered runs to checkout feature-branch
code; change the workflow's ref key to the fixed branch name 'main' (replace the
current ref: ${{ github.ref_name }} with ref: 'main') so the fleet monitor
always checks out scripts from the main branch before merging the PR.
- Around line 88-179: The workflow file is missing the required SOURCE OF TRUTH
header; add a top-of-file comment line exactly matching the repo standard (for
example: "# SOURCE OF TRUTH:
petry-projects/.github/standards/workflows/actions-fleet-monitor.yml") so this
workflow uses the approved template verbatim, or if exempt provide a pointer to
exemption documentation; update the .github/workflows/actions-fleet-monitor.yml
contents to include that header before any YAML content (ensure formatting
matches other workflows like pr-review-mention.yml and dependency-audit.yml).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 969ec339-bb2b-44d0-a877-5b13d890e88a

📥 Commits

Reviewing files that changed from the base of the PR and between 5ed089c and 87237ff.

📒 Files selected for processing (1)
  • .github/workflows/actions-fleet-monitor.yml

Comment thread .github/workflows/actions-fleet-monitor.yml Outdated
Comment thread .github/workflows/actions-fleet-monitor.yml
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Engine ran but made no changes.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (no-changes)

No changes were needed for the open review threads.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87237ff62a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ORG: ${{ env.ORG }}
with:
# Use GITHUB_TOKEN (has explicit issues:write) for label creation and assignment
github-token: ${{ github.token }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use a non-GITHUB_TOKEN token to wake dev-lead

Using github.token here means the issue creation and addLabels calls are performed with GITHUB_TOKEN; GitHub documents that events caused by GITHUB_TOKEN do not start new workflow runs except workflow_dispatch/repository_dispatch, and this repo's Dev-Lead Agent only handles issues: labeled. In the scheduled or reusable monitor runs where a high-failure workflow should wake dev-lead, the issue will be created and labeled but the dev-lead workflow will not run; use the PAT/App token or explicitly dispatch the agent after creating the issue.

Useful? React with 👍 / 👎.

owner: context.repo.owner, repo: context.repo.repo,
title,
body,
labels: ['dev-lead', 'fleet-tracker', 'health-check'],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Target dev-lead issues at the failing repo

For the scheduled monitor, this creates every fleet-tracker issue in .github-private while item.repo can be any org repo, and adding dev-lead hands that issue to scripts/dev-lead-fix-issue.sh, which defaults REPO to GITHUB_REPOSITORY and therefore checks out/patches the issue repository rather than the failing workflow's repository. When a workflow in another repo crosses the threshold, the agent will work in the wrong codebase; create/dispatch the ticket against item.repo or avoid the automation label for centralized tracking issues.

Useful? React with 👍 / 👎.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — human-pr (no-changes)

No changes were needed for this PR.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3094b8e35d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

owner: context.repo.owner, repo: context.repo.repo,
title,
body,
labels: ['dev-lead', 'fleet-tracker', 'health-check'],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add dev-lead after creating the issue

For a brand-new tracker issue, putting dev-lead in the initial issues.create labels does not give this repo's Dev-Lead Agent an event it routes: I checked .github/workflows/dev-lead.yml, which subscribes to issues: labeled, and scripts/dev-lead-intent.sh, which skips any issues action other than labeled. In the first-time high-failure case the issue is opened already labeled, so the agent can miss the handoff; create the issue without dev-lead and then call addLabels for dev-lead, or route labeled/opened events explicitly.

Useful? React with 👍 / 👎.

don-petry and others added 2 commits May 18, 2026 12:07
scripts/fleet_monitor.sh:
- Use exact failure rate (failed*100/total) in jq instead of truncated rate_int
  preventing 10.5% workflows from being silently excluded (Copilot/Codex P2)
- Apply confidence filter: CRITICAL rows with <5 runs become LOW-CONF and are
  excluded, matching fleet_report.sh apply_confidence_filter (Codex P2)
- Handle ERROR sentinel rows: include regardless of rate_int=0 since they
  represent actionable monitor failures (Gemini/Codex)
- Switch to idiomatic jq -Rn with inputs (no -s slurp) (Gemini)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
.github/workflows/actions-fleet-monitor.yml:
- Add SOURCE OF TRUTH header block (CodeRabbit — repo standard)
- Revert checkout ref from github.ref_name back to main (CodeRabbit Critical)
- Fix join('+') → join(' ') in search query so Octokit does not URL-encode
  '+' as '%2B', breaking the find-existing-issue search (CodeRabbit Major)
- Add health-check label to addLabels update path for consistency with
  the create path (Copilot inline)
- Add 'Dispatch dev-lead for critical items' step: github.token events do
  not trigger issues:labeled workflows; use PAT repository_dispatch instead
  so Dev-Lead Agent is explicitly woken (Codex P1)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@don-petry don-petry dismissed stale reviews from coderabbitai[bot] and coderabbitai[bot] May 18, 2026 17:07

All actionable items addressed in latest commit (8c00a45):
• ref: github.ref_name → ref: main (Critical)
• join('+') → join(' ') in search query (Major)
• SOURCE OF TRUTH header added
All three CodeRabbit findings resolved.

@don-petry

Copy link
Copy Markdown
Collaborator Author

✅ All review comments addressed (commit 8c00a45)

Reviewer Finding Resolution
CodeRabbit 🔴 Critical ref: github.ref_name must be reverted to main ✅ Reverted to ref: main
CodeRabbit 🟠 Major join('+') URL-encodes as %2B, breaking issue search ✅ Changed to join(' ')
CodeRabbit Missing SOURCE OF TRUTH header ✅ Added standard header block
Copilot health-check label missing from update path ✅ Added to addLabels call
Copilot join('+') breaks Octokit search query ✅ Fixed (same as above)
Copilot rate_int integer truncation excludes 10.5% workflows ✅ Using exact failed*100/total in jq
Gemini 🔴 ERROR rows have rate_int=0, excluded by filter ✅ Special-cased: ERROR rows always included
Gemini -Rsn + split less idiomatic than -Rn + inputs ✅ Switched to jq -Rn
Codex P1 github.token won't trigger issues:labeled → Dev-Lead Agent won't fire ✅ Added Dispatch dev-lead for critical items step using PAT
Codex P2 Confidence filter not applied (CRITICAL <5 runs creates noise) ✅ LOW-CONF rows excluded before export
Codex P2 rate_int truncation (same as Copilot) ✅ Fixed
Codex P1 ref: main for reusable callers (same as CodeRabbit) ✅ Fixed

@sonarqubecloud

Copy link
Copy Markdown

@don-petry don-petry merged commit 4fcb788 into main May 18, 2026
16 checks passed
@don-petry don-petry deleted the feat/fleet-monitor-high-failure-issues branch May 18, 2026 17:09

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8c00a45f4b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gh api "repos/${{ github.repository }}/dispatches" \
-X POST \
-f event_type=dev-lead-ci-failure \
-f 'client_payload={"source":"fleet-monitor","count":"${{ env.HIGH_FAILURE_COUNT }}"}'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Build the dispatch payload as a JSON object

When HIGH_FAILURE_COUNT is nonzero, this gh api call sends client_payload via -f/--raw-field, which the GitHub CLI docs define as adding a static string parameter, while the repository dispatch endpoint requires client_payload to be an object. In that high-failure path the API call can return a 422 validation error after the tracker issues are created, failing the monitor run and preventing the intended dev-lead dispatch; use nested fields or --input/JSON so client_payload is an object.

Useful? React with 👍 / 👎.

Comment on lines +204 to +205
-f event_type=dev-lead-ci-failure \
-f 'client_payload={"source":"fleet-monitor","count":"${{ env.HIGH_FAILURE_COUNT }}"}'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Dispatch an event the dev-lead intent accepts

For fleet-tracker issues this dispatch still will not wake dev-lead: I checked scripts/dev-lead-intent.sh, and repository_dispatch handling exits with no-pr-number-in-payload before the dev-lead-ci-failure case unless client_payload.pr_number is present, then routes that event to fix-ci rather than the issue handler. Since this payload only contains source and count, high-failure tracker issues are created but the explicit wake-up is skipped; send an issue-oriented event that the classifier handles or include enough context and route it to the issue path.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants