Skip to content

fix(compliance): unbreak daily re-trigger sweep + throttle to one issue per repo#432

Merged
don-petry merged 2 commits into
mainfrom
feat/compliance-audit-retrigger-dev-lead
Jun 10, 2026
Merged

fix(compliance): unbreak daily re-trigger sweep + throttle to one issue per repo#432
don-petry merged 2 commits into
mainfrom
feat/compliance-audit-retrigger-dev-lead

Conversation

@don-petry

@don-petry don-petry commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the daily compliance re-trigger sweep, which had been failing every day since 2026-06-05 and leaving ~107 stale dev-lead compliance issues stuck across the fleet.

Closes #431.

Changes

  1. Unbreak the search query — GitHub now rejects search/issues queries that omit is:issue/is:pull-request with HTTP 422. Added is:issue to the query in scripts/compliance-retrigger.sh.
  2. Throttle to one engagement per repo per run — process issues oldest-first (sort=created&order=asc) and re-trigger at most one issue per repo, deferring the rest to later daily sweeps. A repo already active (open dev-lead PR or in-progress issue) consumes its slot. Prevents the fleet-wide burst (63 concurrent dev-lead runs in dry-run) that caused rebase storms and token exhaustion — the failure mode that stranded the work originally.
  3. Move sweep to 12:00 AM Central (0 5 * * *, 05:00 UTC) so it runs off-peak.

Validation

workflow_dispatch dry-runs on this branch:

Run Result
27272861932 (query fix) ✅ success — query returns backlog (106), 63 would re-trigger
27273144678 (throttle) ✅ success — 5 re-triggered / 3 skipped / 92 deferred (≤1 per repo)

First green retrigger runs since 2026-06-05.

Note on DST

GitHub cron is fixed UTC and does not observe DST; 05:00 UTC = 12:00 AM CDT (summer) / 11:00 PM CST (winter), matching the repo's existing CDT annotation convention.

Context

Part of the Compliance program initiative (GH Project #1 Initiatives) / "Compliance Blitz" effort.

Summary by CodeRabbit

  • Chores
    • Adjusted compliance retrigger workflow schedule from 14:00 UTC to 05:00 UTC
    • Enhanced compliance issue retrigger script with improved error handling and repository throttling logic

Copilot AI review requested due to automatic review settings June 10, 2026 11:40
@don-petry don-petry requested a review from a team as a code owner June 10, 2026 11:40
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: f34f2048-2309-4432-a82e-75e043aebce7

📥 Commits

Reviewing files that changed from the base of the PR and between 9aad093 and fd48a59.

📒 Files selected for processing (3)
  • .github/workflows/compliance-retrigger.yml
  • scripts/compliance-retrigger.sh
  • scripts/lib/dev-lead-retrigger.sh

📝 Walkthrough

Walkthrough

This PR refactors the dev-lead compliance re-triggering system: it converts the dev-lead library to sourced-only with improved ISO-8601 timestamp parsing, adds per-run per-repo throttling and deferred-issues metrics to the compliance script, updates both primary and legacy issue sweeps to paginate and respect engagement limits, and shifts the workflow schedule to 05:00 UTC.

Changes

Compliance Re-triggering Overhaul

Layer / File(s) Summary
Library conversion to sourced-only with timestamp parsing updates
scripts/lib/dev-lead-retrigger.sh
Shebang removed and caller-contract documentation added. In-progress label timestamp selection now uses the first matching labeled event instead of the last. Timestamp parsing replaced with ISO-8601 normalization (Z → +00:00) and Python datetime.fromisoformat() for more robust epoch conversion.
Throttling infrastructure and state tracking
scripts/compliance-retrigger.sh (header and initialization)
Header documentation expanded to describe per-repo engagement limits across both sweeps. New REPO_ENGAGED associative array and ISSUES_DEFERRED counter introduced to track and prevent multiple repo engagements per run.
Primary sweep: pagination and per-repo throttling
scripts/compliance-retrigger.sh (primary sweep)
GitHub search query updated to include is:issue qualifier and --paginate. Error detection refactored to scan all returned page objects for error shapes. Issue loop breaks on first non-stale issue (oldest-first), defers when repo already engaged, consumes engagement slot on re-trigger, and updates metrics.
Legacy sweep: pagination and throttling alignment
scripts/compliance-retrigger.sh (legacy sweep)
Legacy search query updated to require is:issue and paginate. Loop mirrors primary sweep behavior: break on first non-stale, defer if repo already engaged this run, mark repos engaged when skipping due to active dev-lead state.
Metrics tracking and reporting
scripts/compliance-retrigger.sh (output and summary)
Completion log updated to include ISSUES_DEFERRED. Console summary and GitHub step-summary table both extended to report deferred-issues count alongside retriggered and skipped metrics.
Workflow schedule update
.github/workflows/compliance-retrigger.yml
Daily cron schedule moved from 14:00 UTC (0 14 * * *) to 05:00 UTC (0 5 * * *).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

  • petry-projects/.github#431: This PR directly implements the code-level fixes described in this issue—adding is:issue to GitHub search queries, introducing per-repo throttling with engagement tracking and defer counter, and moving the workflow to 05:00 UTC.

Possibly related PRs

  • petry-projects/.github#400: Updates the same dl_dev_lead_active function and compliance-retrigger plumbing that this PR refactors, introducing the timestamp-staleness and re-engagement logic that the throttling infrastructure now builds upon.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/compliance-audit-retrigger-dev-lead

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning
Issues addressed: 0
Files changed: none
Skipped (informational): 1
- "You have reached your Codex usage limits for code reviews": usage limit notification with no code findings — not actionable
```
The bot comment contains no code issues to fix. It is a system-level notification that the Codex reviewer exhausted its quota, not a code review finding. There are zero Tier 1 CI blockers and no open review threads to resolve.

@don-petry don-petry enabled auto-merge (squash) June 10, 2026 11:41

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request migrates the legacy claude label to dev-lead across the compliance audit and re-trigger scripts, introducing a shared helper library dev-lead-retrigger.sh and a one-time migration script. Feedback on these changes highlights a starvation risk in the daily sweep due to missing pagination on the issue search, a potential script crash from fragile ISO 8601 timestamp parsing in the helper library, and an incomplete migration risk caused by the default limit of the GitHub CLI issue list command.

Comment thread scripts/compliance-retrigger.sh Outdated
Comment thread scripts/lib/dev-lead-retrigger.sh Outdated
Comment thread scripts/migrate-claude-label.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes the daily compliance re-trigger sweep by restoring a valid GitHub search/issues query and adding per-repo throttling so dev-lead is re-engaged gradually (oldest-first) without causing bursty concurrent runs. This also standardizes the trigger label to dev-lead (vs legacy claude) and updates audit/retrigger workflows accordingly.

Changes:

  • Update daily re-trigger sweep search + add “≤1 engagement per repo per run” throttling.
  • Introduce shared dev-lead re-trigger helper library and reuse it from both daily sweep and weekly audit.
  • Switch cron schedule for the sweep and update workflows/docs to use dev-lead label (plus a one-time label migration script).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
scripts/migrate-claude-label.sh Adds a one-time org migration script to move open issues from claudedev-lead and optionally delete the old label.
scripts/lib/dev-lead-retrigger.sh New shared helpers to detect active dev-lead work and cycle the trigger label safely.
scripts/compliance-retrigger.sh Fixes the org search query and adds per-repo throttling + shared helper usage.
scripts/compliance-audit.sh Updates audit issue labeling to dev-lead and re-triggers persistent findings via shared helpers.
.github/workflows/compliance-retrigger.yml Moves the scheduled run time to 05:00 UTC (midnight Central during DST).
.github/workflows/compliance-audit-and-improvement.yml Updates guidance/output plumbing to reflect dev-lead labeling and adds issues_retriggered output.

Comment thread scripts/lib/dev-lead-retrigger.sh Outdated
Comment thread scripts/compliance-retrigger.sh
Comment thread scripts/compliance-retrigger.sh Outdated
Comment thread scripts/migrate-claude-label.sh Outdated
Comment thread scripts/lib/dev-lead-retrigger.sh Outdated
…ue per repo

The daily sweep failed every run since 2026-06-05: GitHub now rejects
search/issues queries that omit is:issue/is:pull-request with HTTP 422.
Both the primary and legacy-label sweep queries omitted it. Add is:issue
to both (the primary failure aborted the whole run; ~107 stale dev-lead
issues piled up across the fleet).

Also throttle re-triggering to avoid the fleet-wide burst that stranded
the work in the first place — cycling every stale issue at once queues many
concurrent dev-lead runs in a single repo (rebase storms, token exhaustion):

- One engagement per repo per run, shared across the primary and legacy
  sweeps via a run-scoped REPO_ENGAGED set.
- Process issues oldest-first (sort=created&order=asc) so the most-stuck
  finding per repo is the one re-engaged; the daily cadence drains the rest.
- --paginate both queries so a single repo's large backlog cannot fill the
  first result page and starve every other repo (gemini review).
- Slurp-aware total_count / error-detection for paginated multi-object output.
- break (not continue) at the first non-stale issue — global ascending sort
  means no older issues remain (Copilot review).
- Mark a repo engaged regardless of cycle outcome, so a transient failure
  cannot let a second issue in the same repo fire (Copilot review).

Move the sweep to 12:00 AM Central (05:00 UTC) to run off-peak.

Closes #431.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@don-petry don-petry force-pushed the feat/compliance-audit-retrigger-dev-lead branch from b057e08 to 1b8730a Compare June 10, 2026 12:22
@don-petry don-petry disabled auto-merge June 10, 2026 12:28
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — review-changes (applied)

Changes committed and pushed.

@don-petry don-petry enabled auto-merge (squash) June 10, 2026 12:32
@don-petry don-petry merged commit d6fb9c1 into main Jun 10, 2026
14 of 16 checks passed
@don-petry don-petry deleted the feat/compliance-audit-retrigger-dev-lead branch June 10, 2026 12:32
@sonarqubecloud

Copy link
Copy Markdown

don-petry added a commit that referenced this pull request Jun 11, 2026
…ue per repo (#432)

* fix(compliance): unbreak daily re-trigger sweep + throttle to one issue per repo

The daily sweep failed every run since 2026-06-05: GitHub now rejects
search/issues queries that omit is:issue/is:pull-request with HTTP 422.
Both the primary and legacy-label sweep queries omitted it. Add is:issue
to both (the primary failure aborted the whole run; ~107 stale dev-lead
issues piled up across the fleet).

Also throttle re-triggering to avoid the fleet-wide burst that stranded
the work in the first place — cycling every stale issue at once queues many
concurrent dev-lead runs in a single repo (rebase storms, token exhaustion):

- One engagement per repo per run, shared across the primary and legacy
  sweeps via a run-scoped REPO_ENGAGED set.
- Process issues oldest-first (sort=created&order=asc) so the most-stuck
  finding per repo is the one re-engaged; the daily cadence drains the rest.
- --paginate both queries so a single repo's large backlog cannot fill the
  first result page and starve every other repo (gemini review).
- Slurp-aware total_count / error-detection for paginated multi-object output.
- break (not continue) at the first non-stale issue — global ascending sort
  means no older issues remain (Copilot review).
- Mark a repo engaged regardless of cycle outcome, so a transient failure
  cannot let a second issue in the same repo fire (Copilot review).

Move the sweep to 12:00 AM Central (05:00 UTC) to run off-peak.

Closes #431.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: apply manual instructions [skip ci-relay]

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
don-petry added a commit that referenced this pull request Jun 11, 2026
…ue per repo (#432)

* fix(compliance): unbreak daily re-trigger sweep + throttle to one issue per repo

The daily sweep failed every run since 2026-06-05: GitHub now rejects
search/issues queries that omit is:issue/is:pull-request with HTTP 422.
Both the primary and legacy-label sweep queries omitted it. Add is:issue
to both (the primary failure aborted the whole run; ~107 stale dev-lead
issues piled up across the fleet).

Also throttle re-triggering to avoid the fleet-wide burst that stranded
the work in the first place — cycling every stale issue at once queues many
concurrent dev-lead runs in a single repo (rebase storms, token exhaustion):

- One engagement per repo per run, shared across the primary and legacy
  sweeps via a run-scoped REPO_ENGAGED set.
- Process issues oldest-first (sort=created&order=asc) so the most-stuck
  finding per repo is the one re-engaged; the daily cadence drains the rest.
- --paginate both queries so a single repo's large backlog cannot fill the
  first result page and starve every other repo (gemini review).
- Slurp-aware total_count / error-detection for paginated multi-object output.
- break (not continue) at the first non-stale issue — global ascending sort
  means no older issues remain (Copilot review).
- Mark a repo engaged regardless of cycle outcome, so a transient failure
  cannot let a second issue in the same repo fire (Copilot review).

Move the sweep to 12:00 AM Central (05:00 UTC) to run off-peak.

Closes #431.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: apply manual instructions [skip ci-relay]

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(compliance): daily re-trigger sweep broken (search 422) + fleet-wide burst risk

3 participants