Add code compliance audit template (spec+design → code)#42
Merged
Alan-Jowett merged 2 commits intomicrosoft:mainfrom Mar 20, 2026
Merged
Add code compliance audit template (spec+design → code)#42Alan-Jowett merged 2 commits intomicrosoft:mainfrom
Alan-Jowett merged 2 commits intomicrosoft:mainfrom
Conversation
Add a new template that audits source code against requirements and design documents for specification drift. Detects unimplemented requirements (D8), undocumented behavior (D9), and constraint violations in code (D10). New components: - Protocol: code-compliance-audit — 7-phase methodology for mapping spec claims to code behavior (spec inventory, code inventory, forward/backward traceability, constraint verification) - Template: audit-code-compliance — consumes requirements + code + optional design, produces investigation-report Extended components: - Taxonomy: specification-drift — D8-D10 labels defined, D11-D13 remain reserved for test compliance - Ranking criteria updated to include D8-D10 - Scenarios gallery updated (moved from future to existing) Closes microsoft#37 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new PromptKit template + reasoning protocol to audit source code vs. requirements/design for “code compliance drift”, extending the existing specification-drift taxonomy to cover D8–D10 and documenting the new scenario as an existing capability.
Changes:
- Added
audit-code-compliancetemplate that consumes requirements + code context (+ optional design) and produces an investigation report classified with D8–D10. - Added
code-compliance-auditreasoning protocol defining a phased audit methodology (spec inventory → code inventory → forward/backward tracing → constraint verification → classification → coverage). - Extended
specification-drifttaxonomy with D8–D10 labels and updated docs/manifest registrations accordingly.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/audit-code-compliance.md | New audit template wiring persona/protocols/taxonomy/format plus task instructions and quality checklist. |
| protocols/reasoning/code-compliance-audit.md | New 7-phase reasoning protocol for code↔spec compliance auditing and reporting. |
| taxonomies/specification-drift.md | Adds D8–D10 code-compliance drift labels; updates applicability and ranking criteria. |
| manifest.yaml | Registers the new protocol and template in the component manifest. |
| docs/scenarios.md | Moves the code-compliance audit scenario into “Existing Templates” and documents the assembled components. |
…us scoping - INCONCLUSIVE is now a confidence level (Low), not a competing label - PARTIALLY IMPLEMENTED maps to D8 with Medium confidence - D9 findings use 'None — no matching requirement identified' for spec location since undocumented behavior has no requirement by definition - Focus area scoping applies to code modules, not just requirements Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
abeltrano
added a commit
that referenced
this pull request
May 4, 2026
- ADO reply POST switched to temp-file pattern (--body @reply.json) to handle apostrophes/newlines/backslashes in real reply text; mirrors the GitHub recipe. - pr_reference param doc clarified: URL auto-detect with git-remote fallback and ambiguity prompt (covers #42 / !123 inputs). - All shell command fences in SKILL labeled bash for clarity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Alan-Jowett
pushed a commit
that referenced
this pull request
May 6, 2026
…fidelity protocol (#254) * feat(respond-to-pr-comments): add Azure DevOps Services support Extends the respond-to-pr-comments template, skill, and format to handle Azure DevOps Services PRs alongside GitHub. Platform is auto-detected from the PR URL (with git-remote fallback); the workflow shape is shared and only API recipes branch per platform. - Auto-detect platform from PR URL or git remote (handles SSH and legacy visualstudio.com hosts); prompt on ambiguity, do not guess. - ADO auth uses 'az login' + 'az rest --resource <ADO GUID>' on every call; no Personal Access Token path. - Preserve each platform's native status vocabulary in output (no cross-platform normalization). ADO uses 'fixed' (not 'resolved') per the CommentThreadStatus REST enum. - ADO reply payload uses content + parentCommentId + commentType ('text'); always set parentCommentId, including for PR-wide threads. - Filter ADO system threads (commentType 'system' or system CodeReviewThreadType properties); flag, do not auto-skip, threads with no text comments. - Conservative outdated detection: prefer ADO iteration/items API, fall back to local working tree only when HEAD matches the iteration's source-branch tip; otherwise mark unverified. - GitHub recipe paginates both reviewThreads and inner comments via follow-up cursored queries. - ADO Server / on-prem / TFS / custom hostnames are out of scope — stop with a clear message. - Update format file with per-platform status tables and add byDesign to the closed-state action summary. - Update manifest description to mention ADO Services. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(respond-to-pr-comments): address PR #253 review feedback - ADO reply POST switched to temp-file pattern (--body @reply.json) to handle apostrophes/newlines/backslashes in real reply text; mirrors the GitHub recipe. - pr_reference param doc clarified: URL auto-detect with git-remote fallback and ambiguity prompt (covers #42 / !123 inputs). - All shell command fences in SKILL labeled bash for clarity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(human-voice-fidelity): add opt-in voice-preservation guardrail protocol Adds a new opt-in guardrail protocol that preserves the user's communication style when an agent drafts externally visible text on the user's behalf. Scoped narrowly to user-authored prose; analysis, code, command output, and quoted reviewer text are exempt. Protocol features: - Pluggable voice sources (session samples, prior repo PRs, session history, org tools, explicit style notes), in priority order. - Calibrated style extraction (sentence length, hedging, technical density, openers/closers, characteristic phrases, punctuation). - Per-output self-check that bans em-dashes and a list of AI-tell phrases unless they appear in the user's own samples. - Neutral collaborative default + explicit disclosure when no voice samples are available; never claims voice match without evidence. - Output annotation requiring a Voice Calibration note. Integration: - protocols/guardrails/human-voice-fidelity.md (new, 163 lines) - manifest.yaml: register protocol under guardrails. - respond-to-pr-comments template: add protocol to frontmatter and manifest protocols list; reference the protocol when drafting reviewer replies. Out of scope (deferred): delegation matrix, work-item proposal phase, expanded format schema, and SKILL rewrite from the prior exploration branch — those will land in separate PRs if pursued. Validation: python tests/validate-manifest.py passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(human-voice-fidelity): apply to review-pull-request Adds human-voice-fidelity to the review-pull-request template, the second template that drafts text posted externally under the user's identity. Action mode of review-pull-request POSTs inline review comments and an overall review summary to GitHub via the Reviews API; both bodies are user-voice prose where AI tells (em-dashes, AI-tell phrases) would betray non-human authorship. Changes: - protocols/guardrails/human-voice-fidelity.md: add review-pull-request to applicable_to. - manifest.yaml: add human-voice-fidelity to review-pull-request protocols list. - templates/review-pull-request.md: add to frontmatter; reference the protocol in Phase 5 action-mode step where comment bodies are drafted, with the same scope note (drafted prose only; code/paths /quoted text exempt). Validation: python tests/validate-manifest.py passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Generalize Phase 1 voice-sample sourcing to multiple SCMs Replace the GitHub-only gh recipe in Phase 1 item 2 with a per-platform list (GitHub, Azure DevOps Services, GitLab, Bitbucket Cloud, Gitea/Forgejo) plus a fallback bullet for unspecified SCMs. The underlying intent is unchanged: sample 5-20 recent self-authored comment bodies, preferring inline review-comment bodies over commit messages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Generalize Phase 1 agent-history and style-notes sources to be LLM-agnostic Item 3 was Copilot-CLI-specific; reframe as 'prior agent session history' with examples for GitHub Copilot CLI (session_store_sql), Claude Code (~/.claude/projects JSONL transcripts), and Cursor/Windsurf/IDE-embedded agents, plus a fallback for other agents. Item 5 was copilot-instructions.md-only; expand the example list to also include CLAUDE.md, AGENTS.md, .cursorrules, and .windsurfrules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #254 review feedback (10 threads) Voice-fidelity protocol (4 threads): - Em-dash rule now conditional on user's own samples (T1, T5) - Bullet-list rule moved from hard rules to soft rules (T6) - Add consent and confidentiality requirements before sampling agent transcripts or org-tool history (T9) respond-to-pr-comments template/skill (6 threads): - Soften ADO threads pagination claim and add defensive continuationToken handling (T2) - URL-encode {project} and {repoName} in all ADO az rest example URIs via {projectEnc}/{repoNameEnc} placeholders (T3, T4, T10) - Replace GitLab-style !123 PR id notation with bare 123 or ado:123 prefix (T7) - Rename 'all pending' selector to 'all open' to avoid collision with ADO's distinct 'pending' status (T8) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #254 second-pass review feedback ADO status enum casing (5 threads): - Replace 'lowercase API enum values' wording in template, skill, PATCH instructions, and checklist with 'exact case-sensitive enum values' noting wontFix and byDesign are camelCase Voice fidelity (2 threads): - Tighten source 4 (org communication tools) to require explicit, per-session opt-in; default behavior is to skip - Tighten Consent block: disclose what, approximate volume, and that consent does not carry between sessions - Clarify Voice Calibration note placement: reported in agent chat output / action summary, NOT inserted into the produced format artifact (prevents format drift in pr-comment-responses and investigation-report) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert: remove unrelated cli/specs/audit-2026-03-30* files These were accidentally included via git add -A in 7650489. They are not part of the human-voice-fidelity protocol work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #254 third-pass review feedback (5 threads) - Soften voice-fidelity self-check paraphrases in template, manifest, and review-pull-request to reflect the protocol's conditional rules (avoid restating the rules to prevent future drift) - Switch ADO/GitHub status tables in pr-comment-responses format to API enum literals for consistency with surrounding text - Add human-voice-fidelity reference to SKILL.md draft-reply step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(respond-to-pr-comments): warn against shell escape sequences in reply text Serialization to JSON may preserve shell-specific character escape sequences literally rather than the intended Unicode character. Add guidance to use literal UTF-8 characters directly in reply text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #254 fourth-pass review feedback (6 threads) - Phase 1: make `ado:` prefix an explicit platform override that bypasses remote inspection; carry only the numeric `prId` - Phase 2 GitHub: reference concrete GraphQL fields (`isResolved`/`isOutdated`) instead of a non-existent `state` field - Phase 2 ADO: reuse Phase 1 coordinates instead of re-parsing the URL - Format: reframe GitHub status table as derived workflow labels (`open`/`outdated`/`resolved`); cascade `pending`->`open` in skill - Format: standardize per-thread placeholder examples to lowercase code literals matching the Thread Summary tables - SKILL.md: mirror the `ado:` prefix handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new template that audits source code against requirements and design documents for code compliance drift — gaps between what was specified and what was built. Closes #37.
New Components
Extended Components
New Drift Types (D8–D10)
D11–D13 remain reserved for the future \�udit-test-compliance\ template (#38).
Reused Components
Validation