[syntax-error-quality] Daily Syntax Error Quality Analysis – 2026-04-05

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-04-05
Test Cases: 3
Average Score: 68.3/100
Status: ⚠️ Needs Improvement

Method: Analysis performed via compiler source-code tracing (the sandbox environment restricts binary execution). Error outputs were derived by tracing the full call chains through pkg/parser/, pkg/workflow/, and pkg/console/.

---

Executive Summary

Three workflows were analysed across three error categories. The compiler's schema-validation path (engine typos, invalid permission scopes) produces well-structured, IDE-parseable errors with source context. However, YAML syntax errors expose a structural inconsistency that causes IDEs to navigate to the wrong line, and several patterns introduce avoidable noise that reduces readability.

Key Findings:

• Strengths: "Did you mean?" suggestions for engine typos; rich source context with ^ pointer for schema errors; permissions docs link; YAML error messages translated from cryptic goccy text to plain English
• Weaknesses: YAML parse errors wrap with file:1:1: instead of the actual error line; schema validation errors repeat position information redundantly in both the IDE prefix and the message body; invalid-permissions errors may list valid scopes twice (once from appendKnownFieldValidValuesHint, once from generateSchemaBasedSuggestions)
• Critical Issues: None – all scores are above the critical threshold (55)

---

Test Case Results

Test Case 1: YAML Syntax Error (missing colon) — Score: 63/100 ⚠️

Test Configuration

Workflow: .github/workflows/artifacts-summary.md (97 lines, simple)
Error Type: Category A – Frontmatter YAML syntax error
Error Introduced: Line 9 – engine copilot (: removed → engine copilot)

Predicted Compiler Output

.github/workflows/test-1.md:1:1: error: failed to parse frontmatter:
[9:1] missing ':' after key — YAML mapping entries require 'key: value' format
>  9 | engine copilot
       ^
```

_Source trace_: `frontmatter_content.go:67` wraps the goccy error string in `fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)`. This is NOT a `FormattedParserError`, so `compiler.go:96` calls `formatCompilerError(markdownPath, "error", err.Error(), err)` which hard-codes position `1:1`.

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 18/25 | Good |
| Actionability | 20/25 | Good |
| Context | 13/20 | Limited |
| Examples | 3/15 | Poor |
| Consistency | 9/15 | Inconsistent |
| **Total** | **63/100** | **Acceptable** |

#### Strengths
- ✅ Excellent YAML error translation: "missing ':' after key — YAML mapping entries require 'key: value' format" (from `yaml_error.go` translation table)
- ✅ Goccy-style source context with `^` pointer embedded in message body
- ✅ Correct inner line reference `[9:1]` visible in message

#### Weaknesses
- ❌ **Outer IDE position is wrong**: `file:1:1:` instead of `file:9:1:` — IDE "click-to-jump" navigates to the wrong line. Root cause: `frontmatter_content.go:67` uses `fmt.Errorf` (not `FormattedParserError`), so `compiler.go:96` falls back to `1:1`
- ⚠️ Generic wrapper `"failed to parse frontmatter:"` adds noise without value
- ⚠️ Mixed format: outer `file:N:M:` style and inner `[line:col]` goccy style in same output
- ⚠️ No example of correct syntax (`engine: copilot`)

#### Improvement Suggestions

1. **Fix outer position by returning `FormattedParserError` from frontmatter parser**:
   Instead of `fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)`, extract the line/column from the goccy error and return a `FormattedParserError` with the exact position, so the compiler gets a properly-positioned error at `file:9:1:` not `file:1:1:`.

2. **Remove the "failed to parse frontmatter:" wrapper**:
   The goccy output already contains full context. The wrapper message is redundant noise.

3. **Add a corrected syntax example** (following the `mcp.go` pattern):
   ```
   Correct usage:
     engine: copilot
   ```

</details>

<details>
<summary>Test Case 2: Invalid Engine Name (typo) — Score: 76/100 ✅</summary>

#### Test Configuration

**Workflow**: `.github/workflows/agentic-observability-kit.md` (265 lines, medium)
**Error Type**: Category B – Configuration error (invalid value)
**Error Introduced**: `engine: copiilot` (double-`i` typo)

#### Predicted Compiler Output

```
.github/workflows/test-2.md:11:1: error: 'engine' (line 11, col 1): value must be one of 'claude', 'codex', 'copilot', 'gemini'. Did you mean 'copilot'?
   9 | permissions:
  10 |   contents: read
  11 | engine: copiilot
     ^^^^^^
  12 | strict: true
  13 | tracker-id: agentic-observability-kit
```

_Source trace_: `validateWithSchemaAndLocation` → `cleanOneOfMessage` strips the `oneOf` jargon → `LocateJSONPathInYAMLWithAdditionalProperties` finds the precise line → `generateSchemaBasedSuggestions` computes Levenshtein distance between "copiilot" and enum values (distance 1 to "copilot") → returns `Did you mean 'copilot'?`

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 21/25 | Excellent |
| Actionability | 22/25 | Excellent |
| Context | 17/20 | Good |
| Examples | 5/15 | Minimal |
| Consistency | 11/15 | Generally consistent |
| **Total** | **76/100** | **Good** |

#### Strengths
- ✅ "Did you mean 'copilot'?" — highly actionable, pinpoints the fix instantly
- ✅ Lists all valid engine names in the error message
- ✅ Precise line/column with source context and `^` pointer
- ✅ IDE-parseable format (`file:line:col:`)

#### Weaknesses
- ⚠️ **Redundant position** in message body: `'engine' (line 11, col 1):` duplicates what the IDE-format prefix already shows. When there is only one failure the `formatSchemaFailureDetail` path-prefix adds noise
- ⚠️ "value must be one of" is JSON-schema vocabulary; plainer phrasing (e.g. "unknown engine") would be more accessible
- ⚠️ No pointer to documentation or custom engine reference

#### Improvement Suggestions

1. **Suppress the `'field' (line N, col M):` prefix when there is only a single failure** — it duplicates the IDE prefix and clutters the message
2. **Replace "value must be one of" with plain language**, e.g. `invalid engine 'copiilot'. Valid engines: claude, codex, copilot, gemini`
3. **Add documentation link**: "For custom engines see: https://github.com/github/gh-aw#custom-engines"

</details>

<details>
<summary>Test Case 3: Invalid Permissions Scope — Score: 66/100 ⚠️</summary>

#### Test Configuration

**Workflow**: `.github/workflows/security-compliance.md` (301 lines, complex)
**Error Type**: Category C – Semantic / unknown property
**Error Introduced**: `unknown-scope: read` added to `permissions:` block

#### Predicted Compiler Output

```
.github/workflows/test-3.md:N:3: error: 'permissions/unknown-scope' (line N, col 3):
Unknown property: unknown-scope (Valid permission scopes: actions, all, attestations,
checks, contents, deployments, discussions, id-token, issues, metadata, models,
organization-projects, packages, pages, pull-requests, repository-projects,
security-events, statuses, vulnerability-alerts) See: https://docs.github.com/....
Valid fields are: actions, all, attestations, checks, contents, deployments,
discussions, id-token, issues, metadata, ...
[context lines + ^ pointer]

Source trace: formatSchemaFailureDetail calls appendKnownFieldValidValuesHint (adds the full scopes list + docs link), then calls generateSchemaBasedSuggestions → generateFieldSuggestions (no close match → adds "Valid fields are: …" listing the scopes again up to 10 items).

Evaluation Scores

Dimension	Score	Rating
Clarity	15/25	Unclear
Actionability	19/25	Moderate
Context	16/20	Good
Examples	4/15	Poor
Consistency	12/15	Generally consistent
Total	66/100	Acceptable

Strengths

• ✅ Documentation link to GitHub Actions permissions docs
• ✅ Precise source location with ^ pointer
• ✅ Lists the valid permission scope names

Weaknesses

• ❌ Duplicate valid-scopes list: appendKnownFieldValidValuesHint adds the full scopes list, then generateSchemaBasedSuggestions → generateFieldSuggestions adds it again (as "Valid fields are: …") because no close match is found. The message becomes extremely long
• ⚠️ Redundant position in message body ('permissions/unknown-scope' (line N, col M):)
• ⚠️ No "Did you mean?" for scopes that are even moderately close (threshold may be too strict)
• ⚠️ No example of correct usage

Improvement Suggestions

1. Prevent duplicate scope listing: In formatSchemaFailureDetail, skip the generateSchemaBasedSuggestions call (or suppress its "Valid fields" branch) when appendKnownFieldValidValuesHint has already added valid-values content to the message. A simple guard: check whether message already contains "Valid" before calling the suggestions generator
2. Shorten the scope list in appendKnownFieldValidValuesHint: Cap at the same maxAcceptedFields = 10 limit used by generateFieldSuggestions
3. Add a corrected-syntax example:

   permissions:
     contents: read
     issues: write
   

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	68.3/100
Good (70–84)	1 (TC2)
Acceptable (55–69)	2 (TC1, TC3)
Poor (<55)	0
Below threshold (70)	✅ Yes — issue created

Quality Assessment: ⚠️ Needs Improvement — Average score 68.3/100, below the 70 threshold. Two of three test cases fell in the "Acceptable" range. No critical failures were found, but several patterns consistently degrade developer experience.

---

Priority Improvement Recommendations

🔴 High Priority

1. Fix IDE position for YAML syntax errors (pkg/parser/frontmatter_content.go)

Currently: frontmatter_content.go:67 returns a plain fmt.Errorf, causing the compiler to fall back to file:1:1:.

// CURRENT — loses precise position
return nil, fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)

// IMPROVED — extract line/col from goccy error, return FormattedParserError
line, col := extractLineColFromYAMLError(err)
compilerErr := console.CompilerError{
    Position: console.ErrorPosition{File: filePath, Line: line + frontmatterOffset, Column: col},
    Type:    "error",
    Message: translateYAMLError(yaml.FormatError(err, false, false)),  // message only, no source
    Context: contextLinesAroundLine(source, line, 3),
}
return nil, &FormattedParserError{formatted: console.FormatError(compilerErr)}

Impact: IDE "click to jump" will take developers directly to the offending line instead of line 1.

2. Prevent duplicate valid-scopes output (pkg/parser/schema_compiler.go:formatSchemaFailureDetail)

message = appendKnownFieldValidValuesHint(message, pathInfo.Path)
// Guard: only add suggestions if the hint didn't already add a valid-values list
if !strings.Contains(message, "Valid ") {
    suggestions := generateSchemaBasedSuggestions(...)
    if suggestions != "" {
        message = message + ". " + suggestions
    }
}

Impact: Eliminates the extremely long double-listing of all permission scopes.

🟡 Medium Priority

3. Remove redundant 'field' (line N, col M): prefix for single-failure errors (pkg/parser/schema_compiler.go:validateWithSchemaAndLocation)

// Only prefix with path+position for multi-failure output
message := detailLines[0]
if len(detailLines) > 1 {
    message = "Multiple schema validation failures:\n- " + strings.Join(detailLines, "\n- ")
} else {
    // For single failure, strip the path prefix since IDE format already shows position
    message = stripPathPositionPrefix(detailLines[0])
}

4. Simplify schema jargon in engine error — Replace "value must be one of 'claude', 'codex', 'copilot', 'gemini'" with "invalid engine '%s'. Valid engines: claude, codex, copilot, gemini" by extracting the user-typed value from frontmatter.

🟢 Low Priority (Nice to Have)

5. Add correct-syntax examples following the mcp.go pattern — mcp.go already includes Example:\n YAML blocks in error messages. The same pattern could be adopted for the most common schema validation errors (engine, permissions) to make errors immediately self-sufficient.

6. Remove the "failed to parse frontmatter:" wrapper — Once fix #1 is applied, this wrapper message becomes redundant. The formatted error with position already describes the problem.

---

Implementation Notes

The // Hints removed as per requirements comments in pkg/parser/schema_compiler.go (lines 352 and 380) indicate the CompilerError.Hint field was deliberately removed. If that decision is revisited, the Hint field already exists in pkg/console/console_types.go and is rendered by pkg/console/console.go (with hint: prefix in cyan). This would be the natural place to add correct-syntax examples without cluttering the main message.

The MCP-related errors in pkg/parser/mcp.go serve as a positive example: they include Example:\n YAML blocks that give developers the exact syntax they need. This pattern is worth adopting more broadly for schema-validated fields.

---

References:

• §23999646318

Generated by Daily Syntax Error Quality Check · ● 9.2M · ◷

• expires on Apr 8, 2026, 10:49 AM UTC