[awf] Copilot engine: intermittent 400 'model not supported' for claude-opus-4.6 due to catalog inconsistency

[lpcox]

Problem

~20-25% of runs targeting claude-opus-4.6 fail within ~4 seconds with 400 The requested model is not supported. The same workflow succeeds minutes later, indicating an upstream Copilot model-catalogue inconsistency rather than a configuration bug.

Context

Upstream report: github/gh-aw#35075

The model catalogue varies between runs (30 vs 39 models observed). The Copilot harness logs attempt 1: model not supported — not retrying.

Root Cause

The Copilot CLI's model-validation step checks the catalogue fetched at startup. When the upstream entitlement service returns a reduced catalogue (transient inconsistency), valid models like claude-opus-4.6 appear unsupported. The harness currently does not retry on 400 model not supported.

Proposed Solution

1. In the Copilot harness retry logic, treat 400 model not supported as a transient error and retry up to 3 times with exponential backoff before reporting failure.
2. Alternatively, add a --skip-model-validation flag to bypass pre-flight catalogue checks and let the API respond directly.
3. Log the full model list at debug level when this error occurs to aid diagnosis.

Generated by Firewall Issue Dispatcher · sonnet46 1.2M · ◷

[github-actions]

👋 Hello! I noticed this issue might be related to an existing issue:

• [awf] copilot-harness: Intermittent 400 "model not supported" for claude-opus-4.6 due to upstream model catalogue inconsistency #3902 — [awf] copilot-harness: Intermittent 400 "model not supported" for claude-opus-4.6 due to upstream model catalogue inconsistency (closed)

Both issues describe the same problem: ~20-25% of runs targeting claude-opus-4.6 fail with 400 The requested model is not supported, caused by the upstream Copilot model-catalogue API returning inconsistent results (30 vs 39 models). Both reference the same upstream report and propose the same harness-side retry mitigation.

If #3902 was resolved and the problem has recurred, it may be worth linking this issue to it or re-opening the original.

This is an automated message from the issue duplication detector.

Generated by Issue Duplication Detector for issue #4020 · sonnet46 599.2K · ◷

[lpcox]

Resolved by #3911 (merged 2026-05-27) which adds automatic retry with exponential backoff for Copilot 400 model not supported responses (up to 2 retries). This handles the transient catalogue inconsistency that causes ~20-25% of claude-opus-4.6 requests to fail.