Skip to content

Add a harness-neutral reasoning-effort setting to the spec loop #925

Description

@justinmclean

Summary

Add a SPEC_LOOP_EFFORT setting to the spec loop that takes low,
medium or high, and translate that level into whichever equivalent flag
the selected agent CLI understands. Unset means unset: no flag is added.

Background

tools/spec-loop/loop.sh drives a headless agent CLI through repeated
build, plan, update and consolidate passes. Several of those CLIs expose a
per-invocation control over how hard the model thinks, under different flag
names and different level vocabularies. The loop cannot express that today,
so a mechanical consolidate pass and a hard build pass run at the same
setting.

The framework never binds to one vendor's flag ([PRINCIPLES.md §9](https://github.com/apache/airflow-steward/blob/main/PRINCIPLES.md)),
so low/medium/high is the framework-facing vocabulary and each
harness maps it to its own flag. Where a CLI offers more levels, map high
to its highest sensible value and record that choice in the spec table.
spec_loop_launch_agent already does exactly this kind of translation for
output format, turning one request into --output-format, --format json,
--json, or nothing depending on the harness.

Where to look

  • tools/spec-loop/lib.shspec_loop_launch_agent. The output-format
    handling inside each per-harness branch is the pattern to copy.
  • tools/spec-loop/loop.sh — the SPEC_LOOP_MODEL block shows how a
    setting is read, printed in the startup banner and documented.
  • Related: tools/spec-loop/tests/test_runner_fixtures.sh — asserts on the
    argv the runner builds, including a kiro case asserting --model is
    absent. New assertions go here.

Acceptance criteria

  • SPEC_LOOP_EFFORT=low|medium|high reaches spec_loop_launch_agent
    and is mapped to the right flag for each harness whose CLI has one,
    taking the flag name from that CLI's own --help.
  • A harness with no such knob adds nothing and prints no warning, the
    treatment kiro already gets for --model.
  • Any other value fails at startup with a message naming the accepted
    values, before the first iteration launches.
  • With the variable unset, the argv for every harness is unchanged.
  • test_runner_fixtures.sh covers set-and-mapped,
    set-but-unsupported-harness, and unset, and passes.
  • README.md and the harness table in specs/spec-loop-runner.md
    document the setting and the per-harness mapping.

Estimated effort

~2-3 hours for someone new to the codebase. Most of it is reading each
agent CLI's --help to fill in the mapping; the shell change is small and
has a pattern to copy in the same two files.

Getting started

[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)


This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions