Skip to content

test(surgery): stop the empty-pipeline comparison at the first EOS#760

Merged
inureyes merged 1 commit into
mainfrom
test/728-surgery-cli-eos-truncated-comparison
Jul 12, 2026
Merged

test(surgery): stop the empty-pipeline comparison at the first EOS#760
inureyes merged 1 commit into
mainfrom
test/728-surgery-cli-eos-truncated-comparison

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

empty_surgery_pipeline_matches_baseline_output compared the baseline and empty-surgery-pipeline CLI outputs byte-for-byte over the full fixed -n 20 token budget. mlxcel generate keeps sampling for the whole -n count instead of stopping at EOS, so on qwen2.5-0.5b-4bit both runs already emit <|im_end|> and <|endoftext|> partway through and then continue into a free-running tail that carries no signal for the no-op property under test. On the CUDA/GB10 backend, near-tied logits in that tail let GPU reduction-order variance flip the argmax, making the full-window comparison flaky. This PR truncates the comparison at the first EOS marker instead.

Root cause

Confirmed locally on this machine's Metal backend that the same post-EOS tail exists outside CUDA (mlxcel generate -m models/qwen2.5-0.5b-4bit -p "Hello" -n 20 --temp 0.0 prints ...<|im_end|>\n<|endoftext|>Human: How can I improve my public for the last few tokens), matching the left/right sample in the issue. The property under test (an empty surgery pipeline must not alter output) is sound; only the comparison window was fragile, exactly as the issue diagnosed. Changing the CLI's own EOS-stop behavior is out of scope for this test-only fix.

What changed

  • tests/surgery_cli.rs: added EOS_MARKERS (<|im_end|>, <|endoftext|> — the literal strings decode_generated_text renders since it decodes with skip_special_tokens = false) and truncate_at_first_eos, which truncates a decoded run's text at the end of whichever marker starts earliest, or returns the full text unchanged when no marker is present.
  • The baseline and surgery outputs are now truncated independently with truncate_at_first_eos before the assert_eq!, so the assertion is bounded to the deterministic pre-EOS region.
  • Any genuine pre-EOS divergence between the two runs is still caught: if the text before either run's own first marker differs, the truncated prefixes differ and the assertion fails.
  • If neither run's output reaches EOS within the 20-token budget, the helper falls back to comparing the full window, so the test still asserts something meaningful in that case.

Test plan

  • cargo check --tests (metal, accelerate features)
  • cargo build --release --features metal,accelerate
  • cargo test --release --test surgery_cli empty_surgery_pipeline_matches_baseline_output -- --nocapture — 5 consecutive runs, all passed (test result: ok. 1 passed)
  • cargo test --release --test surgery_cli -- --nocapture — full 3-test suite (including malformed_surgery_yaml_fails_fast and missing_surgery_yaml_fails_fast) passed together
  • cargo fmt --all -- --check
  • python3 scripts/ci/check_cross_repo_refs.py

All validation above ran on this machine's Metal backend against the real models/qwen2.5-0.5b-4bit checkpoint. The issue's GB10/CUDA 5-consecutive-run acceptance criterion still needs confirmation on CUDA hardware, which was not reachable from this session. That said, the fix removes the flake mechanism (post-EOS near-tie argmax under GPU reduction-order variance) platform-independently by construction: the comparison no longer inspects the post-EOS region on any backend, so there is nothing left for reduction-order nondeterminism to disturb.

Closes #728

empty_surgery_pipeline_matches_baseline_output ran the CLI twice over a fixed -n 20 token budget and compared the two stdouts byte-for-byte. mlxcel generate keeps sampling for the full -n count instead of stopping at EOS, so on qwen2.5-0.5b-4bit both runs already emit <|im_end|> and <|endoftext|> partway through, then continue into a free-running tail with no signal for the no-op property under test. On the CUDA/GB10 backend, near-tied logits in that tail let GPU reduction-order variance flip the argmax and make the full-window comparison flaky (issue #728); the same tail is reproducible locally on the Metal backend, just without observed nondeterminism there.

Added EOS_MARKERS (<|im_end|>, <|endoftext|>, matching decode_generated_text's skip_special_tokens = false rendering) and truncate_at_first_eos, applied independently to the baseline and surgery outputs before the equality assertion. Truncating each run's own text at its own first marker bounds the comparison to the deterministic pre-EOS region while still failing loudly on any genuine pre-EOS divergence, and falls back to comparing the full window when no marker appears within the token budget.

Validated on this machine's Metal backend: cargo test --release --test surgery_cli empty_surgery_pipeline_matches_baseline_output passed 5 consecutive times, and the full surgery_cli suite (3 tests) passed together. The GB10/CUDA 5-consecutive-run acceptance criterion from the issue still needs confirmation on CUDA hardware, which was not reachable from this session; the fix removes the flake mechanism (post-EOS near-tie argmax) platform-independently by construction, since it no longer depends on GPU reduction order at all.
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions priority:low Low priority area:surgery Model surgery operations (merge, interpolate, replace, etc.) area:cli Command-line interface / CLI flags status:done Completed labels Jul 12, 2026
@inureyes
inureyes merged commit e63dae0 into main Jul 12, 2026
5 checks passed
@inureyes
inureyes deleted the test/728-surgery-cli-eos-truncated-comparison branch July 12, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli Command-line interface / CLI flags area:surgery Model surgery operations (merge, interpolate, replace, etc.) priority:low Low priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(surgery): empty_surgery_pipeline_matches_baseline_output is flaky on CUDA (post-EOS tail nondeterminism)

1 participant