From 709d371032cfbf7721bec9f7fe9a88b1c89c224b Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Mon, 3 Aug 2026 11:22:52 +0100 Subject: [PATCH 1/2] fix(build): repair the GCC 15 -Werror CPU build, one in code and one scoped GCC 15.2.0 rejects two translation units that older compilers accept, so the CPU build cannot complete on a newer-toolchain host. Both are correct code the optimizer mis-analyses after inlining. A keep-going build enumerated the whole surface first: these two, and only these two, out of 731 targets. hunyuan_a13b.cpp assigned a one-element std::vector from an initializer_list, which routes through _M_insert_range -> _M_copy_aligned, where GCC 15 reports a false -Warray-bounds against the bit-packing word copy. assign(1, true) is the same operation on a different libstdc++ path and needs no suppression at all, so -Werror keeps its full strength there. chat_template.cpp draws -Wfree-nonheap-object from inside the vendored minja parser, so that one demotes exactly that diagnostic, on exactly that unit, for GNU compilers only. third_party/ is already a SYSTEM include, but that covers front-end diagnostics only: this one is raised by the optimizer after inlining and attributed to the translation unit rather than the header, so -isystem cannot reach it. Ordered after vllm_cpp_set_warnings so the target-level -Werror cannot re-promote it, which is the ordering lesson from the MLX header exception. Pre-existing rather than rebase damage: the identical failure reproduces from a clean worktree at bfd51fb84. CI does not cover it, because ubuntu-latest ships an older GCC that raises neither diagnostic. Benchmarked rather than declared not-applicable. The 4B lever was re-run against the pinned oracle (vLLM 0.23.1rc1.dev1511+g555967922, built from source at the parity pin) across the 217-commit rebase plus this repair: 0.9980x total throughput (was 0.9972x on 2026-07-29), TTFT 0.7674x PASS, TPOT 1.1243x still the one failing axis, peak PSS 2.147 against 7.621 GiB. Output is BIT-IDENTICAL to the previous series, 128/128 per arm, and agreement with vLLM is unchanged at 89/128. The 0.0008x ratio move is the denominator (the oracle came in 0.06% slower while we moved +0.017%), so it is a null result, not an improvement. Three interleaved repetitions per arm under one flock, all 18 legs at 0% GPU utilization before starting, clean CUDA rebuild 899/899. CPU tier: 340/340 targets build, suite 329/331. The two failures are pre-existing host-environment issues this repair merely made reachable, since the suite could not be built here at all before: PYTHONHASHSEED=0 exported by the nix dev shell, and a hardcoded /usr/bin/true that does not exist on NixOS. Both recorded as open host-portability gaps. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] --- CMakeLists.txt | 15 +++ docs/BENCHMARKS.md | 59 ++++++++++ docs/STATUS.md | 23 +++- .../qwen35-4b-gcc15fix-20260803.md | 103 ++++++++++++++++++ .../openai/tool_parsers/hunyuan_a13b.cpp | 7 +- 5 files changed, 205 insertions(+), 2 deletions(-) create mode 100644 docs/bench-evidence/qwen35-4b-gcc15fix-20260803.md diff --git a/CMakeLists.txt b/CMakeLists.txt index 2cbd6d4c2..d50fc654e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -779,6 +779,21 @@ target_include_directories(vllm PRIVATE src) # src/vllm/entrypoints/chat_template.cpp. target_include_directories(vllm SYSTEM PUBLIC third_party) vllm_cpp_set_warnings(vllm) +# SYSTEM covers the FRONT-END diagnostics from those vendored headers, but not +# the ones the OPTIMIZER raises after inlining: those are attributed to the TU +# being compiled, not to the header, so -isystem cannot reach them. GCC 15 +# inlines minja::Parser::parseValueExpression (third_party/minja/minja.hpp), +# loses track of the allocation base of a std::vector it is +# destroying, and reports -Wfree-nonheap-object against correct vendored code. +# Demote just that diagnostic, just here: the warning stays visible, -Werror +# stays in force for every other diagnostic and every other TU, and the +# vendored source stays unmodified. Ordered after vllm_cpp_set_warnings so the +# target-level -Werror cannot promote it again (the lesson of the MLX header +# exception below). cmake/CompilerWarnings.cmake documents the same GCC +# false-positive class for the sanitizer lanes. +set_source_files_properties(src/vllm/entrypoints/chat_template.cpp PROPERTIES + COMPILE_OPTIONS + "$<$:-Wno-error=free-nonheap-object>") # --- Vendored BLAKE3 (LMCache MODE-1 lm:// token hash, KV-EXTERNAL-CACHE W1) -- # The official BLAKE3 C implementation (third_party/blake3, upstream tag 1.5.5 # / commit 81f772a; dual CC0-1.0 / Apache-2.0), the same algorithm the `blake3` diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 2b0965c07..823b65fb1 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,5 +1,64 @@ # Benchmarks +## Qwen3.5-4B re-validated across the 217-commit rebase + GCC 15 build repair (2026-08-03, `CLAIM-CPU-GCC15-WERROR-LOCAL`) - MEASURED, 0.9980x total throughput, output BIT-IDENTICAL + +Measured, not assumed. `main` advanced `f3ecbe70d` to `265e3bf98` (217 upstream +commits) and the GCC 15 `-Werror` build repair landed on top as `4e43aa5e`, so +the 4B lever was re-run against the pinned oracle on the identical workload. + +| Axis | ours | vLLM @ pin | ratio | 2026-07-29 | Disposition | +|---|---:|---:|---:|---:|---| +| Total throughput (tok/s) | 6611.397 | 6624.585 | 0.9980x | 0.9972x | FAIL | +| Output throughput (tok/s) | 731.070 | 732.528 | 0.9980x | 0.9972x | FAIL | +| Requests/s | 5.710 | 5.723 | 0.9978x | 0.9971x | FAIL | +| Mean TTFT (ms) | 728.243 | 948.961 | 0.7674x | 0.7701x | PASS | +| Mean TPOT (ms) | 38.160 | 33.942 | 1.1243x | 1.1247x | FAIL | +| Peak PSS (GiB) | 2.147 | 7.621 | 0.2816x | | PASS | +| Peak VRAM (MiB) | 12850.7 | 12832.0 | 1.0015x | | FAIL | + +**Nothing moved.** Every axis reproduces the 2026-07-29 series inside +run-to-run noise, and our generated tokens are BIT-IDENTICAL to it: 128/128 per +repetition within the run, and 128/128 against that series for every arm. The +217-commit advance plus the build repair changed no output on this workload. +Agreement with vLLM stays at 89/128, unchanged, so the residual is the known +near-tie sampling difference and not new drift. + +The published ratio moved 0.0008x, and the honest attribution is the +DENOMINATOR: ours moved +0.017% (ratio-to-historical 1.00017x) while the oracle +came in 0.06% slower than its own previous series. Both are inside noise. Read +this as a null result, not an improvement. + +Still an OPEN GAP under the match-or-beat-every-axis rule: throughput 0.2% +below, TPOT 12.4% above, peak VRAM 18.7 MiB above. TPOT remains owned by +`ENG-ASYNC-SCHED` and needs the async engine loop, which `vllm-bench` does not +drive. Peak host memory is the decisive win (2.147 against 7.621 GiB; stable +PSS 0.760 against 4.442 GiB). + +Oracle: `.venv-vllm-pin`, vLLM `0.23.1rc1.dev1511+g555967922` built from source +at the parity pin, NOT the 0.24.0 release that would understate us. Clean CUDA +rebuild 899/899. Three interleaved repetitions per arm under one +`flock /tmp/gpu`, and all 18 legs recorded 0% GPU utilization before starting, +so the series is uncontended and binding. No 4B result implies support or speed +for the 27B/35B gates, which stay hardware-unavailable here and `PENDING`. + +The build repair itself: the CPU build failed on this host's GCC 15.2.0 under +`-Wall -Wextra -Werror` in exactly two translation units out of 731, both +correct code mis-analysed after inlining, enumerated with a keep-going build +before anything was changed. `hunyuan_a13b.cpp` drew `-Warray-bounds` from the +`std::vector` bit-packing word copy behind a one-element assignment and is +repaired in project code as `assign(1, true)`, with no suppression. +`chat_template.cpp` drew `-Wfree-nonheap-object` from inside the vendored minja +parser and demotes that single diagnostic on that single unit, for GNU only. +Pre-existing rather than rebase damage: the identical failure reproduces at +`bfd51fb84`. CI never saw it, because `ubuntu-latest` ships an older GCC. CPU +suite now 329/331; the two failures are host-environment issues this repair +merely made reachable (`PYTHONHASHSEED=0` exported by the nix dev shell, and a +hardcoded `/usr/bin/true` absent on NixOS), recorded as open host-portability +gaps. + +Evidence and reproduction: +[Qwen3.5-4B GCC 15 fix re-validation](bench-evidence/qwen35-4b-gcc15fix-20260803.md). + ## Laguna-S-2.1-NVFP4 decode — router top-k KERNEL-EFFICIENCY (`VT_LAGUNA_TOPK_SHFL`), BYTE-EXACT, SigmoidTopK 1.67×, −0.57% decode-step GPU (2026-08-03, `CLAIM-LAGUNA-TOPK-SHFL`) With the residual-norm byte-exact floor reached (`CLAIM-LAGUNA-FAST-NORM` below), a fresh `nsys cuda_gpu_kern_sum --cuda-graph-trace=node` 2-length diff (20-vs-70) ranked the remaining small kernels; excluding the at-parity projection GEMVs (`gemvx`, ~69% of decode step — the IDENTICAL cuBLAS kernels vLLM uses) and the Marlin MoE (we win) / attention-compute kernels, the router `SigmoidTopKKernel` was the single largest still-optimizable small kernel (415 µs/step, 1.6%). Same box/model/ids/env as below (`~/laguna-xs-nvfp4`, ids `2,785,9626,377,15360,395`, `VT_LAGUNA_RESIDENT_DECODE=1 VT_LAGUNA_MARLIN_MOE=1 VT_LAGUNA_DECODE_GRAPH=1`), origin/main `e61b4de3`. diff --git a/docs/STATUS.md b/docs/STATUS.md index f2225a2b0..e7d8a126f 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1190,6 +1190,11 @@ pip-installed 0.24.0 release: that release is 1.25% faster than the pin, so the old denominator was understating us. TPOT (+12.4%) is the one real gap. Re-validated 2026-07-29 after rebasing onto 139 upstream commits: 0.9972x / TPOT 1.1247x, every axis inside noise and token-identical to the prior series. +Re-validated again 2026-08-03 across a further 217 upstream commits plus the +GCC 15 build repair: 0.9980x / TPOT 1.1243x, still inside noise and still +bit-identical (128/128 per arm against the 2026-07-29 series). The 0.0008x +ratio move is the denominator, the oracle came in 0.06% slower while we moved ++0.017%, so read it as a null result rather than an improvement. One open lead is on record from the same profiling pass: cuBLASLt resolves Ampere-class GEMM kernels on this Blackwell device. It is unmeasured and may be @@ -1209,7 +1214,23 @@ CI runs both as separate jobs. Verified end to end: the ASan+UBSan lane builds and passes `test_input_batch`, `test_combine_tokens` and `test_arena` with leak detection on. The lanes keep the warnings but drop `-Werror`, because sanitizer instrumentation makes GCC's range and initialization analyses fire inside -libstdc++ on correct code; the plain build is the one that enforces `-Werror`. The lane refuses to configure with the CUDA +libstdc++ on correct code; the plain build is the one that enforces `-Werror`. + +That same analysis class reaches the PLAIN build on a new enough compiler, so +two cases are now handled at their source rather than by relaxing the policy. +GCC 15.2.0 (the nix dev shell on the RTX 5070 Ti host) rejected exactly two +translation units, both correct code mis-analysed after inlining. +`hunyuan_a13b.cpp` drew `-Warray-bounds` from the bit-packing word copy behind +a one-element `std::vector` assignment; that is project code and is now +written as `assign(1, true)`, which takes a different libstdc++ path and needs +no suppression at all. `chat_template.cpp` drew `-Wfree-nonheap-object` from +inside the vendored minja parser, so it demotes that ONE diagnostic to a +warning, on GNU compilers only. `third_party/` being a SYSTEM include does not +cover that case: the optimizer raises it after inlining and attributes it to +the translation unit rather than to the header, so `-isystem` cannot reach it. +`-Werror` remains in force for every other diagnostic and every other unit. CI +sees neither, because `ubuntu-latest` ships an older GCC, so the +newer-toolchain build is a LOCAL gate with no CI coverage. The lane refuses to configure with the CUDA backend on, because a host sanitizer runtime does not instrument nvcc device translation units and reports false positives against the CUDA driver; the CUDA tier's equivalent is `compute-sanitizer`, and `VT_POOL_BYPASS=1` makes the device diff --git a/docs/bench-evidence/qwen35-4b-gcc15fix-20260803.md b/docs/bench-evidence/qwen35-4b-gcc15fix-20260803.md new file mode 100644 index 000000000..9d46c43e6 --- /dev/null +++ b/docs/bench-evidence/qwen35-4b-gcc15fix-20260803.md @@ -0,0 +1,103 @@ +# Qwen3.5-4B re-validated after the 217-commit rebase and the GCC 15 build fix, 2026-08-03 + +Immutable evidence index for the re-measurement of the 4B lever after `main` was +advanced from `f3ecbe70d` to `265e3bf98` (217 upstream commits) and the GCC 15 +`-Werror` build repair landed on top as `4e43aa5e`. + +## Headline + +**Nothing moved.** 217 upstream commits and one build repair later, every axis +reproduces the 2026-07-29 series inside run-to-run noise, and our generated +tokens are **bit-identical** to that series. The ratio reads **0.9980x** total +throughput (was 0.9972x), TTFT **PASS**, TPOT still the one failing axis at +**+12.4%**. + +The published ratio moved by 0.0008x, and the honest attribution is the +DENOMINATOR, not us: our own total throughput moved +0.017% (6610.270 to +6611.397, ratio-to-historical 1.00017x) while the oracle came in 0.06% slower +than its previous series (6628.651 to 6624.585). Both are inside noise. This is +a null result, and it is the useful kind: the rebase did not regress the lever, +and the build fix did not touch this workload. + +## What was measured + +- Ours: `4e43aa5e` on `fix/minja-gcc15-werror-20260803`, clean CUDA rebuild + 899/899, 0 errors, 0 warnings. +- Oracle: `.venv-vllm-pin`, vLLM `0.23.1rc1.dev1511+g555967922`, built from + source at the parity pin. Pin re-checked before the run and unchanged + (`AGENTS.md` still specifies `555967922`), so the same denominator remains + correct. This is NOT the `.venv-vllm` 0.24.0 release, which is 1.25% faster + than the pin and would understate us. +- Build: RelWithDebInfo, `VLLM_CPP_CUDA_ARCHITECTURES=120a`, FlashAttn ON, + Triton AOT ON with regen, vendored arch `sm_120`. +- Three performance repetitions per arm, interleaved, one `flock /tmp/gpu` held + across the whole 18-leg series. +- **All 18 legs recorded 0% GPU utilization** and 539 MiB resident (the X + server) before starting. The series is uncontended and therefore binding. + +## Binding result + +| Axis | ours | vLLM @ pin | ratio | 2026-07-29 ratio | Disposition | +|---|---:|---:|---:|---:|---| +| Total throughput (tok/s) | 6611.397 | 6624.585 | 0.9980x | 0.9972x | FAIL | +| Output throughput (tok/s) | 731.070 | 732.528 | 0.9980x | 0.9972x | FAIL | +| Requests/s | 5.710 | 5.723 | 0.9978x | 0.9971x | FAIL | +| Mean TTFT (ms) | 728.243 | 948.961 | 0.7674x | 0.7701x | PASS | +| Mean TPOT (ms) | 38.160 | 33.942 | 1.1243x | 1.1247x | FAIL | +| Mean ITL (ms) | 38.160 | 33.942 | 1.1243x | | FAIL | +| Peak PSS (GiB) | 2.147 | 7.621 | 0.2816x | | PASS | +| Peak VRAM (MiB) | 12850.7 | 12832.0 | 1.0015x | | FAIL | + +Per-axis spread across the three repetitions is tight: total throughput +6608.16 to 6614.46, mean TPOT 38.15 to 38.17, mean TTFT 726.85 to 730.61. + +The acceptance rule is match or beat on EVERY axis, so this remains an OPEN GAP, +not a pass. Throughput sits 0.2% below the oracle, TPOT 12.4% above, and peak +VRAM 18.7 MiB above. Peak host memory is the one decisive win (2.147 GiB against +7.621 GiB, and stable PSS 0.760 GiB against 4.442 GiB). + +## Output stability + +- Ours is bit-stable within the run: 128/128 identical generated tokens across + all three repetitions, both arms. +- Ours is bit-identical to the 2026-07-29 series: 128/128 for `cpp-on`, + `cpp-off` and the `vllm` arm alike. **The 217-commit advance plus the build + fix changed no output on this workload.** +- Ours versus vLLM agreement is 89/128, UNCHANGED from the 2026-07-29 series. + The residual divergence is the known near-tie sampling difference, not new + drift. + +## Direct-load arm + +The `cpp-off` (no direct loader) arm reproduces as well: 6494.79 tok/s against +its own 2026-07-29 6493.39 (1.00022x). Direct loading remains worth 1.8% total +throughput and drops peak PSS from 8.574 to 2.147 GiB. + +## Interpretation + +TPOT remains owned by `ENG-ASYNC-SCHED`, and the W4 finding stands: closing it +needs the ASYNC engine loop, which `vllm-bench` does not drive. Nothing in this +change set touches it. The build fix altered a CMake warning property and one +`std::vector` assignment in a tool parser that this dense 4B decode +workload never executes, so a null result was the prediction and the measurement +confirms it rather than assuming it. + +## Reproduction + +```sh +nix develop .#cuda --command bash -c 'REQUIRE_TRITON_AOT=1 \ + CPP_BENCH="$PWD/build-nix-cuda-transplant-triton/examples/vllm-bench" \ + CMAKE_CACHE="$PWD/build-nix-cuda-transplant-triton/CMakeCache.txt" \ + VLLM_PYTHON="$PWD/.venv-vllm-pin/bin/python" \ + VLLM_CUDA_HOME="$PWD/.venv-vllm-pin/lib/python3.12/site-packages/nvidia/cu13" \ + flock /tmp/gpu tools/bench/run_qwen35_4b_compare.sh /tmp/qwen35-gcc15fix-4e43aa5ee' + +python3 tools/bench/summarize_qwen35_4b_compare.py \ + --root /tmp/qwen35-gcc15fix-4e43aa5ee \ + --historical-root /tmp/qwen35-postrebase-13906f189 \ + --output /tmp/qwen35-gcc15fix-4e43aa5ee/aggregate.json +``` + +Evidence root `/tmp/qwen35-gcc15fix-4e43aa5ee` (306 files), aggregate +`aggregate.json`. No 4B result implies support or speed for the 27B/35B gate +checkpoints, which remain hardware-unavailable on this host and `PENDING`. diff --git a/src/vllm/entrypoints/openai/tool_parsers/hunyuan_a13b.cpp b/src/vllm/entrypoints/openai/tool_parsers/hunyuan_a13b.cpp index 22e62cc3f..7bced431d 100644 --- a/src/vllm/entrypoints/openai/tool_parsers/hunyuan_a13b.cpp +++ b/src/vllm/entrypoints/openai/tool_parsers/hunyuan_a13b.cpp @@ -200,7 +200,12 @@ std::optional HunyuanA13BToolParser::handle_test_compatibility( d.function.name = function_name; DeltaMessage msg; msg.tool_calls = std::vector{std::move(d)}; - hy_current_tools_sent = {true}; + // assign() rather than `= {true}`: the initializer_list assignment routes + // through vector::_M_insert_range -> _M_copy_aligned, where GCC 15 + // mis-analyses the bit-packing word copy and reports a false + // -Warray-bounds ("forming offset 8 is out of the bounds [0, 8]") against + // a correct one-element assignment. Same semantics, no suppression. + hy_current_tools_sent.assign(1, true); hy_current_tool_id = 0; hy_current_tool_index = 0; if (hy_sent_tools.empty()) { From 75617fb3ac51d15c1ca3dea383b5e35854b3d0c9 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Tue, 4 Aug 2026 11:59:52 +0100 Subject: [PATCH 2/2] ci(sanitize): keep detector builds within runner disk Share one fully instrumented test image instead of force-linking the engine into every sanitizer executable. Keep leak detection binding by using the allocator bypass and remove the minja macro context cycle. Also make the two host-sensitive CPU tests independent of Nix's filesystem layout and inherited Python hash seed. Local GCC 15.2 gates: ASan+UBSan 331/331; TSan 331/331; affected plain -Werror tests 3/3. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/parity-ledger.md | 1 + .../specs/hardening-adoption-2026-07-27.md | 26 ++++++----- .agents/state.md | 44 +++++++++++++++++++ .github/workflows/ci.yml | 4 ++ CMakeLists.txt | 31 +++++++++++-- docs/BENCHMARKS.md | 9 ++-- docs/STATUS.md | 19 +++++--- tests/CMakeLists.txt | 11 ++++- tests/tools/test_gdn_packed_component.py | 5 ++- tests/tools/test_online_gate_client.py | 3 +- third_party/minja/minja.hpp | 10 ++++- 11 files changed, 137 insertions(+), 26 deletions(-) diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 1156d1926..606340574 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -886,3 +886,4 @@ Columns: | 2026-07-31 (`SERVE-C-ABI` W0 contract spike; `CLAIM-SERVE-C-ABI-SPIKE`; CPU-only records/docs) | Accepted `.agents/specs/c-api-library.md` for the already-shipped original C packaging layer: complete scope, vLLM semantic chain/deviation, ABI v10/19-symbol baseline, ownership/error/version/dispatch rules, exact code/test anchors, gates, dependencies, risks, and W1-W5 follow-ons. Also fixes the verified stale public `VLLM_ABI_VERSION 9` labels in README/USAGE to the source-of-truth v10 and adds the missing v10 usage-table entry. No production/test/CMake source changed. | Pinned vLLM `555967922` has no C ABI; behavior beneath the adapter remains owned by its vLLM-derived engine rows. The flat ABI is the recorded llama.cpp-style packaging deviation and may translate, never reimplement, policy. | **CPU/records gate only; benchmark NOT APPLICABLE.** Focused C11/C++/dlopen/export gate passed 3/3 after explicitly building `vllm_shared`; five record checkers pass. `check-agent-record` reports the base tree's same six missing closing-commit objects (`444ea9d7`, `7a3f04b2`, `164453a2`), none in this row/diff. Row stays `ANCHOR-BACKFILL` because all-symbol dlsym coverage (chat symbols currently omitted), historical-layout compatibility, allocation-failure no-throw proof, lifetime sanitizer stress, and a standalone real-model C consumer remain W1-W5. | | 2026-07-31 (`CLAIM-CPU-GCC12-WERROR-PORTABILITY`; maintenance, rows `QUANT-GGUF-KEEPQ-LOADER` + `KV-OFFLOAD`; lifecycle unchanged) | Removes two GCC 12 production-library `-Werror` blockers without suppressions: the GGUF prefault keeps the same one-byte-per-page volatile XOR but uses simple assignment, and the KV filesystem tier builds the identical `...tmp` suffix with append operations inside its thread-local initializer. No API, algorithm, default, CUDA, fixture, or golden change. | Behavior remains grounded in the accepted loader and KV-persistence leaf specs: llama.cpp mmap prefault intent and vLLM `tiering/fs/io.py` unique temporary-file publication. This is compiler portability, not a parity-surface change. | RED: GCC 12 failed first at `qwen3_5_gguf_weights.cpp:49` (`-Wvolatile`), then at `fs_io.cpp:66` (`-Wrestrict`). GREEN: production `vllm` and focused test targets build clean; focused CTest 2/2 (`test_gguf_keep_quant`, `test_kv_offload_fs`). Full all-target build is PARTIAL at 42% on unrelated test-only GCC 12 `-Wrestrict` diagnostics in `test_deepseek_v2_paged_engine.cpp` and `test_glm4_moe_lite_paged_engine.cpp`; no full-CTest claim. Benchmark NOT APPLICABLE. | | 2026-08-01 (`SERVE-CLI-CHAT` W0 contract spike; `CLAIM-SERVE-CLI-CHAT-SPIKE`; CPU-only records/spec) | Accepts `.agents/specs/cli-chat-complete.md`, corrects the inventory from “no direct commands” to the actual pinned `chat`/`complete` surface, and decomposes a dual-mode port: exact remote OpenAI HTTP/SSE commands plus preservation of the existing in-process invocation as a compatibility alias. No production, test, CMake, model, kernel, fixture, or generated file changes. | Pinned vLLM `5559679229`: command registration `vllm/entrypoints/cli/main.py:17-37,73-98`; model/auth resolution and stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312`. The local compatibility baseline is `examples/cli/main.cpp:1-207`. | CPU record/doc gates only; benchmark `NOT APPLICABLE`, `benchmark_binding=false`. Implementation remains absent and the row moves `INVENTORIED` -> `SPIKE`. W1-W5 name parse, transport, complete, chat, and packaging gates, including fake-server request/SSE transcript parity, Release `-Werror`, ASan+UBSan, and TSan. | +| 2026-08-04 (`HARDEN-DETECTOR-LANES` PR #28 CI repair; `CLAIM-HARDEN-SANITIZER-DISK`; Ordino task `t-e19dc73f`; CPU-only, lifecycle unchanged; closing commit: this checkpoint) | Repairs the hosted ASan+UBSan build's filesystem exhaustion without weakening detector coverage. Sanitizer tests share one internal fully instrumented engine image instead of force-linking another static copy into every executable, and `-g1` retains file/line traces without full type/local-variable DWARF. CI enables the existing `VT_POOL_BYPASS=1` exact-allocation/real-free detector mode. The newly reachable leak survey removes a real minja `MacroNode` ownership cycle by weakly capturing the context that owns the callable. It also closes the two Nix-only suite portability gaps: resolve `true` and the active Python executable instead of assuming `/usr/bin`, and remove inherited `PYTHONHASHSEED` only from the unconfigured control suite. Default build linkage and packaged C ABI exports remain unchanged. | No vLLM behavioral counterpart: this is local build/test infrastructure plus a vendored minja lifetime repair. Anchors: `CMakeLists.txt`, `tests/CMakeLists.txt`, `.github/workflows/ci.yml`, `third_party/minja/minja.hpp`, `tests/tools/test_gdn_packed_component.py`, and `tests/tools/test_online_gate_client.py`. Remote root-cause evidence: GitHub run `30819266647`, job `91704728276`, 99 MiB free then `ld: No space left on device`. | **PASS.** GCC 15.2.0 ASan+UBSan full suite **331/331** under leak detection and pool bypass; TSan full suite **331/331** under pool bypass; affected plain GCC 15 `-Werror` suites **3/3**. ASan+UBSan tree **93 GiB -> 5.6 GiB** (about 94% smaller); TSan tree **1.9 GiB**. `benchmark_binding=false`, performance **NOT APPLICABLE**. Hosted PR confirmation is the next external gate; `continue-on-error` stays until that confirmation. | diff --git a/.agents/specs/hardening-adoption-2026-07-27.md b/.agents/specs/hardening-adoption-2026-07-27.md index e049fb0a5..fe598aaa2 100644 --- a/.agents/specs/hardening-adoption-2026-07-27.md +++ b/.agents/specs/hardening-adoption-2026-07-27.md @@ -56,9 +56,10 @@ separate jobs. Design points, all from the guide: project code, and the plain build — the one that DOES enforce `-Werror` — stays clean. Letting a false positive in a system header stop the runtime detectors would defeat the point. -- Landed `continue-on-error: true` because the lanes have never run: their FIRST - run is a survey, and a pre-existing finding must not block unrelated work. The - closing step of this row is triaging that first run and REMOVING the flag. +- `continue-on-error: true` remains until the repaired PR validates the two + lanes on the hosted runner. The first complete local survey is now triaged: + both full suites pass, and removing the flag after remote confirmation is the + closing step of this row. **The lane paid for itself before it was even committed.** Its first real build failed on `-Werror=unused-function`: `AsyncDeviceMirrorEnvDefault()`, added by the @@ -71,11 +72,16 @@ compiles the other `cfg` branches. **Verified end to end**, not merely configured: `address,undefined` and `thread` both configure; the two guards fire (a lane value outside the allowlist, and a -lane with the CUDA backend on, are both `FATAL_ERROR`); the ASan+UBSan lane -builds `test_input_batch`, `test_combine_tokens` and `test_arena` clean and all -three PASS under `ASAN_OPTIONS=detect_leaks=1 UBSAN_OPTIONS=print_stacktrace=1` -(25/25 + 183 asserts, 7/7 + 14, 4/4 + 18), which includes the new W4 op-log -cases. The plain CPU build was re-verified clean and green in the same pass. +lane with the CUDA backend on, are both `FATAL_ERROR`). On GCC 15.2.0 the full +ASan+UBSan suite passes 331/331 under +`ASAN_OPTIONS=detect_leaks=1:strict_string_checks=1`, +`UBSAN_OPTIONS=print_stacktrace=1`, and `VT_POOL_BYPASS=1`; the full TSan suite +also passes 331/331 with the bypass. The detector-specific shared test image +reduces the ASan+UBSan build from 93 GiB to 5.6 GiB, while keeping file/line +traces with `-g1`; TSan is 1.9 GiB. The survey also removed the minja macro +context ownership cycle that leak detection exposed. The plain GCC 15 `-Werror` +build and the affected `test_serve_low_tools`, `test_none_hash_determinism`, and +`test_chat_template` cases pass 3/3. ### 2. Allocator-blind-spot bypass (§4) @@ -121,8 +127,8 @@ reproduce here, which is worth recording so a later reader does not re-audit: ## Follow-up rows -1. Triage the first `sanitize-cpu` run and remove `continue-on-error` (closes - this row). +1. Confirm both repaired `sanitize-cpu` jobs on the hosted PR runner, then + remove `continue-on-error` (closes this row). 2. Validate the three residual `atoi` sites. 3. A concurrency matrix per §7 (independent concurrent engines, callback re-entry, forced worker failure). diff --git a/.agents/state.md b/.agents/state.md index 4c3604f4b..a531caefe 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -34536,3 +34536,47 @@ unchanged. A RED-first source regression test pins that ownership boundary. This is build portability only and changes no runtime, model lifecycle, correctness result, or benchmark disposition. Darwin consumer CI remains the binding AppleClang verification. + +## 2026-08-04 - PR #28 sanitizer CI disk and leak repair + +Ordino task `t-e19dc73f`, PR #28, branch +`fix/minja-gcc15-werror-20260803`. GitHub run `30819266647`, job +`91704728276`, failed during the ASan+UBSan **Build** at about 60%: the hosted +runner reported 99 MiB free and `ld` returned `No space left on device` while +linking `test_mtp_speculator` and `test_rejection_sampler`. There was no +sanitizer runtime finding in that run. Each test force-linked the full static +instrumented engine, duplicating hundreds of MiB about 330 times. + +The sanitizer configuration now emits `-g1` and links helper-generated tests to +one internal, unversioned `libvllm_sanitize_test_shared.so` that retains the +whole static archive and its registrars. The packaged `libvllm.so` and default +static test linkage are unchanged. The ASan+UBSan build fell from 93 GiB to +5.6 GiB (shared image 289,836,680 bytes; `test_version` 3,025,144 bytes), and +the TSan build is 1.9 GiB. CI runs the existing `VT_POOL_BYPASS=1` detector mode +so the intentionally retained scratch pool uses real frees. The now-reachable +leak survey found one genuine minja ownership cycle: `MacroNode` stored a +callable in a context while strongly capturing that same context. A weak +capture preserves invocation lifetime through the caller's parent chain and +releases the graph after rendering. + +Two previously recorded Nix host-portability failures are also closed. The +packed-component test resolves `true` from `PATH`, its clean-environment import +uses the current absolute Python executable, and CTest removes an inherited +`PYTHONHASHSEED` only for the suite whose control arm explicitly requires no +seed configuration. + +Binding local evidence on GCC 15.2.0: + +- ASan+UBSan full build, then `UBSAN_OPTIONS=print_stacktrace=1 + ASAN_OPTIONS=detect_leaks=1:strict_string_checks=1 VT_POOL_BYPASS=1 ctest + --test-dir build-nix-sanitize-g1 --output-on-failure`: **331/331 PASS**. +- TSan full build, then the same CTest command against + `build-nix-sanitize-thread-shared`: **331/331 PASS**. +- Plain GCC 15 `-Werror` affected-target build and focused CTest + (`test_serve_low_tools`, `test_none_hash_determinism`, + `test_chat_template`): **3/3 PASS**. + +No model, kernel, runtime default, CUDA path, correctness lifecycle, or +benchmark headline changed. Performance is NOT APPLICABLE. Next external gate: +push the checkpoint through the Ordino action-request path and confirm both +hosted sanitizer jobs before removing their first-survey `continue-on-error`. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c43103ab8..3abd68b62 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -240,4 +240,8 @@ jobs: env: UBSAN_OPTIONS: print_stacktrace=1 ASAN_OPTIONS: detect_leaks=1:strict_string_checks=1 + # The production DevicePool deliberately retains scratch blocks. Its + # detector lane uses exact allocations and real frees so ASan can + # distinguish that cache from a leak and can see use-after-free. + VT_POOL_BYPASS: "1" run: ctest --test-dir build-sanitize --output-on-failure diff --git a/CMakeLists.txt b/CMakeLists.txt index d50fc654e..509df70df 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -142,9 +142,12 @@ endif() # # -fno-sanitize-recover=all makes a finding ABORT instead of print-and-continue, # so ctest FAILS the lane rather than burying the report inside passing output. -# Frame pointers plus -g keep the traces attributable; the lane keeps the -# project's -ffp-contract=off numerics so a bit-identity test still means what it -# means in the production build. +# Frame pointers plus -g1 keep file/line traces attributable without emitting +# full type/local-variable DWARF. Every test statically links the instrumented +# library, so full -g duplicates hundreds of MiB into each executable; the +# ASan+UBSan GitHub runner exhausted its filesystem before the suite finished +# linking. The lane keeps the project's -ffp-contract=off numerics so a +# bit-identity test still means what it means in the production build. set(VLLM_CPP_SANITIZE "OFF" CACHE STRING "Host sanitizer lane: OFF | address | undefined | address,undefined | thread") set_property(CACHE VLLM_CPP_SANITIZE PROPERTY STRINGS @@ -166,7 +169,7 @@ if(NOT VLLM_CPP_SANITIZE STREQUAL "OFF") $<$:-fsanitize=${VLLM_CPP_SANITIZE}> $<$:-fno-sanitize-recover=all> $<$:-fno-omit-frame-pointer> - $<$:-g>) + $<$:-g1>) # -Werror itself is dropped for this lane in cmake/CompilerWarnings.cmake, where # the per-target warning flags are set; see the rationale there. add_link_options(-fsanitize=${VLLM_CPP_SANITIZE}) @@ -1425,6 +1428,26 @@ if(UNIX AND NOT APPLE) "LINKER:--version-script,${CMAKE_CURRENT_SOURCE_DIR}/cmake/vllm_export.map") endif() +# Sanitizer-only test linkage. Every normal test force-links the complete static +# `vllm` archive so its registrars are retained. With ASan+UBSan that makes each +# executable hundreds of MiB and duplicates the same instrumented engine across +# the whole suite; GitHub's hosted runner ran out of disk while linking. Keep one +# unversioned internal shared image of the fully instrumented archive instead. +# This is deliberately separate from packaged `vllm_shared`: that library's +# version script must continue exporting only the stable C ABI, while the tests +# need the C++ internals. Production/default builds never create this target. +if(NOT VLLM_CPP_SANITIZE STREQUAL "OFF") + add_library(vllm_sanitize_test_shared SHARED "${_vllm_shared_stub}") + add_library(vllm::sanitize_test_shared ALIAS vllm_sanitize_test_shared) + target_link_libraries(vllm_sanitize_test_shared PRIVATE vllm) + target_include_directories(vllm_sanitize_test_shared INTERFACE + "$") + target_include_directories(vllm_sanitize_test_shared SYSTEM INTERFACE + "$") + target_compile_definitions(vllm_sanitize_test_shared INTERFACE + "$") +endif() + # make install => a consumable libvllm.so + libvllm.a + vllm.h (llama.cpp-style). install(TARGETS vllm vllm_shared ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 823b65fb1..b800ec12f 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -6644,9 +6644,12 @@ configuration - it reinstates the per-op `cudaMalloc`/`cudaFree` device-sync storm the pool exists to remove. Rationale and the item-by-item transfer decision: [.agents/specs/hardening-adoption-2026-07-27.md](../.agents/specs/hardening-adoption-2026-07-27.md). The lanes are verified end to end (both configure, both guards fire, ASan+UBSan -builds and passes three CPU suites with leak detection on), and their first real -build already caught a `-Werror=unused-function` break in the W4 commit that -would also have failed the existing `build-test-cpu` job. +and TSan each pass the full 331-test CPU suite). ASan+UBSan ran with +`detect_leaks=1`, `strict_string_checks=1`, and `VT_POOL_BYPASS=1`; TSan ran the +same suite with the pool bypass. The repaired ASan+UBSan build occupies 5.6 GiB +instead of the pre-repair 93 GiB, and the TSan build occupies 1.9 GiB. This is a +build/test-layout result only: no model, kernel, runtime default, GPU path, or +timing path changed, so performance remains **NOT APPLICABLE**. ## `M3c-1` CUDA-path neutrality on GB10 (2026-07-27) - NO MEASURABLE REGRESSION diff --git a/docs/STATUS.md b/docs/STATUS.md index e7d8a126f..f12a5b3ac 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1210,11 +1210,20 @@ a cooldown rather than before it. Alongside the default build, `-DVLLM_CPP_SANITIZE=address,undefined` and `-DVLLM_CPP_SANITIZE=thread` build the CPU tier under the dynamic detectors, and -CI runs both as separate jobs. Verified end to end: the ASan+UBSan lane builds -and passes `test_input_batch`, `test_combine_tokens` and `test_arena` with leak -detection on. The lanes keep the warnings but drop `-Werror`, because sanitizer -instrumentation makes GCC's range and initialization analyses fire inside -libstdc++ on correct code; the plain build is the one that enforces `-Werror`. +CI runs both as separate jobs. Current binding local result on GCC 15.2.0 is +**331/331 PASS in both lanes**, with ASan leak detection enabled. PR #28's +hosted ASan+UBSan failure was a build-filesystem exhaustion, not a sanitizer +finding: every test had force-linked another complete instrumented static +engine, leaving 99 MiB before `ld` failed. Sanitizer tests now load one internal +shared instrumented image, and `-g1` retains file/line attribution without full +type/local-variable DWARF. The ASan+UBSan tree fell from 93 GiB to 5.6 GiB +(about 94%); the TSan tree is 1.9 GiB. CI also sets the existing +`VT_POOL_BYPASS=1` detector mode so deliberate scratch-cache retention is not +reported as a leak. The full survey found and fixed one real leak, minja macros +strongly captured the same context that owned their callable. The lanes keep +the warnings but drop `-Werror`, because sanitizer instrumentation makes GCC's +range and initialization analyses fire inside libstdc++ on correct code; the +plain build is the one that enforces `-Werror`. That same analysis class reaches the PLAIN build on a new enough compiler, so two cases are now handled at their source rather than by relaxing the policy. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 16aa6b3f0..52a68978c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -17,7 +17,11 @@ set_tests_properties(test_serve_low_tools PROPERTIES function(vllm_cpp_add_test name) add_executable(${name} ${ARGN}) - target_link_libraries(${name} PRIVATE vllm::vllm vllm_test_main) + if(TARGET vllm::sanitize_test_shared) + target_link_libraries(${name} PRIVATE vllm::sanitize_test_shared vllm_test_main) + else() + target_link_libraries(${name} PRIVATE vllm::vllm vllm_test_main) + endif() vllm_cpp_set_warnings(${name}) add_test(NAME ${name} COMMAND ${name}) endfunction() @@ -207,6 +211,11 @@ vllm_cpp_add_test(test_single_type_kv_cache_manager vllm/v1/test_single_type_kv_ vllm_cpp_add_test(test_kv_cache_coordinator vllm/v1/test_kv_cache_coordinator.cpp) vllm_cpp_add_test(test_kv_cache_manager vllm/v1/test_kv_cache_manager.cpp) vllm_cpp_add_test(test_none_hash_determinism vllm/v1/test_none_hash_determinism.cpp) +# This suite explicitly exercises configured and unconfigured hash seeding in +# child processes. Insulate its unconfigured control arm from a developer shell +# that happens to export Python's seed (the Nix dev shell exports 0). +set_tests_properties(test_none_hash_determinism PROPERTIES + ENVIRONMENT_MODIFICATION "PYTHONHASHSEED=unset:") vllm_cpp_add_test(test_prefix_cache_stats vllm/v1/test_prefix_cache_stats.cpp) vllm_cpp_add_test(test_prometheus_metrics vllm/v1/test_prometheus_metrics.cpp) vllm_cpp_add_test(test_kv_offload_cpu vllm/v1/test_kv_offload_cpu.cpp) diff --git a/tests/tools/test_gdn_packed_component.py b/tests/tools/test_gdn_packed_component.py index c7d235aef..ca3155401 100644 --- a/tests/tools/test_gdn_packed_component.py +++ b/tests/tools/test_gdn_packed_component.py @@ -1078,8 +1078,11 @@ def _write_complete_evidence(root: pathlib.Path, *, packed_better: bool = True) class GdnPackedComponentTest(unittest.TestCase): @classmethod def setUpClass(cls) -> None: + true_binary = shutil.which("true") + if true_binary is None: + raise unittest.SkipTest("the platform has no 'true' executable") cls.cuda_compiler_patch = mock.patch.object( - gdn_component, "DGX_CUDA_COMPILER", pathlib.Path("/usr/bin/true") + gdn_component, "DGX_CUDA_COMPILER", pathlib.Path(true_binary) ) cls.cuda_compiler_patch.start() diff --git a/tests/tools/test_online_gate_client.py b/tests/tools/test_online_gate_client.py index f1716af66..5b8ebadc4 100644 --- a/tests/tools/test_online_gate_client.py +++ b/tests/tools/test_online_gate_client.py @@ -1462,7 +1462,8 @@ def test_trace_driver_sets_h1d_plan_environment_before_manifest(self) -> None: "-c", definitions + '"${benchmark_clean_env[@]}" "${h1d_plan_env[@]}" ' - "python3 -c 'import tools.bench.online_gate'", + + shlex.quote(sys.executable) + + " -c 'import tools.bench.online_gate'", ], env=polluted, check=True, diff --git a/third_party/minja/minja.hpp b/third_party/minja/minja.hpp index f414cca9c..3dc121419 100644 --- a/third_party/minja/minja.hpp +++ b/third_party/minja/minja.hpp @@ -1063,7 +1063,15 @@ class MacroNode : public TemplateNode { void do_render(std::ostringstream &, const std::shared_ptr & macro_context) const override { if (!name) throw std::runtime_error("MacroNode.name is null"); if (!body) throw std::runtime_error("MacroNode.body is null"); - auto callable = Value::callable([this, macro_context](const std::shared_ptr & call_context, ArgumentsValue & args) { + // The callable is stored in macro_context itself. Capturing that context + // strongly creates a permanent cycle (Context -> Value -> callable -> + // Context) after rendering any template with a macro. Callers keep the + // context alive through call_context's parent chain, so a weak capture + // preserves the lifetime needed during invocation without leaking it. + std::weak_ptr weak_macro_context = macro_context; + auto callable = Value::callable([this, weak_macro_context](const std::shared_ptr & call_context, ArgumentsValue & args) { + auto macro_context = weak_macro_context.lock(); + if (!macro_context) throw std::runtime_error("Macro context has expired"); auto execution_context = Context::make(Value::object(), macro_context); if (call_context->contains("caller")) {