Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .agents/benchmark-record.md
Original file line number Diff line number Diff line change
Expand Up @@ -11639,3 +11639,79 @@ hazard-A handling (double-buffer `exec_state_` so freeing the prior StepInputs
doesn't race the in-flight forward) and hazard-C (block-table device buffer). Not
a "small scoped drain" — it is the device-resident-sampled-tokens architecture.
Evidence: `dgx:~/work/q35-regrid/evidence/raw/35/ours/c16-r{1,2,3}-{abdrain,abevent}.json`.

## Qwen3.5-4B revalidation across the 217-commit rebase and GCC 15 repair (2026-08-03)

`CLAIM-CPU-GCC15-WERROR-LOCAL`. Measured, not assumed. `main` advanced from
`f3ecbe70d` to `265e3bf98` (217 upstream commits) and the GCC 15 `-Werror`
repair landed on top, so the 4B lever was rerun against the pinned oracle on
the identical workload.

| Axis | ours | vLLM at pin | ratio | 2026-07-29 | Disposition |
|---|---:|---:|---:|---:|---|
| Total throughput (tok/s) | 6611.397 | 6624.585 | 0.9980x | 0.9972x | FAIL |
| Output throughput (tok/s) | 731.070 | 732.528 | 0.9980x | 0.9972x | FAIL |
| Requests/s | 5.710 | 5.723 | 0.9978x | 0.9971x | FAIL |
| Mean TTFT (ms) | 728.243 | 948.961 | 0.7674x | 0.7701x | PASS |
| Mean TPOT (ms) | 38.160 | 33.942 | 1.1243x | 1.1247x | FAIL |
| Peak PSS (GiB) | 2.147 | 7.621 | 0.2816x | | PASS |
| Peak VRAM (MiB) | 12850.7 | 12832.0 | 1.0015x | | FAIL |

Every axis reproduced the 2026-07-29 series inside run-to-run noise. Our
generated tokens were bit-identical, 128/128 within each repetition and
128/128 against the earlier series for every arm. Agreement with vLLM remained
89/128, so the residual stayed the known near-tie sampling difference. The
0.0008x ratio movement came from the denominator: ours moved +0.017% while the
oracle ran 0.06% slower than its previous series. This was a null result, not
an improvement.

The open gaps remained total/output throughput, requests/s, TPOT, and peak
VRAM. Peak host memory remained the decisive win. Oracle:
`.venv-vllm-pin`, vLLM `0.23.1rc1.dev1511+g555967922`, built from source at the
parity pin rather than the faster 0.24.0 release. A clean CUDA build passed
899/899. Three interleaved repetitions per arm ran under one `flock /tmp/gpu`;
all 18 legs observed 0% GPU utilization before starting. No 4B result implies
support or speed for the 27B/35B gates.

Evidence and full reproduction:
[Qwen3.5-4B GCC 15 fix revalidation](../docs/bench-evidence/qwen35-4b-gcc15fix-20260803.md).

## PR #28 sanitizer CI disk and leak repair (2026-08-04)

`HARDEN-DETECTOR-LANES`, infrastructure, performance NOT APPLICABLE. GitHub run
`30819266647`, job `91704728276`, did not report a sanitizer finding. The
hosted runner reached 99 MiB free and `ld` failed with `No space left on
device`, because every sanitizer test force-linked another complete
instrumented static engine.

Sanitizer tests now share one internal instrumented image and emit `-g1`. The
ASan+UBSan build footprint fell from 93 GiB to 5.6 GiB; the TSan tree occupies
1.9 GiB. CI uses `VT_POOL_BYPASS=1`, so intentionally cached scratch blocks are
really freed during detector runs. The complete leak survey then found and
fixed a genuine minja ownership cycle: a `MacroNode` callable strongly captured
the context that owned the callable. Two Nix-only test environment assumptions
were also removed.

Binding GCC 15.2 local evidence: ASan+UBSan **331/331 PASS** with
`detect_leaks=1`, `strict_string_checks=1`, and `VT_POOL_BYPASS=1`; TSan
**331/331 PASS**; the affected plain `-Werror` tests **3/3 PASS**. No model,
kernel, runtime default, GPU path, or timing path changed, so no performance
number is applicable. Exact commands and binary sizes are retained in
`.agents/state.md` under "PR #28 sanitizer CI disk and leak repair".

## PR #28 post-upstream-merge re-gate (2026-08-05)

Merged `upstream/main` at `35542ad3a` into the PR branch and repeated the full
GCC 15.2 detector lanes. ASan+UBSan with `detect_leaks=1`,
`strict_string_checks=1`, and `VT_POOL_BYPASS=1` passed **333/333**; TSan with
`VT_POOL_BYPASS=1` passed **333/333**. The merged ASan+UBSan tree is 5.7 GiB
and the TSan tree remains 1.9 GiB.

The first merged ASan survey exposed a stale exact error-message expectation:
the architecture list in `test_model_loader_gguf` omitted upstream's newly
registered `KimiLinearForCausalLM`. Updating that expectation restored the
full lane, and the focused plain GCC tests (`test_chat_template`,
`test_none_hash_determinism`, `test_serve_low_tools`, and
`test_model_loader_gguf`) passed **4/4**. This integration repair changes no
model, kernel, runtime default, GPU path, or timing path, so performance remains
NOT APPLICABLE.
1 change: 1 addition & 0 deletions .agents/parity-ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -886,3 +886,4 @@ Columns:
| 2026-07-31 (`SERVE-C-ABI` W0 contract spike; `CLAIM-SERVE-C-ABI-SPIKE`; CPU-only records/docs) | Accepted `.agents/specs/c-api-library.md` for the already-shipped original C packaging layer: complete scope, vLLM semantic chain/deviation, ABI v10/19-symbol baseline, ownership/error/version/dispatch rules, exact code/test anchors, gates, dependencies, risks, and W1-W5 follow-ons. Also fixes the verified stale public `VLLM_ABI_VERSION 9` labels in README/USAGE to the source-of-truth v10 and adds the missing v10 usage-table entry. No production/test/CMake source changed. | Pinned vLLM `555967922` has no C ABI; behavior beneath the adapter remains owned by its vLLM-derived engine rows. The flat ABI is the recorded llama.cpp-style packaging deviation and may translate, never reimplement, policy. | **CPU/records gate only; benchmark NOT APPLICABLE.** Focused C11/C++/dlopen/export gate passed 3/3 after explicitly building `vllm_shared`; five record checkers pass. `check-agent-record` reports the base tree's same six missing closing-commit objects (`444ea9d7`, `7a3f04b2`, `164453a2`), none in this row/diff. Row stays `ANCHOR-BACKFILL` because all-symbol dlsym coverage (chat symbols currently omitted), historical-layout compatibility, allocation-failure no-throw proof, lifetime sanitizer stress, and a standalone real-model C consumer remain W1-W5. |
| 2026-07-31 (`CLAIM-CPU-GCC12-WERROR-PORTABILITY`; maintenance, rows `QUANT-GGUF-KEEPQ-LOADER` + `KV-OFFLOAD`; lifecycle unchanged) | Removes two GCC 12 production-library `-Werror` blockers without suppressions: the GGUF prefault keeps the same one-byte-per-page volatile XOR but uses simple assignment, and the KV filesystem tier builds the identical `.<pid>.<counter>.tmp` suffix with append operations inside its thread-local initializer. No API, algorithm, default, CUDA, fixture, or golden change. | Behavior remains grounded in the accepted loader and KV-persistence leaf specs: llama.cpp mmap prefault intent and vLLM `tiering/fs/io.py` unique temporary-file publication. This is compiler portability, not a parity-surface change. | RED: GCC 12 failed first at `qwen3_5_gguf_weights.cpp:49` (`-Wvolatile`), then at `fs_io.cpp:66` (`-Wrestrict`). GREEN: production `vllm` and focused test targets build clean; focused CTest 2/2 (`test_gguf_keep_quant`, `test_kv_offload_fs`). Full all-target build is PARTIAL at 42% on unrelated test-only GCC 12 `-Wrestrict` diagnostics in `test_deepseek_v2_paged_engine.cpp` and `test_glm4_moe_lite_paged_engine.cpp`; no full-CTest claim. Benchmark NOT APPLICABLE. |
| 2026-08-01 (`SERVE-CLI-CHAT` W0 contract spike; `CLAIM-SERVE-CLI-CHAT-SPIKE`; CPU-only records/spec) | Accepts `.agents/specs/cli-chat-complete.md`, corrects the inventory from “no direct commands” to the actual pinned `chat`/`complete` surface, and decomposes a dual-mode port: exact remote OpenAI HTTP/SSE commands plus preservation of the existing in-process invocation as a compatibility alias. No production, test, CMake, model, kernel, fixture, or generated file changes. | Pinned vLLM `5559679229`: command registration `vllm/entrypoints/cli/main.py:17-37,73-98`; model/auth resolution and stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312`. The local compatibility baseline is `examples/cli/main.cpp:1-207`. | CPU record/doc gates only; benchmark `NOT APPLICABLE`, `benchmark_binding=false`. Implementation remains absent and the row moves `INVENTORIED` -> `SPIKE`. W1-W5 name parse, transport, complete, chat, and packaging gates, including fake-server request/SSE transcript parity, Release `-Werror`, ASan+UBSan, and TSan. |
| 2026-08-04 (`HARDEN-DETECTOR-LANES` PR #28 CI repair; `CLAIM-HARDEN-SANITIZER-DISK`; Ordino task `t-e19dc73f`; CPU-only, lifecycle unchanged; closing commit: this checkpoint) | Repairs the hosted ASan+UBSan build's filesystem exhaustion without weakening detector coverage. Sanitizer tests share one internal fully instrumented engine image instead of force-linking another static copy into every executable, and `-g1` retains file/line traces without full type/local-variable DWARF. CI enables the existing `VT_POOL_BYPASS=1` exact-allocation/real-free detector mode. The newly reachable leak survey removes a real minja `MacroNode` ownership cycle by weakly capturing the context that owns the callable. It also closes the two Nix-only suite portability gaps: resolve `true` and the active Python executable instead of assuming `/usr/bin`, and remove inherited `PYTHONHASHSEED` only from the unconfigured control suite. Default build linkage and packaged C ABI exports remain unchanged. | No vLLM behavioral counterpart: this is local build/test infrastructure plus a vendored minja lifetime repair. Anchors: `CMakeLists.txt`, `tests/CMakeLists.txt`, `.github/workflows/ci.yml`, `third_party/minja/minja.hpp`, `tests/tools/test_gdn_packed_component.py`, and `tests/tools/test_online_gate_client.py`. Remote root-cause evidence: GitHub run `30819266647`, job `91704728276`, 99 MiB free then `ld: No space left on device`. | **PASS.** GCC 15.2.0 ASan+UBSan full suite **331/331** under leak detection and pool bypass; TSan full suite **331/331** under pool bypass; affected plain GCC 15 `-Werror` suites **3/3**. ASan+UBSan tree **93 GiB -> 5.6 GiB** (about 94% smaller); TSan tree **1.9 GiB**. `benchmark_binding=false`, performance **NOT APPLICABLE**. Hosted PR confirmation is the next external gate; `continue-on-error` stays until that confirmation. |
26 changes: 16 additions & 10 deletions .agents/specs/hardening-adoption-2026-07-27.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ separate jobs. Design points, all from the guide:
project code, and the plain build — the one that DOES enforce `-Werror` — stays
clean. Letting a false positive in a system header stop the runtime detectors
would defeat the point.
- Landed `continue-on-error: true` because the lanes have never run: their FIRST
run is a survey, and a pre-existing finding must not block unrelated work. The
closing step of this row is triaging that first run and REMOVING the flag.
- `continue-on-error: true` remains until the repaired PR validates the two
lanes on the hosted runner. The first complete local survey is now triaged:
both full suites pass, and removing the flag after remote confirmation is the
closing step of this row.

**The lane paid for itself before it was even committed.** Its first real build
failed on `-Werror=unused-function`: `AsyncDeviceMirrorEnvDefault()`, added by the
Expand All @@ -71,11 +72,16 @@ compiles the other `cfg` branches.

**Verified end to end**, not merely configured: `address,undefined` and `thread`
both configure; the two guards fire (a lane value outside the allowlist, and a
lane with the CUDA backend on, are both `FATAL_ERROR`); the ASan+UBSan lane
builds `test_input_batch`, `test_combine_tokens` and `test_arena` clean and all
three PASS under `ASAN_OPTIONS=detect_leaks=1 UBSAN_OPTIONS=print_stacktrace=1`
(25/25 + 183 asserts, 7/7 + 14, 4/4 + 18), which includes the new W4 op-log
cases. The plain CPU build was re-verified clean and green in the same pass.
lane with the CUDA backend on, are both `FATAL_ERROR`). On GCC 15.2.0 the full
ASan+UBSan suite passes 331/331 under
`ASAN_OPTIONS=detect_leaks=1:strict_string_checks=1`,
`UBSAN_OPTIONS=print_stacktrace=1`, and `VT_POOL_BYPASS=1`; the full TSan suite
also passes 331/331 with the bypass. The detector-specific shared test image
reduces the ASan+UBSan build from 93 GiB to 5.6 GiB, while keeping file/line
traces with `-g1`; TSan is 1.9 GiB. The survey also removed the minja macro
context ownership cycle that leak detection exposed. The plain GCC 15 `-Werror`
build and the affected `test_serve_low_tools`, `test_none_hash_determinism`, and
`test_chat_template` cases pass 3/3.

### 2. Allocator-blind-spot bypass (§4)

Expand Down Expand Up @@ -121,8 +127,8 @@ reproduce here, which is worth recording so a later reader does not re-audit:

## Follow-up rows

1. Triage the first `sanitize-cpu` run and remove `continue-on-error` (closes
this row).
1. Confirm both repaired `sanitize-cpu` jobs on the hosted PR runner, then
remove `continue-on-error` (closes this row).
2. Validate the three residual `atoi` sites.
3. A concurrency matrix per §7 (independent concurrent engines, callback
re-entry, forced worker failure).
Expand Down
44 changes: 44 additions & 0 deletions .agents/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -34537,6 +34537,50 @@ This is build portability only and changes no runtime, model lifecycle,
correctness result, or benchmark disposition. Darwin consumer CI remains the
binding AppleClang verification.

## 2026-08-04 - PR #28 sanitizer CI disk and leak repair

Ordino task `t-e19dc73f`, PR #28, branch
`fix/minja-gcc15-werror-20260803`. GitHub run `30819266647`, job
`91704728276`, failed during the ASan+UBSan **Build** at about 60%: the hosted
runner reported 99 MiB free and `ld` returned `No space left on device` while
linking `test_mtp_speculator` and `test_rejection_sampler`. There was no
sanitizer runtime finding in that run. Each test force-linked the full static
instrumented engine, duplicating hundreds of MiB about 330 times.

The sanitizer configuration now emits `-g1` and links helper-generated tests to
one internal, unversioned `libvllm_sanitize_test_shared.so` that retains the
whole static archive and its registrars. The packaged `libvllm.so` and default
static test linkage are unchanged. The ASan+UBSan build fell from 93 GiB to
5.6 GiB (shared image 289,836,680 bytes; `test_version` 3,025,144 bytes), and
the TSan build is 1.9 GiB. CI runs the existing `VT_POOL_BYPASS=1` detector mode
so the intentionally retained scratch pool uses real frees. The now-reachable
leak survey found one genuine minja ownership cycle: `MacroNode` stored a
callable in a context while strongly capturing that same context. A weak
capture preserves invocation lifetime through the caller's parent chain and
releases the graph after rendering.

Two previously recorded Nix host-portability failures are also closed. The
packed-component test resolves `true` from `PATH`, its clean-environment import
uses the current absolute Python executable, and CTest removes an inherited
`PYTHONHASHSEED` only for the suite whose control arm explicitly requires no
seed configuration.

Binding local evidence on GCC 15.2.0:

- ASan+UBSan full build, then `UBSAN_OPTIONS=print_stacktrace=1
ASAN_OPTIONS=detect_leaks=1:strict_string_checks=1 VT_POOL_BYPASS=1 ctest
--test-dir build-nix-sanitize-g1 --output-on-failure`: **331/331 PASS**.
- TSan full build, then the same CTest command against
`build-nix-sanitize-thread-shared`: **331/331 PASS**.
- Plain GCC 15 `-Werror` affected-target build and focused CTest
(`test_serve_low_tools`, `test_none_hash_determinism`,
`test_chat_template`): **3/3 PASS**.

No model, kernel, runtime default, CUDA path, correctness lifecycle, or
benchmark headline changed. Performance is NOT APPLICABLE. Next external gate:
push the checkpoint through the Ordino action-request path and confirm both
hosted sanitizer jobs before removing their first-survey `continue-on-error`.

## 2026-08-03 - MLX system-header dependency boundary

LocalAI Darwin consumer run `30783379823` showed that translation-unit warning
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -291,4 +291,8 @@ jobs:
env:
UBSAN_OPTIONS: print_stacktrace=1
ASAN_OPTIONS: detect_leaks=1:strict_string_checks=1
# The production DevicePool deliberately retains scratch blocks. Its
# detector lane uses exact allocations and real frees so ASan can
# distinguish that cache from a leak and can see use-after-free.
VT_POOL_BYPASS: "1"
run: ctest --test-dir build-sanitize --output-on-failure
Loading
Loading