Dissertation port: corpus ingestion (Marker/bboxes/integrity) + Lanham prose-style analytics#49
Open
ProfessorVR wants to merge 45 commits into
Open
Conversation
…le train) Port ste-bah#2 — structured PDF ingestion wired into run_pdf_ingest_pipeline: - archon-ingest-ext: Marker JSON -> Block parser; token-aware, bbox-carrying chunker (chars/4 code-point parity, pairwise undersized-merge); is_real_table gate + HTML->grid; Bekker/page-number locator strip+capture. - archon-docs: additive doc_chunk_spatial / doc_chunk_hashes / doc_locators satellites; block_chunking (ChunkOut->ChunkArtifact, id format preserved, bbox capture); provenance_chunks (commit_hash -> chunks_root -> extract_text_spatial record on ALL ingest, fills provenance_record_id; verify_chunks_root tamper check); marker_source (device-agnostic subprocess/http/pre-extracted); reprocess clear-path covers new satellites. - archon-policy: PdfPolicy.chunker (default token_aware) + marker_sidecar/device. - scripts: device-agnostic archon_marker_sidecar.py (auto cuda->mps->cpu, with --selftest) + chunk_parity_check.py. token_aware is the default (page_anchor kept as fallback); integrity runs on all ingest; Marker is used when a sidecar is configured, else a flat-text fallback. Intentional, documented deviations from the Python reference: corrected page_end (chunk spans only pages it actually contains); Bekker regex broadened to catch 4-digit Aristotle numbers (e.g. 1147a). Reviewed via a 5-lens adversarial pass (7 findings fixed: byte->code-point tokens, chained->pairwise merge, reprocess satellite cleanup, restored Princeton/Bollingen title markers, Marker-path artifact hash, zero-chunk guard, source-file provenance hash). Tests: archon-ingest-ext 29, archon-docs 189, archon-policy 5; cargo build --bin archon clean. SITREP: plans/ARCHON-PORT2-COMPLETE-2026-06-25.md (in the dissertation repo). Also includes port #1 from the prior session: `archon style train` first-class subcommand (new archon-lanham crate + cli_args/command/dispatch wiring). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… embeddings
B2 — Marker as a local, device-agnostic sidecar (standalone, no WRAITH):
- PdfPolicy.marker_python runs the sidecar via an isolated venv interpreter
(system python may be too new for torch/marker). marker 1.10.2 JSON matches the
Rust parser directly (bbox + /page/N/). Verified on Mac: ocr_engine=marker,
coord_space=marker real bboxes in doc_chunk_spatial.
I2 — cross-modal CLIP image embeddings (text<->image), local/standalone:
- embed_fastembed: CLIP ViT-B/32 vision (embed_image via embed_bytes) + text
(embed_image_query) encoders (fastembed 4.9.1, shared 512-dim), lazy-loaded;
image_dimension().
- ensure_vec_schema(dim, image_dim): vec_page_images is sized to the IMAGE
dimension (512 CLIP), independent of the text dim (fixes a hardcode that broke
the multimodal test); all call sites updated.
- ingest: image OCR is NON-FATAL — a failed OCR (e.g. text-less game frames) no
longer aborts the document; the image still gets CLIP-embedded.
- pdf_image_enrichment: PDF embedded figures are CLIP-embedded too (per-figure
key {page_id}-imgN) so they're visually searchable alongside standalone images.
- retrieval_image::search_images + `archon docs search-images <text>`: text->image
cross-modal search over vec_page_images (CLIP-text query); resolve_page strips
the figure suffix.
- ocr/rapid: ARCHON_RAPIDOCR_PYTHON auto-detects the ~/.archon-marker-venv helper
venv (which bundles rapidocr) so figure OCR works standalone, no env needed.
Verified live on the Mac: standalone frame -> "Image embeddings: 1" (CLIP vision
model downloaded); cross-modal search ranks frames by text query. Tests 189/29/5
green single-threaded (the parallel suite has pre-existing global-provider
test-isolation flakiness, unrelated to this change — both flaky tests pass isolated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…leak, OCR status, search guard) Found by an adversarial 5-lens wiring/bug audit of 594ecb5 (each finding independently verified against the code; the HIGH one empirically reproduced). - schema: vec_page_images dim-mismatch MIGRATION (HIGH). A DB created by a pre-CLIP build sized vec_page_images to the TEXT dim (768); `:create` is a no-op when the relation exists, so 512-dim CLIP image vectors were rejected forever (silent — only a warning, image embeddings never persisted). ensure_vec_page_images now reads the existing dim via `::columns` and, on a mismatch, drops the HNSW index then the relation and recreates it at the image dim. Regression test added (reproduces the old failure, asserts the fix). - reprocess: clear vec_page_images (MEDIUM). remove_generated_rows never deleted image/figure embeddings → orphaned/stale vectors that `search-images` still returned after reprocess. Now `:rm`'s all "page-{document_id}-" keyed rows (page-level + per-figure {page_id}-imgN). - ingest: keep the OCR-run status Failed on the non-fatal image path (LOW). The failed run was being relabeled Completed by the shared completion update; now gated by image_ocr_failed so doc_ocr_runs provenance stays accurate (the document still proceeds to CLIP embedding). - retrieval_image: guard search_images for a missing vec_page_images relation (LOW). `search-images` on a fresh/never-indexed DB now returns empty (friendly message) instead of a hard "relation not found" error, mirroring the text path. Tests: archon-docs 190 green single-threaded (+1 migration regression test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t planner New leaf crate `archon-accel` (Port ste-bah#2 device layer). RUNTIME (not compile-feature- gated) detection of CUDA / Apple-Metal-unified / CPU plus *free* memory, and a pure, deterministic placement planner that fits the Marker sidecar to free VRAM. - detect.rs: degrade-safe probe (nvidia-smi free VRAM, WSL-aware; Apple unified via sysinfo; CPU fallback). Never panics; always yields a valid report. - report.rs: AcceleratorReport + best_gpu() (selects by FREE, not card size). - placement.rs: plan_placement/plan_marker_ingest. Load-bearing rule `free < footprint -> CPU`; GPU placements ALWAYS carry per-doc OOM->CPU; force_marker_device + memory_budget_mb overrides; surya batch-cap env mapping; dormant multi-consumer seam (whisper/frame-VLM) for the video round. - 13 tests incl. the verified co-tenancy case (32607 MiB total / 139 MiB free -> CPU); examples/probe.rs prints the live report + plan for the host. Additive: nothing consumes it yet (PR-C wires it into ingest). Gates green: build, clippy -D warnings, fmt, file-size (max 379 <= 500). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ity mode
Lock the 6 intentional Rust-vs-Python ingestion divergences (structured TableGrid vs
Python flatten; chunks_root/commit_hash Rust-only; cleaning_version none vs clean_v1;
anchored running-head vs unanchored inline locators; page_end last-page-contained;
broadened Bekker regex) + the 4 must-match chunk constants as EXPECTED, not drift.
Every anchor source-verified.
Correction: the Bekker-broadening gain is vs layout_analyzer.py:54
`_BEKKER_RE=^\s*\d{2,3}[a-b]?\d{0,2}\s*$` (caps at 3 digits, misses 1147a), NOT
markdown_chunker.py:303 (\d{2,4}, already reaches 4) — register documents both.
scripts/chunk_parity_check.py: add an additive `--marker-json <path>` mode to print the
Python reference chunk table from a real Marker JSON dump (for PR-D corpus diffing). The
no-arg invocation is byte-identical to baseline (sha256 match, exit 0).
.gitignore: re-include /docs/ingestion/ (matches the existing committed-subtree pattern).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
… OOM→CPU retry Wire archon-accel's free-VRAM placement planner into the live PDF ingest path. The Marker sidecar's device + surya batch caps are now resolved from the host's FREE VRAM (not a static string): marker_device None/"auto" -> planner-chosen, explicit cuda|mps|cpu forces it. The load-bearing correctness guarantee is the per-doc GPU-OOM -> CPU retry. - archon-docs/marker_source.rs: from_policy() resolves placement via archon_accel::detect() + plan_marker_ingest(); MarkerSource::Subprocess now carries the resolved env (TORCH_DEVICE, surya batch caps, PYTORCH_CUDA_ALLOC_CONF); run_sidecar() classifies a torch-OOM (sidecar exit 42 or stderr signature) and retries the document on CPU. + overrides_from_policy + 2 host-agnostic tests. (archon-docs lib: 192 tests green.) - archon-accel: marker_cpu_fallback_env() (CPU env for the OOM retry) + re-export. - archon-policy: PdfPolicy.marker_memory_budget_mb override (clamps usable free VRAM). - sidecar archon_marker_sidecar.py: --device "auto" now triggers real auto-detect (was forwarded verbatim → an invalid torch device); torch-OOM is caught and exits 42 so the Rust caller can retry on CPU. ingest_pdf.rs unchanged (from_policy is the chokepoint). Build + tests green; the touched code is clippy-clean (pre-existing >7-arg lints in untouched files are CI-advisory). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
scripts/bbox_jitter_diff.py — run the SAME PDF's Marker JSON on multiple devices (5090 / 5070 / Mac-MPS / CPU), align blocks by id, and report the per-coordinate delta distribution plus the per-bucket quantization unify-rate. This is the measurement that decides whether rounding bbox coords before hashing can make spatial_hash device-independent (so re-ingestion is portable) or whether geometry stays verify-by-recompute-only. Stdlib only; handles bbox or polygon. Smoke-tested on the selftest fixture (identical dumps → 0px delta, 100% unify). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The lanham (port #1) and ingestion-port (port ste-bah#2) feature crates, plus this session's device-adaptive additions, were committed without running `cargo fmt`, leaving the branch dirty under `cargo fmt --all -- --check` — the CI fmt gate (dtolnay/rust-toolchain@stable, which runs on PRs to main). Normalize the whole branch to canonical rustfmt (verified: local rustfmt 1.9.0 / rustc 1.96.0 is the same stable line CI installs, and the existing `style: cargo fmt` commits stay clean under it). Pure formatting; no semantic change. 19 files: archon-docs (9), archon-ingest-ext (4), archon-lanham (3), src/command (3). Verified post-fmt: `cargo fmt --all -- --check` exit 0; `cargo build --bin archon` clean; archon-ingest-ext 29 (parity golden gate) + archon-accel 13 + archon-policy 5 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…(PR-D) crates/archon-ingest-ext/examples/chunk_dump.rs — reads a Marker JSON dump, runs parse_marker_str + chunk_blocks_default, and prints the Rust chunk table (idx, page_start, page_end, text_len, text_head). Mirrors scripts/chunk_parity_check.py --marker-json so the Rust port diffs directly against the Python reference on real corpus PDFs. No native deps; builds on WSL + Mac. Validated on a real Mac (marker-pdf 1.10.2, MPS) Marker JSON of a 13pp article: Rust and Python produce 10 identical chunks (every text_len byte-exact); the sole diff is the documented page_end correction (divergence ste-bah#5). Confirms the port is faithful on real hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…4 MiB) Real GPU memory measured on the RTX 5070 Laptop (torch 2.11.0+cu128, surya-ocr 0.17.1, 13pp doc): torch peak reserved ~5956 MiB (peak allocated ~5194, + ~0.5 GiB CUDA context). The prior 5120 estimate was nearly exact on *allocated* but undercounted the reserved pool + context. Bump ModelFootprintTable.marker_mb 5120 -> 6144 so the fit decision reflects real VRAM: on an idle 8 GB card (7822 free) Marker still fits Generous (need 6144+1536=7680, and it ran there), while a busier 8 GB card now correctly drops to Reduced/CPU instead of risking OOM. Adjusted 3 test free-memory thresholds to the new footprint. surya peak also scales with the configured batch caps (measured at surya defaults); MPS peak still pending (Mac was asleep). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…both platforms PR-D Mac measurement: torch MPS driver-allocated ~6089 MiB for the same 13pp doc — essentially matches the RTX 5070 CUDA reserved peak (~5956 MiB). Marker's footprint is ~6 GiB on both CUDA and MPS, so the 6144 marker_mb tuned in c36edef covers either platform; no logic change. Doc comment updated (removes the "MPS pending" caveat). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Placement adapted DOWN (fit smaller cards / CPU) but "Generous" was a fixed batch-cap ceiling — a 16 GB or 32 GB card ran the same surya batches as a barely-fitting 8 GB card, leaving VRAM and throughput on the table. Make it adapt UP too, safely: - SuryaTier gains Ample + Max; decide_one picks the tier by FREE VRAM bands (>=need -> Generous, >=12 GiB -> Ample, >=20 GiB -> Max), so a bigger card runs bigger batches. - marker_env_for(device, tier) centralizes the device+tier -> env mapping. Generous stays the measured ~6 GiB config; Ample/Max are PROVISIONAL upscales. - marker_env_ladder() builds the per-doc OOM->retry-smaller sequence: start at the chosen tier, step DOWN GPU tiers on torch-OOM, CPU only as the last resort. marker_source::fetch_json now iterates this ladder (advancing only on OOM), replacing the single OOM->CPU jump — so an over-eager upscale on a hard doc drops to a smaller batch tier instead of all the way to CPU. archon-accel 17 tests (4 new: tier-scaling, batch monotonicity, ladder step-down, cpu-only); archon-docs 192 lib tests green; clippy -D warnings + fmt clean. The Ample/Max caps + AMPLE/MAX VRAM thresholds are provisional — calibrate with mem_probe.py; the ladder makes over-estimates safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…alibration) PR-D calibration on the 5090 (marker-pdf 1.10.2 / surya 0.17.1) disproved the premise of the batch-size tiers: Marker's VRAM is INERT to batch size (batch 1..256 identical) and instead tracks DOCUMENT SIZE — surya's OCR encoder processes the whole document's line-crops at once. Three points (13pp->5956, 129pp->9424, 578pp->8470 MiB reserved) show a bounded, SATURATING footprint, not page-linear-unbounded (578pp used less than 129pp; page resolution drives the top). - Revert 3b0e490's Ample/Max upward tiers + retry-smaller-batch ladder (they can't work: bigger batch != more VRAM; smaller batch != OOM relief). SuryaTier -> {Gpu,Cpu,None}; GPU leaves batch to surya defaults, CPU caps it to bound RAM. marker_env_ladder = [GPU, CPU]. - Add marker_footprint_mb(pages) = min(6000 + 30*pages, 10240): fits all 3 points, conservative cap, OOM->CPU backstop. Thread the already-computed extract_result.page_count through from_policy -> marker_env_ladder -> the placement footprint. decide_one: GPU iff free >= need. - headroom 1536 -> 512 (footprint now includes context + is conservative). - scripts/mem_probe.py: the VRAM-peak calibration tool that produced the model. Net: placement now depends on BOTH free VRAM AND document size. A small doc runs on an 8 GB card's GPU; a 300pp doc routes to CPU there (no wasted GPU-OOM cycle) and to GPU on a 16 GB+ card. archon-accel 14 tests, archon-docs 192 lib tests, clippy -D warnings + fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Batch size is VRAM-inert (PR-D), so the only lever to fit a document larger than a card's free
VRAM onto its GPU is to process it in page-range slices. Verified on the 5090: marker-pdf's
--page-range restricts the run AND keeps ABSOLUTE page ids (chunk 0-6 -> pages 0..6, chunk 7-12
-> pages 7..12), so slice block-streams concatenate in page order with no re-offset.
archon-accel:
- marker_chunk_pages(usable_mb): inverse of the footprint model (shared FLOOR/PER_PAGE/CAP consts)
-> the largest page-range whose footprint fits free VRAM.
- MarkerChunk { page_range, attempts } + marker_ingest_plan(report, overrides, pages): a whole-doc
chunk when it fits (or forced / no-GPU / CPU), else contiguous GPU chunks sized to free VRAM;
each chunk keeps its own [GPU, CPU] OOM ladder. +7 host-agnostic tests (8GB splits 300pp into 7,
16GB runs whole, Apple splits on mps, tiny-free / forced-cpu stay whole-doc CPU).
archon-docs (marker_source):
- Subprocess now carries Vec<MarkerChunk>; blocks_for runs each chunk (--page-range when set)
through run_chunk's ladder and concatenates the parsed blocks; from_policy uses marker_ingest_plan.
sidecar:
- --page-range START-END (0-indexed inclusive) -> PdfConverter page_range config.
archon-accel 20 tests, archon-docs 192 tests, clippy(touched files)+fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
`cargo run -p archon-docs --example chunk_ingest -- <pdf> <sidecar> <python> <budget_mb> <pages>` forces a small VRAM budget so marker_ingest_plan splits the doc into page-range GPU chunks, runs the real MarkerSource::blocks_for over them, and checks the concatenated blocks span every page. Hardware-gate tool for the small-card / Apple-MPS chunking path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…shareable template Ingestion runs Marker (device-adaptive placement + page-range chunking + real bboxes) only when [policy.docs.pdf].marker_sidecar is set; otherwise it silently falls back to pdftotext text-only chunks. This turns it on: - .archon/policy.toml (now gitignored): active marker_sidecar + marker_python for THIS machine — real absolute paths, kept out of git so they never upload. - .archon/policy.example.toml (tracked): the shareable GitHub template — same config with the Marker keys as commented placeholders + instructions, header marked "EXAMPLE TEMPLATE". Others: cp it to policy.toml and set their paths. - .gitignore: ignore .archon/policy.toml (machine-specific), track the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ml enables Marker `cargo run -p archon-policy --example effective_policy` loads the effective policy for the cwd (system -> user -> workspace .archon/policy.toml) and prints the resolved PDF/Marker settings, so you can confirm device-adaptive Marker ingestion is the DEFAULT on a given machine (config file honored). Verified on WSL (CUDA) + Mac (MPS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Adds docs/ingestion/hardware-vram-guide.md — device-adaptive placement, MEASURED footprints (Marker 6-10 GB page-scaled; qwen2.5vl:7b 14-22 GB by context), a what-runs-on-what-VRAM table, the sequential-pipeline + VLM-residency notes, constrained-host tuning (num_ctx / keep_alive / CPU offload), and per-tier configs — including the locked 8 GB laptop policy (Marker on GPU + VLM on CPU). policy.example.toml: recommend LOCAL qwen2.5vl (cross-version; llama3.2-vision's mllama arch is dropped in current Ollama) with a pointer to the guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…_gpu
Adds three optional knobs to [policy.docs.vlm.ollama] so constrained hosts can keep
the VLM from starving Marker of VRAM:
- num_ctx (u32) — smaller context window = much less KV-cache VRAM (default 32K
is wasteful for image description)
- keep_alive (str) — "0" unloads the model immediately after a doc's images, freeing
VRAM for the next doc's Marker; "5m" is Ollama's default
- num_gpu (u32) — 0 forces CPU (Marker owns the GPU on <=8 GB hosts)
Wired through models.rs (OllamaVlmPolicy + Default) -> loader_docs.rs (Raw + merge) ->
vlm/ollama.rs (provider fields, from_policy, request body). num_ctx/num_gpu nest under
"options"; keep_alive is top-level. All three are Option and OMITTED from the request
body when unset, so behavior is byte-identical to today (the pre-existing exact-match
wiremock test still passes; two new tests cover the set/omit paths). 0 is meaningful
(force CPU) and distinct from unset — hence Option, not a u32 sentinel.
Also: repoint repository_policy_template_parses_all_vlm_provider_fields at the committed
.archon/policy.example.toml (the real policy.toml is now gitignored) + document the knobs.
archon-docs vlm::ollama 8 tests, archon-policy 15 tests green; build + fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ntegrity) Image-OCR (pdf-image-ocr-*) and VLM-description chunks were persisted AFTER the document's chunks_root Merkle root was sealed, so image/figure-derived text was excluded from the tamper-evidence root (and silently from verify_chunks_root). Now they are re-folded in: enrich_pdf_images returns the image ChunkArtifacts and the pipeline re-runs persist_chunk_integrity over the text+image UNION after enrichment. - enrich_pdf_images / persist_image_result / persist_ocr_result / persist_vlm_result thread a &mut Vec<ChunkArtifact> accumulator; persist_image_ocr_chunks and persist_vlm_description now return their chunks. - ingest_pdf keeps the early text-only seal (so a fatal VLM error still leaves a valid text root), captures it in TextSeal, then re-seals over the union — but ONLY when the doc had a text seal AND produced image chunks. chunks_root sorts commits, so a no-image ingest is byte-identical to before; the second seal is an idempotent upsert of the same record. - Test: seal text-only, re-fold over text+image, assert the root changes, verifies over the superset, and tampering an image-OCR chunk now flips verify (was silently excluded before). Follow-up (noted in code): image-only/scanned PDFs (empty full_text) still get no root — needs a synthetic artifact/record. archon-docs 195 lib tests, clippy (touched files) + fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Self-scanned book PDFs store one full-page raster per page + an OCR text layer. The text layer triggers Marker (which OCRs the pages WITH bboxes = the real content), but the image-enrichment path then treated each full-page scan as a "figure" — re-OCRing it (duplicate chunks, now folded into chunks_root) and VLM-ing it (useless "a page of text"). Skip enrichment for these. Detection (is_scanned_page_images): >=70% of pages have exactly one LARGE (min side >=1000px) embedded image. Distribution is the signal, not raw count: born-digital docs cluster figures (King&Salvo = 17 imgs across 8/17 pages, ~24%) while a scanned Uexkull has one per page (100%). enrich_pdf_images returns early (page metadata still marked) when detected; non-scanned docs (born-digital figures) enrich exactly as before. Follow-up: figure-region VLM (describe Marker-detected figure bboxes) for figures baked INTO scanned pages — a separate feature. +4 unit tests; archon-docs 199 lib tests, clippy(touched files)+fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ement
Enabling figure VLM needs BOTH [policy.workers] vlm = "allow-local" (the coarse master
gate) AND [policy.docs.vlm] enabled=true — a "deny" on the worker gate silently overrides
the detailed config ("denied by policy.workers.vlm" in ingest logs). Surfaced by the King
pipeline test. Also note `tesseract-ocr` is needed for image/figure OCR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Refine the full-page-scan detector: an embedded image counts as a page-scan only if it is large AND page-shaped (long/short side ratio in ~1.2-1.6 — the Letter/A4/book range, orientation-agnostic). The prior size-only check false-positived on a large square diagram or wide chart appearing on most pages, wrongly flagging the doc as scanned and skipping real figures. New is_page_scale() + a per-page "exactly one page-scan and nothing else" rule feed the same 70% doc-level ratio. (ste-bah#3 is already satisfied by the doc-level binary: a born-digital doc enriches ALL its images — including any full-page infographic — because the skip only fires for scanned_book docs.) +3 unit tests (large square / wide chart per page are NOT scans; page-shaped sizes ARE). Follow-up remains true page-dimension coverage % via the page MediaBox. clippy+fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The 1.6 upper bound (calibrated on Letter 1.29 / A4 1.41) was too tight for taller book formats: Uexkull's real scans measure ~1.58 with crops up to ~1.61, so a chunk of its pages slipped the gate and would have been wrongly enriched. Widen to [1.2, 1.7] (covers Letter, A4, US Legal 1.65, and book formats; a >1.7 portrait image is a tall figure, not a page). Verified against the real Uexkull image dims + a 20-page book-scan detection test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Before a single-PDF ingest, classify how the image-enrichment path will treat the doc (SCANNED BOOK -> skip N page-scans, or BORN-DIGITAL -> enrich M figures) and print a LOUD report with a warning, so a misclassification (like the Uexkull aspect-gate edge) is caught BEFORE any OCR/VLM. When interactive (stdin is a TTY) and not --yes, prompt to proceed; non-interactive/batch/-y auto-proceeds (the report is still logged). - pdf::classify_pdf_enrichment(path): lightweight -- pdfimages -list (dims) + pdfinfo (pages), no byte extraction, no Marker; reuses the pipeline's own is_scanned_page_images detector so the report matches what actually happens. - `docs ingest` gains -y/--yes; the confirm follows the existing auth.rs prompt pattern. Verified: King -> BORN-DIGITAL (17 figures), Uexkull -> SCANNED BOOK (281 scans skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
… vs aspect) Add a coverage-based scanned-book detector — image drawn-size / page MediaBox — run A/B against the shipped pixel-dims aspect heuristic. A `scan_detector` policy knob (default "aspect") selects the active detector; both always run for the pre-ingest report and a loud divergence log, so coverage can be validated on the corpus before any default flip. - pdf_scan.rs (new): ScanDetector, CoverageVerdict, classify_by_coverage (per-page coverage summed and capped at 1.0; unusable ppi / missing page dims defer to the aspect page-scale test and flag low-confidence), classify_scan A/B wrapper, and page_dimensions/get_media_box via lopdf (inheritance walk + depth-10 cycle guard + array guard + Integer-or-Real numeric parsing). - pdf.rs: keep x-ppi/y-ppi/size from `pdfimages -list`; classify_pdf_enrichment returns both detector verdicts. - enrich_pdf_images gains scanned_override so coverage mode drives the skip decision; the aspect default path stays byte-identical. - archon-policy: PdfPolicy.scan_detector + loader wiring (the field was dropped by the Raw-struct layering) with known-value validation. - docs.rs: pre-ingest banner shows both verdicts, peak coverage, and a divergence warning. - scripts/coverage_oracle.py: pypdfium2 dry-run oracle validating the ppi-proxy against true placement rects. lopdf 0.36 added. archon-docs coverage/lopdf tests + archon-policy loader tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Adversarial review found the coverage/aspect pre-ingest report classified on the raw `pdfimages -list`, while the ingest pipeline filters and gates embedded images — so the report could promise "SCANNED BOOK, enrichment skipped" while the pipeline enriched a different (filtered) image set. Apply the pipeline's gate inside the classifier: honor extract_embedded_images and filter by min_image_dimension + min_image_bytes (via the size-column proxy for the extracted-PNG size — a conservative approximation that only ever drops small images, never real page-scans, so it cannot wrongly flip a scanned book to born-digital). No object dedup in the report path: `pdfimages` already lists a shared XObject once per page, which is the per-page granularity the coverage sum needs. Coverage mode now drives the pipeline from classify_scan's selected verdict — including its aspect fallback when page dims are unreadable — so the report and the pipeline never disagree in coverage mode. The aspect default path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Add a "union" scan_detector: a page is a scan if the aspect page-scale test OR the coverage >= 0.80 test flags it. The corpus dry-run (161 PDFs) showed neither base detector alone is complete — aspect misses low-DPI scans (below its 1000px pixel floor; 29 corpus docs) and coverage misses margin-cropped scans (text-block-only images that fill ~0.73 of the page; Konstan). Their union classifies every divergent corpus doc correctly: Konstan -> SCANNED (via aspect), the 29 low-DPI books -> SCANNED (via coverage), and every agreeing doc unchanged. - classify_by_union + UnionVerdict; factor coverage_per_page out of classify_by_coverage so both share the per-page coverage map, and add aspect_scan_page_set mirroring is_scanned_page_images' per-page rule. - ScanDetector::Union + parse/validation; resolve() selects it. - ingest_pdf: any non-aspect detector resolves the verdict from the path (coverage + union); aspect stays on the in-memory default, unchanged. - policy.example.toml documents "union" as the recommended setting for the corpus flip. Default remains "aspect" (opt-in). Validated end-to-end: Konstan (438pp, coverage peak 67%) now classifies SCANNED BOOK under union. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The corpus dry-run (161 PDFs) validated union as strictly better than either base detector — it catches the 29 low-DPI scans aspect misses AND the margin-cropped scans coverage misses — so promote it from opt-in to the shipped default. - PdfPolicy::default().scan_detector = "union"; the loader keeps the default on an unknown value. - ScanDetector::parse falls back to Union (not Aspect), so a typo degrades to the best detector rather than a weaker one. - policy.example.toml ships scan_detector = "union". Every PDF ingest now resolves the scanned-book verdict from the file (pdfimages + lopdf page dims); when page dims are unreadable it falls back to the shipped aspect heuristic, so the decision degrades gracefully. "aspect" / "coverage" remain selectable for A/B. 228 archon-docs lib + archon-policy tests green; validated end-to-end (no config → union active). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…(PR-E/C3) An image-only PDF (page scans, no text layer) was classified SCANNED and its enrichment skipped — but with no text layer there is nothing for Marker to OCR, so under the union default the doc got NO content at all (a regression of the old "chunks but no root" gap). Even when image-OCR did run, the chunks had no integrity root. - Gate the scanned-book skip on whether a text seal was produced: scanned + text layer -> skip (Marker/text owns the pages; unchanged); scanned + NO text layer -> OCR the page scans (the only content) with VLM forced off (a full-page scan is a page reproduction, not a discrete figure, and VLM over 100+ pages is wasteful). - Widen the content-extraction gate so Marker runs on image-only pages when configured — its surya OCR reads scanned pages into real bbox text + a normal chunks_root. - Synthetic chunks_root: image-only docs with no text seal now seal their image-OCR chunks under a synthetic OCR artifact, for the same tamper-evidence a text doc gets. - The pre-ingest banner distinguishes "SCANNED BOOK (text layer present) -> skipped" from "IMAGE-ONLY SCAN (no text layer) -> WILL be OCR'd" via a lightweight pdftotext probe. Validated: Frede (9pp image-only) now yields 55 OCR chunks + a root (was 0). New integration test (+ a pdfinfo mock hook in the test guard). 229 archon-docs lib tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…/C4, opt-in)
Scanned books bake figures INTO the page scans, so there is no discrete
embedded image for the VLM to describe and the page scans are skipped
entirely. This adds an opt-in path that crops Marker's detected FIGURE
regions from a page render and VLM-describes them.
- archon-ingest-ext: FigureRegion + parse_marker_figures — a SEPARATE walk
from parse_marker_json (text blocks) so chunk parity is byte-identical;
collects Figure/Picture/Image blocks (page + bbox).
- marker_source: blocks_and_figures_for — one sidecar run yields both the
text blocks and the figure regions; blocks_for delegates to it.
- pdf_figure_vlm: render the page (pdftoppm -singlefile), crop the bbox
(points -> px, top-left origin, NO y-flip; the `image` crate), VLM-describe
the crop, persist via persist_vlm_description; the descriptions join the
image chunks so they fold into chunks_root with everything else.
- policy: [policy.docs.pdf] figure_region_vlm (default false) + loader wiring.
Validated on the 5090 (Hullman, qwen2.5vl:7b): 7 figure regions across 3
pages, 7/7 described accurately — the descriptions cite exact figure titles
("Poll Dancing", "A Peek Into Netflix Queues", "Mapping America: Every City,
Every Block", "Organizational Chart of the House Democrats' Health Plan"),
proving the crops land precisely on the figures. Unit tests for the
coordinate mapping + figure parsing; 233 archon-docs lib + 31 ingest-ext
tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…/V-3, V-4) Given a (dissertation) quote, find where it lives in the corpus — document, page(s), and per-fragment bboxes — so a citation can be verified against the source and a PDF highlight drawn. Catches misquotes: exact confirms verbatim, fuzzy surfaces near-misses (OCR/transcription drift) with a similarity score, no match means it is not in the corpus. - quote_verify.rs (V-3): locate_quote (FTS-narrow to candidate docs, match each) + find_fragment_bboxes (within a doc). Matching is whitespace/punctuation-normalized (smart quotes, dashes, soft hyphens, collapsed whitespace) with an offset map back to the ORIGINAL text, so the returned span is verbatim source and a quote crossing a chunk boundary attributes to every chunk it spans. Fuzzy uses approximate-substring alignment (Sellers) so a drifted quote still resolves with a score. - docs.rs / cli_args (V-4): `archon docs verify-quote "<quote>" [--doc <id>] [--limit N] [--json]` — prints EXACT / FUZZY nn% / NOT FOUND, the source file + page(s) + bbox(es), and the verbatim source span to compare. Validated on the working DB (King): a verbatim phrase -> EXACT, p6 bbox [49.8,352.2,546.9,692.6]; a one-word swap -> FUZZY 92%; a nonsense quote -> NOT FOUND; --json carries the structured location + bbox. 9 unit tests (normalization + offset map + Sellers + cross-chunk + DB-backed exact/fuzzy/ absent/no-bbox); 242 archon-docs lib tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
… (flake fix) reprocess_preserves_source_and_kb_membership intermittently failed only under the full parallel test run (~40%): a concurrent PDF test's mock OCR output leaked into the reprocessed doc's chunks, flipping the content assertion. Root cause: the reprocess tests were plain #[tokio::test] (non-serial) but ingest transitively touches the PROCESS-GLOBAL OCR/VLM/embedding provider registries (and cozo's shared in-memory engine), so they raced the serial PDF tests that set those globals. Tag both reprocess tests with #[serial_test::serial(docs_global_state)] — the same group the provider- mutating PDF tests already use — so they never overlap. Verified: 7/7 clean full-suite runs (242 passed each). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
`docs verify-integrity [--doc <id>] [--json]` recomputes each document's
chunks_root (Merkle-style root over the per-chunk commit hashes) and compares
it to the sealed extract_text_spatial provenance record — surfacing the V-1
tamper-evidence that previously had no CLI. Per doc it reports INTACT / MISMATCH
/ NO-RECORD plus the covered chunk count and the record id; --json emits
{all_pass, documents[]} for scripting. A doc may carry more than one ocr_text
artifact (text + image union under C3), so it passes if ANY sealed root matches
the current chunk set.
scripts/spotcheck_corpus.sh is a self-driving post-flip verification harness: it
runs verify-integrity over all docs, samples per-doc inspect (chunks/OCR/images/
provenance), probes hybrid retrieval, and auto-derives a real verbatim phrase
from a search hit to exercise verify-quote (locate + fuzzy + absent), confirming
marker bboxes / coord_space on born-digital docs.
Validated on the live King + Uexküll docs: both chunks_root INTACT, quote-verify
returns a marker bbox on an exact-ish locate and FUZZY 93% on a perturbed phrase.
verify_chunks_root's mismatch branch is already covered by unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8sSHgWg4nHyfwYgXt1RRV
…+ integrity guards
Wire a persistent Marker server (surya models loaded once) through the existing
MarkerSource::Http transport, add VRAM-adaptive VLM-enrichment concurrency, and
harden the ingest integrity path. Built to re-ingest the dissertation corpus
with real bboxes without paying the per-document surya-model-reload cost.
Persistent Marker (no per-doc reload):
- scripts/archon_marker_server.py (NEW): FastAPI/uvicorn server, loads surya once
at startup, serves POST /convert with output byte-identical to the subprocess
sidecar; GET /health {status,models_loaded}; CUDA-OOM -> empty_cache -> CPU
retry ladder; convert_lock serializes model access.
- scripts/archon_marker_core.py (NEW): shared resolve_device + run_marker core
imported by BOTH the sidecar and the server, so the two transports cannot drift.
- archon_marker_sidecar.py: refactored to import the core (behavior-preserving;
--selftest stays torch-free).
- policy.docs.pdf.marker_url selects the Http transport (overrides marker_sidecar);
when unset, the subprocess path is byte-for-byte unchanged.
Adaptive parallel VLM enrichment:
- auto_image_workers(): derive worker count from FREE VRAM (reserve model weights
+ headroom, serial floor of 1, unified-memory cap of 2).
- --jobs <auto|N> flag; interactive prompt when auto; explicit N rejected if out
of 1..=16 rather than silently clamped. DB writes stay serial (single writer);
only the stateless OCR+VLM work fans out.
Integrity guards (so GPU starvation degrades VISIBLY, not silently):
- Http transport: a Marker error / empty result / timeout is a HARD failure
(document Failed -> sources_failed), never a silent bbox-less COORD_NONE that
still counts as "Ingested".
- /health preflight before an ingest run when marker_url is set (bounded retry
for the model-load window; hard-error if never ready).
- 900s per-convert request timeout; end-of-run COORD_MARKER vs COORD_NONE tally.
Verified: cargo build clean, 251 tests pass; two adversarial-review passes;
byte-identical subprocess/server parity; corpus re-ingest of 126 PDFs -> 124 docs
(123/124 with real marker bboxes; content-corruption audit 124/124 clean;
quote-verify EXACT with a marker bbox confirmed end-to-end).
Known follow-ups (see task notes): the VLM VRAM reserve is a hardcoded constant
that under-estimates ollama's actual allocation (~22.8GB observed vs 6.5 assumed)
and the 8GB-laptop/Mac paths are only unit-tested, not run on real hardware;
the ingestion path still lacks a hang-watchdog, and there is a tokio
runtime-drop-on-exit hang on index/model-status/reprocess.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8sSHgWg4nHyfwYgXt1RRV
The Apple detector reported the Metal accelerator's free memory as the instantaneous free-RAM snapshot minus an OS reserve (`ram_free - 6144`). On a 24GB Mac with ~11GB free at launch that is ~5112 MB — below the planner's 6000 MB surya floor (`marker_chunk_pages` returns None) — so Marker was demoted to CPU despite 24GB of unified memory. The instantaneous snapshot is the wrong budget model for unified memory, where the GPU grows into the whole shared pool. `apple_unified_gpu_budget_mb` now takes max(instantaneous-free-minus-OS, total-pool-minus-OS-minus-coresident-VLM). The VLM term (6800 MB = measured ollama qwen2.5vl:7b Metal footprint) keeps surya and the resident VLM from fighting over the shared pool. On the 24GB Mac the budget rises 5112 -> 11632 MB, above the floor, so Marker places on MPS; 16GB/8GB machines still resolve below the floor and correctly stay on CPU. The budget is only a fit-GATE, not an allocation — surya still allocates its real ~6GB. Validated on the target 24GB Mac (native arm64): Marker launched --device mps, surya completed, 1 COORD_MARKER / 0 COORD_NONE, verify-integrity intact, no OOM. Adversarially reviewed (clean, no blockers). Non-macos/CUDA detect path is untouched. 4 new unit tests cover the 24/16/8GB + plenty-free cases. Known follow-up (see doc comment): the free-independent `unified` term can over-report on a large machine under heavy non-VLM pressure, and a jetsam SIGKILL is not ladder-catchable — bound the term to a fraction of free and classify signal-kills as CPU-retryable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ls on CPU Two hardening fixes from the ultracode review of the marker-MPS budget (follow-ups tracked on task ste-bah#12). Apple unified-memory budget (crates/archon-accel/src/detect.rs): - apple_unified_gpu_budget_mb's `unified` term was free-INDEPENDENT (total - OS - VLM), so on a large machine under heavy non-VLM pressure it over-reported wildly (64GB total / 3GB free -> ~51GB "MPS-viable"). Bound it to at most 1.5x actual free RAM (APPLE_UNIFIED_OVERCOMMIT 3/2). The validated 24GB Mac case is unchanged (still 11632 -> MPS); the 64GB/3GB case now yields 4608 (< the 6000 surya floor -> CPU). Two new tests cover the bound. Marker signal-kill -> CPU retry (crates/archon-docs/src/marker_source.rs): - A marker sidecar terminated by a signal (out.status.code() == None, e.g. a jetsam/OOM-killer SIGKILL that leaves no exit-42 and no OOM stderr) was classified SidecarError::Other, so run_chunk returned Err WITHOUT trying the CPU rung. Add SidecarError::Killed; run_sidecar returns it when code() is None; run_chunk treats it like Oom and advances the GPU->CPU ladder. Two new subprocess tests (a fake sidecar that SIGKILLs on GPU, succeeds on CPU) prove the ladder advances and that a last-rung kill surfaces a clear error. cargo build clean; archon-accel 20 tests + archon-docs marker_source 10 tests pass. Adversarially reviewed (no blockers on this fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…outs Resolve the observed parallel-VLM-enrichment HANG (archon at 0 CPU jiffies during image enrichment): a wedged external call (OCR subprocess or VLM) left a JoinSet worker blocked forever, so docs.join_next().await never returned and the whole ingest stalled with no recovery (task ste-bah#10). Per-image wall-clock backstop (crates/archon-docs/src/pdf_image_enrichment.rs): - Wrap process_image (OCR + VLM for one image) in tokio::time::timeout in BOTH the serial and JoinSet paths (env ARCHON_PDF_IMAGE_TIMEOUT_SECS, default 600s -- a LOOSE backstop well above legitimate VLM time, fires only on a true hang). A timed-out image is recorded as a per-image SKIP (record_image_timeout: progress line + failure counter + warning), never fatal to the document. The JoinSet task hands its ImageWork back on timeout (JoinSet gives no task identity at join time). Defensive: break the drain loop when no work remains instead of spinning. OCR subprocess timeouts (crates/archon-docs/src/ocr/{provider,rapid,local}.rs): - Every previously un-timed `Command...output().await` now uses kill_on_drop(true) + tokio::time::timeout so a wedged binary errors out instead of hanging: RapidOCR, tesseract, and pdftotext use ocr_timeout() (per-image/text, env ARCHON_OCR_TIMEOUT_SECS default 120s); the WHOLE-document pdftoppm render uses a separate, generous pdf_render_timeout() (env ARCHON_PDF_RENDER_TIMEOUT_SECS default 1800s) -- it rasterizes every page in one invocation, so a 300-600pp scan must not be clamped by the 120s per-page bound (that regression was caught in review before commit). cargo build clean; archon-docs full suite 254 tests pass (incl. new OCR-timeout tests). Adversarially reviewed; the review's one blocker (pdftoppm mis-bounded by the per-page timeout) is fixed here via the separate render timeout. Known follow-up: the marker CONVERSION subprocess (run_sidecar) still has no timeout/kill_on_drop, so a wedged marker (MPS deadlock / surya spin) can hang a worker; run_chunk should treat an elapse like the Killed rung and advance to CPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…recovers The Subprocess marker path (run_sidecar) called cmd.output().await with no timeout and no kill_on_drop, so a marker that HANGS mid-conversion (MPS driver deadlock / surya spin / GPU contention — an observed Mac corpus-flip failure mode) never returned and the ingest worker blocked forever with no CPU fallback. The A2 signal-kill/OOM ladder only handles sidecars that EXIT; a hang never reaches it. The Http transport already bounds converts (HTTP_CONVERT_TIMEOUT_SECS=900) — this brings the Subprocess path to parity and completes the "fully debug the ingestion function" goal (with tasks ste-bah#10/ste-bah#12). - SidecarError::TimedOut { device }, alongside Oom/Killed. - marker_timeout(device): per-attempt wall clock. GPU marker is fast, so a long GPU run is a hang → tight default 900s (ARCHON_MARKER_GPU_TIMEOUT_SECS) catches it and the ladder falls to CPU. CPU marker of a large page-range chunk is legitimately slow → generous default 3600s (ARCHON_MARKER_CPU_TIMEOUT_SECS); it is the last rung, so an elapse there hard-fails the chunk exactly as a true hang would. - run_sidecar: cmd.kill_on_drop(true) + tokio::time::timeout(marker_timeout, output()); on elapse the child is killed (and reaped by tokio's process driver) and TimedOut is returned. Success path unchanged. - run_chunk: a TimedOut rung records `last` and advances the GPU->CPU ladder, mirroring Oom/Killed. Two real-subprocess tests (a bash sidecar that sleeps 30s on GPU, succeeds on CPU): a 1s GPU budget advances to CPU in ~1s (not 30s), and a last-rung 1s CPU timeout surfaces a "timed out … device=cpu" error. Both wrap the call in a 10s guard so removing the timeout fails the test rather than hanging it. cargo build clean; archon-docs full suite 256 tests pass (12 marker_source incl. the 2 new). Adversarially reviewed (no blockers). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etup guide Document the corpus-ingestion + prose-style-analytics additions this fork makes over upstream archon (3 new crates: archon-accel, archon-ingest-ext, archon-lanham; archon-docs heavily extended). - docs/dissertation-port/README.md — improvements vs original archon (both subsystems), the `archon docs` / `archon style` CLIs, an at-a-glance comparison table, and an honest inventory of experimental/incomplete parts. - docs/dissertation-port/platform-agnostic-design.md — one Marker core on MPS/CUDA/CPU, archon-accel device abstraction, free-VRAM placement + page-range chunking + OOM→CPU ladder + Apple unified budget, byte-identical cross-platform chunk parity, verify-by-recompute, and the device-agnostic hang-hardening timeouts; validated-fleet table. - docs/dissertation-port/dependencies-and-setup.md — full dependency list (Rust + LIBCLANG, embedded Cozo/RocksDB/fastembed, the Marker Python venv per platform, Ollama VLM, poppler/rapidocr/tesseract), a minimal policy.toml, a step-by-step runbook, and a tunable-env-var table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… not pending work
The multi-consumer GPU arbiter the ConsumerKind::{Whisper,FrameVlm} seam would
feed is intentionally unbuilt: media-path GPU consumers run sequentially (Marker
frees VRAM before the VLM; video ASR -> frame-VLM would too), so single-consumer
free-VRAM placement already covers the real cases. Reword so the limitations
inventory reads as a design decision, not unfinished work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates the prime directive, engineering rules, pipeline integrity, memory, and CI-gate sections into a shorter reference doc.
default_memory_data_dir/default_config_path resolve through dirs::data_dir()/dirs::config_dir(), which derive from $HOME on every platform and only consult $XDG_DATA_HOME/$XDG_CONFIG_HOME on Linux. On macOS the test's XDG overrides were a no-op, so the smoke test read the machine's real durable memory graph (accumulated counts) instead of a blank slate, failing the 0/30, 0/20, 0/3 gate assertions. Overriding $HOME isolates the CLI on both platforms. Found via a clean-clone build+test run of this branch on a Mac with real archon usage history — verified passing there and on WSL.
Trackpad/mouse-wheel scroll should work out of the box regardless of WSL detection. Users who need native terminal text selection can opt out with ARCHON_TUI_MOUSE_CAPTURE=0. Also gitignore /parity-ref/, a local portability copy of the god-agent markdown_chunker.py used by scripts/chunk_parity_check.py on machines that can't reach the WSL reference tree directly.
This reverts the mouse-capture-default change from 0477386. Keeping WSL-only default mouse capture as originally implemented; not wanted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds corpus ingestion and prose-style analytics to Archon — the
dissertation port. Delta vs the base is ~94 files, +13.7k lines, introducing
three new workspace crates and two end-user capabilities.
📖 Start here:
docs/dissertation-port/README.md· platform-agnostic design
· dependencies & setup
New crates
archon-accelarchon-ingest-extarchon-lanhamCorpus ingestion (
archon docs …)PDF → Marker (surya neural layout/OCR) block tree with real per-block
bounding boxes → token-aware structure-aware chunking → per-image OCR + optional
VLM figure descriptions → embedded vector index → tamper-evident integrity
seal. Replaces the original flat-
pdftotextpath.Highlights:
Marker footprint, page-range chunking to fit small cards, GPU→CPU OOM ladder,
bounded Apple unified-memory budget.
server that loads the ~6 GB surya models once (byte-identical output, shared
core).
chunks_rootseal on all ingest,docs verify-integrity,COORD_MARKER/COORD_NONEtally, and strict-fail (no silent bbox-lessdegradation on the HTTP path).
render / per-image wall-clock timeouts (all
kill_on_drop), plussignal-kill→CPU retry — no wedged call can hang an ingest.
Prose-style analytics (
archon style train)archon-lanhammeasures a writing sample along six Lanham axes (noun/verb,parataxis/hypotaxis, periodic/running, voice, register, opacity) plus tacit
rhetorical figures, and renders the result as an enforceable Archon output-style
that is injected into the system prompt so the model drafts in the trained voice.
Pure Rust, fully offline, golden-tested byte-for-byte against the TS reference.
Validation
Exercised on real hardware — MacBook 24 GB (Apple MPS), RTX 5070 8 GB laptop
(CUDA), RTX 5090 32 GB. The dissertation corpus flip landed 124 documents,
123 with real Marker bboxes, integrity 124/124 clean. Every ingestion-path
change in this session was implemented, adversarially reviewed, and committed:
3d4ec06persistent Marker HTTP server + adaptive parallel ingest + integrity guards41683e1Marker → Apple-Silicon MPS via unified-memory budget5451071bound Apple budget + retry marker signal-kills on CPU5de215bper-image + OCR-subprocess hang backstops002bab4bound the marker conversion subprocess9655ccfdocs: dissertation-port overview, platform-agnostic design, setupKnown limitations
Documented honestly in the README's "experimental" inventory — e.g. Lanham POS/
clause features are deferred ("L2"), the research-pipeline
style_applierisstill a stub, and the multi-consumer GPU arbitration seam (whisper/frame-VLM) is
dormant pending the video port.
🤖 Generated with Claude Code