diff --git a/contexts/design/README.md b/contexts/design/README.md index d055829..5d4209e 100644 --- a/contexts/design/README.md +++ b/contexts/design/README.md @@ -30,6 +30,7 @@ implementation must preserve. | Preprocess | [Page-aware multimodal PDF parsing](preprocess/pdf.md) | | RAG | [Page-aware document chunking and retrieval](rag/document.md) | | Library | [Local knowledge storage and meaning-based search](library/local.md) | +| Mind | [Build and retrieve from a page-preserving structure tree](mind/retrieval.md) | | Operations | [Public operation naming](operations/naming.md) | | Operations | [Orchestration and construction altitude](operations/orchestration.md) | diff --git a/contexts/design/knowledge/paper.md b/contexts/design/knowledge/paper.md index e1182a9..f0fe077 100644 --- a/contexts/design/knowledge/paper.md +++ b/contexts/design/knowledge/paper.md @@ -92,4 +92,4 @@ Canonical paper models do not implement `embedding_text()` and do not select ret `LegacyPaper` retains the pre-V1 `TreeKnowledge` shape only so existing version-2 databases and the bundled legacy example can be opened. It is not exported as `Paper`, is not produced by `paper_flow`, and is not part of the V1 paper contract. -There is no `PaperTree` in V1. A future tree or PageIndex artifact must be independently versioned, linked to an exact source and inputs, and added only with a real retrieval use case and migration plan. +There is no `PaperTree` in V1. A future tree or PageIndex artifact must be independently versioned, linked to an exact source and inputs, and added only with a real retrieval use case and migration plan. The accepted plan for that artifact is `PaperStructureTree`, a paper-artifact binding of a shared `StructureTree` base; see [Build and retrieve from a page-preserving structure tree](../mind/retrieval.md). diff --git a/contexts/design/library/local.md b/contexts/design/library/local.md index f698ef4..7862436 100644 --- a/contexts/design/library/local.md +++ b/contexts/design/library/local.md @@ -127,9 +127,9 @@ Reads fail closed when canonical hashes, counts, IDs, membership, lineage, sourc ## PageIndex Boundary -The library is canonical storage plus collection-wide semantic search, not a vector database abstraction. A future PageIndex path may first select a paper through `search()` and then navigate a separate, independently versioned tree artifact through an opinionated operation under `quantmind.rag`. +The library is canonical storage plus collection-wide semantic search, not a vector database abstraction. A future PageIndex path may first select candidate nodes through `search()` and then retrieve over a separate paper structure-tree artifact. That agentic, library-backed retrieval is owned by `quantmind.mind`, and `resolve()` is extended to the structure-tree kind. See [Build and retrieve from a page-preserving structure tree](../mind/retrieval.md). -PageIndex is not required to use `LocalKnowledgeLibrary.search()` or private LlamaIndex vector ranking. Paper Flow V1 deliberately stores chunks and a cited summary without defining a paper tree. +PageIndex is not required to use `LocalKnowledgeLibrary.search()` or private LlamaIndex vector ranking. Paper Flow V1 deliberately stores chunks and a cited summary without defining a paper tree. A structure tree is stored as a source-linked paper artifact whose canonical persistence needs no embeddings; building per-node projections is an explicit later step that then enables hybrid semantic-plus-agentic retrieval without a second index. ## Out of Scope @@ -137,5 +137,5 @@ PageIndex is not required to use `LocalKnowledgeLibrary.search()` or private Lla - implicit persistence inside flows; - answer synthesis or agent memory; - merging distinct canonical identities; -- a V1 paper tree or PageIndex navigation API; +- a V1 paper tree or PageIndex retrieval API; - treating rebuildable projections as canonical knowledge. diff --git a/contexts/design/mind/retrieval.md b/contexts/design/mind/retrieval.md new file mode 100644 index 0000000..3bca1b2 --- /dev/null +++ b/contexts/design/mind/retrieval.md @@ -0,0 +1,315 @@ +# Build and retrieve from a page-preserving structure tree + +## Quick Summary + +- **Purpose**: Define how a source-first paper gains a validated structure tree + and how a reasoning agent traverses it, without embeddings replacing reasoning. +- **Read when**: Designing or changing PageIndex-style tree construction, the + `mind` retrieval package, or hybrid semantic-plus-agentic retrieval. +- **Status**: Planned. No implementation exists on `master`. This page records + the accepted design and refines issue #95 for the source-first Paper Flow V1 + model shipped in #120; the `quantmind.mind` package does not exist yet. +- **Core rule**: One tree implementation. A shared `StructureTree` base carries + structure, traversal, and the integrity gate; each document type subclasses it + for identity. The paper structure tree is a derived, source-linked artifact whose model + returns only a draft while code validates every node. Retrieval reasons over + titles and summaries; embeddings are a coarse pre-filter added later, never a + replacement. +- **Canonical models**: [Paper source and artifact design](../knowledge/paper.md). +- **Related work**: issues #122 (context), #95 (feature request), #71 (`mind` + scaffold), #120 (source-first Paper Flow V1); prior art `VectifyAI/PageIndex` + (MIT). + +## Contents + +- [Motivation](#motivation) +- [Design at a Glance](#design-at-a-glance) +- [Ownership](#ownership) +- [Structure Tree Base and Paper Binding](#structure-tree-base-and-paper-binding) +- [Build Pipeline](#build-pipeline) +- [Retrieval](#retrieval) +- [Multi-Model Compatibility](#multi-model-compatibility) +- [Hybrid Search Compatibility](#hybrid-search-compatibility) +- [Boundaries and Import Contracts](#boundaries-and-import-contracts) +- [Verification Slice](#verification-slice) +- [Out of Scope](#out-of-scope) + +## Motivation + +Vector retrieval assumes the passage most similar to a query in embedding space +is the most relevant one. For long, structured financial documents that +assumption breaks: near-identical passages differ on a threshold or exception; +fixed-size chunking fragments a table; a cross-reference such as "see Item 7A" +shares no similarity with its target; and a stateless retriever cannot use prior +reasoning to decide where to look. + +Reasoning-based retrieval reframes the problem as relevance classification over a +document's real structure: an agent reads a tree of section titles and +summaries, picks a branch, drills down, and lazily loads leaf text with exact +page provenance. `quantmind.knowledge` already records this as the purpose of +`TreeKnowledge`, and embeddings there "act as a coarse pre-filter, never as a +replacement for that reasoning." + +## Design at a Glance + +The build spine is solid; the dotted branch is the later hybrid path that adds +embeddings. Each package owns one stage, and every deterministic or code-owned +stage carries no model call. + +```mermaid +flowchart TD + PD["ParsedDocument (page-aware)"] + subgraph PRE["preprocess — deterministic"] + OUT["outline signals: TOC / headings / page offset"] + end + subgraph FLW["flows"] + DRAFT["draft structuring agent (model, private draft)"] + end + subgraph KNW["knowledge"] + FD["PaperStructureTree.from_draft() (code): mint ids, links, citations + validate() gate"] + end + subgraph LIB["library"] + PUT["put artifact: paper_artifacts (no embeddings)"] + PROJ["per-node projections (embeddings)"] + SRCH["search(SemanticQuery)"] + RES["resolve(locator): page-cited content"] + end + subgraph MND["mind"] + NAV["retrieve(structure, question): single-pass / agentic"] + end + PD --> OUT + OUT --> DRAFT + DRAFT --> FD + FD --> PUT + PUT --> NAV + NAV -->|get_node_content| RES + PUT -.->|P2 projections| PROJ + PROJ -.->|candidates| SRCH + SRCH -.->|seed_locators| NAV +``` + +## Ownership + +Each existing package keeps its responsibility; only agentic traversal introduces +a new owner, `mind`. No shared runtime is moved and no second store is added. + +| Owner | Responsibility | +|---|---| +| `quantmind.preprocess` | Emit deterministic outline signals (heading candidates, table-of-contents pages, printed-to-physical page offset) from a parsed document. No LLM calls. | +| `quantmind.knowledge` | Add the `StructureTree` structural base (reused by a refactored `TreeKnowledge`) plus a `PaperStructureTree(StructureTree)` artifact whose `from_draft` constructor mints identity and runs the shared integrity gate. | +| `quantmind.flows` | Run one draft-structuring agent and call the knowledge constructor. Reuses `flows._runner` unchanged. Persistence stays explicit. | +| `quantmind.library` | Persist the artifact through the existing paper artifact tables and extend `resolve()` to the new kind. Per-node projections are a separate later step. | +| `quantmind.mind` | Traverse the tree with the Agents SDK and return node evidence. May request a semantic shortlist from `library`. | + +`quantmind.rag` is unchanged: it stays deterministic chunking and BM25 with no +LLM dependency and hosts no PageIndex draft producer. + +## Structure Tree Base and Paper Binding + +The tree structure is shared across document types; the identity binding is not. +The design factors the two apart so the codebase keeps exactly one tree, not two. + +```mermaid +classDiagram + class StructureTree { + <> + +UUID root_node_id + +Map nodes + +root() + +children_of() + +walk_dfs() + +find_path() + +validate() + } + class TreeNode { + +UUID node_id + +UUID parent_id + +str title + +str summary + } + class Citation { + +int page + +UUID node_id + } + class BaseKnowledge { + <> + } + class TreeKnowledge + class PaperStructureTree { + +PaperArtifactKind artifact_kind + +UUID source_revision_id + +str producer_config_hash + +str content_hash + +from_draft() + } + class PaperSourceRevision { + <> + } + class PaperChunkSet + + StructureTree <|-- TreeKnowledge + BaseKnowledge <|-- TreeKnowledge + StructureTree <|-- PaperStructureTree + StructureTree *-- TreeNode : nodes + TreeNode *-- Citation : citations + PaperStructureTree ..> PaperSourceRevision : source_revision_id + PaperStructureTree ..> PaperChunkSet : lineage +``` + +`StructureTree` is a structural base — a plain `BaseModel` with no +`BaseKnowledge` identity: `root_node_id: UUID`, `nodes: dict[UUID, TreeNode]`, the +traversal surface (`root()`, `children_of()`, `walk_dfs()`, `find_path()`), and +the `validate()` integrity gate. It carries no `id`, `as_of`, or `source`, so a +subclass adds whatever identity its storage model needs without a second +competing identity. The existing `TreeKnowledge` is refactored to +`TreeKnowledge(BaseKnowledge, StructureTree)` and reuses the same nodes, +helpers, and gate instead of defining its own. + +A structure tree is a derived artifact — rebuildable from a source plus a +producer configuration, like `PaperChunkSet` — not canonical knowledge, so the +paper binding is a paper artifact rather than a `TreeKnowledge` stored as a +knowledge item. `PaperStructureTree(StructureTree)` adds: + +- `artifact_kind = PaperArtifactKind.STRUCTURE_TREE` and `schema_version`; +- `source_revision_id` binding it to an exact `PaperSourceRevision`; +- a `producer` config (model, prompt version, input chunk-set id, instructions + hash, structuring bounds) and its `producer_config_hash`; +- a `content_hash` over the canonical tree and lineage to the input + `PaperChunkSet` via `paper_artifact_lineage`. + +A stable, source-and-producer-derived id makes an unchanged re-run idempotent and +versions a changed configuration rather than overwriting it, exactly as the other +paper artifacts behave. A future document type adds its own `StructureTree` +subclass with its own source binding; nothing paper-specific leaks into the base. + +Page ranges reuse `Citation`: a node spanning pages 5-8 carries four +`Citation(page=5..8)` entries on `TreeNode.citations`. No `end_page` field or new +range rule is added. A leaf does not copy chunk text; `TreeNode.content` stays +empty and provenance is a citation pointing at the owning chunk, so source text +is never duplicated into the tree. + +## Build Pipeline + +Construction mirrors `paper_flow`: deterministic work in code, one bounded model +call for the draft, and code-owned identity and validation. + +1. **Outline signals (`preprocess`, deterministic).** From the page-aware + `ParsedDocument`, detect table-of-contents pages, heading candidates, and the + printed-to-physical page offset. Emit ordered, page-anchored signals; make no + LLM call. +2. **Draft structuring (`flows`, one agent).** An agent proposes a private draft + hierarchy from outline signals and chunk summaries: titles, nesting, per-node + draft summaries, and candidate chunk indices per node. The draft chooses no + ids, links, or canonical citations, exactly like the summary draft. +3. **Canonicalization and integrity gate (`knowledge`).** + `PaperStructureTree.from_draft(chunk_set, *, producer, draft)` mints node + ids, builds parent/child links, and resolves each node's `Citation` entries + from the chunk set, then runs the shared `StructureTree.validate()` gate. The + gate rejects any tree that is not single-rooted and acyclic with every node + reachable, bidirectional parent/child consistency, unique sibling positions, + no orphan, every cited page within the source, and every child's cited pages + contained in its parent's. A low structuring-quality signal falls back to a + flat single-level tree rather than an unverified hierarchy. +4. **Persistence (`library`).** Store the artifact through the paper artifact + tables. Persistence needs no embeddings. + +Steps 1, 3, and 4 have no model dependency and are testable without a network. + +## Retrieval + +Retrieval lives in `quantmind.mind.retrieval` and returns node evidence, never +a synthesized answer: + +```text +retrieve(structure, question, *, library, cfg, seed_locators=None) + -> list[RetrievalEvidence] +``` + +`structure` is any `StructureTree` — a `PaperStructureTree` today. Retrieval is +written against the base, so a future document type reuses it unchanged. Leaf +content is resolved through the existing `LocalKnowledgeLibrary.resolve()`, +extended to the new artifact kind, so no parallel page-resolver concept is added. +Two grains are supported: + +- **Single-pass selection.** Serialize the tree with leaf text stripped (ids, + titles, summaries, hierarchy), make **one** model call for the relevant node + ids, then resolve their page-cited content in code. +- **Agentic traversal.** Expose two SDK `@function_tool` functions — + `get_document_structure()` (tree without leaf text) and + `get_node_content(node_ids)` (page-cited leaf text via `resolve()`) — and let an + Agent decide, turn by turn, which node to open and when it has enough evidence. + +Retrieval calls `agents.Runner.run(...)` with its own `RunConfig` directly; it +does not import `flows._runner`. Whole-tree serialization is bounded by a +structure token budget over titles and summaries. + +## Multi-Model Compatibility + +- **Rely on the SDK.** `cfg.model` (a plain string, including a + `litellm//` value) flows unchanged into the SDK `Runner`, + which already routes multiple providers. No provider-resolution wrapper is + added. +- **Capability requirement.** Draft structuring needs reliable structured output + and agentic traversal needs tool-calling; a provider that lacks the capability a + stage needs is unsupported for that stage. Tests cover at least one non-OpenAI + model across both stages. +- **Embeddings.** The hybrid step depends only on the library's existing + `_EmbeddingProvider` seam and on `SemanticQuery` / `SemanticHit`, never on a + specific vendor. + +## Hybrid Search Compatibility + +Hybrid retrieval — shortlist nodes by semantic search, then reason over the +shortlist — is a later, explicit step that reuses locator identity end to end. It +is the only part that needs an embedding provider; build and pure-agentic +retrieval do not. + +- Building per-node projections is an explicit later step. Once built, + `search(SemanticQuery(...))` returns `SemanticHit` values that already carry a + full `ArtifactLocator`; the design keeps the locator and never collapses a hit + to a bare `node_id`. +- Seeds are validated locators: `retrieve(..., seed_locators=...)`. Single-tree + mode constrains the query with `tree_id = artifact.id` and rejects any seed + whose artifact id does not match, so a hit from another tree, a flat item, or a + chunk cannot leak in. Cross-artifact traversal keys seeds by the full + `(artifact_id, node_id)` locator. +- Embeddings stay a coarse pre-filter: the agent may leave the seeded subtree, and + a hit never becomes an answer without the reasoning step. + +Pure-agentic retrieval passes no seeds; hybrid adds one shortlist step in front +of the same primitive. + +## Boundaries and Import Contracts + +Placement satisfies the existing `import-linter` contracts, which already forbid +`library` and `rag` from importing `quantmind.mind`. When `mind` is implemented, +add one contract pinning `mind -> library -> knowledge`: `mind` may import +`knowledge`, `library`, `configs`, and `utils`, but not `flows`, `magic`, `rag`, +or `preprocess`; `flows` may import `mind`. Nothing moves out of `flows`, and +`rag` keeps its imports-only-`preprocess` rule. + +## Verification Slice + +Offline tests use fixed generated PDFs and a fake structuring provider. They +cover a clean table of contents, a missing table of contents, a printed +page-number reset, and an in-body cross-reference; integrity rejection of cyclic, +orphaned, and out-of-page-range trees; stable ids and idempotent re-runs; citation +resolution to correct pages; reopen behavior; and, once projections exist, seeded +hybrid retrieval with locator validation. + +A bounded live slice builds a structure tree over one exact arXiv revision with a +real model, traverses it single-pass and agentically, and prints the selected +titles, resolved page-cited content, and citation pages. It runs at least one +non-OpenAI model to exercise the capability requirement. + +## Out of Scope + +- a nested `TreeKnowledge` inside an artifact, a second parallel tree + implementation, or a `PaperTree` on `Paper`; +- a `Citation.end_page` field or a separate page-resolver concept; +- a shared runtime module or moving `flows._runner`; +- a public retriever, vector-store, provider registry, or query-engine hierarchy; +- a second persistence or semantic-index layer; +- answer synthesis or agent memory inside the retrieval primitive; +- corpus-level virtual nodes or query-time tree reconstruction; +- knowledge-graph construction. diff --git a/contexts/design/preprocess/pdf.md b/contexts/design/preprocess/pdf.md index 4e29959..a3124de 100644 --- a/contexts/design/preprocess/pdf.md +++ b/contexts/design/preprocess/pdf.md @@ -31,6 +31,6 @@ Preprocessing ends after producing `ParsedDocument`. It does not chunk, index, r - [`quantmind.rag`](../rag/document.md) converts the parsed value into LlamaIndex-backed chunks and page-aware retrieval evidence. - [`paper_flow`](../flow/paper.md) uses the preserved source pages to build an exact source revision and a canonical chunk set before generating a cited summary. -- A future PageIndex adapter may consume the same ordered pages to propose independently versioned navigation evidence. +- A future PageIndex adapter may consume the same ordered pages to propose independently versioned outline signals for a structure tree. Flattened Markdown remains a compatibility view produced from the preserved pages. It is not the primary parsing result. diff --git a/contexts/design/rag/document.md b/contexts/design/rag/document.md index 5d4903e..500448e 100644 --- a/contexts/design/rag/document.md +++ b/contexts/design/rag/document.md @@ -56,7 +56,7 @@ The conversion replaces parser paths with canonical source asset IDs and validat Document-local RAG and collection search have different responsibilities. [`LocalKnowledgeLibrary`](../library/local.md) stores canonical sources and artifacts in SQLite and privately uses LlamaIndex for collection-wide embedding ranking. It does not persist transient `ParsedDocumentHit` values. -Paper Flow V1 defines no paper tree. A future PageIndex implementation may live under `quantmind.rag` as another opinionated document operation. It may consume `ParsedDocument` or an exact paper source and return a bounded draft or navigation evidence, while canonical IDs, links, citations, and source-backed text remain code-owned. +Paper Flow V1 defines no paper tree. A PageIndex-style structure tree is not a RAG operation: deterministic outline signals live in `quantmind.preprocess`, building the persisted canonical tree lives in `quantmind.flows`, and library-backed agentic retrieval lives in `quantmind.mind`. `quantmind.rag` stays deterministic chunking and BM25 with no LLM dependency and hosts no draft producer. See [Build and retrieve from a page-preserving structure tree](../mind/retrieval.md). Canonical IDs, links, citations, and source-backed text remain code-owned throughout. ## What This Package Does Not Abstract