feat(mind): add PageIndex-style structure retrieval

.agents/skills/quantmind-dev/references/develop-components.md

@@ -13,10 +13,18 @@ apply throughout.
    adding any import. `lint-imports` enforces these; if your design needs a
    forbidden import, the design is wrong — restructure, do not work around
    the contract.
-3. **Implement small and flat.** Pure functions over classes; `Protocol`
-   over ABC; no meaningless wrappers (a method must add logic, abstraction,
-   or a side effect beyond the call it wraps). No premature abstractions —
-   extract shared code when the second real caller appears, not before.
+3. **Implement small and flat.** Use functions for self-contained stateless
+   transformations. Use a small service class to bind, at construction, the
+   immutable `cfg`/policy/dependency that must stay constant across calls; the
+   operand is passed per call. Binding `cfg` for batch reproducibility alone
+   justifies a class (`PaperFlow(cfg).build(input)`,
+   `AgenticRetriever(cfg).retrieve(...)`), and the cfg *type* may select the
+   shape/strategy (typed dispatch, not a hierarchy). Do not add a class only as a
+   namespace, and do not store an implicit mutable "current input" or "current
+   result". Use `Protocol` over ABC and avoid framework-style class hierarchies.
+   No meaningless wrappers (a method must add logic, abstraction, or a side effect
+   beyond the call it wraps). No premature abstractions — extract shared code when
+   the second real caller appears, not before.
 4. **Add the unit test and the example** (sections below).
 5. **Update the public surface** if needed: package exports, the relevant
    design or guide, and the catalog in `docs/README.md`. Update the root
@@ -36,6 +44,8 @@ apply throughout.
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
 | `quantmind/rag/` | `preprocess` only |
+| `quantmind/library/` | `knowledge` only |
+| `quantmind/mind/` | `knowledge`, `configs`, `utils` (retrieval is library-free; the `library` edge is reserved for the future collection path, not single-tree `retrieve`) |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -75,20 +85,51 @@ apply throughout.
 
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
-- Public operations are `async def` functions, not classes; state passes
-  as arguments and side effects are explicit.
+- Public operations are intent-oriented async callables. Prefer a function for a
+  one-off transform; use a config-bound service class when a batch must run under
+  one fixed setting — bind `cfg` at construction (`PaperFlow(cfg)`), pass only the
+  operand per call (`build(input)`), and let the cfg *type* select the knowledge
+  shape. Instances must not hide the active input or result as mutable state, and
+  side effects stay explicit.
+- **A pipeline is pure processing: `input → self-contained artifact`.** It does
+  not bind a `library`, persist, or retrieve. Producing the artifact fully —
+  including any embeddings it carries — is the whole job. Persistence (`library`
+  dump/load) and retrieval (`mind`) are downstream steps the caller chooses.
+- Do not expose a half-finished intermediate (parse-only, source-only) as a
+  public flow; that is a component seam owned by `preprocess`/`rag`/`knowledge`.
 - Semantic operations use the OpenAI Agents SDK directly (`Agent`,
   `@function_tool`, `output_type=`); deterministic operations do not add an
   LLM. Never wrap `from agents import ...` in a facade.
 - Fan-out goes through `batch_run` (bounded concurrency, error policy);
   `batch_run` rejects `memory=` at the signature layer by design.
 
+### `quantmind/library/` — local persistence and semantic retrieval
+
+- Persist canonical knowledge and derived artifacts; dump and load, do not
+  transform. A self-contained artifact (e.g. a structure tree carrying its own
+  node text and any embeddings) round-trips through `put` / `open_*` to an
+  identical value. Do not make one artifact's content depend on refilling from
+  another at read time.
+- Keep SQLite and LlamaIndex types private; import from `knowledge` only.
+- Do not add a vector-store, retriever, provider, or backend registry.
+
 ### `quantmind/mind/` — cognitive layer
 
-- Lands via the Agents SDK migration (tracking: #71). Backends implement
-  the `Memory` Protocol with granular `tools()`, `mcp_servers()`,
-  `run_hooks()`, `reset()` — each may return an empty list; do not force
-  MCP on every implementation.
+- **Memory**: backends implement the `Memory` Protocol with granular `tools()`,
+  `mcp_servers()`, `run_hooks()`, `reset()` — each may return an empty list; do
+  not force MCP on every implementation.
+- **Retrieval (pure agentic)**: `AgenticRetriever(RetrievalCfg)` reasons over one
+  **self-contained** structure (`Retrievable`, a `StructureTree` today) with an
+  LLM agent and returns `RetrievalEvidence` values whose `content` comes from the
+  structure — no `library`, no query-time refill. The `locator` field is optional
+  provenance for cross-artifact fusion, never the path to the content. `mind` is
+  the **reasoning** layer; **mechanical** retrieval (semantic vector search /
+  BM25) is `quantmind.library` / `quantmind.rag`, and a future hybrid path
+  composes them (a semantic shortlist seeding this reasoning pass). Use the Agents
+  SDK directly (`Agent`, `@function_tool`, `output_type=`, its own `RunConfig`);
+  do not import `flows._runner`, and do not build a retriever/query-engine
+  hierarchy — one agentic strategy, no strategy registry. See
+  `contexts/design/mind/retrieval.md`.
 
 ### `quantmind/utils/`
 
@@ -101,10 +142,12 @@ A public operation is complete only when all of these agree:
 
 1. A stage and name consistent with `contexts/design/operations/naming.md`.
 2. Typed input and config models, exported from `quantmind.configs`.
-3. One intent-oriented `async def` operation exported from `quantmind.flows`,
-   with its result contract exported from the canonical owning layer.
-4. Offline success and failure tests plus a magic-introspection test when the
-   operation follows the `(input, *, cfg)` convention.
+3. One intent-oriented async function, small service class, or document-scoped
+   handle exported from `quantmind.flows`, with its result contract exported
+   from the canonical owning layer.
+4. Offline success and failure tests for the public callable, plus a
+   magic-introspection test when a function follows the `(input, *, cfg)`
+   convention.
 5. One runnable common-path example under `examples/<module>/`.
 6. A relevant design or guide and one row in `docs/README.md`.
 

.claude/skills/quantmind-dev/references/develop-components.md

@@ -13,10 +13,18 @@ apply throughout.
    adding any import. `lint-imports` enforces these; if your design needs a
    forbidden import, the design is wrong — restructure, do not work around
    the contract.
-3. **Implement small and flat.** Pure functions over classes; `Protocol`
-   over ABC; no meaningless wrappers (a method must add logic, abstraction,
-   or a side effect beyond the call it wraps). No premature abstractions —
-   extract shared code when the second real caller appears, not before.
+3. **Implement small and flat.** Use functions for self-contained stateless
+   transformations. Use a small service class to bind, at construction, the
+   immutable `cfg`/policy/dependency that must stay constant across calls; the
+   operand is passed per call. Binding `cfg` for batch reproducibility alone
+   justifies a class (`PaperFlow(cfg).build(input)`,
+   `AgenticRetriever(cfg).retrieve(...)`), and the cfg *type* may select the
+   shape/strategy (typed dispatch, not a hierarchy). Do not add a class only as a
+   namespace, and do not store an implicit mutable "current input" or "current
+   result". Use `Protocol` over ABC and avoid framework-style class hierarchies.
+   No meaningless wrappers (a method must add logic, abstraction, or a side effect
+   beyond the call it wraps). No premature abstractions — extract shared code when
+   the second real caller appears, not before.
 4. **Add the unit test and the example** (sections below).
 5. **Update the public surface** if needed: package exports, the relevant
    design or guide, and the catalog in `docs/README.md`. Update the root
@@ -36,6 +44,8 @@ apply throughout.
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
 | `quantmind/rag/` | `preprocess` only |
+| `quantmind/library/` | `knowledge` only |
+| `quantmind/mind/` | `knowledge`, `configs`, `utils` (retrieval is library-free; the `library` edge is reserved for the future collection path, not single-tree `retrieve`) |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -75,20 +85,51 @@ apply throughout.
 
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
-- Public operations are `async def` functions, not classes; state passes
-  as arguments and side effects are explicit.
+- Public operations are intent-oriented async callables. Prefer a function for a
+  one-off transform; use a config-bound service class when a batch must run under
+  one fixed setting — bind `cfg` at construction (`PaperFlow(cfg)`), pass only the
+  operand per call (`build(input)`), and let the cfg *type* select the knowledge
+  shape. Instances must not hide the active input or result as mutable state, and
+  side effects stay explicit.
+- **A pipeline is pure processing: `input → self-contained artifact`.** It does
+  not bind a `library`, persist, or retrieve. Producing the artifact fully —
+  including any embeddings it carries — is the whole job. Persistence (`library`
+  dump/load) and retrieval (`mind`) are downstream steps the caller chooses.
+- Do not expose a half-finished intermediate (parse-only, source-only) as a
+  public flow; that is a component seam owned by `preprocess`/`rag`/`knowledge`.
 - Semantic operations use the OpenAI Agents SDK directly (`Agent`,
   `@function_tool`, `output_type=`); deterministic operations do not add an
   LLM. Never wrap `from agents import ...` in a facade.
 - Fan-out goes through `batch_run` (bounded concurrency, error policy);
   `batch_run` rejects `memory=` at the signature layer by design.
 
+### `quantmind/library/` — local persistence and semantic retrieval
+
+- Persist canonical knowledge and derived artifacts; dump and load, do not
+  transform. A self-contained artifact (e.g. a structure tree carrying its own
+  node text and any embeddings) round-trips through `put` / `open_*` to an
+  identical value. Do not make one artifact's content depend on refilling from
+  another at read time.
+- Keep SQLite and LlamaIndex types private; import from `knowledge` only.
+- Do not add a vector-store, retriever, provider, or backend registry.
+
 ### `quantmind/mind/` — cognitive layer
 
-- Lands via the Agents SDK migration (tracking: #71). Backends implement
-  the `Memory` Protocol with granular `tools()`, `mcp_servers()`,
-  `run_hooks()`, `reset()` — each may return an empty list; do not force
-  MCP on every implementation.
+- **Memory**: backends implement the `Memory` Protocol with granular `tools()`,
+  `mcp_servers()`, `run_hooks()`, `reset()` — each may return an empty list; do
+  not force MCP on every implementation.
+- **Retrieval (pure agentic)**: `AgenticRetriever(RetrievalCfg)` reasons over one
+  **self-contained** structure (`Retrievable`, a `StructureTree` today) with an
+  LLM agent and returns `RetrievalEvidence` values whose `content` comes from the
+  structure — no `library`, no query-time refill. The `locator` field is optional
+  provenance for cross-artifact fusion, never the path to the content. `mind` is
+  the **reasoning** layer; **mechanical** retrieval (semantic vector search /
+  BM25) is `quantmind.library` / `quantmind.rag`, and a future hybrid path
+  composes them (a semantic shortlist seeding this reasoning pass). Use the Agents
+  SDK directly (`Agent`, `@function_tool`, `output_type=`, its own `RunConfig`);
+  do not import `flows._runner`, and do not build a retriever/query-engine
+  hierarchy — one agentic strategy, no strategy registry. See
+  `contexts/design/mind/retrieval.md`.
 
 ### `quantmind/utils/`
 
@@ -101,10 +142,12 @@ A public operation is complete only when all of these agree:
 
 1. A stage and name consistent with `contexts/design/operations/naming.md`.
 2. Typed input and config models, exported from `quantmind.configs`.
-3. One intent-oriented `async def` operation exported from `quantmind.flows`,
-   with its result contract exported from the canonical owning layer.
-4. Offline success and failure tests plus a magic-introspection test when the
-   operation follows the `(input, *, cfg)` convention.
+3. One intent-oriented async function, small service class, or document-scoped
+   handle exported from `quantmind.flows`, with its result contract exported
+   from the canonical owning layer.
+4. Offline success and failure tests for the public callable, plus a
+   magic-introspection test when a function follows the `(input, *, cfg)`
+   convention.
 5. One runnable common-path example under `examples/<module>/`.
 6. A relevant design or guide and one row in `docs/README.md`.
 

.github/workflows/e2e.yml

@@ -25,6 +25,12 @@ on:
       - 'quantmind/knowledge/paper.py'
       - 'quantmind/library/**'
       - 'scripts/verify_pdf_rag_e2e.py'
+      - 'scripts/verify_structure_e2e.py'
+      - 'quantmind/mind/**'
+      - 'quantmind/configs/retrieval.py'
+      - 'quantmind/configs/structure.py'
+      - 'quantmind/flows/paper/**'
+      - 'quantmind/preprocess/outline.py'
       - 'pyproject.toml'
   schedule:
     - cron: "17 3 * * *"
@@ -44,6 +50,7 @@ jobs:
     outputs:
       news: ${{ steps.filter.outputs.news }}
       paper_flow: ${{ steps.filter.outputs.paper_flow }}
+      structure: ${{ steps.filter.outputs.structure }}
 
     steps:
       - name: Checkout
@@ -77,6 +84,19 @@ jobs:
               - 'quantmind/preprocess/format/pdf.py'
               - 'quantmind/rag/**'
               - 'pyproject.toml'
+            structure:
+              - '.github/workflows/e2e.yml'
+              - 'scripts/verify_structure_e2e.py'
+              - 'quantmind/mind/**'
+              - 'quantmind/configs/retrieval.py'
+              - 'quantmind/configs/structure.py'
+              - 'quantmind/flows/paper/**'
+              - 'quantmind/knowledge/paper.py'
+              - 'quantmind/knowledge/_tree.py'
+              - 'quantmind/library/**'
+              - 'quantmind/preprocess/outline.py'
+              - 'quantmind/preprocess/format/pdf.py'
+              - 'pyproject.toml'
 
   news:
     needs: changes
@@ -151,3 +171,47 @@ jobs:
       - name: Run live Paper Flow V1 E2E
         if: steps.openai.outputs.available == 'true'
         run: .venv/bin/python scripts/verify_pdf_rag_e2e.py
+
+  structure:
+    needs: changes
+    if: github.event_name != 'pull_request' || needs.changes.outputs.structure == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    env:
+      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+          cache-dependency-glob: pyproject.toml
+
+      - name: Create virtual environment
+        run: uv venv
+
+      - name: Install project runtime dependencies
+        run: uv pip install --python .venv/bin/python -e .
+
+      - name: Check OpenAI credential
+        id: openai
+        shell: bash
+        run: |
+          if [[ -n "${OPENAI_API_KEY}" ]]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+            echo "::notice::Skipping structure-retrieval E2E because OPENAI_API_KEY is not configured."
+          fi
+
+      - name: Run live structure-retrieval E2E
+        if: steps.openai.outputs.available == 'true'
+        run: .venv/bin/python scripts/verify_structure_e2e.py

AGENTS.md

@@ -39,7 +39,7 @@ handoff all come from `openai-agents`.
 | `quantmind/rag/` | Opinionated LlamaIndex document chunking and retrieval — depends only on `preprocess` |
 | `quantmind/flows/` | Apex layer: public library operations (`paper_flow`, `collect_news`, `batch_run`) |
 | `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` |
-| `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) |
+| `quantmind/mind/` | Pure-agentic reasoning layer — memory + agentic (reasoning-based) retrieval where an LLM decides; mechanical retrieval (similarity / BM25) lives in `rag` / `library` |
 | `quantmind/utils/` | Logger only — keep it that way |
 
 The pre-migration agent runtime was removed and archived on the
@@ -74,8 +74,15 @@ the user explicitly authorizes it — fix the underlying issue instead.
 
 ## Architecture Constraints (stable)
 
-1. **Library, not framework** — functions over classes, `Protocol` over ABC,
-   no plugin registries, no hook discovery, no CLI.
+1. **Library, not framework** — use functions for self-contained stateless
+   transformations and small service classes that bind, at construction, the
+   immutable `cfg`/policy/dependency that must stay constant across calls; the
+   runtime operand is passed per call. Binding `cfg` for batch reproducibility
+   alone justifies a class (`PaperFlow(cfg).build(input)`,
+   `AgenticRetriever(cfg).retrieve(structure, q)`), and the cfg *type* may select the
+   shape/strategy (typed dispatch, not a class hierarchy). Keep canonical values
+   free of runtime service state; use `Protocol` over ABC, with no
+   framework-style class hierarchies, plugin registries, hook discovery, or CLI.
 2. **RAG data plane, not framework** — use LlamaIndex directly inside
    `quantmind.rag`; keep upstream types private and do not add retriever,
    vector-store, provider, or backend registries.
@@ -93,7 +100,23 @@ the user explicitly authorizes it — fix the underlying issue instead.
    side effect beyond the call it wraps; otherwise inline it.
 8. **Name public operations by intent** — follow
    `contexts/design/operations/naming.md`; use stage verbs, and reserve
-   `pipeline` for deliberate multi-stage composition.
+   `pipeline` for deliberate multi-stage composition. `flow` as a verb or
+   `*_flow` function name is banned; `Flow` as a noun on a document handle
+   (`PaperFlow`) is allowed.
+9. **Pipelines produce self-contained artifacts** — a `flows` pipeline is pure
+   processing (`input → artifact`) and returns a value usable *and storable*
+   without a store; it does not bind a `library`, persist, or retrieve.
+   `library` only dumps and loads (`put(artifact)` / `open_*`, round-tripping to
+   an identical value); `mind` only retrieves, returning evidence **values**
+   (content included) with any locator as optional provenance. A self-contained
+   artifact carries its own text (and any embeddings) plus the minimal
+   provenance metadata (`as_of` + a light source ref) needed to persist and
+   time-query it standalone — never a reference refilled from a store. Keep that
+   provenance metadata out of the artifact's `id` / `content_hash` (identity
+   stays reproducible); share it via a light provenance base, not full
+   `BaseKnowledge`. Accept modest redundancy to keep artifacts self-contained.
+   Half-finished intermediates stay component seams, not public flows. See
+   `contexts/design/operations/orchestration.md`.
 
 ## Tests and Examples