feat(library): local semantic knowledge library MVP

[keli-wen]

Feature Summary

Add an opinionated local semantic knowledge library for QuantMind's typed financial knowledge.

The public surface is deliberately small:

• LocalKnowledgeLibrary
• SemanticQuery
• SemanticHit

It persists canonical BaseKnowledge, indexes both knowledge items and tree nodes, performs financial-time-aware semantic search, and returns QuantMind evidence with source and citation metadata.

This is the first retrieval priority. It establishes the semantic baseline before PageIndex-style agentic navigation in #95.

Motivation

QuantMind defines embedding_text() on knowledge items and tree nodes, but it does not provide embedding generation, a durable semantic index, or a semantic query API.

A semantic baseline has broader demand than long-document navigation:

1. It works across flat cards, whole documents, and tree nodes.
2. It provides fast cross-document candidate retrieval.
3. It gives feat: page-preserving TreeKnowledge build + agentic navigation #95 a measurable baseline and future hybrid shortlist.
4. It provides one stable financial-domain API for agent tools and examples.

QuantMind must not become another generic RAG framework. The differentiating work is mapping typed financial knowledge to auditable search evidence, not provider registries, vector database abstractions, or answer generation.

Scope

Package boundary

Introduce a top-level quantmind/library/ package:

quantmind/library/
├── __init__.py
├── _types.py
├── _projection.py
├── _ports.py
├── _embed.py
└── local.py

Layer semantics:

• knowledge/ defines immutable canonical schemas and remains free of I/O.
• library/ persists, indexes, and searches canonical knowledge collections.
• mind/ consumes the library through agent tools; the library never depends on mind.
• flows/ may write extracted results into the library.
• preprocess/ remains independent and does not become a storage service.

Update import-linter so library may depend on knowledge and low-level external libraries, but not on mind, flows, magic, or preprocess.

Private provider/backend seams are allowed only when they isolate external side effects or enable deterministic tests. A helper that only forwards a call must be inlined.

Canonical and derived data

• Canonical knowledge is the source of truth.
• Embeddings and searchable records are derived and rebuildable, even when stored in the same SQLite file.
• Raw PDF, HTML, and media retention is caller-owned and outside V1.
• FlattenKnowledge: one searchable record from embedding_text().
• TreeKnowledge: one document/root record plus one record per non-root TreeNode; the item record represents the root once with node_id=None.
• Stable target identity distinguishes an item from one of its nodes.
• Index metadata records model, dimensions, projection hash, source content hash, and schema version.
• Re-putting unchanged knowledge is idempotent and does not call the embedding provider again. This guarantee is scoped to the same canonical item ID; cross-extraction deduplication needs a separate stable-ID contract.
• Changed projection/model/schema metadata invalidates only affected vectors.

V1 uses one private local implementation. SQLite plus exact cosine ranking with the existing NumPy dependency is sufficient. Do not expose a backend contract before a second real backend exists.

Financial time semantics

BaseKnowledge.as_of is the information cutoff represented by an item, not necessarily the time the source became observable. Add optional BaseKnowledge.available_at and keep the query filters distinct:

• as_of_before: information-cutoff filtering;
• available_at_before: source-availability filtering for no-look-ahead research.

Flows should populate available_at from source publication time. If only fetch time is known, SourceRef.fetched_at is a conservative upper bound. When available_at_before is provided, records with unknown availability are excluded rather than assumed safe.

Other V1 filters are limited to canonical fields:

• item_types
• source_kinds
• confidence
• tags
• tree_id
• top_k

Proposed API

from quantmind.library import LocalKnowledgeLibrary, SemanticQuery

library = await LocalKnowledgeLibrary.open(
    ".quantmind/library.db",
    embedding_model="text-embedding-3-small",
)

try:
    await library.put(paper)
    hits = await library.search(
        SemanticQuery(
            text="management expects capital expenditure to increase",
            item_types=["earnings", "paper"],
            as_of_before=information_cutoff,
            available_at_before=research_cutoff,
            top_k=10,
        )
    )
    item = await library.get(hits[0].item_id)
finally:
    await library.close()

A SemanticHit carries:

item_id
node_id | None
item_type
score
matched_text
as_of
available_at | None
source
citations

Provider- or database-specific response types must not leak through the public API. matched_text is the exact projection used for ranking; get(item_id) returns the validated canonical BaseKnowledge so callers can resolve node IDs, paths, and full content. delete(item_id) transactionally removes the canonical item and all derived root/node records.

Non-goals

• Public Embedder, VectorStore, Retriever, or backend frameworks
• Backend registry or multiple vector-database adapters
• ANN/HNSW in V1
• Reranking or RAG answer generation
• Automatic semantic/PageIndex routing
• PageIndex tree navigation (feat: page-preserving TreeKnowledge build + agentic navigation #95)
• Knowledge Graph construction
• Generic Document + metadata: dict APIs
• Raw source artifact storage in V1
• Cross-extraction deduplication for separately generated UUIDs

Acceptance Criteria

• Public API contains only the domain-level library/query/hit types.
• Flat items, tree roots, and tree nodes are indexed at the documented grain without indexing the root twice.
• Hits include the exact matched projection text, and get() resolves item/node evidence from canonical knowledge.
• as_of_before and available_at_before have distinct tested semantics.
• Availability filtering excludes unknown or post-cutoff records and does not claim that as_of alone prevents look-ahead.
• Source, confidence, tag, type, and tree filters work together.
• Scores are deterministic for a fake embedder and ordered best-first.
• Re-putting the same canonical item ID is idempotent.
• Changed model/dimension/text/schema metadata invalidates affected embeddings.
• Deleting knowledge removes its root and node index records without conflating canonical deletion with derived-index rebuilds.
• get() and delete() have explicit not-found and stale-data behavior.
• Dimension mismatches and corrupt index data fail clearly.
• No network is required by unit tests; provider/backend seams remain private.
• Unit tests live under tests/library/ and inherit unittest.TestCase or unittest.IsolatedAsyncioTestCase.
• A single simple usage example lives under examples/library/.
• scripts/verify.sh passes.

Related Issues

• Relates to Tracking: Migrate Agent layer to OpenAI Agents SDK #71 for the longer-term agent layer.
• Establishes the semantic baseline for feat: page-preserving TreeKnowledge build + agentic navigation #95.
• The paper-flow integration follows after paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91 is fixed in feat(example): paper extraction to semantic knowledge search #112.

Implementation Considerations

Breaking Changes

• This feature introduces breaking changes.
• This feature is backward compatible; available_at is additive and optional.

Dependencies

• This requires a new public retrieval framework dependency.
• This can use existing dependencies for the MVP.