feat: release DocPull 6.4.0 website snapshots

README.md

@@ -183,6 +183,7 @@ docpull standards-pack rfc:9110 -o packs/standard
 docpull dataset-pack ./metrics.csv -o packs/dataset
 docpull dataset-pack 'https://data.example.org/rows.json?$limit=100' -o packs/dataset
 docpull relationship-pack example.com --subject "Example Brand" -o packs/relationships
+docpull website-pack example.com -o packs/website
 docpull transcript-pack ./meeting.vtt -o packs/transcript
 docpull wiki-pack wiki:Web_scraping -o packs/wiki
 docpull brand-pack example.com -o packs/brand
@@ -196,6 +197,9 @@ docpull export packs/docs --format openai-vector-jsonl -o exports/openai.jsonl
 docpull export packs/docs --format cursor-rules -o .cursor/rules --skill-name docs
 ```
 
+The canonical baseline-aware website acquisition contract is documented in
+[Website snapshots](docs/website-snapshot.md).
+
 Use `docpull ci --prepare` to validate a project or standalone pack in CI.
 
 Install optional extras as needed:
@@ -648,7 +652,7 @@ part of the package release contract.
   `docpull parse`, `docpull openapi-pack`, `docpull feed-pack`,
   `docpull paper-pack`, `docpull repo-pack`, `docpull package-pack`,
   `docpull standards-pack`, `docpull dataset-pack`, `docpull transcript-pack`,
-  `docpull wiki-pack`, `docpull brand-pack`, `docpull product-pack`,
+  `docpull wiki-pack`, `docpull website-pack`, `docpull brand-pack`, `docpull product-pack`,
   `docpull styleguide-pack`, `docpull image-pack`, `docpull policy-pack`,
   `docpull pack validate`,
   `docpull pack audit`, `docpull export`,

docs/CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [6.4.0] - 2026-07-16
+
+### Added
+- Add `website-pack` across CLI, Python workflow API, and MCP. The workflow
+  emits the pinned `website.snapshot.v1` schema, stable URL-based document
+  identities, content-hash versions, page roles, source authority, OKF/raw
+  representations, bounded optional visuals, portable-v3 manifests, and
+  recursively verifiable artifact hashes.
+- Add verified-baseline diff states for added, changed, unchanged, removed,
+  failed, and blocked pages, plus golden/tamper fixtures.
+
+### Changed
+- Let brand, product, policy, styleguide, image, and relationship workflows
+  consume existing packs without network fallback. Product extraction now
+  keeps trial metadata separate from ordinary price fields and excludes
+  testimonial copy from feature evidence.
+
 ### Removed
 - Remove the standalone DocPull website, its Next.js workspace, and web-only
   CI/release gates. GitHub, PyPI, and the MCP Registry remain the canonical

docs/competitor-tracker-integration.md

@@ -1,6 +1,6 @@
 # Competitor-tracker integration contract
 
-Pin DocPull `6.3.0` and validate `intelligence.bundle.v1.json` against
+Pin DocPull `6.4.0` and validate `website.snapshot.v1.json` and `intelligence.bundle.v1.json` against
 `intelligence-bundle.v1.schema.json` at import time.
 
 ## Producer flow

docs/context-pack-contract-v3.md

@@ -60,6 +60,11 @@ write the agent/eval sidecars after parsing.
 
 ## Typed Knowledge Lanes
 
+`website-pack` is the canonical bounded website lane. In addition to the v3
+corpus, it writes `website.snapshot.v1.json`, explicit baseline states, OKF and
+optional visual representations, and recursive artifact hashes. See
+[Website snapshots](website-snapshot.md).
+
 Known-source lanes use the same v3 raw contract: `openapi-pack`,
 `feed-pack`, `paper-pack`, `repo-pack`, `package-pack`, `standards-pack`,
 `dataset-pack`, `transcript-pack`, and `wiki-pack` all write

docs/contracts.md

@@ -27,6 +27,13 @@ generic contracts.
 
 ## Cross-repository contracts
 
+### `website.snapshot.v1`
+
+The canonical website-intelligence acquisition artifact. It binds stable
+document/version identities, entity and authority metadata, page roles,
+baseline states, OKF/raw/visual representations, portable-v3 manifests, and a
+canonical snapshot hash. See [Website snapshots](website-snapshot.md).
+
 ### `workflow.request.v1`
 
 A scheduler-neutral invocation containing a stable request ID, workflow name,

docs/surface-contract.md

@@ -48,7 +48,7 @@ DocPull targets capability alignment, not 1:1 flag parity. MCP should not mirror
 | Refresh/score/diff/audit context packs | `docpull refresh`, `docpull pack score`, `docpull pack sources`, `docpull pack diff`, `docpull pack audit` | Pack helper modules, `refresh_pack`, `audit_pack` | `refresh_pack`, `pack_score`, `pack_diff`, `audit_pack` | Core-aligned |
 | Build pack citations/entities/search/briefs | `docpull pack citations`, `docpull pack entities`, `docpull pack search`, `docpull pack brief` | `build_citation_map`, `extract_pack_entities`, `search_pack`, `build_research_brief` | `pack_citations`, `pack_entities`, `pack_search`, `pack_brief` | Core-aligned |
 | Prepare full pack intelligence bundle | `docpull pack prepare` | `prepare_pack` in `docpull.pack_tools` | `pack_prepare` | Core-aligned |
-| Evidence-pack workflow protocol | `brand-pack`, `product-pack`, `styleguide-pack`, `image-pack`, `screenshot-pack`, `policy-pack`, `relationship-pack`, `dataset-pack` | concrete `build_*_pack` builders plus `WorkflowRequest`, `run_workflow`, `async_run_workflow` | `workflow_run` plus dedicated evidence tools | Core-aligned |
+| Evidence-pack workflow protocol | `website-pack`, `brand-pack`, `product-pack`, `styleguide-pack`, `image-pack`, `screenshot-pack`, `policy-pack`, `relationship-pack`, `dataset-pack` | concrete `build_*_pack` builders plus `WorkflowRequest`, `run_workflow`, `async_run_workflow` | `workflow_run` plus dedicated evidence tools including `website_pack` | Core-aligned |
 | Tracker import bundle | `docpull pack intelligence-bundle` (`company-brain` alias) | `build_intelligence_bundle` (`build_company_brain_bundle` alias) | `intelligence_bundle` | Core-aligned |
 | Context CI and eval-grade context | `docpull ci`, `docpull pack prepare --eval-grade`, `docpull pack validate --level eval` | `run_context_ci`, `validate_pack_contract`, pack preparation helpers | Not exposed yet | Core-aligned |
 | Build/query local source graphs | `docpull graph build`, `docpull graph status`, `docpull graph query`, `docpull graph neighbors`, `docpull graph refresh` | `build_graph`, `load_graph`, `graph_status`, `query_graph`, `graph_neighbors`, `refresh_graph` | `graph_build`, `graph_status`, `graph_query`, `graph_neighbors`, `graph_refresh` | Core-aligned |

docs/website-snapshot.md

@@ -0,0 +1,58 @@
+# Website snapshots
+
+`website-pack` is DocPull's canonical acquisition workflow for downstream
+competitive-intelligence systems. It writes a portable-v3 artifact whose entrypoint is
+`website.snapshot.v1.json` and whose complete file set is pinned by
+`artifact.manifest.json`.
+
+```bash
+docpull website-pack https://example.com -o packs/example-website
+docpull website-pack https://example.com -o packs/example-next \
+  --baseline-pack packs/example-website
+```
+
+The default bounded crawl is 50 pages and depth 3. It classifies home, product,
+pricing, documentation, trust, legal, changelog, support, and other pages. Every
+active page has a stable canonical-URL `document_id`, a content-hash
+`document_version`, source authority, entity reference, page role, and OKF
+representation. Raw HTML is on by default. At most eight key-page screenshots
+are included when `DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` and a compatible
+local browser runtime are available; missing browser capability produces a
+typed warning and does not invalidate text evidence.
+
+## Baseline states
+
+A baseline must pass recursive manifest, reference, pack/run identity, schema,
+and snapshot-hash verification before use. The new snapshot explicitly reports
+`added`, `changed`, `unchanged`, `removed`, `failed`, and `blocked` document
+states. Unchanged documents remain in `documents.ndjson`; removed documents
+remain in the snapshot manifest without pretending they were acquired again.
+
+`validate_website_snapshot_pack(path)` verifies the current manifest and every
+referenced representation. Files outside the manifest are ignored, so stale
+files from an older output directory cannot alter current-run identity.
+
+## Artifact layout
+
+- `website.snapshot.v1.json` — canonical snapshot identity, baseline, options,
+  documents, coverage, and representation references.
+- `documents.ndjson` — the active evidence corpus used by downstream analysis.
+- `okf/` — one OKF concept per active document.
+- `raw/` — selected source HTML when raw capture is enabled.
+- `screenshots/` and `brand-assets/` — bounded optional visual evidence.
+- `corpus.manifest.json`, `current-run.manifest.json`,
+  `coverage.manifest.json`, and `provenance.manifest.json` — portable-v3
+  inventory, run, coverage, and provenance layers.
+- `artifact.manifest.json` — sorted relative paths, byte counts, media types,
+  and SHA-256 hashes for the exact current artifact.
+
+The bundled schema is `website-snapshot.v1.schema.json`. Its file SHA-256 is
+written into every snapshot as `schema_sha256`, allowing consumers to pin the
+exact producer contract.
+
+## Local replay
+
+`brand-pack`, `product-pack`, `policy-pack`, `styleguide-pack`, `image-pack`,
+and `relationship-pack` accept an existing DocPull pack path. Local-pack mode
+does not fetch the network or download referenced assets. This makes one
+website acquisition reusable across deterministic extraction lanes.

plugin/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "docpull",
-  "version": "6.3.0",
+  "version": "6.4.0",
   "description": "Pull public web sources into Claude Code. Indexes static and server-rendered sites as local Markdown with conditional-GET caching, then exposes them as MCP tools. Local, browser-free, no API keys.",
   "author": {
     "name": "Raintree Technology",

plugin/.codex-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "docpull",
   "name": "docpull",
-  "version": "6.3.0",
+  "version": "6.4.0",
   "description": "Pull public web sources into Codex as local, searchable Markdown through docpull's MCP server.",
   "skills": "./skills/",
   "mcpServers": "./.mcp.json",

plugin/README.md

@@ -17,9 +17,9 @@ for the boundary between the plugin's MCP tools and the broader CLI/SDK.
 ## What you get
 
 <!-- docpull:mcp-tools:start -->
-- **MCP server** (35 tools):
+- **MCP server** (36 tools):
   - Read: `fetch_url`, `list_sources`, `list_indexed`, `grep_docs`, `read_doc`, `pack_score`, `pack_diff`, `pack_citations`, `pack_entities`, `pack_search`, `pack_brief`, `graph_status`, `graph_query`, `graph_neighbors`, `validate_policy`, `serve_pack_status`
-  - Write: `render_url`, `ensure_docs`, `workflow_run`, `brand_pack`, `product_pack`, `styleguide_pack`, `image_pack`, `screenshot_pack`, `policy_pack`, `relationship_pack`, `intelligence_bundle`, `refresh_pack`, `audit_pack`, `pack_prepare`, `graph_build`, `graph_refresh`, `export_pack`, `add_source`, `remove_source`
+  - Write: `render_url`, `ensure_docs`, `workflow_run`, `website_pack`, `brand_pack`, `product_pack`, `styleguide_pack`, `image_pack`, `screenshot_pack`, `policy_pack`, `relationship_pack`, `intelligence_bundle`, `refresh_pack`, `audit_pack`, `pack_prepare`, `graph_build`, `graph_refresh`, `export_pack`, `add_source`, `remove_source`
   - All read tools advertise `readOnlyHint` so hosts that auto-approve safe tools won't prompt for them.
 <!-- docpull:mcp-tools:end -->
 - **Claude Code slash commands**:
@@ -39,7 +39,7 @@ MCP server is available:
 ```bash
 pip install 'docpull[mcp]'          # or: pipx install 'docpull[mcp]'
                                     #     uv tool install 'docpull[mcp]'
-docpull --version                   # should print 6.3.0 or newer
+docpull --version                   # should print 6.4.0 or newer
 docpull mcp --help                  # confirm the MCP subcommand is wired
 ```
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "docpull"
-version = "6.3.0"
+version = "6.4.0"
 dynamic = []
 description = "Declare, sync, diff, and lock context dependencies for AI agents"
 readme = {file = "README.md", content-type = "text/markdown"}

server.json

@@ -8,12 +8,12 @@
     "source": "github"
   },
   "websiteUrl": "https://github.com/raintree-technology/docpull",
-  "version": "6.3.0",
+  "version": "6.4.0",
   "packages": [
     {
       "registryType": "pypi",
       "identifier": "docpull",
-      "version": "6.3.0",
+      "version": "6.4.0",
       "transport": {
         "type": "stdio"
       }

src/docpull/__init__.py

@@ -15,7 +15,7 @@
 
 from .surface import PUBLIC_SDK_EXPORTS
 
-__version__ = "6.3.0"
+__version__ = "6.4.0"
 
 _LAZY_EXPORTS: dict[str, tuple[str, str]] = {
     **{
@@ -41,9 +41,11 @@
             "build_repo_pack",
             "build_standards_pack",
             "build_styleguide_pack",
+            "build_website_pack",
             "build_transcript_pack",
             "build_wiki_pack",
             "capture_screenshot_pack",
+            "validate_website_snapshot_pack",
         )
     },
     **{
@@ -59,6 +61,8 @@
             "SourceAuthority",
             "WorkflowRequest",
             "WorkflowResult",
+            "WebsiteSnapshot",
+            "WebsiteSnapshotDocument",
             "bundled_schema_path",
             "write_contract_schemas",
         )
@@ -198,8 +202,10 @@ def __dir__() -> list[str]:
         build_standards_pack,
         build_styleguide_pack,
         build_transcript_pack,
+        build_website_pack,
         build_wiki_pack,
         capture_screenshot_pack,
+        validate_website_snapshot_pack,
     )
     from .contracts import (
         ArtifactManifest,
@@ -210,6 +216,8 @@ def __dir__() -> list[str]:
         RelationshipCandidate,
         RelationshipPack,
         SourceAuthority,
+        WebsiteSnapshot,
+        WebsiteSnapshotDocument,
         WorkflowRequest,
         WorkflowResult,
         bundled_schema_path,

src/docpull/cli.py

@@ -1661,6 +1661,7 @@ def main(argv: list[str] | None = None) -> int:
         return run_monitor_cli(raw_argv[1:])
     if raw_argv and raw_argv[0] in {
         "brand-pack",
+        "website-pack",
         "product-pack",
         "styleguide-pack",
         "image-pack",
@@ -1676,9 +1677,11 @@ def main(argv: list[str] | None = None) -> int:
             run_relationship_pack_cli,
             run_screenshot_pack_cli,
             run_styleguide_pack_cli,
+            run_website_pack_cli,
         )
 
         workflow_runners = {
+            "website-pack": run_website_pack_cli,
             "brand-pack": run_brand_pack_cli,
             "product-pack": run_product_pack_cli,
             "styleguide-pack": run_styleguide_pack_cli,

src/docpull/context_packs/__init__.py

@@ -25,6 +25,8 @@
     **{name: (".transcript", name) for name in ("async_build_transcript_pack", "build_transcript_pack")},
     **{name: (".wiki", name) for name in ("async_build_wiki_pack", "build_wiki_pack")},
     "capture_screenshot_pack": (".visuals", "capture_screenshot_pack"),
+    "build_website_pack": (".website", "build_website_pack"),
+    "validate_website_snapshot_pack": (".website", "validate_website_snapshot_pack"),
 }
 
 __all__ = list(PUBLIC_CONTEXT_PACK_EXPORTS)
@@ -59,6 +61,7 @@ def __dir__() -> list[str]:
     from .styleguide import build_styleguide_pack
     from .transcript import async_build_transcript_pack, build_transcript_pack
     from .visuals import build_image_pack, capture_screenshot_pack
+    from .website import build_website_pack, validate_website_snapshot_pack
     from .wiki import async_build_wiki_pack, build_wiki_pack
 
 

src/docpull/context_packs/brand.py

@@ -29,6 +29,7 @@
     homepage_url_for_domain,
     is_free_or_disposable_email_domain,
     likely_internal_pages,
+    pages_from_local_pack,
     public_url,
     quote_markdown,
     resolve_url,
@@ -92,7 +93,14 @@ def build_brand_pack(
     max_pages: int = MAX_BRAND_PAGES,
 ) -> dict[str, Any]:
     """Build a local brand profile from public website evidence."""
-    domain = _resolve_brand_domain(domain_or_url, email=email, allow_free_email=allow_free_email)
+    local_pages = pages_from_local_pack(domain_or_url)
+    domain = (
+        domain_from_input(local_pages[0].url)
+        if local_pages
+        else _resolve_brand_domain(domain_or_url, email=email, allow_free_email=allow_free_email)
+    )
+    if not domain:
+        raise ContextPackError("Could not resolve a domain from local brand-pack evidence.")
     policy = ensure_policy_for_domain(policy, domain)
     output_dir = output_dir.resolve()
     run = ContextPackRun(
@@ -101,14 +109,20 @@ def build_brand_pack(
         policy=policy,
         input_value=email or domain_or_url,
     )
-    home_url = homepage_url_for_domain(domain)
-    home_pages = fetch_pages_blocking([home_url], run=run, max_pages=1)
-    if not home_pages:
-        raise ContextPackError(f"Could not fetch homepage for {domain}.")
-    candidate_urls = likely_internal_pages(home_pages[0], domain, max_pages=max_pages)
-    pages = fetch_pages_blocking(candidate_urls, run=run, max_pages=max_pages)
-    if not pages:
-        pages = home_pages
+    if local_pages is not None:
+        pages = local_pages[:max_pages]
+        if not pages:
+            raise ContextPackError("Local brand-pack input contains no readable documents.")
+        download_assets = False
+    else:
+        home_url = homepage_url_for_domain(domain)
+        home_pages = fetch_pages_blocking([home_url], run=run, max_pages=1)
+        if not home_pages:
+            raise ContextPackError(f"Could not fetch homepage for {domain}.")
+        candidate_urls = likely_internal_pages(home_pages[0], domain, max_pages=max_pages)
+        pages = fetch_pages_blocking(candidate_urls, run=run, max_pages=max_pages)
+        if not pages:
+            pages = home_pages
 
     profile = _extract_brand_profile(pages, domain=domain, name_hint=name, ticker=ticker)
     assets = _extract_brand_assets(