Skip to content

Add versioned v1 backfill artifact contracts#3681

Open
cotti wants to merge 1 commit into
mainfrom
feat/backfill-artifact-contracts
Open

Add versioned v1 backfill artifact contracts#3681
cotti wants to merge 1 commit into
mainfrom
feat/backfill-artifact-contracts

Conversation

@cotti

@cotti cotti commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the versioned v1 contracts for the six document families the changelog backfill pipeline passes between its stages, so components can interoperate deterministically and plans can be content-addressed. Contracts and serialization only — no CLI surface changes (docs/cli-schema.json is untouched), no behavior changes to existing changelog code, and everything is new files under:

  • src/services/Elastic.Changelog/Backfill/ — the contracts (namespace Elastic.Changelog.Backfill), with a README.md giving a plain-language tour
  • tests/Elastic.Changelog.Tests/Backfill/ — the tests

The six document families

Family What it is
inventory The census: which products and release-note sources exist, where they came from, and what we decided about each (classification, cutoff, adoption state, allowlist status, substitutions, filename convention, unresolved items)
overrides Manual corrections an operator feeds into planning, each with a required reason
semantic-model The release notes reduced to their meaning (ProductRelease shape from the epic: category family, optional precise type / breaking-change subtype, title, description, impact/action, highlight, product references, canonical PR/issue URLs, areas, source locations) plus structured triage diagnostics
plan Exactly what we intend to create in S3 (create-bundle, create-amend, create-supplemental-bundle, skip-existing, manual-review, conflict), pinned to all inputs: source commits, input document hashes, current-state snapshot, optional enrichment snapshot, and the deployed scrubber allowlist identity (content SHA-256 and/or deployment commit)
provenance The evidence trail: which field of which entry got which value, from which rung of the evidence ladder, at what confidence
ledger What actually happened when a plan was applied: plan hash, input refs, created-object hashes, every attempted action with its outcome, registry state, verification result

Identity types are first-class: canonical PR URL, canonical issue URL, and synthetic file identity (stable name + checksum, the thing amend retractions match on). PR/issue references must always be full canonical URLs — bare numbers are rejected by validation.

Envelopes and schema versioning

Every persisted document is wrapped in an envelope carrying artifact (the family name) and schema_version. BackfillDocuments.Deserialize<T> checks both before parsing the payload and throws BackfillDocumentException with an actionable message on: invalid JSON, a non-object top level, a missing header field, an unknown artifact name, a different document kind than requested, or a schema version other than the one this code supports (exact match per family, all currently 1). There is no silent best-effort parsing. Deserialized documents are treated as external input: required fields are enforced by the serializer, semantic validation runs on every read and write and reports all problems at once with their location (e.g. releases[0]: entries[2]: …), and documents over a 64M-character safety bound are refused.

Canonicalization and hashing

A document's identity is SHA-256 over the UTF-8 bytes of the canonical form of its envelope, written sha256: + 64 lower-case hex. Canonical form (CanonicalJson):

  • object keys sorted by ordinal comparison; duplicate keys rejected
  • no insignificant whitespace
  • \r\n / \r inside strings normalized to \n
  • null object properties dropped (absent ≡ null); null array items kept
  • array order preserved (it is meaning); dictionaries serialize as JSON objects, so key sorting makes insertion order irrelevant
  • numbers kept verbatim (all contract numbers are integers); dates/timestamps are invariant ISO strings, ledger timestamps must be UTC

The pretty-printed on-disk form never affects the hash: reading a file back and re-hashing gives the same answer.

AOT

All serialization goes through the source-generated BackfillJsonContext (snake_case properties, kebab-case string enums via JsonStringEnumMemberName, nulls omitted). No reflection; the AOT publish of docs-builder is warning-free with these types included.

Test plan

All verified locally:

  • Round-trip serialize → deserialize → equivalent (and hash-equal) for all six families with representative fixtures
  • Determinism: same document always hashes the same; dictionary insertion order and formatting/line-ending differences do not change the hash; any semantic change does; list order remains meaningful; algorithm pinned against a known SHA-256 vector
  • Version mismatch (newer and older), wrong document kind, unknown kind, missing header fields, invalid JSON, oversized input — all fail explicitly with actionable errors
  • Envelope/field validation: missing required JSON fields, bare PR numbers, malformed hashes, truncated commit SHAs, set-override without value, duplicate override ids, amend without parent, non-UTC ledger timestamps, and more
  • dotnet format clean · ./build.sh build --skip-dirty-check green · dotnet publish -c Release (AOT) zero trim/AOT warnings · tests/Elastic.Changelog.Tests: 873 passed, 0 failed (67 new) · docs/cli-schema.json untouched

Part of elastic/docs-eng-team#672

Note: issue #672 also resolves where checked-in inventories, overrides, plans, provenance, and ledgers live long term; that placement discussion can follow up on this PR without changing these contracts.

Defines the six document families the changelog backfill pipeline passes
between its stages (inventory, overrides, semantic model, plan, provenance,
ledger), each wrapped in a small envelope carrying the document kind and
schema version so readers fail fast and explicitly on documents they do not
understand. Adds canonical JSON serialization and deterministic SHA-256
hashing so plans are content-addressed, plus first-class identity types
(canonical PR/issue URLs, synthetic file identities).

Contracts and serialization only: no CLI surface changes, no behavior
changes to existing changelog code.

Part of elastic/docs-eng-team#672
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant