Refactor layer cache index#49
Merged
Merged
Conversation
Layer indexing now emits self-contained, deterministic LayerArtifacts (ordered entry ops + gzip checkpoints + decompressed hash) that are merged sequentially in layer order. Artifacts are cacheable via the new LayerIndexCache interface keyed by layer digest, so unchanged layers skip the registry pull and decompression entirely. Layers index concurrently (default 4), root timestamps derive from the image config created time for byte-reproducible metadata, and the content cache warm prefers a size-aware existence check. Co-authored-by: Cursor <cursoragent@cursor.com>
verify pushes a synthetic image (whiteouts, opaque dirs, hardlinks, symlinks, multi-checkpoint file) to an in-memory registry and asserts sequential, parallel, and layer-index-cached runs produce identical output that matches independently computed overlay ground truth. bench measures cold vs warm indexing of a real image. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Gunzip dominates indexing cost; klauspost decompression is ~30% faster than stdlib (180 -> 234 MiB/s sequential on a 922 MiB image). bench now reports cold-sequential vs cold-parallel vs warm with throughput, and verifies outputs are identical across all modes. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
OCI layer digests are the sha256 of the compressed bytes, so compressed blobs can live in the content-addressed cache directly under the digest. Indexing now reads layer bytes from the content cache when present (verified against the digest) and warms them on registry pulls, so a layer the cluster has seen is never re-pulled, with no path-addressed metadata needed. Co-authored-by: Cursor <cursoragent@cursor.com>
Indexing now emits two info lines per image (start + summary with cache hit counts); cache hit/store/skip details move to debug, and the content-cache existence check in the store path is compacted. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the test-flavored clip-harness layout with a cleaner package: reference image construction and the in-memory registry live in image.go, index runs and comparison in index.go, and the verify/bench commands read top-down. Binary renamed to bin/harness. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.