Skip to content

feat(sync): dedupe unchanged blob payloads via ETag and content hash #1988

Description

@toddbaert

Blob sync (core/pkg/sync/blob/blob_sync.go:82-95) dedupes on ModTime only. Any bucket op that bumps ModTime without changing bytes (re-upload, tag/ACL edit, replication, terraform re-apply) causes a redundant republish.

HTTP sync already handles this correctly using ETag + SHA-3 body hash (core/pkg/sync/http/http_sync.go:180-201, 208-223).

Proposed change

Bring blob to parity with HTTP:

  • Use attrs.ETag when available (skip fetch on match).
  • Fall back to existing ModTime check.
  • After fetch, SHA-3 the body; skip publish if unchanged, as a backup for lack of etag or ModTimes with no material changes.

Test

Unit tests with mock bucket covering:

  • ETag unchanged -> no fetch, no publish.
  • ETag changed, ModTime bumped, body identical -> fetch but no publish (SHA guard).
  • ETag absent, ModTime unchanged -> no fetch, no publish.
  • ETag absent, ModTime bumped, body identical -> fetch but no publish (SHA guard).
  • ETag absent, ModTime bumped, body changed -> fetch and publish.
  • ETag changed, body changed -> fetch and publish.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions