Commit shards incrementally within a month

[tamnd]

Why

Publish uploaded a whole month's shards in one hub commit after the entire .zst finished processing. A big month (2021-05 submissions is ~64 shards, comments months run to hundreds) takes hours, so:

• the first commit and any proof the pipeline works were hours away
• a restart mid-month redid the month from shard zero, re-uploading everything
• every shard staged on disk until the single commit

What

Stream shards to the hub in batches of --commit-every (default 8, ARCTIC_COMMIT_EVERY) as they are written:

• ProcessFileStream adds a StartShard and a per-shard callback to both the go and duckdb engines; ProcessFile/ProcessFileTo stay as thin wrappers.
• After each batch, publish-progress.json records how far the month got (shards, records, bytes) and the uploaded shards are removed locally, keeping disk low on the comments months.
• A restart reads the marker and resumes after the committed shards. The marker is engine-tagged because the two engines draw shard boundaries differently, so a cross-engine marker is ignored and the month restarts clean.
• The ledger row and README still land once, at month end; then the marker is cleared.
• --commit-every 0 keeps the old behavior (one commit at month end).

Tests

• ProcessFileStream resume skips already-committed shards, emits only the rest, and the resumed shard holds the same rows.
• Progress marker round-trips; the engine guard rejects a cross-engine marker.
• End-to-end publish with a fake uploader: shards land in batches, local copies are dropped, the marker tracks then clears, and one ledger row is written; a seeded marker resumes from the right shard.

Default go build/vet/test and -race pass; -tags duckdb builds clean. (The pre-existing TestProcessFileComments failure under -tags duckdb is unrelated — it asserts go-engine skip counts while running the duckdb engine, and fails the same way on main.)