feat: streaming per-period Icechunk ingest — full #64 implementation#144

Draft

turban wants to merge 81 commits into

mainfrom

feat/icechunk-ingest

Commits on May 20, 2026

feat: per-period Icechunk ingest — protocol, orchestrator, ERA5-Land plugin
turban
committed
fix: per-period commit and hourly period filter in ERA5-Land plugin
turban
committed
fix: strip zarr v2 Blosc encoding before writing to icechunk store
turban
committed
fix: pin time encoding to hours epoch so hourly appends land on correct timestamps
turban
committed
feat: store-based sync for Icechunk — use read_committed_period_ids as source of truth
turban
committed
feat: rechunk Icechunk store in-place after initial ingest (time: 1 → time: N)
turban
committed
feat: CHIRPS3 and WorldPop IngestionPlugin implementations
turban
committed
feat: Icechunk support in data accessor, zarr-serving, STAC and resample
turban
committed
feat: GridSpec.extra_dims and expire_snapshots after ingest
turban
committed
docs: update for IngestionPlugin, Icechunk ingest, and built-in plugins
turban
committed
refactor: remove ingestion.function — ingestion.plugin is the only path

turban
and
claude
committed
fix: strip CF encoding attrs (add_offset, scale_factor) from WorldPop datasets

turban
and
claude
committed
fix: detect actual data CRS for STAC proj:code and coverage spatial_wgs84

turban
and
claude
committed
fix: pass overwrite flag through _create_icechunk_artifact to upsert

turban
and
claude
committed
fix: Icechunk zarr key serving and STAC zarr_format detection

turban
and
claude
committed
fix: WorldPop nodata handling — mask -99999 to NaN during ingest

turban
and
claude
committed
chore: remove display.nodata from worldpop dataset

turban
and
claude
committed
Code cleaning
turban
committed
fix: resolve all mypy and pyright lint errors

turban
and
claude
committed
fix: treat empty icechunk skeleton as first write in orchestrator

turban
and
claude
committed
fix: update tests and guard _detect_dataset_crs after encoding/nodata refactors

turban
and
claude
committed
fix: guard math.isnan call against None for pyright

turban
and
claude
committed
fix: address Copilot PR review — resource leaks, prefer_zarr, GridSpec, plugin errors

turban
and
claude
committed
fix: strip CF attrs from zarr encoding in rechunk_store

turban
and
claude
committed
fix: address second Copilot PR review — task leak, subscript, buffer, empty store, bbox

turban
and
claude
committed
fix: address third Copilot PR review — cursor safety, overwrite, ds.close, request_scope

turban
and
claude
committed
fix: transforms, expire_snapshots timing, sync duplicates, and protocol cleanup

turban
and
claude
committed
feat: derive sync availability from plugin.periods() instead of latest_available_function

turban
and
claude
committed
feat: remove chirps3/worldpop latest_available_function — plugin.periods() owns availability

turban
and
claude
committed
refactor: remove legacy latest_available_function dispatch — plugin.periods() is the only availability path

turban
and
claude
committed
refactor: delete climate_api/providers package — no longer contains anything

turban
and
claude
committed
fix: address fourth Copilot PR review — task leak, GeoZarr attrs, Protocol, unused param

turban
and
claude
committed
feat: multiscale pyramid support for Icechunk ingest

turban
and
claude
committed
chore: remove dhis2eo dependency — never imported

turban
and
claude
committed
refactor: remove legacy build_dataset_zarr and /build_zarr HTTP route

turban
and
claude
committed
refactor: remove legacy netcdf/flat-zarr code paths after Icechunk migration

turban
and
claude
committed
docs: remove automatic reprojection references — plugin CRS is the stored CRS

turban
and
claude
committed
fix: address Copilot review — pyramid append guard, WorldPop validation, commit_batch_size docs

turban
and
claude
committed
feat: POST /ingestions Prefer: respond-async + progress callbacks

turban
and
claude
committed
fix: pyramid map rendering — strip sharding, serve root for zoom selection

turban
and
claude
committed
docs: update STAC pyramid asset href behaviour for Icechunk stores

turban
and
claude
committed
docs: add Icechunk section to zarr_and_geozarr.md

turban
and
claude
committed
fix: handle bare group paths in Icechunk zarr serving; fix legacy remote_zarr format

turban
and
claude
committed
refactor: simplify examples — spatial dims are always x/y

turban
and
claude
committed
refactor: remove _upgrade_legacy_record

turban
and
claude
committed
refactor: remove legacy code — prefer_zarr, NETCDF format, artifact.path, dead functions

turban
and
claude
committed
refactor: sync IngestionPlugin protocol, enumerate_periods utility, clean public API

turban
and
claude
committed
fix: always read committed periods from store on resume; sync plan_sync_dataset with store

turban
and
claude
committed
style: remove trailing whitespace from blank lines in test_datasets.py

turban
and
claude
committed
style: ruff format

turban
and
claude
committed
fix: resolve mypy errors in store, sync_engine, and services

turban
and
claude
committed
fix: resolve pyright errors — remove stale path= field, suppress side-effect imports

turban
and
claude
committed
fix: update FakePlugin/EmptyPlugin in tests to use sync periods()

turban
and
claude
committed
fix: correct async→sync docs; guard open_or_create_repo against missing paths

turban
and
claude
committed
fix: address review findings — remove load_cursor, fix ERA5 probe, clarify docs

turban
and
claude
committed
fix: expose format as "zarr" in ZarrListing responses

turban
and
claude
committed
feat: live progress tracking in manage UI (ingest and sync)

turban
and
claude
committed
fix: remove extent field from ingest form (single-extent instances)

turban
and
claude
committed
fix: add SSE headers and keepalive to unblock browser buffering

turban
and
claude
committed
perf: cache ERA5-Land remote store on plugin instance

turban
and
claude
committed
fix: add python-multipart as explicit dependency

turban
and
claude
committed
fix: expand scalar time coord before passing to xstac

turban
and
claude
committed
fix: hourly end date normalizes to T23 when given a bare date

turban
and
claude
committed
fix: get_time_dim checks actual dims, not just coordinates

turban
and
claude
committed
fix: detect WGS84 from coordinate units when spatial_ref is absent

turban
and
claude
committed
fix: detect pyramid Icechunk stores for correct STAC zarr href

turban
and
claude
committed
fix: reduce CHIRPS3 max_concurrency to 1 to avoid rate-limit bans

turban
and
claude
committed
fix: handle timeless datasets in coverage_from_open_dataset

turban
and
claude
committed
fix: detect completed timeless stores in read_committed_period_ids

turban
and
claude
committed
fix: allow null temporal coverage for static (timeless) datasets

turban
and
claude
committed
fix: handle timeless datasets in STAC collection building

turban
and
claude
committed
fix: skip xstac for timeless stores, build spatial cube:dimensions manually

turban
and
claude
committed
fix: guard _override_temporal_extent_from_artifact against None start/end

turban
and
claude
committed
fix: expose pyramid root URL to zarr-layer for multiscale selection

turban
and
claude
committed
feat: palette LUT support for categorical datasets

turban
and
claude
committed
Merge remote-tracking branch 'origin/main' into feat/icechunk-ingest
turban
committed
fix: resolve ruff lint errors after merge with main

turban
and
claude
committed
fix: resolve all mypy type errors

turban
and
claude
committed
fix: initialise da=None before retry loop to satisfy pyright

turban
and
claude
committed
docs: remove outdated single-CRS constraint from zarr_and_geozarr.md

turban
and
claude
committed
docs: move Store layout on disk section after Icechunk section

turban
and
claude
committed