feat: extensible transforms pipeline for zarr build#86
Closed
turban wants to merge 51 commits into
Closed
Conversation
Add a display block to each built-in dataset template (colormap, value range, nodata) and surface it through the STAC Render extension on every published collection. Add a /maps endpoint that serves a single-page map viewer: it reads the STAC catalog to list available datasets, loads the Render and Datacube metadata to configure a ZarrLayer (MapLibre + @carbonplan/zarr-layer), and builds a time slider from cube:dimensions. No tile server or build step required — the browser reads the Zarr store directly via the existing /zarr HTTP range endpoint. Closes #66
Implements issue #72. Adds a server-rendered management page at GET /manage that lets operators ingest and sync datasets without needing to know API endpoint details or dataset template IDs. - GET /manage renders a Jinja2 page with an ingest form (template dropdown, start/end dates, extent pre-filled) and a status table with per-dataset Sync buttons; flash messages show success or error after each operation - POST /manage/ingest handles the ingest form and redirects back to /manage - POST /manage/sync handles the sync form and redirects back to /manage - Landing page gains an "Available dataset templates" card listing all registered templates and a Manage link in the Explore section
Pass opacity: 0.75 to ZarrLayer so the basemap shows through the data layer. Also wire renders.nodata from the STAC collection into ZarrLayer's fillValue so dataset-specific nodata pixels render as transparent.
Shows a gradient bar with min/max labels and units when a dataset is selected. Legend is built from the same colormap and clim range used by the data layer, sourced from the STAC renders block.
Replaces the OSM raster style with OpenFreeMap's positron vector style. The data layer is inserted before the first symbol layer so country borders, road labels, and place names always render on top of the climate data.
Replaces the OSM raster style with OpenFreeMap's positron vector style. The data layer is inserted before the first symbol layer so country borders, road labels, and place names always render on top of the climate data.
This reverts commit a21e7ca.
This was referenced May 9, 2026
Merged
feat: map viewer at /maps with STAC-backed display metadata
Separate deployment instances (e.g. Norway vs Sierra Leone) sharing
the same DOWNLOAD_DIR would silently reuse each other's NetCDF/Zarr
cache files because the prefix was keyed only on dataset id. Add an
optional extent_id suffix so each extent gets its own cache namespace.
Validate bbox against a dataset's declared coverage field before
downloading, returning HTTP 400 early instead of a confusing
provider-level error. Add coverage: {lat: [-50, 50]} to chirps3.yaml
since CHIRPS3 does not cover latitudes above 50°N (e.g. Norway).
Aligns dataset YAML schema with OGC API Collections by replacing the custom coverage.lat/lon block with extents.spatial.bbox (OGC [xmin, ymin, xmax, ymax] format) and adding extents.temporal with begin, end, trs, and resolution fields. _validate_spatial_coverage now reads extents.spatial.bbox directly, which covers both axes in one check without separate lat/lon keys. All three dataset templates receive extents blocks.
Each configured instance must now declare data_dir in climate-api.yaml. The API raises a clear error at startup if a config file is present but data_dir is not set, rather than silently falling back to a shared XDG directory that another instance might also use. Resolution order for the data directory: 1. CACHE_OVERRIDE env var — preserved for Docker/CI backward compat 2. data_dir from CLIMATE_API_CONFIG — required when config is present 3. XDG default — only used when no config file is configured Extent_id remains in cache filenames to support future multi-extent configurations within a single instance.
…lback Data directory resolution now uses data_dir from climate-api.yaml (required when a config file is present) with a clean XDG fallback. The legacy CACHE_OVERRIDE environment variable is gone from all resolver functions, tests, and .env.example.
Clarifies the distinction from data_dir (runtime storage) — templates_dir points at user-supplied YAML templates and will cover both dataset and processing templates going forward.
templates_dir now acts as a root directory. Dataset templates go in templates_dir/datasets/, leaving room for processing/ and other template types alongside it without structural changes.
- Validate request bbox against extents using the env fallback (DOWNLOAD_BBOX) when no explicit bbox is provided, so coverage checks apply to all request paths - Guard against malformed template extents.spatial.bbox (non-list or wrong length) to avoid a 500 on user-supplied templates - Update get_data_dir() docstring to accurately describe the None-on-missing-file behaviour introduced for CI safety
…e-validation fix: scope cache files by extent_id and validate spatial coverage
Contributor
Author
|
Included in #87 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79.
Summary
_UNIT_CONVERSIONSdict andpre_processlist with a singletransformspipeline in the dataset YAML{function, params}dict), resolved at runtime the same wayingestion.functionworksclimate_api/transforms/with two built-in transforms:convert_unitsanddeaccumulate_era5era5_land.yamlto usetransforms:for both temperature and precipitation datasetsUsage
External transforms from
dhis2eoor any other package can be referenced by dotted path without changes to core code.Test plan
uv run pytest tests/test_transforms.py— 12 new tests covering unit conversion, deaccumulation, pipeline execution, and edge casesuv run pytest— full suite passesStacked on #85. Restores PR #80, which was accidentally merged then reverted.