Skip to content

SDAP-505: Support DOMS insitu API#299

Merged
stephenykp merged 3 commits into
apache:developfrom
stephenykp:SDAP-505b
Mar 6, 2024
Merged

SDAP-505: Support DOMS insitu API#299
stephenykp merged 3 commits into
apache:developfrom
stephenykp:SDAP-505b

Conversation

@stephenykp
Copy link
Copy Markdown
Member

  • Cast platform to string. SDAP requires platform to be a string (db col is str) but the DOMS insitu API seems to return an integer.
  • Add mechanism for not supporting insitu schema. As far as I know no schema endpoint exists in DOMS, so in this case we'd set the schema endpoint to None.
  • Removed cloud insitu nodes from conf. Added DOMS insitu nodes

Testing

All testing was done on a local instance of SDAP where Cass/Solr are running locally via Docker. I ingested AVHRR_OI_L4_GHRSST_NCEI 2018-08-01 - 2018-08-16 data prior to running this test.

Test query:

localhost:8083/match_spark?primary=AVHRR_OI_L4_GHRSST_NCEI&secondary=SAMOS&startTime=2018-08-01T00%3A00%3A00Z&endTime=2018-08-01T23%3A59%3A59Z&b=-122%2C%2026%2C%20-114%2C%2034&platforms=30&depthMin=-1&depthMax=1&tt=43200&rt=18750&matchOnce=true&prioritizeDistance=true

Job completes successfully. Results (JSON /cdmsresults endpoint)

{
	"executionId": "fe87f866-a415-494e-851a-7a666fdfe5f3",
	"data": [{
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-117.375",
		"lat": "32.625",
		"point": "Point(-117.375 32.625)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 90, 50]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 23.899993896484375,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "30",
			"device": null,
			"lon": "-117.2368",
			"lat": "32.7076",
			"point": "Point(-117.237 32.708)",
			"time": 1533081240,
			"depth": -99999,
			"fileurl": null,
			"id": "WSQ2674_20180731v20001_1434-99999.0@-99999.0",
			"source": "SAMOS",
			"secondary": [{
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 71,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 23.2,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}]
	}, {
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-117.125",
		"lat": "32.625",
		"point": "Point(-117.125 32.625)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 90, 51]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 24.410003662109375,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "30",
			"device": null,
			"lon": "-117.2367",
			"lat": "32.7076",
			"point": "Point(-117.237 32.708)",
			"time": 1533077340,
			"depth": -99999,
			"fileurl": null,
			"id": "WSQ2674_20180731v20001_1369-99999.0@-99999.0",
			"source": "SAMOS",
			"secondary": [{
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 72.8,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 23.2,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}]
	}],
	"params": {
		"primary": "AVHRR_OI_L4_GHRSST_NCEI",
		"matchup": "SAMOS",
		"startTime": 1533081600,
		"endTime": 1533167999,
		"bbox": "-122, 26, -114, 34",
		"timeTolerance": 43200,
		"radiusTolerance": 18750,
		"platforms": "30",
		"parameter": null,
		"depthMin": -1,
		"depthMax": 1
	},
	"bounds": {},
	"count": 2,
	"details": {
		"timeToComplete": 9,
		"numSecondaryMatched": 2,
		"numPrimaryMatched": 2,
		"numUniqueSecondaries": 2,
		"pageNum": 1,
		"pageSize": 1000
	}
}

Have not yet tested NCAR insitu endpoint.

@stephenykp
Copy link
Copy Markdown
Member Author

Updated NCAR insitu api url, tested like so:

localhost:8083/match_spark?primary=AVHRR_OI_L4_GHRSST_NCEI&secondary=ICOADS Release 3.0&startTime=2018-08-01T00%3A00%3A00Z&endTime=2018-08-01T23%3A59%3A59Z&b=-122%2C%2026%2C%20-114%2C%2034&platforms=30&depthMin=-1&depthMax=1&tt=43200&rt=18750&matchOnce=true&prioritizeDistance=true

Job completes successfully. Results (JSON /cdmsresults endpoint w/pageSize=2)

{
	"executionId": "8185d41f-c3af-4291-bbaa-ab46e80abe7f",
	"data": [{
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-117.125",
		"lat": "32.625",
		"point": "Point(-117.125 32.625)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 90, 51]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 24.410003662109375,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "WSQ2674",
			"device": null,
			"lon": "-117.2",
			"lat": "32.7",
			"point": "Point(-117.200 32.700)",
			"time": 1533081600,
			"depth": -99999,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP9J5K@-99999.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature",
				"cf_variable_name": "air_temperature",
				"variable_value": 24.7,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature",
				"cf_variable_name": "dew_point_temperature",
				"variable_value": 19.9,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 74.7,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 23.3,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed",
				"cf_variable_name": "wind_speed",
				"variable_value": 1.5,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction",
				"cf_variable_name": "wind_from_direction",
				"variable_value": 270,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "northward_wind",
				"cf_variable_name": "northward_wind",
				"variable_value": 1.5,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}, {
			"platform": "WSQ2674",
			"device": null,
			"lon": "-117.2",
			"lat": "32.7",
			"point": "Point(-117.200 32.700)",
			"time": 1533081600,
			"depth": 0,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP9J5K@0.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure",
				"cf_variable_name": "air_pressure",
				"variable_value": 1013.4,
				"variable_unit": null
			}, {
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 9,
				"variable_unit": null
			}]
		}]
	}, {
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-120.375",
		"lat": "27.875",
		"point": "Point(-120.375 27.875)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 71, 38]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 21.07000732421875,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "VRDK7",
			"device": null,
			"lon": "-120.3",
			"lat": "27.8",
			"point": "Point(-120.300 27.800)",
			"time": 1533099600,
			"depth": -99999,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=P32YWX@-99999.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature",
				"cf_variable_name": "air_temperature",
				"variable_value": 21,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature",
				"cf_variable_name": "dew_point_temperature",
				"variable_value": 16.4,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 75,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 25,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature",
				"cf_variable_name": "wet_bulb_temperature",
				"variable_value": 18,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}, {
			"platform": "VRDK7",
			"device": null,
			"lon": "-120.3",
			"lat": "27.8",
			"point": "Point(-120.300 27.800)",
			"time": 1533099600,
			"depth": 0,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=P32YWX@0.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure",
				"cf_variable_name": "air_pressure",
				"variable_value": 1017.1,
				"variable_unit": null
			}, {
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 9,
				"variable_unit": null
			}]
		}]
	}],
	"params": {
		"primary": "AVHRR_OI_L4_GHRSST_NCEI",
		"matchup": "ICOADS Release 3.0",
		"startTime": 1533081600,
		"endTime": 1533167999,
		"bbox": "-122, 26, -114, 34",
		"timeTolerance": 43200,
		"radiusTolerance": 18750,
		"platforms": "30",
		"parameter": null,
		"depthMin": -1,
		"depthMax": 1
	},
	"bounds": {},
	"count": 2,
	"details": {
		"timeToComplete": 10,
		"numSecondaryMatched": 44,
		"numPrimaryMatched": 20,
		"numUniqueSecondaries": 28,
		"pageNum": 1,
		"pageSize": 2
	}
}

RKuttruff
RKuttruff previously approved these changes Mar 6, 2024
Copy link
Copy Markdown
Contributor

@RKuttruff RKuttruff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.
I'm assuming the new NCAR endpoint re-adds insitu support after the CDMS project went down?

@stephenykp
Copy link
Copy Markdown
Member Author

Looks good to me. I'm assuming the new NCAR endpoint re-adds insitu support after the CDMS project went down?

Yes, we are now pointing to the DOMS insitu API FSU and NCAR are hosting since the AWS api is no longer running


INSITU_API_ENDPOINT = 'https://doms.jpl.nasa.gov/insitu/1.0/query_data_doms_custom_pagination'
INSITU_API_SCHEMA_ENDPOINT = 'https://doms.jpl.nasa.gov/insitu/1.0/cdms_schema'
INSITU_API_SCHEMA_ENDPOINT = None
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of None would it work if we pull the schema from https://doms.jpl.nasa.gov/in_situ_schema.json ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try that out!

@stephenykp
Copy link
Copy Markdown
Member Author

@ngachung Good news! It appears to have worked.

Tested like so:

localhost:8083/match_spark?primary=AVHRR_OI_L4_GHRSST_NCEI&secondary=ICOADS Release 3.0&startTime=2018-08-01T00%3A00%3A00Z&endTime=2018-08-01T23%3A59%3A59Z&b=-122%2C%2026%2C%20-114%2C%2034&platforms=30&depthMin=-1&depthMax=1&tt=43200&rt=18750&matchOnce=true&prioritizeDistance=true
Job completes successfully. Results (JSON /cdmsresults endpoint w/pageSize=2)
{
	"executionId": "b28d7755-e8f6-418d-a48d-0adef8d9bc62",
	"data": [{
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-116.625",
		"lat": "30.125",
		"point": "Point(-116.625 30.125)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 80, 53]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 20.839996337890625,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "VRRD7",
			"device": null,
			"lon": "-116.7",
			"lat": "30.0",
			"point": "Point(-116.700 30.000)",
			"time": 1533124800,
			"depth": -99999,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP8UR2@-99999.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature",
				"cf_variable_name": "air_temperature",
				"variable_value": 21,
				"variable_unit": "C"
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature",
				"cf_variable_name": "dew_point_temperature",
				"variable_value": 19.4,
				"variable_unit": "C"
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 90.6,
				"variable_unit": "1"
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 22,
				"variable_unit": "C"
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature",
				"cf_variable_name": "wet_bulb_temperature",
				"variable_value": 20,
				"variable_unit": "C"
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed",
				"cf_variable_name": "wind_speed",
				"variable_value": 7.7,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction",
				"cf_variable_name": "wind_from_direction",
				"variable_value": 320,
				"variable_unit": "degree"
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "eastward_wind",
				"cf_variable_name": "eastward_wind",
				"variable_value": -5.9,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "northward_wind",
				"cf_variable_name": "northward_wind",
				"variable_value": 4.9,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}, {
			"platform": "VRRD7",
			"device": null,
			"lon": "-116.7",
			"lat": "30.0",
			"point": "Point(-116.700 30.000)",
			"time": 1533124800,
			"depth": 0,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP8UR2@0.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure",
				"cf_variable_name": "air_pressure",
				"variable_value": 1012.7,
				"variable_unit": "hPa"
			}, {
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 9,
				"variable_unit": null
			}]
		}]
	}, {
		"platform": "orbiting satellite",
		"device": "radiometers",
		"lon": "-116.625",
		"lat": "29.875",
		"point": "Point(-116.625 29.875)",
		"time": 1533081600,
		"depth": null,
		"fileurl": "20180801120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
		"id": "9793ad5b-5d06-3163-95b2-b5f020280044[[0, 79, 53]]",
		"source": "AVHRR_OI_L4_GHRSST_NCEI",
		"primary": [{
			"variable_name": "analysed_sst",
			"cf_variable_name": "sea_surface_temperature",
			"variable_value": 20.670013427734375,
			"variable_unit": null
		}],
		"matches": [{
			"platform": "VRRD7",
			"device": null,
			"lon": "-116.7",
			"lat": "30.0",
			"point": "Point(-116.700 30.000)",
			"time": 1533124800,
			"depth": 0,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP8UR2@0.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure",
				"cf_variable_name": "air_pressure",
				"variable_value": 1012.7,
				"variable_unit": "hPa"
			}, {
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 9,
				"variable_unit": null
			}]
		}, {
			"platform": "VRRD7",
			"device": null,
			"lon": "-116.7",
			"lat": "30.0",
			"point": "Point(-116.700 30.000)",
			"time": 1533124800,
			"depth": -99999,
			"fileurl": null,
			"id": "https://rda.ucar.edu/php/icoadsuid.php?uid=YP8UR2@-99999.0",
			"source": "ICOADS Release 3.0",
			"secondary": [{
				"variable_name": "air_pressure_quality",
				"cf_variable_name": "air_pressure_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "air_temperature",
				"cf_variable_name": "air_temperature",
				"variable_value": 21,
				"variable_unit": "C"
			}, {
				"variable_name": "air_temperature_quality",
				"cf_variable_name": "air_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "dew_point_temperature",
				"cf_variable_name": "dew_point_temperature",
				"variable_value": 19.4,
				"variable_unit": "C"
			}, {
				"variable_name": "dew_point_temperature_quality",
				"cf_variable_name": "dew_point_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "relative_humidity",
				"cf_variable_name": "relative_humidity",
				"variable_value": 90.6,
				"variable_unit": "1"
			}, {
				"variable_name": "relative_humidity_quality",
				"cf_variable_name": "relative_humidity_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_salinity_quality",
				"cf_variable_name": "sea_water_salinity_quality",
				"variable_value": 9,
				"variable_unit": null
			}, {
				"variable_name": "sea_water_temperature",
				"cf_variable_name": "sea_water_temperature",
				"variable_value": 22,
				"variable_unit": "C"
			}, {
				"variable_name": "sea_water_temperature_quality",
				"cf_variable_name": "sea_water_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wet_bulb_temperature",
				"cf_variable_name": "wet_bulb_temperature",
				"variable_value": 20,
				"variable_unit": "C"
			}, {
				"variable_name": "wet_bulb_temperature_quality",
				"cf_variable_name": "wet_bulb_temperature_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_speed",
				"cf_variable_name": "wind_speed",
				"variable_value": 7.7,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "wind_speed_quality",
				"cf_variable_name": "wind_speed_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "wind_from_direction",
				"cf_variable_name": "wind_from_direction",
				"variable_value": 320,
				"variable_unit": "degree"
			}, {
				"variable_name": "wind_from_direction_quality",
				"cf_variable_name": "wind_from_direction_quality",
				"variable_value": 1,
				"variable_unit": null
			}, {
				"variable_name": "eastward_wind",
				"cf_variable_name": "eastward_wind",
				"variable_value": -5.9,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "northward_wind",
				"cf_variable_name": "northward_wind",
				"variable_value": 4.9,
				"variable_unit": "m s-1"
			}, {
				"variable_name": "wind_component_quality",
				"cf_variable_name": "wind_component_quality",
				"variable_value": 1,
				"variable_unit": null
			}]
		}]
	}],
	"params": {
		"primary": "AVHRR_OI_L4_GHRSST_NCEI",
		"matchup": "ICOADS Release 3.0",
		"startTime": 1533081600,
		"endTime": 1533167999,
		"bbox": "-122, 26, -114, 34",
		"timeTolerance": 43200,
		"radiusTolerance": 18750,
		"platforms": "30",
		"parameter": null,
		"depthMin": -1,
		"depthMax": 1
	},
	"bounds": {},
	"count": 2,
	"details": {
		"timeToComplete": 10,
		"numSecondaryMatched": 44,
		"numPrimaryMatched": 20,
		"numUniqueSecondaries": 28,
		"pageNum": 1,
		"pageSize": 2
	}
}

@ngachung ngachung self-requested a review March 6, 2024 20:36
Copy link
Copy Markdown
Contributor

@ngachung ngachung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm. Thank you!!

@stephenykp stephenykp merged commit 2c92cff into apache:develop Mar 6, 2024
RKuttruff added a commit that referenced this pull request Jun 24, 2024
* SDAP-493: Pagination improvements (#282)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* SDAP-500: Improvements to SDAP Asynchronous Jobs (#287)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update changelog

* rename key to executionID

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-499: Add page number to default filename for matchup output (#288)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* changelog

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-506: STAC Catalog for Matchup outputs (#291)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* stac catalog

* Updated openapi spec

* move stac endpoints to matchup tag in openapi spec

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* fix bug where still-running jobs failed /job endpoint due to missing metadata

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update changelog

* re-add removed changelog entry

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-508 report sat spatial extents (#295)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-505: Support DOMS insitu API (#299)

* support DOMS insitu API

* update NCAR insitu url

* add endpoint for getting insitu units

* SDAP-472 - data-access overhaul to support multiple simultaneous data backends (#294)

* Separated NTS backends

* n/a

* More nts backend stuff

* Working(?) np backend

* Working(?) np backend

* gitignore ini

* ASF headers

* First functioning test of 2 simultaneous backends

* Removed accidentally committed ini files

* Working zarr backend ds list

+ datasets are no longer case sensitive
+ handling for failed zarr ds opens (bad path, bad creds, &c)

* Capture and handle NTS requests routed to backend that doesn't (yet) support them

* analysis setup fails to find VERSION.txt when building locally

* Implemented more NTS functions in zarr backend

* Added misc backend time metrics record field in NCSH

* fixes

* Dynamic dataset management

* Dynamic dataset management

* Dataset management

* Timeseriesspark support

* Update backend dict on dataset mgmt query

* Fixes and improvements

* Adapted matchup to work with zarr backends

* Zarr support

- Distinct slices of time is now default
- No longer assuming+shaping as multivar tiles unless needed

* DDAS adjustments

* find_tile_by_polygon_and_most_recent_day_of_year impl

* Don't sel by time if neither max nor min time are given

* Fix not calling partial when needed

* Pinned s3fs and fsspec versions

* Fixed some dependencies to ensure image builds properly + s3fs works

* Config override for backends

* Deps update

* Add metadata from Zarr collection to /list

* Zarr: Probe lat order and flip if necessary

* Strip quotes from variable names

CM can sometimes publish with extra quotes resulting in KeyErrors

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add params to dataset management handler classes

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* stac catalog

* Updated openapi spec

* move stac endpoints to matchup tag in openapi spec

* SDAP-507 - Changes to remove geos sub-dependency

* SDAP-507 - Changelog

* SDAP-507 - Changes to remove geos sub-dependency

* SDAP-507 - Changelog

* delete instead of comment out

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* deleted disabled endpoint files

* fix bug where still-running jobs failed /job endpoint due to missing metadata

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Moved changelog entries

* SDAP-472 changelog entries

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: skorper <stepheny.k.perez@gmail.com>

* Merge master into develop post-1.2.0 release (#305)

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* domspurge patch (#280)

* Fix delete query to account for PK update

* Fixed mistake in domspurge readme

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Patch: Made conda dep changes accidentally omitted from geos PR

* SDAP-511 - Switch package manager to poetry (#301)

* Redid branch w/ correct base

I tried to just rebase but it didn't work properly

* Changelogs

* Delete old dockerfile

* Removed old conda install files

* Removed version.txt to just use pyproject.toml

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* pyproject.toml dependency updates for Zarr

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>

* SDAP 515 - Improved handling of unreachable remote SDAPs (#308)

* Improved error handling to account for any reason a remote is not reachable

* Further fixes for unreachable remote SDAPs

* Improved logging

* SDAP-513 - Configurable Solr init pod image in helm (#306)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Added some missing ASF headers and removed some unneeded files (#302)

* Remove IDE files

* Added more missing headers

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Update requirements.txt (#309)

* Updates to openapi (#310)

* Disable the try it now button

* Added additional spark algorithms

* Updates to openapi to include additional algorithms

* SDAP-518 - Collection Config Docs (#311)

* Initial work on CC docs

* Add collections to index toctree

* YAML highlighting

* NetCDF section

* remove incubation msg from intro.rst

* Added recs for gridded tile size

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* data-access patch (#313)

* Patches to backend mgmt and zarr backend

- Code cleanup
- Added dask as dependency, so it will be leveraged with Zarr datasets
- Fixed creation of tile times array from Zarr data to ensure its in seconds from 1970-01-01. Original method has been seen to produce incorrect results from bad assumptions

* Changelog

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Changed SDAP startup behavior to wait for all datasets to be prepared before accepting HTTP requests (#314)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update quickstart with updated solr command (#298)

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>

* Update release files

- 1.3.0 header in changelogs
- Removed incubation DISCLAIMER
- Removed incubation language from NOTICE and updated URLs
- Bumped versions
- Made copyright year current
- Poetry re-lock

* Fix Dockerfile for DISCLAIMER rm

---------

Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: skorper <stepheny.k.perez@gmail.com>
Co-authored-by: Kevin <kevinmarlis@gmail.com>
RKuttruff added a commit that referenced this pull request Nov 4, 2024
* SDAP-493: Pagination improvements (#282)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* SDAP-500: Improvements to SDAP Asynchronous Jobs (#287)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update changelog

* rename key to executionID

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-499: Add page number to default filename for matchup output (#288)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* changelog

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-506: STAC Catalog for Matchup outputs (#291)

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* stac catalog

* Updated openapi spec

* move stac endpoints to matchup tag in openapi spec

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* fix bug where still-running jobs failed /job endpoint due to missing metadata

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update changelog

* re-add removed changelog entry

---------

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-508 report sat spatial extents (#295)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-505: Support DOMS insitu API (#299)

* support DOMS insitu API

* update NCAR insitu url

* add endpoint for getting insitu units

* SDAP-472 - data-access overhaul to support multiple simultaneous data backends (#294)

* Separated NTS backends

* n/a

* More nts backend stuff

* Working(?) np backend

* Working(?) np backend

* gitignore ini

* ASF headers

* First functioning test of 2 simultaneous backends

* Removed accidentally committed ini files

* Working zarr backend ds list

+ datasets are no longer case sensitive
+ handling for failed zarr ds opens (bad path, bad creds, &c)

* Capture and handle NTS requests routed to backend that doesn't (yet) support them

* analysis setup fails to find VERSION.txt when building locally

* Implemented more NTS functions in zarr backend

* Added misc backend time metrics record field in NCSH

* fixes

* Dynamic dataset management

* Dynamic dataset management

* Dataset management

* Timeseriesspark support

* Update backend dict on dataset mgmt query

* Fixes and improvements

* Adapted matchup to work with zarr backends

* Zarr support

- Distinct slices of time is now default
- No longer assuming+shaping as multivar tiles unless needed

* DDAS adjustments

* find_tile_by_polygon_and_most_recent_day_of_year impl

* Don't sel by time if neither max nor min time are given

* Fix not calling partial when needed

* Pinned s3fs and fsspec versions

* Fixed some dependencies to ensure image builds properly + s3fs works

* Config override for backends

* Deps update

* Add metadata from Zarr collection to /list

* Zarr: Probe lat order and flip if necessary

* Strip quotes from variable names

CM can sometimes publish with extra quotes resulting in KeyErrors

* removed resultSizeLimit param from matchup

* Add # of primaries/avergae secondaries to job output

* rename to executionId

* update changelog

* add totalSecondaryMatched field to /job output

* num unique secondaries addition

* updated docs to use correct sea_water_temperature param name

* bugfix

* fix division by zero bug

* add params to dataset management handler classes

* add page number to default filename for matchup output

* pagination improvements

* removed debugging line

* changelog

* Update helm cassandra dependency (#289)

* Update helm cassandra dependency

* Bump default cassandra PV to 4

* Bump default cassandra PV to 4 in tools

* Changelog

* Fixed small documentation issue

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* stac catalog

* Updated openapi spec

* move stac endpoints to matchup tag in openapi spec

* SDAP-507 - Changes to remove geos sub-dependency

* SDAP-507 - Changelog

* SDAP-507 - Changes to remove geos sub-dependency

* SDAP-507 - Changelog

* delete instead of comment out

* Revert "Update helm cassandra dependency (#289)"

This reverts commit 1e8cc4e.

* deleted disabled endpoint files

* fix bug where still-running jobs failed /job endpoint due to missing metadata

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Moved changelog entries

* SDAP-472 changelog entries

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: skorper <stepheny.k.perez@gmail.com>

* Merge master into develop post-1.2.0 release (#305)

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* domspurge patch (#280)

* Fix delete query to account for PK update

* Fixed mistake in domspurge readme

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Patch: Made conda dep changes accidentally omitted from geos PR

* SDAP-511 - Switch package manager to poetry (#301)

* Redid branch w/ correct base

I tried to just rebase but it didn't work properly

* Changelogs

* Delete old dockerfile

* Removed old conda install files

* Removed version.txt to just use pyproject.toml

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* pyproject.toml dependency updates for Zarr

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>

* SDAP 515 - Improved handling of unreachable remote SDAPs (#308)

* Improved error handling to account for any reason a remote is not reachable

* Further fixes for unreachable remote SDAPs

* Improved logging

* SDAP-513 - Configurable Solr init pod image in helm (#306)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Added some missing ASF headers and removed some unneeded files (#302)

* Remove IDE files

* Added more missing headers

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Update requirements.txt (#309)

* Updates to openapi (#310)

* Disable the try it now button

* Added additional spark algorithms

* Updates to openapi to include additional algorithms

* SDAP-518 - Collection Config Docs (#311)

* Initial work on CC docs

* Add collections to index toctree

* YAML highlighting

* NetCDF section

* remove incubation msg from intro.rst

* Added recs for gridded tile size

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* data-access patch (#313)

* Patches to backend mgmt and zarr backend

- Code cleanup
- Added dask as dependency, so it will be leveraged with Zarr datasets
- Fixed creation of tile times array from Zarr data to ensure its in seconds from 1970-01-01. Original method has been seen to produce incorrect results from bad assumptions

* Changelog

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Changed SDAP startup behavior to wait for all datasets to be prepared before accepting HTTP requests (#314)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* update quickstart with updated solr command (#298)

Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com>

* 1.3.0 to develop (#317)

* Update .asf.yaml (#293)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* domspurge patch (#280)

* Fix delete query to account for PK update

* Fixed mistake in domspurge readme

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Patch: Made conda dep changes accidentally omitted from geos PR

* SDAP-511 - Switch package manager to poetry (#301)

* Redid branch w/ correct base

I tried to just rebase but it didn't work properly

* Changelogs

* Delete old dockerfile

* Removed old conda install files

* Removed version.txt to just use pyproject.toml

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Update release files

- 1.3.0 header in changelogs
- Removed incubation DISCLAIMER
- Removed incubation language from NOTICE and updated URLs
- Bumped versions
- Made copyright year current
- Poetry re-lock

* Fix Dockerfile for DISCLAIMER rm

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>

* SDAP-469 - Support for data with elevation (#276)

* SDAP-469 - Configured timeseriesspark to have the option to bound by depth

* Elevation fetch & masking

untested

* Elevation subsetting

* fixes

* Check if there's elevation when pulling the tile data

* Elevation masking for tiles with no elevation does nothing

* Add elevation fields to collection creation script

* Name elevation fields to be dynamically typed correctly

* Elevation array shape + add to nexuspoint

* Add elev to data in bounds result

* Improved handling of elev arguments & masking

* tile elev in nexusproto backend

* Docker fix

* Additional algorithm support for SDAP 469 3d data (#10)

* Added support for default elevation value of 0 when parameter not provided

* Added support for setting min and max elevation when using single elevation arg

* Added support for data ingested with elevation

* Fixed logic for elevation clause when min and max elevations are equivalent

* Fixed logic for elevation clause when min and max elevations are equivalent

* Bug fix for missing elevation parameters

* Reverted timeAvgMapSpark to use NexusRequestObject

* Fixed bug with order of arguments

* Commented out saving netcdf files

* Bug fix for how numpy arrays are handled

* Cleaned up logic for handling the different arg cases

* Reworked elevation clause logic and added to polygon search

* Added elevation args to find_tiles_in_polygon

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: Kevin <kevinmarlis@gmail.com>

* SDAP-522 - Fixes for broken endpoints found by work for SDAP-521 (#319)

* Implemented fixes

* Changelog

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Error handling on ds listing for zarr (#320)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-319 - Fixed typo (#322)

* Added forgotten changelog entry (#321)

Forgot from PR #276

* [SDAP-492] STV-FIS tomogram data visualization support (#323)

* SDAP-469 - Configured timeseriesspark to have the option to bound by depth

* Elevation fetch & masking

untested

* Elevation subsetting

* fixes

* Check if there's elevation when pulling the tile data

* Elevation masking for tiles with no elevation does nothing

* SDAP-492 - Added dependencies + alg classes

* Initial subsetting

Still need to org and plot the data

* Image rendering

* Lon + Lat tomo fetch impls

* Add elevation fields to collection creation script

* Name elevation fields to be dynamically typed correctly

* Elevation array shape + add to nexuspoint

* WIP: Optimizations for lateral slicing (lon only so far)

Untested

* Elevation array shape + add to nexuspoint

* Fixes

* bugfix

* Use actual extent in extent param

* Minor improvements

* More tomogram work

* Render profile tomo image results using `plt.pcolormesh` instead of `plt.imshow`

* SIGNIFICANT improvement of tomogram processing time

* Add elev to data in bounds result

* Add elev to data in bounds result

* Added GIF renderer

* 3D tomogram viz endpoint

* Add labels

* Colorbars

* Add basemap below tomo render

* Additional arguments and render types

* Simple CSV renderer - point cloud only

* Improved handling of elev arguments & masking

* tile elev in nexusproto backend

* poetry re-lock after deps update from merge

* minor updates

* Docker fix

* Speed up 3d result build

* poetry relock

* Vertical truncation of tomograms by RH98/GND height maps

* Updates:

- Disabled elev tomo endpoint
- Added param to render tomograms in vertically cumulative percentiles. (ie, cumulative return power for each voxel column)
- Added NetCDF renderer for tomo3d endpoint

* Tomo improvements

* Fix for tomo 3d basemap being inverted by latitude

* poetry lock

* Simplified profile tomo API + added NC output formatter

* Added multi-slice capability to 2d plotting

* Add cbar min/max option to tomo profile rendering

+ better nodata error handling

* Support for arbitrary line slicing of tomogram data

Currently does not support multislicing

* Remove accidentally committed test code

* Add slicing along line to NTS w/ nexusproto implementation

+ tomogram arb slicing tries to use this with bbox subsetting as a fallback

* Improve performance of elevation binning

* Improve performance of data gridding

* Update CL + remove LGPL 3 indirect dependency

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* [SDAP-497] - Release build script (#290)

* Build script

* Build script

* Rebased onto 1.2.0 release branch to avoid messing up changelog

* Retry failed builds/pushes once to avoid time loss from transient errors

* Moved changelog entry

* Start of src pull changes

* mv build script

* Better ASF pulling

* Build from GitHub

* Simple docs

* Some cleanup, renaming, deleting unused stuff etc

* Update build.py

* Added build from ASF archive

* Updates and fixes

* Updates for post-grad changes

* Add support for custom nexusproto builds from git

* Build tool updates

* Update changelog

* Implement build from local directory

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-520 - Guide for evaluating SDAP release candidates (#318)

* SDAP-520 Added RC eval guide to RTD

* remove incubator

* add to toctree

* Attempt to fix the many Sphinx warnings on build

* Changelog

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* added primitive match return function to CDMS reader (#260)

* added primitive match return function

* Updated function to be used with generic use cases

* Updated documentation for assemble functions

* Updated documentation, removed xarray dependency

* merged with develop, updated changelog

* SDAP-470 Fixed CHANGELOG.md (#324)

* SDAP-526 - Upgrade canopy and ground masking (#326)

* Upgrade canopy and ground masking

* Changelog update

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Patch to add geopy dep that was forgotten (#328)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-527: Fixed creation of execution status Cassandra table (#329)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* SDAP-521 - Improved SDAP testing suite (#325)

* SDAP-520 Added RC eval guide to RTD

* remove incubator

* add to toctree

* Granule download script for testing

* Move & update test script

Made current with what was deployed for the CDMS project. Will need extensive editing.

* Test script pruning + guide start

* Updates

* Updates

* Guide for install and run

* Attempt to fix the many Sphinx warnings on build

* Fix bad ref

* Fix bad ref

* Fix bad ref (third time's the charm?)

* Removal of ingested test data

* Reduced datainbounds L2 test bbox to ease memory footprint

* Revert "Reduced datainbounds L2 test bbox to ease memory footprint"

This reverts commit 46cf5ad.

* Update docs for missing test collection

* SDAP-521 Updated quickstart and test guide. (#327)

* SDAP-521 Updated quickstart and test guide.

* SDAP-521 Updated solr start up env variables to be consistent with helm chart.

* SDAP-521 Updated README.md

---------

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: Nga Chung <17833879+ngachung@users.noreply.github.com>

* SDAP-529: Added configuration for verbose logging for collection manager in the Helm chart (#330)

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>

* Bump versions for release

* Some more version fixes

* Some more version fixes

* Fix doc typo

* I'm having issues with the venv. This needs more testing but shouldn't be a release blocker

* Final release dates to changelog

* Moved incorrect CL entry to proper section

---------

Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
Co-authored-by: skorper <stepheny.k.perez@gmail.com>
Co-authored-by: Kevin <kevinmarlis@gmail.com>
Co-authored-by: alovett-COAPS <113188918+alovett-COAPS@users.noreply.github.com>
Co-authored-by: Nga Chung <17833879+ngachung@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants