opendap error with version 1.7.0

[aragong]

Hello,

We are suffering this error when we read a netcfd throgh opendap with new version 1.7.0, with a netcdf4 version rollback to version 1.6.5 the error dissapeard, hope it helps!

import xarray as xr

xr.open_dataset('https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc')

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('[https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc',)](https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc%27,)), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '85e04132-c9c8-42a9-a6da-0516ae5e8dc1']
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 645, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 408, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 355, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 417, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 411, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: '[https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc'](https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc%27)

[ocefpaf]

I can reproduce that error on Linux:

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url

It is probably a regression b/c we are using the built-in curl library to build it. I'm working on a docker image that will have the same libraries as the previous wheels.

[aragong]

I can reproduce that error on Linux:

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url

It is probably a regression b/c we are using the built-in curl library to build it. I'm working on a docker image that will have the same libraries as the previous wheels.

same error when launch this inside a dockered API... thanks!

[ocefpaf]

Looks like we need these unreleased commits:

Unidata/netcdf-c#2690

and the previous wheel was built against latest code instead of a release:

MacPython/netcdf4-python-wheels@b611fcd

I'm modifying the new wheel config to do the same but it would be nice to get a new release of netcdf-c from upstream.

[ocefpaf]

@aragong can you confirm that it indeed fixes the problem when you used 1.6.5? Or give more context? I tried:

$ pip freeze
certifi==2024.6.2
cftime==1.6.4
netCDF4==1.6.5
numpy==1.26.4

and then your URL

$ ncinfo "https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc"
Traceback (most recent call last):
  File "/envs/TEST/bin/ncinfo", line 8, in <module>
    sys.exit(ncinfo())
  File "/envs/TEST/lib/python3.10/site-packages/netCDF4/utils.py", line 551, in ncinfo
    f = Dataset(filename)
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: 'https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc'

However, the URL used to test this regression in the wheels do pass:

$ ncinfo "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/m2.hamtide11a.nc"                            
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format DAP2):
    history: FERRET V6.62 03-Feb-10
    title: HAMTIDE11a: m2 ocean tide
    Conventions: CF-1.0
    dimensions(sizes): LAT(1441), LON(2881)
    variables(dimensions): float64 LON(LON), float64 LAT(LAT), float32 RE(LAT, LON), float32 IM(LAT, LON), float32 AMPL(LAT, LON), float32 PHAS(LAT, LON)
    groups: 

Or maybe that URL is changes every hour and I should be using https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061400.nc now?

[aragong]

We fix the problem only changing the netcdf version to the previous one. 1.6.5

Pip uninstall netcdf4 -y
Pip install netcdf4==1.6.5

No idea about certs and wheels @ocefpaf ... Sorry

[ocefpaf]

@aragong, while there is a problem with the latest wheel, I do believe that the URL you provided above it no longer valid. Can you check? I can confirm that the new wheel will work for the previous URLs that had that issue but I cannot test against the URL you reported above.

[jswhit]

Looks like we need these unreleased commits:

Unidata/netcdf-c#2690

and the previous wheel was built against latest code instead of a release:

MacPython/netcdf4-python-wheels@b611fcd

I'm modifying the new wheel config to do the same but it would be nice to get a new release of netcdf-c from upstream.

Ah yes - I completely forgot about this, sorry. Since 4.9.3 has not been released yet, we do need to build against github develop.

[dopplershift]

@WardF is working hard to getting 4.9.3 across the finish line right now.

[aragong]

Sorry for the delay, I was out of the office!

Still having the same error I test the following code:

pip uninstall netcdf4 -y
pip install netcdf4 --no-cache-dir

import xarray as xr

xr.open_dataset('https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc')

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '392d7c27-b1ce-44e3-88e0-20555338cc3c']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/api.py", line 573, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 646, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 409, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 356, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 418, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 412, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2470, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2107, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: 'https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc'

@ocefpaf @jswhit I do not see any recent patch version...
Do you spect the problem to be solved or I should wait for the next patch version?

Thank you for your time!

[aragong]

@aragong, while there is a problem with the latest wheel, I do believe that the URL you provided above it no longer valid. Can you check? I can confirm that the new wheel will work for the previous URLs that had that issue but I cannot test against the URL you reported above.

It is operational data, you can check the catalag here: https://ihthredds.ihcantabria.com/thredds/catalog/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/catalog.html

sorry for the delay in my response!

[ocefpaf]

It will be fixed when a new release, or post release, is issued.Ultimately, the problem is not with netcdf4-python but with the underlying netcdf-c. If you are in a hurry you can re-build the wheel with latest netcdf-c from git source, or wait for a new release that will have the fix in it.

[aragong]

It will be fixed when a new release, or post release, is issued.Ultimately, the problem is not with netcdf4-python but with the underlying netcdf-c. If you are in a hurry you can re-build the wheel with latest netcdf-c from git source, or wait for a new release that will have the fix in it.

Thank you very much for the clarification!