[Unexpected Behaviour] `open_group` raises `AssertionError` if array w/o group exists

[FabricioArendTorres]

Zarr version

3.0.8

Numcodecs version

0.15.1

Python Version

3.11.11

Operating System

Linux, Dockerfile based on python:3.11-slim

Installation

poetry

Description

I think there is some unexpected behavior that occurs if one

1. creates an array in the root of a store
2. attempts to open a group in the root of this store in r or r+ mode.

I would have expected some meaningful error, such as the zarr.errors.ContainsArrayError, or FileNotFoundError, but one just gets a very vague AssertionError.
I would like to avoid catching such a broad error in my code.

Below you can find a reproducible test code (tested with uv run) that highlights the expected and actually occuring behaviour.

Note on Stores:
I originally stumbled upon it when using the object store, and could directly reproduce it with the memory store, so it should be independent of the store.

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
#   "pytest>=8.3",
# ]
# ///
# This script automatically imports the development branch of zarr to check for issues
import pytest
import zarr.errors
import zarr.storage


def test_open_group_in_dir_with_array():
    store = zarr.storage.MemoryStore()

    # this passes
    with pytest.raises(FileNotFoundError):
        zarr.open_group(store, mode="r+")

    # now create an array within root
    zarr.create_array(store, shape=(2,), dtype=float)

    # this somehow raises an assertion error ?
    # I would expect zarr.errors.ContainsArrayError or FileNotFoundError.
    # with pytest.raises(AssertionError):
    with pytest.raises((zarr.errors.ContainsArrayError, FileNotFoundError)):
        zarr.open_group(store, mode="r+")


def test_create_group_in_dir_with_array():
    store = zarr.storage.MemoryStore()

    # this passes
    with pytest.raises(FileNotFoundError):
        zarr.open_group(store, mode="r+")

    # now create an array within root
    zarr.create_array(store, shape=(2,), dtype=float)

    with pytest.raises(zarr.errors.ContainsArrayError):
        tmp = zarr.create_group(store)


if __name__ == "__main__":
    pytest.main([__file__])

Additional output

The test_open_group_in_dir_with_arrray fails with the following stacktrace:

==================================================================== FAILURES ====================================================================
_______________________________________________________ test_open_group_in_dir_with_array ________________________________________________________

    def test_open_group_in_dir_with_array():
        store = zarr.storage.MemoryStore()
    
        # this passes
        with pytest.raises(FileNotFoundError):
            zarr.open_group(store, mode="r+")
    
        # now create an array within root
        zarr.create_array(store, shape=(2,), dtype=float)
    
        # this somehow raises an assertion error ?
        # I would expect zarr.errors.ContainsArrayError or FileNotFoundError.
        # with pytest.raises(AssertionError):
        with pytest.raises((zarr.errors.ContainsArrayError, FileNotFoundError)):
>           zarr.open_group(store, mode="r+")

zarr_bug.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/_compat.py:43: in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/api/synchronous.py:529: in open_group
    sync(
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/core/sync.py:163: in sync
    raise return_result
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/core/sync.py:119: in _runner
    return await coro
           ^^^^^^^^^^
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/api/asynchronous.py:828: in open_group
    return await AsyncGroup.open(
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/core/group.py:583: in open
    return cls._from_bytes_v3(
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/core/group.py:647: in _from_bytes_v3
    return cls.from_dict(store_path, group_metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/vscode/.cache/uv/environments-v2/zarr-bug-4f65dcfdf7520214/lib/python3.11/site-packages/zarr/core/group.py:656: in from_dict
    metadata=GroupMetadata.from_dict(data),
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'zarr.core.group.GroupMetadata'>
data = {'attributes': {}, 'chunk_grid': {'configuration': {'chunk_shape': [2]}, 'name': 'regular'}, 'chunk_key_encoding': {'c...ion': {'endian': 'little'}, 'name': 'bytes'}, {'configuration': {'checksum': False, 'level': 0}, 'name': 'zstd'}], ...}

    @classmethod
    def from_dict(cls, data: dict[str, Any]) -> GroupMetadata:
        data = dict(data)
>       assert data.pop("node_type", None) in ("group", None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError

[d-v-b]

agreed that an assertion error here is not very helpful. I'll have a look at this today

[TomNicholas]

I just ran into this - it's very ugly.

I'm surprised that all stores do not have a root group by definition??

This also makes it impossible to use .tree on some stores:

In [2]: z = zarr.create_array(shape=(40, 50), chunks=(10, 10), dtype='f8', store=MemoryStore())
   ...: z
Out[2]: <Array memory://4386248768 shape=(40, 50) dtype=float64>

In [3]: zarr.open_group(z.store).tree()

raises AssertionError.

AFAICT there is no way to get the root group, and no other way to call .tree() other than on a Group instance.

[dstansby]

Isn't it possible (and reasonable) to have a store with just a single array and no groups, as in the code example above? In which case trying to open a group should error (but in a nicer way than an AssertionError).

[TomNicholas]

I personally think that's a bad design choice, and that there should always be a root group. But I understand that that is a spec-level problem.

Separately I would still expect to be able to use .tree on such a group-less store. But if .tree is only available on Group and not all stores have a Group then .tree cannot always be available. That's an example of why not having a root group is a bad design IMO - you forgo the useful property that methods on groups in the tree are available at all levels.

[d-v-b]

Not saying the root array concept is a good idea or not, but requiring a root group would complicate any API for directly creating an array without first defining a group, which is a pattern a lot of people use.

That being said, we can still define a tree method on arrays that does what you expect. the tree method on the group could even just be defined as calling the tree method on its members.