Zarr v2 took care in DirectoryStore to use atomic writes by renaming files into place, which ensures that writes to disk are either complete or not written at all:
|
# write to temporary file |
|
# note we're not using tempfile.NamedTemporaryFile to avoid restrictive file permissions |
|
temp_name = file_name + "." + uuid.uuid4().hex + ".partial" |
|
temp_path = os.path.join(dir_path, temp_name) |
|
try: |
|
self._tofile(value, temp_path) |
|
|
|
# move temporary file into place; |
|
# make several attempts at writing the temporary file to get past |
|
# potential antivirus file locking issues |
|
retry_call(os.replace, (temp_path, file_path), exceptions=(PermissionError,)) |
|
|
|
finally: |
|
# clean up if temp file still exists for whatever reason |
|
if os.path.exists(temp_path): # pragma: no cover |
|
os.remove(temp_path) |
I don't see any similar logic in Zarr v3, which just seems to using Python's file interface (via open):
|
with path.open("r+b") as f: |
|
f.seek(start) |
|
# write takes any object supporting the buffer protocol |
|
f.write(value.as_buffer_like()) |
This would be useful to still support in Zarr v3.
Per @rabernat, this was almost certainly not intentional.
Originally posted by @shoyer in #3410
Zarr v2 took care in DirectoryStore to use atomic writes by renaming files into place, which ensures that writes to disk are either complete or not written at all:
zarr-python/zarr/storage.py
Lines 1146 to 1161 in f1978dd
I don't see any similar logic in Zarr v3, which just seems to using Python's file interface (via
open):zarr-python/src/zarr/storage/_local.py
Lines 52 to 55 in c9509ee
This would be useful to still support in Zarr v3.
Per @rabernat, this was almost certainly not intentional.
Originally posted by @shoyer in #3410