Add persistence examples using shelve/dbm, lmdb

[alimanfoo]

As discovered in #129, the Python standard shelve module can be used for persistent storage as it provides the MutableMapping interface. This is a good alternative to using the file system directly as it avoids lots of files. It's also a good alternative to using a Zip file which needs care when writing to avoid duplicate entries. Similar benefits probably apply to LMDB which has a MutableMapping interface provided by zict.

It would be good to add some examples using shelve and LMDB to the tutorial.

[alimanfoo]

xref #133 - using dbm directly rather than shelve is probably a better example, as this would bypass the pickling/unpickling of values that shelve does, which in the case of Zarr is unnecessary as values are always bytes objects. However, dbm databases cannot currently be used directly as a store because they return keys as bytes objects.

[alimanfoo]

Btw just as a reminder, LMDB supports returning buffer objects which provide direct access to it's own memory-mapped data, which should reduce the number of memory copies required. It would be interesting to benchmark this against other key-value stores to see if this provides an noticeable performance improvement (it may not if compression/decompression is the bottleneck, although previously I have found reducing memory copies can make a difference). Would require some tweak to zict's LMDB wrapper to enable the buffer support.

[jakirkham]

LMDB is definitely pretty slick from what I have read. Another kind of similar database to look at that might be nice is UnQLite.

Agree that pickling/unpickling done by shelve is unnecessary, but would be curious what the overhead is like. If it's not large, having something from the standard library that is easy to setup is nice.

[alimanfoo]

Re shelve I did a very quick benchmark and performance was same as using the file system (i.e. DirectoryStore) so I suspect pickling/unpickling adds little overhead compared to the cost of compression/decompression. The other potential downside with shelve is that pickling reduces portability of data across Python versions. Hence if it was possible to use the Python dbm module directly instead of shelve (which is a very thin wrapper around dbm) that would be preferable.

…

On Mon, Feb 27, 2017 at 4:42 PM, jakirkham ***@***.***> wrote: LMDB is definitely pretty slick from what I have read. Another kind of similar database to look at that might be nice is UnQLite. Agree that pickling/unpickling done by shelve is unnecessary, but would be curious what the overhead is like. If it's not large, having something from the standard library that is easy to setup is nice. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/131#issuecomment-282775333>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QgMzZJJ4i3uROzBkVZM2kat6NG9hks5rgvzdgaJpZM4MLzgt> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[alimanfoo]

One thing to watch out for with standard dbm module is that Python will fall back to a "dumb" dbm implementation if it cannot find gdbm or ndbm. The dumb implementation seems pretty fast for writing and reading in some quick benchmarks, but deleting was pathologically slow. Also I cannot find any way to add gdbm support when using Python from conda (on ubuntu there is python3-gdbm). But maybe not worth the effort to resolve if more modern key-value stores are available. On Mon, Feb 27, 2017 at 4:54 PM, Alistair Miles <alimanfoo@googlemail.com> wrote:

…

Re shelve I did a very quick benchmark and performance was same as using the file system (i.e. DirectoryStore) so I suspect pickling/unpickling adds little overhead compared to the cost of compression/decompression. The other potential downside with shelve is that pickling reduces portability of data across Python versions. Hence if it was possible to use the Python dbm module directly instead of shelve (which is a very thin wrapper around dbm) that would be preferable. On Mon, Feb 27, 2017 at 4:42 PM, jakirkham ***@***.***> wrote: > LMDB is definitely pretty slick from what I have read. Another kind of > similar database to look at that might be nice is UnQLite. > > Agree that pickling/unpickling done by shelve is unnecessary, but would > be curious what the overhead is like. If it's not large, having something > from the standard library that is easy to setup is nice. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <https://github.com/alimanfoo/zarr/issues/131#issuecomment-282775333>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAq8QgMzZJJ4i3uROzBkVZM2kat6NG9hks5rgvzdgaJpZM4MLzgt> > . > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: ***@***.*** Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721 <+44%201865%20287721>

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[jakirkham]

How big were your data and chunks?

...pickling reduces portability of data across Python versions.

As long as we stick to version 0, which is the default, we should be fine.

Hence if it was possible to use the Python dbm module directly instead of shelve (which is a very thin wrapper around dbm) that would be preferable.

Agreed. Though a potential downside is sometimes it is tricky to get a fully built dbm module. I think we (conda-forge) don't support the GNU stuff (gdbm). Perhaps we could try though. ( conda-forge/python-feedstock#125 )

[alimanfoo]

On Mon, Feb 27, 2017 at 5:05 PM, jakirkham ***@***.***> wrote: How big were your data and chunks?

About 4Mb uncompressed. It was a very quick-and-dirty benchmark tho, don't take my word for anythin.

...pickling reduces portability of data across Python versions. As long as we stick to version 0, which is the default, we should be fine.

I thought pickle version 3 was default on Python 3, which is not compatible with Python 2?

Hence if it was possible to use the Python dbm module directly instead of shelve (which is a very thin wrapper around dbm) that would be preferable. Agreed. Though a potential downside is sometimes it is tricky to get a fully built dbm module. I think we (conda-forge) don't support the GNU stuff (gdbm). Perhaps we could try though. ( conda-forge/python-feedstock# 125 <conda-forge/python-feedstock#125> )

Yeah, I wouldn't want to cause more work for you guys, you've got so much on already. That said it would be nice to have a "recommended/preferred" key-value store for use with Zarr, which is reasonably performant and very easy to install (if not available in standard library), i.e., the path of least resistance for general users who want the convenience of data stored in a single file but don't necessarily need absolute blazing maximum performance.

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/131#issuecomment-282782749>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QsejqYMQ6uVMQ5M-lI0BXiOeUWx2ks5rgwJSgaJpZM4MLzgt> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[jakirkham]

I thought pickle version 3 was default on Python 3, which is not compatible with Python 2?

Yep, sorry, I misspoke. So we could still force it to 0.

Yeah, I wouldn't want to cause more work for you guys, you've got so much on already.

If it's helpful, we can explore as time allows. The main thing to notice is these might be tricky things to get working on Windows.

That said it would be nice to have a "recommended/preferred" key-value store for use with Zarr, which is reasonably performant and very easy to install (if not available in standard library)

I would guess that LMDB is going to actually be a good mix of these. We already have a Python package for this for all platforms at conda-forge. They also build wheels for Windows that they have on PyPI. Building on Unix is straightforward.

It might be worthwhile to have shelve if we want something that works with the standard library. While dbm might be more performant, getting a fully featured build of this for all platforms may be more work and still be less performant than LMDB.