ListArray with 64bit instead of 32bit offset

[maartenbreddels]

I'm starting to appriciate Arrow more and more, and many new ideas I have with vaex fully align with arrow. I've recently been adding in string support in vaex and I am trying to follow the arrow specs, however have a bit of an issue with the int32 limitation of ListArray

This limits each element in the list to be ~2GB in size (which is not crazy), but also complicates matters when dealing with string concatenation, having to switch to a ChunkedArray. The memory saving by doing 32bit is not huge, and I am not sure the penalty of code complexity is worth it.

I was wondering if the 32bit limitation is set in stone, or is something that could change, and if maybe other people have strong opinions about.

cc @xhochy

[wesm]

There is https://issues.apache.org/jira/browse/ARROW-750 about having binary and string arrays with 64 bit offsets.

Having arrays with > 2GB elements or binary arrays with > 2GB of data would be considered an anti-pattern in the context of database systems, regardless of whether the offsets are 32- or 64-bit. So in light of these it doesn't make sense to have the default type be 64-bit capable if this capability is seldom used

I opened https://issues.apache.org/jira/browse/ARROW-4810 as I think it would be useful to have the option of representing such data in memory

[wesm]

We can discuss further either on the mailing list or on those JIRA issues