[Python] Support C stream interface of single arrays

[asfimport]

It seems that the C stream interface in pyarrow currently requires the array to be a StructArray.

I do not see this constraint in the spec (https://arrow.apache.org/docs/format/CStreamInterface.html).

The error I get when I pass an Int32Array to it (declared on the schema):

Invalid: Cannot import schema: ArrowSchema describes non-struct type int32

It would be nice to support everything, like the C data interface.

Reporter: Jorge Leitão / @jorgecarleitao

_{Note: This issue was originally created as ARROW-15747. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Which API function are you invoking?

[asfimport]

Jorge Leitão / @jorgecarleitao:
arrow.ipc.RecordBatchReader._import_from_c (this may be the issue); I also tried RecordBatchStreamReader.

For reference the code is I am using is here: jorgecarleitao/arrow2#857, this line: https://github.com/jorgecarleitao/arrow2/pull/857/files#diff-721d8214320dcf310ab55f0de12a4ed43e910df50f959b8c0660ef1a73aa7950R51

I am getting this by passing an empty iterator (i.e. one where the returned array has always been released.

[asfimport]

Antoine Pitrou / @pitrou:
RecordBatchReader reads record batches at the name implies. Only struct arrays can be converted to record batches.

[asfimport]

Jorge Leitão / @jorgecarleitao:
I see. Thanks.

Is there an API for anything else in Python? I could only find this one in the docs? https://arrow.apache.org/docs/python/api/ipc.html

Regardless, I am struggling to make this compatible - when I export it to the C stream interface an empty iterator, the sequence of events I get:

• get_next called

• empty ArrowArray (i.e. released, no buffers, no children) created on the stack

• core dump (on the call to write the empty ArrowArray to the pointer passed to get_next)

  I.e. we are being unable to write to the pointer passed to get_next (allocated by C++).

  If I do nothing to the pointer when the iterator is empty (i.e. not write a released array), the same happens when trying to write to the ArrowSchema pointer passed to get_schema - core dump on the call to write.

  Note that this is different from anything else so far, in the C data interface we have been allocating from Rust, pass the (ArrowArray,ArrowSchema) pointers to C++ via the _import_from_c.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Ok, I made it work! There is no guarantee that the pointer from C is aligned with Rust's representation, so a different call is needed.

Other than that, the only thing I was unable to find was an API in Python to write arrays other than StructArray / RecordBatch

[asfimport]

Antoine Pitrou / @pitrou:

Other than that, the only thing I was unable to find was an API in Python to write arrays other than StructArray / RecordBatch

Yes, because basically this wraps the C++ APIs which only know about RecordBatchReader. Is it important for you to write/read any kind of array?

[asfimport]

Dewey Dunnington / @paleolimbot:
I noticed that there was no support for importing a non-struct array in C++ when writing stream support in the geoarrow/narrow R package prototypes; however, I never found a use for them and couldn't see how they would be useful in the R, Python, or C++ implementations. I assumed that as soon as a practical use-case (i.e., other than for the sake of completeness) was found, that they would be implemented at that point.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Any array can be wrapped on a StructArray, so technically we just have to remember to wrap it on the struct :)

I see three main reasons to support the simple array:

1. an implementation may not support StructArray over the c data interface but still support simpler types.
2. when we want to stream a "Table" where the chunks are not row-aligned, e.g.

c1: [10 rows][20 rows][20 rows]
c2: [20 rows][30 rows]

it is cleaner to have an iterator of chunks of those types, instead of wrapping and unwrapping each of them on StructArray.

3. StructArrays with sliced children over the c data interface are tricky. If we wrap a sliced array over struct, and pass it over the c stream interface, some implementations may not support it.

[asfimport]

Antoine Pitrou / @pitrou:
Use case 2 (heterogenous chunking) is easily addressed by redoing the chunking. That's what Arrow C++ does when you want to get a RecordBatchReader out of a Table. I agree with use cases 1 and 3.

There are two ways this could be added to Arrow C++ (and PyArrow):

1. return a RecordBatchReader that would read batches of a single column

2. add a facility like RecordBatchReader but on Arrays

   The first approach is easier and perhaps less elegant. Also, the second approach would allow to implement an export function, which would be a bit clunky under the first approach.

[jorisvandenbossche]

I think this was addressed recently by @paleolimbot adding ExportChunkedArray/ImportChunkedArray for the stream interface? (#39455)

[paleolimbot]

It definitely provides a means by which to consume or produce streams of non-struct arrays, although it requires that they are 100% materialized up front (or that it is trivial to completely materialize them on consumption). That PR added infrastructure (e.g., the ArrayStreamArrayReader) that could be generalized (e.g., a ChunkedArrayReader or something). If there's vague consensus on what that should look like in Arrow C++ (or if it should be there or not) I'm happy give a go at the implementation.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.