Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar

[corasaurus-hex]

Is your feature request related to a problem or challenge?

Datafusion currently supports registering files in the Arrow IPC file format as tables:

    ctx.register_arrow("my_table", "file.arrow", ArrowReadOptions::default())
        .await
        .unwrap();

    ctx.sql("SELECT * FROM my_table LIMIT 10")
        .await
        .unwrap()
        .show()
        .await
        .unwrap();

You can also just reference the file path from SQL in datafusion-cli:

> SELECT * FROM 'file.arrow' LIMIT 10;

You cannot, however, do the same with files in the Arrow IPC stream format. You get the error:

called `Result::unwrap()` on an `Err` value: ArrowError(ParseError("Arrow file does not contain correct footer"), None)

Describe the solution you'd like

I would love if register_arrow supported files in the Arrow IPC stream format, or if another equivalent function would be added to do the same. Additionally, it would be great if datafusion-cli could just reference the files by name in the same way it can for the alternative Arrow IPC format.

Describe alternatives you've considered

1. Convert from the stream format to the file format and then query as shown above.
2. Read all the record batches into memory and then register it as MemTable.
3. Add a new StreamProvider impl and use a StreamTable.

There are probably others, too, but none as simple as just being able to register the arrow file with register_arrow or referencing the file directly in datafusion-cli.

Additional context

I'm interested in taking a crack at this feature but, assuming y'all are interested in it, I would love some implementation guidance.

Thanks for your time!

[alamb]

I reread the format guide again, and I think the Arrow file format is an extension of the IPC (stream) format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format

It looks like the file format simply has a few extra header bytes and then a footer. Otherwise it is the same as the stream. format

So we should be able to support both formats fairly straightforwardly

It may be possible to simply use the StreamReader rather than the FileReader in the underlying code for ArrowFormatFactory
https://github.com/apache/datafusion/blob/2741c60ad9d637e5014d201802abb25562449491/datafusion/core/src/datasource/file_format/arrow.rs#L110-L109

[coracuity]

a StreamReader can only read an Arrow IPC Stream file, and can't read an Arrow IPC File... file... (this naming is hard), unfortunately, so we'd need to try both and see which works?

[coracuity]

I think, from a superficial reading of the code, I would need to extend ArrowOpener and ArrowFormat to determine the underlying IPC variant and use the appropriate reader. I know the file format starts with the bytes ARROW and so that might be a good enough check.

[alamb]

That sounds right to me

[corasaurus-hex]

I took a crack at this today and got pretty far but I'm running up against a couple instances of low-level arrow decoding that feel out of place. I'd love your thoughts on maybe upstreaming something or coming up with a better abstraction to handle the need to do selective low-level decoding. I'm willing to put in the work, I just don't want to put a lot of labor into something that might get rejected for going a bad direction.

1. ArrowOpener: I'd need to duplicate this decoding/fetching for IPC streams as well. It feels like there's a missing abstraction somewhere for this...

2. infer_schema_from_file_stream: this was a pretty straight-forward one to solve (here's how I did it), but still feels like something that might be best to have in arrow-rs itself?

You can see my WIP branch diff here. I'm not sure if I should be trying to combine things like I am or if I should be creating entirely separate handling for arrow_stream vs arrow_file.

[alamb]

Thanks @corasaurus-hex

I agree it looks like we can improve the code in this area

What I would suggest is:

1. Proceed with your PR, even if it isn't ideal (e.g. replicate ArrowOpener if needed), and ensure all the code is well tested
2. File any relevant upstream tickets to improve the APIs

I should be trying to combine things like I am

That sound great in theory, but I am not really sure until we see what it looks like. By default, I would defer to the person implementing the feature (you)

Also, as long as we have good test coverage of the feature, we can always refactor / improve the code as follow on PRs

Thank you for all your help in this area

[jdcasale]

Any movement on this? Happy to help with dev or review cycles if it's blocked.

Kind of a shame that if I want to read an ipc file (a buffer file or something) I have to hydrate the whole file into memory by calling read_batches().

[corasaurus-hex]

Still just my WIP branch. I haven't had a chance to pick it up again, work has been crazy, but I planned on it this weekend barring something coming up. Would love your thoughts on the direction!

[corasaurus-hex]

would love a review if you have time #18457

[jdcasale]

Will take a look tonight (don't have merge permissions, but I can give a thorough review to help those who do)