[R] Expose Azure Blob Storage filesystem

[asfimport]

I'd like to see the R arrow package be able to interface with the Azure Blob Storage file system from the AzureStor package.

 

In python, pyarrow and adlfs work together so I'd like for AzureStor and arrow under R to also work together.

Reporter: Dean MacGregor

Related issues:

• [C++] Filesystem implementation for Azure Blob Storage (depends upon)

_{Note: This issue was originally created as ARROW-16791. Please see the migration documentation for further details.}

[asfimport]

Will Jones / @wjones127:
PyArrow and adlfs work because we implement a compatibility layer between Python fsspec filesystems and PyArrow filesystems.

IIUC you are suggesting we could implement some similar integration in R? Does there exist a standard protocol analogous to fsspec?

[asfimport]

Dean MacGregor:
I don't think so but I don't really know enough about how pyarrow or r arrow works behind the scenes to answer your question.

 

I just know azure blob works in pyarrow, that R arrow works with S3 (or at least the documentation is there) and I'm assuming that the difference between s3 and azure isn't too big. 

 

 

[asfimport]

Dean MacGregor:
I found that in arrow/r/src/filesystem.cpp, on lines 38-40 there's a comment block which says to uncomment them for AzureBlobFileSystem when R6 classes are made.  However, when looking further, it seems that there is more in the cpp file for s3 and GCS than just that reference but nothing else for Azure so I guess that comment should be more of a TODO since, unless I'm missing a lot (which is all too possible) there's much more needed for Azure to work than just an R6 class, no?

 

To that end, Microsoft has https://github.com/Azure/azure-sdk-for-cpp for using azure in cpp.

 

I don't know cpp enough (really, at all) to be able to extend MS's code into something usable for arrow but just wanted to put the link in here in case it's helpful.

[asfimport]

Neal Richardson / @nealrichardson:
Thanks, yeah, that's being added to the C++ library in ARROW-2034. Once that is merged, we can add R bindings.

[drdrjacobs]

Now that #18014 is closed, can this be implemented?

[kjtheron]

Any news on this development. Having the Azure Cloud storage implemented in arrow for R would be amazing.

[ran-codes]

My company is tied to Azure. This would be amazing - please implement!

[ran-codes]

Are you open to pull requests for this?

[marberts]

I'm also interested in using arrow with Azure from R. @jonkeane are you open to a PR for this?

[jonkeane]

Yes, if you would like to submit a PR, feel free. If you get stuck anywhere let us know and we can get you some pointers. the GCS + AWS filesystems are both good places to start with patterns.