[C++] Dataset APIs require s3:ListBucket Permissions

[asfimport]

Hi team, first time posting an issue so I apologize if the format is lacking. My original comment is on ARROW-13685 Github Issue here. 

Long story short, our environment is super locked down, and while my application has permission to write data against an s3 prefix, I do not have the ListBucket permission nor can I add it. This does not prevent me from using the "individual" file APIs like pq.write_table but the bucket validation logic in the "dataset" APIs breaks when trying to test for the bucket's existence. 

pq.write_to_dataset(pa.Table.from_batches([data]), location, filesystem=s3fs)

OSError: When creating bucket '<my bucket>': AWS Error [code 15]: Access Denied

The same is true for the generic pyarrow.dataset APIs. My understanding is the bucket validation logic is part of the C++ code, not the Python API. As a Pythonista who knows nothing of C++ I am not sure how to resolve this problem.
 
Would it be possible to disable the bucket existence check with an optional key word argument? Thank you for your time!
 

Reporter: Jonny Fuller
Assignee: Sanjiban Sengupta / @sanjibansg

Related issues:

• [C++] S3Filesystem shouldn't create new buckets by default (relates to)

PRs and other links:

• GitHub Pull Request #12701

_{Note: This issue was originally created as ARROW-15892. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Which permissions do you have in that S3 environment exactly?

[asfimport]

Will Jones / @wjones127:
I think the ListBucket permission grants access to list objects in the bucket, which I think is required for a lot of dataset operations.

From Amazon's docs: 

For example, the s3:ListBucket permission allows the user to use the Amazon S3 GET Bucket (List Objects) operation.
The error you have sounds more like it's from not having s3:CreateBucket? Which I would agree is something that shouldn't be necessary.

[asfimport]

Jonny Fuller:
My understanding is that the API tries to create a bucket to see if it exists as part of its standard workflow. The only permissions I have are PutObject within a prefix boundary (ie put object at /myapp/namespace/). I'm looking for a way to bypass the bucket checking step and simply write the data as if I was using the write_table API. 

I should add, I cannot provide the detailed permissions because they are provided by a temporary STS session. To my knowledge there is no way to ask STS "what permissions does this assumed role have?" ie you have to actually make requests to see if they are denied. I know I can PutObject because write_table ie writing a single parquet file works, as do other single file operations. Only when trying to use the higher level dataset (aka multiple file partition APIs) does my operation fail due to access control issues as detailed above. 

[asfimport]

Weston Pace / @westonpace:
I can reproduce this and I agree that the permission in question is probably ListBucket. In this case it isn't the logic around checking bucket existence exactly that is causing the problem. In the datasets module we always call CreateDir since the function succeeds if the directory already exists.

In s3fs, if the directory is a bucket (e.g. if it is foo and not foo/my_dataset), we call CreateBucket without doing any existence checking at all (then if we get a "bucket already exists" error we just return ok). That explains why the error you are getting is "OSError: When creating bucket" and not "OSError: When testing for existence of bucket"

It should be straightforward to add an option to completely disable CreateDir calls in datasets. This would be a little bizarre from a local filesystem perspective but makes perfect sense in an object store where directories don't need to be created. You would also need to make sure you specify existing_data_behavior=overwrite_or_ignore

[asfimport]

Weston Pace / @westonpace:
Actually, I suppose this means that a user might need the CreateBucket permission (assuming they don't disable CreateDir calls entirely with the aforementioned flag) in order to use write_dataset if the base directory for their dataset is a bucket.

In other words, today, the call write_dataset(tab, 'foo', filesystem=s3fs, format='parquet') requires both CreateBucket and ListBucket permission and the call write_dataset(tab, 'foo/my_dataset', filesystem=s3fs, format='parquet') only requires ListBucket permission (assuming the foo bucket exists). We could probably improve on that as well.

[asfimport]

Will Jones / @wjones127:
@westonpace based on my prior experience with S3, I'm actually surprise it would call CreateBucket at all. IMO it should probably error if the bucket doesn't exist by default; buckets are entities that usually have lots of governance around them (tags, permissions, etc), so making it easy for someone to accidentally create one via a typo seems bad. 

[asfimport]

Jonny Fuller:
Wow thanks for following up so quickly team. I'm definitely with @wjones127. I would want to be very specific about creating buckets.

[asfimport]

Weston Pace / @westonpace:
I think this is one of those things that is just a bit leaky when you have a filesystem abstraction that covers both local and remote filesystems. The flag I described above (don't create directories) ought to prevent any buckets from being created. We could also add a configuration parameter to the S3FileSystem itself (e.g. "safe mode") that would error out if it were asked to create a bucket. That would help if the user made a typo for example. I think that probably belongs in a different JIRA so feel free to open.

[asfimport]

Antoine Pitrou / @pitrou:
Hmm, I think adding a general S3 option is a good idea. I agree it may not make sense generally to allow implicit creation of buckets (but some S3-compatible backends may have different rules, so it should still be possible).

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 12701
#12701