Support ETag headers

[daniellehanks]

Split from this issue.

Requesting support for If-Match/If-None-Match headers for GCS via the Python client library.

My use case:
My API is a thin layer over GCS (for storing profile images), and I only want to actually read the image data and serve it to the client if the actual content is different from their cache. They will specify an If-None-Match header to my API (as they won't know anything about GCS or generation numbers, kind of the point), so I just need to essentially pass this along to GCS. I don't want to either do the extra round tripping of reading the current file to check the ETag and then use the generation number from that in a subsequent request, or not use the client library.

[daniellehanks]

My team is about at the point where we are needing this. We would love to consider volunteering a contribution if we could get a design/owner partner on the Google side. Would that be something you all would consider? TIA @tseaver

[tseaver]

@andrewsg, @frankyn, @cojenco PTAL.

[tritone]

Thanks for the proposal @daniellehanks (and sorry for the slow response). We don't support these via most of the other GCS client libraries and I am following up with the service team to determine if there's a good reason for that. Will update you soon when I have more information.

[tseaver]

@daniellehanks Can you clarify the usecase, here? Do you want to pass an etag (a list, for If-None-Match, technically) to Blob.reload? Or Blob.exists? Or do you want to pass it to e.g. Blob.download_to_file?

What would you expect to happen in the case that there was no different ETag? Blob.exists could return False, while we'd be more likely to raise an error for the 304 Not Modified response for Blob.reload or Blob.download_*.

[daniellehanks]

Can it just behave exactly like the meta/generation parameters? Those raise google.api_core.exceptions.NotModified and google.api_core.exceptions.PreconditionFailed, which are exactly what I would expect.

As for my specific use case, it would likely look something like:

from google.api_core import exceptions
# Boilerplate instantiation of client/bucket omitted.
# Constructing filename and etag omitted, but they come from my client's request to my backend.
...
blob = _bucket.blob(filename)
try:
    # We may use one of the other download methods.
    data = blob.download_as_bytes(if_etag_not_match=etag)
except exceptions.NotFound:
    # Return 404 response to my client.
except exceptions.NotModified:
    # Return 304 response to my client.
except exceptions.GoogleAPIError:
    # Return 500 response to my client.
    
# Return 200 with data to my client with blob.etag as etag. Though it looks like I have to get the etag as a separate call to `reload`? Trying to do this in a single call.

Without the etag functionality:

blob = _bucket.blob(filename)
try:
    blob.reload()
    if blob.etag == etag:
        # Return 304 response to my client.
    # Knowing the etag matches, we can use the generation from the same metadata.
    # We may use one of the other download methods.
    data = blob.download_as_bytes(if_generation_not_match=blob.generation)
except exceptions.NotFound:
    # Return 404 response to my client.
except exceptions.NotModified:
    # This shouldn't happen since we checked the etag earlier.
    # Return 304 response to my client.
except exceptions.GoogleAPIError:
    # Return 500 response to my client.
    
# Return 200 with data to my client with blob.etag as etag.

As I went through the exercise of writing this prototype, it looks like calling reload() is the only way to get the metadata of the current object (e.g. etag)? If that is the case, then both solutions above have to call reload() at some point, though the first one with the proposed etag feature would only have to call it if we didn't get a 304. Though the second one doesn't have to call download if the etag matches. Sounds like a wash if I'm understanding everything correctly and assuming fetching metadata and fetching file contents, even if 304 or 412, are all class B operations.

[tritone]

Thanks for the example, and apologies for the slow response.

As you note from your example, this is only going to save you network calls if you are able to store the ETag client-side and will have it on hand when you attempt to download the file. Will this be possible for your use case? Otherwise, I don't see how this will be useful at all -- obviously the ETag (and generation number) you get from blob.reload will still be current when you make your download call immediately after that.

Assuming this is the case, I discussed with the team and we are open to accepting a PR that adds an if_none_match kwarg to the blob.download_* methods only (we'd rather not add it more broadly for now). This would take the value and add it as a header to the outgoing request, and (as you say) raise a google.api_core.exceptions.NotModified if we get a 304 from the server. It'd also require adding a system test to validate that this works correctly for a matching and non-matching ETag. Does this sound reasonable to you?

Alternatively, you can also potentially store the generation number locally when the object is first created or accessed, and follow the same pattern with that using the existing if_generation_match.

[daniellehanks]

I'm not storing the etags in my backend. The client would be caching them and using them to make conditional requests to my API. My backend is basically just a middle man abstracting the storage layer (and the GCS-specific concept of generations in favor of etags).

Is my understanding correct that there is no way to get the etag from GCS via a single call to one of the download_* functions? Both solutions above make 1 network call in the match case (the first to download_* conditional on etag, the second to reload) and 2 in the non-match case (the first to get the etag which actually isn't even guaranteed to match what we just downloaded as there may have been a concurrent modification, the second to actually do the download).

If my above understanding is correct (that the GCS APIs, not just the client libraries, don't support getting the file content and metadata in a single call), then seems like implementing the etag functionality doesn't actually buy me anything in terms of number of network calls. I guess I had just assumed (incorrectly) that the metadata on the blob would be populated on a call to download_* (and, indeed, some fields like hash and crc32 are, but those are based on the file content I presume).

[daniellehanks]

One more question to clarify: Is a conditional call to download that results in a 304 or 412 still billed as a class B operation? If they are not billed, then adding the etag functionality could save us a lot of billed operations.

[tritone]

Ah, so, if I understand correctly:

• For the first time downloading the file you will download without any precondition. You'll take the ETag or generation from the response and cache it.
• For subsequent downloads you'll use the ETag or generation that you had from the initial download to set a precondition, and update the cached data for downloads that succeed.

Is that accurate?

I think I was also confused about the blob data getting updated from the download_* methods -- you definitely would need this for preconditions (generation or ETag) to provide any value here. I verified that the JSON API does return certain metadata including ETag and generation in the response headers for GET requests with alt=media. I work mostly in Go, and in that GCS library we expose this data to the user via ReaderObjectAttrs. @tseaver @andrewsg @cojenco any idea why we don't have a way of exposing this in Python? Would it be reasonable to add this for attributes which are available via the response headers?

As far as your billing question, I would have to do some research to follow up on this, but I would suspect that you are still charged for the class B operation (but not for the data egress that a successful request would entail obviously).

[daniellehanks]

Yes, your summary of what I am doing is correct.

I found on the billing page that 307, 4XX, and 5XX are generally not billed, but that doesn't include 304 so I would assume same as you that the operation is billed but no network egress.

I wasn't able to quickly curl GCS in between meetings today because auth is hard :). Glad you were able to verify the metadata is in the headers on alt=media requests. Can I amend that to my feature request? :D.

If etag (and whatever else is available in the headers that already has an existing property on the blob class that can be easily mapped) could be populated on the download call, and conditional etag support were added to the download methods, that would reduce a call in the case of a not-match as well as probably simplify the code a decent amount. If it's not too much work (like ~<500 LOC or a weekend), I'd love to contribute.