v3.2: How to use `style` with `multipart` (and headers/cookies?) [percent-encoding issues]

[handrews]

It seems that in 3.1, we expanded the use of the Encoding Object's style/explode/ allowReserved fields to multipart/form-data, when previously they only applied to application/x-www-form-urlencoded. This was done in PR #2066.

However, these fields are based on RFC6570 and apply URI percent-encoding to both the names and the values of form fields. While it's clear enough how to convert from the name=value&name=value to putting names in the Content-Disposition and values in the part bodies, it is rarely the case that percent-encoding is appropriate. But we do not call attention to that.

Currently (3.1.1) the only thing we say about this is in Appendix E.2, where we note that:

The form-data media type allows arbitrary text or binary data in its parts, so percent-encoding is not needed and is likely to cause interoperability problems unless the Content-Type of the part is defined to require it.

But we do not point out that style automatically applies percent-encoding.

What was the intention here? AFAICT, you would almost never want to use these fields unless you are doing something very specific that was guaranteed to not trigger any percent-encoding. It's not even entirely clear to me how binary parts would work.

I feel like we should give some implementation guidance here, but my personal advice would be "don't, this is a footgun you don't want to touch."

[handrews]

@mikekistler @darrelmiller @webron y'all were involved in that PR, can you clarify the intent?

[handrews]

An alternative would be to say "what we meant was: Handle style as for form-urlencoded, then percent-decode the results and restructure in form-data format based on each part's media type rules."

This would contradict Appendix E, but we could probably get away with it? But I have no idea how tools have implemented this, or even if they have. This whole area is under-implemented from what I can see, often due to confusion over points like this.

[mikekistler]

I think we do not want URI percent-encoding of either names or values in multipart/form-data. So anywhere this is implied or might be inferred, I would agree that we should clarify this.

[handrews]

@mikekistler Do we think that fits with the compatibility on the grounds that it was never intended to imply percent-encoding?

I admit that sits a little oddly with me as we also use style/explode with in: header and in: cookie... altough it occurs to me that until PRs #4593 and #4699 we did not. I made those changes more beause we had forbidden allowReserved with in: path even though it was clearly useful there.

Was allowReserved forbidden with in: header and in: cookie because the intention was not to have the percent-encode step for those in values? That would make those fields a lot more valuable for in: header in particular. (in: cookie is still fatally flawed for using the wrong delimiter, unless it was intended to also involve a delimiter substitution but really that is stretching my credulity a bit).

Of course, if percent-encoding was never intended to apply to in: header or in: cookie then the 3.0.4/3.1.1 Appendix D is bad advice. 🤦 And maybe we should walk that back and acknowledge an error? Ugh.

[mikekistler]

Yes, I think it was never intended that names or values of form fields should be percent-encoded. But I could be mistaken about this. I did a quick scan of RFC 7578 and the only mentions of percent-encoding are for filenames in MIME-type headers, so I feel relatively confident that field names and values are not percent-encoded.

[handrews]

Issues and PRs involved in this in the past:

• added cookie parameter #655 (added cookie to in list, but no discussion of percent-encoding and this pre-dates escape/allowReserved)
• URI Template support in parameters #804 (added allowReserved, among other things, but initially it was escape and maybe worked differently?)
• Updates to parameter and requestBody #884 (sets the style defaults as being specific to the in value)

From #804:

In its initial commit, it added escape:

escape | boolean | Determines whether the parameter value should have unallowed characters percent-encoded based the parameter location. The default value is true.

Note that this seems to completely disable percent-encoding if it is set to false, and includes "based on the parameter location" (which presumaby meant some sort of different rules for different in values, but what? in: query and in: path would use the component-specific percent-encoding rules, but was the intention here that escaping not apply to in: header and in: cookie?

By the time the PR merged, this had become allowReserved:

allowReserved | boolean | Determines whether the parameter value should allow reserved characters, as defined by RFC3986 :/?#[]@!$&'()*+,;= to be included without percent-encoding. This property only applies to parameters with an in value of query. The default value is false.

This now only partially disabled percent-encoding. AFAICT, forbidding it in in: path was about forbidding multi-path expansions (although by the time 3.0 went out, that was forbidden separately under Path Templating). But making it not apply to in: header and in: cookie feels like it could have been the replacement for "based on the parameter location"? @darrelmiller do you remember the intent here?

[handrews]

@mikekistler the issue is that all of the OAS wording around style/explode/allowReserved is in terms of RFC6570, which was designed for URLs and does percent-encode things. So your statement makes sense but there was nothing in the OAS to say so, which is what I'm grappling with. And also for headers and cookies- the forbidding of allowReserved was strange, but if they were never intended to have percent-encoding in the first place, it would not be.

[handrews]

And while we're at it, if the intent was to modify the URI-specific RFC6570 requirements for in: header and in: cookie, was in: cookie, style: form intended to use the Cookie header's ; delimiter (foo=abc;bar=def) instead of the query string's & delimiter (foo=abc&bar=def)? Otherwise it's not very useful.