Add transport-agnostic response decompression and Accept-Encoding negotiation

[OmarAlJarrah]

Summary

sdk-core has no Content-Encoding handling, and neither reference transport decompresses response bodies or negotiates Accept-Encoding in a uniform way. As a result, the same SDK Request against a server that returns Content-Encoding: gzip produces different response bytes depending on which transport is installed: the OkHttp transport transparently inflates, while the java.net.http transport hands back the raw compressed bytes (and a broken deserialize). This issue proposes a single, transport-agnostic content-coding capability in core — a streaming, zero-dependency decoder plus an opt-in negotiation/decode pipeline step — and converging both reference transports on it so a gzip response round-trips identically everywhere.

This is simultaneously a missing platform capability (consumers have no supported way to request and transparently decode compressed responses) and the fix for a real cross-transport non-determinism.

Problem

The SDK is a platform: generated service clients, downstream SDKs, and application code build on top of sdk-core and choose a transport as an installable unit. The product spec makes cross-transport equivalence a headline guarantee — "the correctness-sensitive decisions … are made once, in the core, so every transport behaves identically" (docs/product-spec.md §1 value proposition, restated in the §4 domain-model preamble). Today that guarantee does not hold for compressed responses.

Concretely, in the current source:

• Core has nothing. The only content-coding references in sdk-core are the header-name constants HttpHeaderName.ACCEPT_ENCODING (http/common/HttpHeaderName.kt:62) and HttpHeaderName.CONTENT_ENCODING (:98), and they are used only by the logging allowlist HttpInstrumentationOptions.DEFAULT_ALLOWED_HEADERS (companion object; the two constants are listed at http/pipeline/steps/HttpInstrumentationOptions.kt:142 and :147). "gzip" otherwise appears only in comments in http/pipeline/steps/BodyPreview.kt:19,164. There is no java.util.zip, Inflater, or GZIPInputStream usage anywhere in the main sources — no decode path exists.

• OkHttp (SDK-managed) silently inflates. OkHttpTransport.Builder.build() (sdk-transport-okhttp/.../OkHttpTransport.kt:496-513) disables native redirect following (:503-504, since followRedirects defaults to false) and retryOnConnectionFailure(false) (:511) so the SDK pipeline is the single authority — but it leaves OkHttp's transparent gzip on. Because the SDK sends no Accept-Encoding, OkHttp's bridge adds Accept-Encoding: gzip itself, then transparently inflates the response and strips Content-Encoding/Content-Length before the SDK's ResponseAdapter ever sees it.

• java.net.http does not. The JDK transport never sends Accept-Encoding (its RequestAdapter sets no such header), and ResponseAdapter.adapt(...) (sdk-transport-jdkhttp/.../internal/ResponseAdapter.kt:65,79-80) wraps the raw response InputStream via Io.provider.source(body) with no Content-Encoding inspection. If a server (or an intermediary) emits Content-Encoding: gzip — solicited or not — the caller gets compressed bytes and a Content-Encoding: gzip header still attached.

The consequences for platform consumers:

1. Non-determinism they cannot see or control. A generated client that works against the OkHttp transport can return garbage (or fail to deserialize) the moment it is run on the JDK transport, purely because a server compressed the response. The response headers also differ (OkHttp strips Content-Encoding; the JDK keeps it), so header-driven logic diverges too.
2. A capability gap with no clean workaround. A consumer cannot portably ask for compression: setting Accept-Encoding: gzip on the SDK request actually disables OkHttp's transparent inflation (OkHttp only auto-inflates when it added the header itself), so the caller then gets raw bytes on both transports and must hand-roll decompression against the SDK's BufferedSource/ResponseBody. Every downstream SDK would reinvent this.
3. It is a core-primitive gap, so it is platform-wide. No generated client or downstream SDK can fix it below itself; the fix has to live in the platform, and until it does, every consumer and every generated client inherits the divergence.

Proposed approach

Keep the decode capability in core (the correctness decision made once), keep core dependency-free, and make the transports defer to it.

1. A Content-Encoding-aware body decorator in sdk-core. A BufferedSource/ResponseBody wrapper that decodes gzip and deflate (zlib) using java.util.zip (GZIPInputStream / InflaterInputStream). This is JDK standard library, so it adds zero runtime dependencies and preserves SEAM-1. It is Java-8-safe and streaming — it wraps the underlying source lazily and never drains it up front, preserving the lazily-read, non-pre-buffered body contract (TRANSPORT-25, SEAM-11). It must:

   • handle a multi-valued Content-Encoding (e.g. deflate, gzip) by decoding codings in the correct reverse order;
   • pass through identity and any unknown/unsupported coding untouched rather than corrupting the body (a value the core cannot decode is surfaced as-is, so an opt-in codec adapter can layer on top);
   • after decoding, present the body with Content-Encoding removed and length reported as unknown (-1), so downstream stages and the caller see a coherent decoded body;
   • keep codings that require a third-party runtime (Brotli br, Zstandard zstd) out of core. Those belong in a separate opt-in adapter module that plugs the same decorator seam, mirroring how transports and codecs are separately installable units (SEAM-1, §2 minimal-footprint-core paragraph).

2. An opt-in pipeline step (sync HttpStep plus its async mirror) that sets Accept-Encoding on the outbound request and applies the decorator to the inbound response. It is off by default (no wire-behavior change unless a consumer opts in). It skips SSE (text/event-stream) and other streaming/unknown-length responses, and body-less responses (204/304), and is a no-op when the response carries no decodable Content-Encoding (so a BYO transport that already stripped the header is never double-decoded). The natural placement is around the SERDE region of the existing Stage ordering (PRE_SERDE/SERDE/POST_SERDE), since the body must be decoded before deserialization reads it.

3. Converge both reference transports so the body reaching core is exactly what the server sent.

   • OkHttp (SDK-managed only): stop relying on OkHttp's native transparent gzip, so core is the single decode authority — the same "SDK pipeline is the single authority" rationale already used to disable native redirects/retry (TRANSPORT-1, TRANSPORT-2; OkHttpTransport.kt:503-511). In practice OkHttp suppresses transparent gzip whenever an explicit Accept-Encoding is present, so an SDK-managed client sends an explicit value (the negotiation step's value when enabled, otherwise identity) and lets core decode.
   • java.net.http: the negotiation step supplies Accept-Encoding on the outbound path; the inbound decorator handles decoding.
   • BYO clients are untouched. The SDK never reconfigures a caller-supplied native client; if a BYO OkHttp inflates and strips Content-Encoding itself, the core step simply sees no encoding and does nothing.

4. Codify the contract in the spec. Add a TRANSPORT-* conformance requirement (§17, following the existing TRANSPORT-1..TRANSPORT-30) stating that an SDK-managed transport must not apply native content-coding transformations that diverge from the core decode path, and that a compressed response must decode to identical bytes across conforming transports — with a conformance note (stream a gzip body through each transport, assert byte-identical decoded output). This forces future ports to normalize the behavior rather than inherit their native client's defaults.

Scope and non-goals

In scope:

• gzip + deflate decode in core via java.util.zip; multi-coding and unknown-coding handling.
• An opt-in Accept-Encoding negotiation + inbound-decode step (sync + async).
• Converging the two reference transports so the server's raw bytes reach core on SDK-managed clients; BYO clients left native.
• A new TRANSPORT-* conformance requirement plus tests.

Non-goals:

• Request-body compression (outbound Content-Encoding on what the SDK sends). Separate concern; can follow later.
• Brotli / Zstandard or any coding needing a third-party runtime — deferred to an opt-in adapter module, never core.
• Changing default wire behavior: the negotiation step stays opt-in; the only default change is that SDK-managed OkHttp no longer performs its own transparent gzip (it delegates to core), which is behavior-preserving for decoded bytes.
• Decoding SSE / streaming / unknown-length bodies.

Acceptance criteria

• sdk-core gains a streaming Content-Encoding decoder over BufferedSource/ResponseBody for gzip and deflate, using only java.util.zip — no new runtime dependency, Java-8 bytecode, apiCheck/apiDump updated.
• The decoder handles a multi-valued Content-Encoding list in the correct order and passes an unknown/unsupported coding through untouched.
• A decoded response exposes the body with Content-Encoding removed and length -1, and reads lazily without pre-buffering (a multi-megabyte gzip body round-trips byte-exact and streams; TRANSPORT-25/SEAM-11).
• An opt-in step (sync + async) sets Accept-Encoding outbound and decodes inbound; disabled by default; skips SSE, body-less, and non-decodable responses.
• SDK-managed OkHttp no longer depends on native transparent gzip; a gzip response yields byte-identical decoded bodies and consistent Content-Encoding/length metadata through both reference transports (cross-transport parity test).
• BYO native clients are not reconfigured and are never double-decoded.
• A new TRANSPORT-* requirement in docs/product-spec.md §17 captures content-coding normalization and cross-transport parity, with a conformance note; the §17 conformance checklist is updated.
• Tests cover: gzip and deflate round-trip, multi-coding, unknown-coding pass-through, streaming byte-exactness, SSE left untouched, and the two-transport parity case.

References

• docs/product-spec.md:
  • §1 value proposition — "so every transport behaves identically" (restated in the §4 domain-model preamble). This capability is the parity fix.
  • SEAM-1 (dependency-free core; optional capabilities are separately installable units depending on core plus at most one third-party library) — justifies java.util.zip in core and Brotli/Zstd in an opt-in adapter.
  • SEAM-11 and TRANSPORT-25 — response body is a lazily-read, non-pre-buffered stream; the decoder must preserve this.
  • TRANSPORT-1 / TRANSPORT-2 — precedent for disabling native-client features on SDK-managed clients so the SDK pipeline is the single authority.
  • REDIR-5 — the spec already treats Content-Encoding as a request header (removed on a followed 303), confirming it is part of the modeled surface.
  • New TRANSPORT-* content-coding parity requirement to be added in §17 (after TRANSPORT-30).
• Code:
  • sdk-transport-okhttp/src/main/kotlin/org/dexpace/sdk/transport/okhttp/OkHttpTransport.kt — Builder.build() (:496-513), where native transparent gzip is currently left on; native redirect (:503-504) and retry (:511) are already disabled.
  • sdk-transport-jdkhttp/src/main/kotlin/org/dexpace/sdk/transport/jdkhttp/internal/ResponseAdapter.kt — adapt(...) raw-stream wrap with no Content-Encoding handling (:65, :79-80); the sibling RequestAdapter.kt sends no Accept-Encoding.
  • sdk-core/src/main/kotlin/org/dexpace/sdk/core/http/common/HttpHeaderName.kt — ACCEPT_ENCODING (:62), CONTENT_ENCODING (:98).
  • sdk-core/src/main/kotlin/org/dexpace/sdk/core/http/pipeline/steps/HttpInstrumentationOptions.kt — DEFAULT_ALLOWED_HEADERS (companion object; the two constants at :142 and :147), the only current consumers of those constants.
  • sdk-core/src/main/kotlin/org/dexpace/sdk/core/http/pipeline/Stage.kt — Stage ordering (PRE_SERDE=900 / SERDE=1000, reserved and currently unused / POST_SERDE=1100) for step placement.