Add span-based Deflate, ZLib and GZip encoder/decoder APIs by iremyux · Pull Request #123145 · dotnet/runtime

iremyux · 2026-01-13T18:46:21Z

This PR introduces new span-based, streamless compression and decompression APIs for Deflate, ZLib, and GZip formats, matching the existing BrotliEncoder/BrotliDecoder pattern.

New APIs

DeflateEncoder / DeflateDecoder
ZLibEncoder / ZLibDecoder
GZipEncoder / GZipDecoder

These classes provide:

Instance-based API for streaming/chunked compression with Compress(), Decompress(), and Flush()
Static one-shot API via TryCompress() and TryDecompress() for simple scenarios
GetMaxCompressedLength() to calculate buffer sizes

Closes #62113
Closes #39327
Closes #44793

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

src/libraries/System.IO.Compression/ref/System.IO.Compression.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibDecoder.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

…mpressedLength

AraHaan · 2026-01-22T23:44:38Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+                CompressionLevel.Fastest => ZLibNative.CompressionLevel.BestSpeed,
+                CompressionLevel.NoCompression => ZLibNative.CompressionLevel.NoCompression,
+                CompressionLevel.SmallestSize => ZLibNative.CompressionLevel.BestCompression,
+                _ => throw new ArgumentOutOfRangeException(nameof(compressionLevel)),


This would fail on valid native compression levels not covered by the CompressionLevel enum. Instead I think it should check if the value is is < -1 or > 9 to throw out of range instead.

Also to add on to the above, now those who want compression levels that just happen to == a value in the CompressionLevel enum will now not be able to use those compression levels either. Perhaps a solution to this is to expose a version of the ctor with CompressionLevel and a version with int that gets casted to ZLibNative.CompressionLevel after a range check.

src/libraries/System.IO.Compression/tests/DeflateZLibGZipEncoderDecoderTests.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.

src/native/libs/System.IO.Compression.Native/pal_zlib.c

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.

src/native/libs/System.IO.Compression.Native/pal_zlib.c

Copilot · 2026-03-22T18:14:57Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+        public static long GetMaxCompressedLength(long inputLength)
+        {
+            ArgumentOutOfRangeException.ThrowIfNegative(inputLength);
+            ArgumentOutOfRangeException.ThrowIfGreaterThan(inputLength, uint.MaxValue);
+
+            // compressBound() returns the upper bound for zlib-wrapped deflate output.
+            // For raw deflate (no header/trailer) this slightly overestimates, which is safe.
+            return (long)Interop.ZLib.compressBound((uint)inputLength);
+        }


GetMaxCompressedLength relies on Interop.ZLib.compressBound which currently returns a 32-bit value. For sufficiently large inputLength, the true bound can exceed 32 bits, leading to truncation and an incorrect (too small) maximum length. Either tighten the allowed inputLength range to what the underlying bound can represent safely, or switch the native + interop compressBound plumbing to a wider return type and add managed overflow checks (similar to ZstandardEncoder.GetMaxCompressedLength).

It feels weird to accept long as input but then throw if it is outside of 32-bit uint range. Can we make larger numbers work?

src/libraries/System.IO.Compression/src/System/IO/Compression/ZLibCompressionOptions.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateDecoder.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

…ZLibCompressionOptions.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/libraries/System.IO.Compression/tests/Zstandard/ZstandardEncoderDecoderTests.cs:25

ZstandardUtils.WindowLog_Max is platform-dependent (31 on 64-bit, 30 on 32-bit). Hardcoding MaxWindowLog=31 and InvalidWindowLogTooHigh=32 will likely fail on 32-bit test runs. Consider using Environment.Is64BitProcess (or referencing ZstandardUtils.WindowLog_Max) to set MaxWindowLog and the corresponding invalid-too-high value.

        protected override int MinWindowLog => 10;
        protected override int MaxWindowLog => 31;
        protected override int InvalidQualityTooLow => -(1 << 17) - 1;
        protected override int InvalidQualityTooHigh => 23;
        protected override int InvalidWindowLogTooLow => 9;
        protected override int InvalidWindowLogTooHigh => 32;

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZLibCompressionOptions.cs

src/libraries/System.IO.Compression/tests/Zstandard/ZstandardEncoderDecoderTests.cs

rzikm · 2026-03-26T15:05:00Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+        private static int ResolveWindowBits(int windowLog, CompressionFormat format)
+        {
+            // zlib-ng rejects windowBits 8 for raw deflate and gzip; classic zlib silently upgrades to 9.
+            if (format != CompressionFormat.ZLib)
+            {
+                windowLog = Math.Max(windowLog, 9);
+            }
+
+            return format switch
+            {
+                CompressionFormat.Deflate => -windowLog,
+                CompressionFormat.ZLib => windowLog,
+                CompressionFormat.GZip => windowLog + 16,
+                _ => throw new ArgumentOutOfRangeException(nameof(format))
+            };
+        }
+


Can we move this to some shared class (ZLibNative?) and reuse this in DeflateStream etc?

rzikm · 2026-03-26T15:08:18Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+        public static long GetMaxCompressedLength(long inputLength)
+        {
+            ArgumentOutOfRangeException.ThrowIfNegative(inputLength);
+            ArgumentOutOfRangeException.ThrowIfGreaterThan(inputLength, uint.MaxValue);
+
+            // compressBound() returns the upper bound for zlib-wrapped deflate output.
+            // For raw deflate (no header/trailer) this slightly overestimates, which is safe.
+            return (long)Interop.ZLib.compressBound((uint)inputLength);
+        }


It feels weird to accept long as input but then throw if it is outside of 32-bit uint range. Can we make larger numbers work?

rzikm · 2026-03-26T15:32:19Z

src/libraries/System.IO.Compression/tests/CompressionStreamUnitTests.Deflate.cs

Can we revert this whitespace change?

rzikm · 2026-03-26T15:32:38Z

src/libraries/System.IO.Compression/tests/CompressionStreamUnitTests.Gzip.cs

Can we revert this whitespace change?

rzikm · 2026-03-26T15:32:43Z

src/libraries/System.IO.Compression/tests/CompressionStreamUnitTests.ZLib.cs

                }
            }, testScenario.ToString()).Dispose();
        }
+


Can we revert this whitespace change?

rzikm · 2026-03-26T15:34:30Z

src/libraries/Common/tests/System/IO/Compression/ZLibEncoderDecoderTestBase.cs

+
+namespace System.IO.Compression
+{
+    public abstract class ZLibEncoderDecoderTestBase : EncoderDecoderTestBase


This file can be moved to System.IO.Compression.Tests, as it is only used by its tests.

iremyux and others added 3 commits January 12, 2026 20:46

Add Zlib Encoder and Decoder classes

e402958

Add tests

95ac930

Merge branch 'dotnet:main' into 62113-zlib-encoder-decoder

88acc57

iremyux self-assigned this Jan 13, 2026

iremyux added the area-System.IO.Compression label Jan 13, 2026

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/ref/System.IO.Compression.cs Outdated Show resolved Hide resolved

This was referenced Jan 13, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

rzikm self-requested a review January 14, 2026 11:06

iremyux added 2 commits January 15, 2026 11:28

Use CompressionLevel instead of int

6f1ab80

Use CompressionLevel instead of int

8926759

rzikm reviewed Jan 16, 2026

View reviewed changes

iremyux added 2 commits January 19, 2026 10:51

Introduce seperate encoders and decoders

bb2d89c

Update the tests accordingly

a21a800

iremyux changed the title ~~[WIP] Add span-based ZlibEncoder and ZlibDecoder APIs~~ [WIP] Add span-based Deflate, ZLib and GZip encoder/decoder APIs Jan 19, 2026

iremyux added 4 commits January 19, 2026 11:35

Remove ZlibEncoder and decoder with lower L

4b87dcd

Add ZLibEncoder and Decoder with capital L

aabc3a6

Change GetMaxCompressedLengt's inputLength to long

cfb3d7e

"File is required to end with a single newline character"

f031eed

This was referenced Jan 19, 2026

[android][arm64] System.Net.Sockets.Tests.SendTo_SyncForceNonBlocking.Datagram_UDP_ShouldImplicitlyBindLocalEndpoint fails with NetworkUnreachable #120526

Open

browser-wasm linux LibraryTests build failure - NETSDK1022: Duplicate 'Content' items #123237

Closed

Add Interop layer for zlib's compressBound and use it inside GetMaxCo…

1d18eec

…mpressedLength

build-analysis bot mentioned this pull request Jan 20, 2026

Assertion failures in System.Tests.ConvertTests.TryFromBase64Chars on Mono MiniJIT and Interpreter across multiple Linux and macOS platforms #123389

Closed

AraHaan reviewed Jan 22, 2026

View reviewed changes

rzikm reviewed Jan 23, 2026

View reviewed changes

src/libraries/System.IO.Compression/tests/DeflateZLibGZipEncoderDecoderTests.cs Outdated Show resolved Hide resolved

rzikm reviewed Jan 23, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs Outdated Show resolved Hide resolved

iremyux added 7 commits March 21, 2026 20:04

Address review comments related to tests

4aaaace

Removed GetMaxCompressedLength_FitsActualOutput test

4f3946a

Removed regions

cd7cef9

Remove Encoder_WithOptions test

e06c1d0

Fix windowlog=8 failure for gzip and deflate

e66a00b

Address review feedback on tests

89510f8

Fix tests

8c07952

Copilot AI review requested due to automatic review settings March 21, 2026 20:57

Copilot started reviewing on behalf of iremyux March 21, 2026 20:58 View session

Copilot AI reviewed Mar 21, 2026

View reviewed changes

src/native/libs/System.IO.Compression.Native/pal_zlib.c Show resolved Hide resolved

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs Show resolved Hide resolved

iremyux added 2 commits March 21, 2026 22:16

Use NoCompressionMemLevel for NoCompression level

7fd0591

Rename ZlibEncoderDecoderTests.cs to ZLibEncoderDecoderTests.cs

b7b29ac

iremyux requested review from alinpahontu2912 and rzikm March 21, 2026 21:18

Remove extra tests

a75b24f

Copilot AI review requested due to automatic review settings March 22, 2026 18:07

Copilot started reviewing on behalf of iremyux March 22, 2026 18:09 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

This was referenced Mar 22, 2026

[browser] Wasm.Build.Tests timeout - Timed out after 10s waiting for 'WASM EXIT' #116697

Open

Android arm32 device not found (armeabi-v7a architecture unavailable) #125440

Open

alinpahontu2912 reviewed Mar 24, 2026

View reviewed changes

iremyux added 3 commits March 24, 2026 21:11

Add ResolveWindowBits helper method

b795f31

Add CompressionNative_CompressBound to callhelpers-pinvoke.cpp

f4b24c2

Simplify errorCode switch

94c95dc

Copilot AI review requested due to automatic review settings March 25, 2026 12:05

Update src/libraries/System.IO.Compression/src/System/IO/Compression/…

9b39f4d

…ZLibCompressionOptions.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of iremyux March 25, 2026 12:06 View session

iremyux requested a review from alinpahontu2912 March 25, 2026 12:06

Copilot AI reviewed Mar 25, 2026

View reviewed changes

rzikm reviewed Mar 26, 2026

View reviewed changes

Conversation

iremyux commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AraHaan Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iremyux commented Jan 13, 2026 •

edited

Loading

AraHaan Feb 4, 2026 •

edited

Loading