zlib.error: Error -3 while decompressing data: invalid distance too far back

[EdwardYGLi]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: <macosx catalina 10.15.7>
• Poetry version: <1.1.8>

Issue

When trying to install some packages on my mac under a virtualenv. I keep getting :

  zlib.error: Error -3 while decompressing data: invalid distance too far back

I've tried to pip install the package manually to get by it but that only works sometimes. Now this has really become a blocker for me as I an unable to update my dependencies as I run into this error every time. I've tried brew reinstalling zlib and other zlib deps to no avail.

[clintonroy]

Please always try to give as much logs as you can. I suspect this is not an issue with zlib itself, but a corrupted download.

If you're having problems with pip as well, poetry will run into the same issue.

Try something like 'poetry cache clear --all pypi' to clear your downloads and try again.

[EdwardYGLi]

@clintonroy I tried the poetry cache clear --all pypi and is still getting the error, pip installing these packages works for me. Previously I had to pip install manually and then run poetry install to finish remaining packages, but recently this error still pops up even if I pip installed the package in the virtual env already. . Here is the full log

Installing dependencies from lock file

Package operations: 214 installs, 0 updates, 0 removals

  • Installing numpy (1.19.5): Failed

  EnvCommandError

  Command ['/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/bin/pip', 'install', '--no-deps', 'file:///Users/edwardli/Library/Caches/pypoetry/artifacts/56/eb/6a/742f1d115d9229ff778b439581e41b19dbae8121fdfff5c6290b579805/numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl'] errored with the following return code 2, and output: 
  Processing /Users/edwardli/Library/Caches/pypoetry/artifacts/56/eb/6a/742f1d115d9229ff778b439581e41b19dbae8121fdfff5c6290b579805/numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl
  Installing collected packages: numpy
  ERROR: Exception:
  Traceback (most recent call last):
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
      status = self.run(options, args)
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
      return func(self, options, args)
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 393, in run
      installed = install_given_reqs(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 77, in install_given_reqs
      requirement.install(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 787, in install
      install_wheel(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 810, in install_wheel
      _install_wheel(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 653, in _install_wheel
      file.save()
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 420, in save
      shutil.copyfileobj(f, dest)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 202, in copyfileobj
      buf = fsrc_read(length)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/zipfile.py", line 940, in read
      data = self._read1(n)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/zipfile.py", line 1016, in _read1
      data = self._decompressor.decompress(data, n)
  zlib.error: Error -3 while decompressing data: invalid distance too far back
  WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
  You should consider upgrading via the '/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/bin/python -m pip install --upgrade pip' command.
  

  at /usr/local/Cellar/poetry/1.1.6/libexec/lib/python3.9/site-packages/poetry/utils/env.py:1075 in _run
      1071│                 output = subprocess.check_output(
      1072│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1073│                 )
      1074│         except CalledProcessError as e:
    → 1075│             raise EnvCommandError(e, input=input_)
      1076│ 
      1077│         return decode(output)
      1078│ 
      1079│     def execute(self, bin, *args, **kwargs):

  • Installing protobuf (3.17.3): Failed

  EnvCommandError

  Command ['/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/bin/pip', 'install', '--no-deps', 'file:///Users/edwardli/Library/Caches/pypoetry/artifacts/da/16/58/39025383d35aa8857945fb78e0716001e527e7dee801db8994234468da/protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl'] errored with the following return code 2, and output: 
  Processing /Users/edwardli/Library/Caches/pypoetry/artifacts/da/16/58/39025383d35aa8857945fb78e0716001e527e7dee801db8994234468da/protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl
  Installing collected packages: protobuf
  ERROR: Exception:
  Traceback (most recent call last):
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
      status = self.run(options, args)
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
      return func(self, options, args)
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 393, in run
      installed = install_given_reqs(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 77, in install_given_reqs
      requirement.install(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 787, in install
      install_wheel(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 810, in install_wheel
      _install_wheel(
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 653, in _install_wheel
      file.save()
    File "/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 420, in save
      shutil.copyfileobj(f, dest)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 202, in copyfileobj
      buf = fsrc_read(length)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/zipfile.py", line 940, in read
      data = self._read1(n)
    File "/Applications/Xcode-12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/zipfile.py", line 1016, in _read1
      data = self._decompressor.decompress(data, n)
  zlib.error: Error -3 while decompressing data: invalid distance too far back
  WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
  You should consider upgrading via the '/Users/edwardli/Library/Caches/pypoetry/virtualenvs/projects-Qe-Yzjkt-py3.8/bin/python -m pip install --upgrade pip' command.
  

  at /usr/local/Cellar/poetry/1.1.6/libexec/lib/python3.9/site-packages/poetry/utils/env.py:1075 in _run
      1071│                 output = subprocess.check_output(
      1072│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1073│                 )
      1074│         except CalledProcessError as e:
    → 1075│             raise EnvCommandError(e, input=input_)
      1076│ 
      1077│         return decode(output)
      1078│ 
      1079│     def execute(self, bin, *args, **kwargs):

[clintonroy]

Can you manually unzip /Users/edwardli/Library/Caches/pypoetry/artifacts/56/eb/6a/742f1d115d9229ff778b439581e41b19dbae8121fdfff5c6290b579805/numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl ?

Does the hash and size match what's on pypi? https://pypi.org/project/numpy/1.19.5/#files

Does poetry cache clear delete it?

[EdwardYGLi]

https://pypi.org/project/numpy/1.19.5/#files

Hey @clintonroy, Thanks for getting back to me on this. I can manually unzip it, but there seems to be some bad sectors. Additionally, the hash is different from pypi, the file was corrupt. This is where it gets interesting. When I try calling poetry cache clear pypi --all this time, it gives me no cache found. I tried doing poetry cache clear . -all and it shows 0 entries. Then I tried using poetry cache list and it says no caches found. But the file is still there. Seems the only way to clear this cache is rm -rf ~/Library/Caches/pypoetry/artifacts. Must be a bug with poetry cache clear then. After manually deleting the cache, it seems to work now. It's also a good idea to allow poetry install without using the cache, similar to pip.

[dimbleby]

such cases should be fixed since #6186

[neersighted]

Duplicate #5142.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.