Install on Heroku fails with pip error

[cfra]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name:

• Poetry version: 1.1.7

• pyproject.toml

Issue

We use Poetry on Heroku to install our packages. For this, we list poetry==1.1.7 as sole dependency in our requirements.txt, and have a post-compile hook that looks like this:

python -m poetry config virtualenvs.create false
python -m poetry install --no-dev 2>&1

Additionally, we invalidate the cache in the pre-compile hook to ensure we always start with a clean environment and don't have any old packages lying around:

echo "taint cache" > "$CACHE_DIR/.heroku/requirements.txt"

All in all this seems to work fine, however, we spuriously encounter errors like these:

-----> Running post-compile hook

-----> Installing dependencies with Poetry...

      Skipping virtualenv creation, as specified in config file.

      Installing dependencies from lock file

      

      Package operations: 91 installs, 7 updates, 4 removals

      

        • Removing packaging (20.9)

        • Removing pexpect (4.8.0)

        • Removing ptyprocess (0.7.0)

        • Removing pyparsing (2.4.7)

        • Updating six (1.16.0 -> 1.15.0)

        • Updating cffi (1.14.6 -> 1.14.5)

        • Installing jmespath (0.10.0)

        • Installing python-dateutil (2.8.1)

        • Updating urllib3 (1.26.6 -> 1.26.5)

        • Installing botocore (1.20.34)

        • Updating certifi (2021.5.30 -> 2020.12.5)

        • Installing chardet (3.0.4)

        • Updating cryptography (3.4.7 -> 3.4.6)

        • Installing defusedxml (0.7.1)

        • Installing et-xmlfile (1.0.1)

        • Updating idna (3.2 -> 2.10)

        • Installing multidict (5.1.0)

      

        EnvCommandError

      

        Command ['/app/.heroku/python/bin/python', '-m', 'pip', 'install', '--no-deps', '/app/.cache/pypoetry/artifacts/36/92/90/7beba42c3a6efbfdc30dc864c33286fa23817b3942b21ccaf319f18ed3/multidict-5.1.0-cp38-cp38-manylinux2014_x86_64.whl'] errored with the following return code 1, and output: 

        Processing /app/.cache/pypoetry/artifacts/36/92/90/7beba42c3a6efbfdc30dc864c33286fa23817b3942b21ccaf319f18ed3/multidict-5.1.0-cp38-cp38-manylinux2014_x86_64.whl

        Installing collected packages: multidict

        ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build_8c19dde7/.heroku/python/lib/python3.8/site-packages/~dna-3.2.dist-info'

        

        WARNING: You are using pip version 20.2.4; however, version 21.1.3 is available.

        You should consider upgrading via the '/app/.heroku/python/bin/python -m pip install --upgrade pip' command.

This error is not specific to a single package, we saw it happen for different packages.

The error always is the No such file or directory for the .dist-info.

The file/directory that is not found always has the first character substituted by ~ which might be a name constructed by pip's AdjacentTempDirectory.

As the error is only occurring spuriously and as I have no interactive access to the build environment, I am not really sure how I should best debug this further.

[cfra]

This seems somewhat similar to #2658

[finswimmer]

Hello @cfra,

please try to delete this ~dna-3.2.dist-info and run again. I still have no idea where this ~ prefixed folder comes from, but saw it from time to time by other people 🤔

fin swimmer

[cfra]

@finswimmer this is a build on Heroku, not an interactive environment.

I cannot modify the environment and rerun from a specific step, I can just rerun the complete build.

Also, I cannot delete the ~dna-3.2.dist-info directory, as the directory is not there, as can be seen from the "No such file or directory error" in the log.

As for where the ~ is coming from:

Pip uses a class AdjacentTempDirectory to shuffle things around when it installs packages.

That class creates a temporary directory adjacent to another path.

For this, it replaces the first character of the path with ~.

Specifically, looking at my error log above, AdjacentTempDirectory will create a directory ~dna-3.2.dist-info when it is asked to create a directory adjacent to idna-3.2.dist-info.

Also, note that immediately prior to the error that happens when attempting to install multidict, we are updating idna.

So my assumption would be that the update and the install are being run in parallel which is apparently unsafe.

I think the pip updating idna creates a temporary directory ~dna-3.2.dist-info that is collected by the pip installing multidict, however the pip updating idna removes/renames that directory so that it just "vanishes" from the perspective of the pip installing multidict, leading to the "No such file or directory error" that can be seen above.

[uSpike]

I mentioned it in #4568 but I am seeing this issue as well. @cfra analysis looks sound. It seems that executing multiple pip operations in parallel where packages have shared dependencies is not safe for parallel execution.

[allen-munsch]

Just stopping by to corroborate a similar error, however, not using poetry, and rather just using a standard pip install, and private pypi, and custom wheeled directory install.

-----> Building on the Heroku-20 stack
-----> Using buildpacks:
       1. https://github.com/heroku/heroku-buildpack-cli
       2. https://github.com/heroku/heroku-buildpack-python
       -----> heroku-cli app detected
=== Fetching and vendoring Heroku CLI into slug
=== Installing profile.d script
 ›   Warning: Our terms of service have changed: 
 ›   https://dashboard.heroku.com/terms-of-service
 heroku/7.59.4 linux-x64 node-v12.21.0
=== Heroku CLI installation done
-----> Python app detected
-----> Using Python version specified in runtime.txt
 !     Python has released a security update! Please consider upgrading to python-3.9.12
       Learn More: https://devcenter.heroku.com/articles/python-runtimes
       -----> No change in requirements detected, installing from cache
-----> Using cached install of python-3.9.10
-----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
-----> Installing SQLite3
-----> Installing requirements with pip
       Looking in indexes: https://<authtoken>@pypi.fury.io/somespam/
       Processing /tmp/pip-install-xxx/dependencies_ba123xxx/wheels/httplib2-0.20.2-py3-none-any.whl
       ERROR: Could not install packages due to an OSError: [Errno 2] 
No such file or directory: '/tmp/pip-install-xxx/dependencies_ba123xxx/wheels/httplib2-0.20.2-py3-none-any.whl'
 !     Push rejected, failed to compile Python app.
 !     Push failed

a hacky setup.py

#!/usr/bin/env python
import os
from setuptools import setup, find_packages
build_dir = '/home/jm/pycharm_projects/python-wheels'
with open('requirements.txt') as f:
    requirements = f.read().replace(build_dir, os.getcwd()).split('\n')
setup(name='dependencies',
      version='1.1',
      install_requires=requirements,
)

requirements.txt

--index-url https://<auth_token>:@pypi.fury.io/somespam/
dependencies==1.1

[cfra]

Just stopping by to corroborate a similar error, however, not using poetry, and rather just using a standard pip install, and private pypi, and custom wheeled directory install.

Thank you for providing this input. I believe your error to be unrelated to this on though.

[neersighted]

Fixed by #6186

[cfra]

@neersighted I don't think this is addressed by #6186.

[neersighted]

Ah, shoot, clicked on the wrong tab. Indeed this is not fixed by #6186, however, I don't think it should be happening with the new installer as we now invoke pip with a single dep and --no-deps -- if someone can reproduce on 1.2.x, that would be very helpful.