- Ubuntu 18.04:v1.0.2 (docker container)
- Seen on multiple versions (issue seems to be in dependency)
- pyproject.toml not particularly relevant
Issue
We have a build script that runs the poetry install command concurrently to install into virtual environments for multiple different python versions. This fails sporadically in CI with an exception from the cachy library:
STDOUT: Updating dependencies
STDOUT: Resolving dependencies...
STDOUT:
STDOUT: ValueError
STDOUT:
STDOUT: invalid literal for int() with base 10: b''
STDOUT:
STDOUT: at /usr/local/lib/python3.6/dist-packages/cachy/stores/file_store.py:65 in _get_payload
STDOUT: 61│
STDOUT: 62│ with open(path, 'rb') as fh:
STDOUT: 63│ contents = fh.read()
STDOUT: 64│
STDOUT: → 65│ expire = int(contents[:10])
STDOUT: 66│
The multiple concurrent poetry install commands are using the same cache directory, and it looks like the code in the cachy library is not thread safe - https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py.
It appears that if one process is performing a cache write at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L102-L103 while another is doing a read at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L62-L63, the read could result in reading an empty file before the contents are written, and then L65 attempts to cast an empty set of bytes to an integer and throws.
Ideally the cache would be thread safe, but documentation saying that concurrent installs are not safe would be an alternative.
A possible workaround that we will attempt is to set the cache-dir in the config to different directories for different concurrent installs.
UPDATE:
The attempted workaround of setting a specific cache-dir in the config does not work, likely due to the issue described in #2445 meaning the repository cache is still shared. As a workaround we are now going to try setting both the poetry cache-dir and XDG_CACHE_HOME to different directories for different installs
-vvvoption).-vvvIssue
We have a build script that runs the
poetry installcommand concurrently to install into virtual environments for multiple different python versions. This fails sporadically in CI with an exception from the cachy library:The multiple concurrent
poetry installcommands are using the same cache directory, and it looks like the code in thecachylibrary is not thread safe - https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py.It appears that if one process is performing a cache write at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L102-L103 while another is doing a read at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L62-L63, the read could result in reading an empty file before the contents are written, and then L65 attempts to cast an empty set of bytes to an integer and throws.
Ideally the cache would be thread safe, but documentation saying that concurrent installs are not safe would be an alternative.
A possible workaround that we will attempt is to set the
cache-dirin the config to different directories for different concurrent installs.UPDATE:
The attempted workaround of setting a specific
cache-dirin the config does not work, likely due to the issue described in #2445 meaning the repository cache is still shared. As a workaround we are now going to try setting both the poetrycache-dirandXDG_CACHE_HOMEto different directories for different installs