Poetry doesn't find a compatible version of python, even though one is available

[tungol]

• Poetry version: Introduced in poetry 1.4.0
• Python version: Python 3.7, 3.10, 3.11 are all involved
• OS version and name: MacOS and Debian
• pyproject.toml:

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Running with -vvv did not produce additional output.

Issue

In my project, I have the python dependency python = "~=3.10". I happen to have python 3.11 available on my development laptop, so I use that there. As part of our build pipeline, the pyproject.toml and poetry.lock files are copied into a Debian docker image which has a default python of 3.7, and has 3.10 but not 3.11 available. Prior to Poetry 1.4.0 this worked fine - Poetry would note that 3.7 wasn't supported by the project and then discover and use 3.10 automatically:

#24 0.801 The currently activated Python version 3.7.4 is not supported by the project (~=3.10).
#24 0.801 Trying to find and use a compatible version. 
#24 0.833 Using python3.10 (3.10.8)

On Poetry 1.4.0 and higher, that doesn't happen. Instead, I get an error because 3.11 isn't available:

#23 0.844 The currently activated Python version 3.7.4 is not supported by the project (~=3.10).
#23 0.844 Trying to find and use a compatible version. 
#23 0.885 
#23 0.885 [Errno 2] No such file or directory: 'python3.11': 'python3.11'

I believe this behavior was introduced by #7221 , although I haven't read through the associated MR in detail yet.

[dimbleby]

poetry isn't simply inventing python3.11 out of thin air, there must be something in your docker image that makes it think that's a good idea.

perhaps you have copied an unwanted envs.toml into the container?

Anyway, if you can't figure it out yourself and are hoping that someone else will help - please provide a way to reproduce the issue.

[tungol]

Hm, I made the assumption that 3.11 was recorded in the lock file somewhere, but I just made a minimal reproduction and that's definitely not the case so I'm really not sure where it's coming from. Here's what I have to reproduce:

pyproject.toml

[tool.poetry]
name = "test_case"
version = "1.0"
description = ""
authors = []
	
[tool.poetry.dependencies]
python = "~=3.10"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Dockerfile

FROM ubuntu:focal

RUN apt-get update
RUN apt-get install -y --no-install-recommends software-properties-common

RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install -y --no-install-recommends python3.9 python3.9-venv python3.10 python3.10-venv

RUN python3.9 -m ensurepip

ARG POETRY_VERSION=1.4.2

ENV PATH=/root/.local/bin:$PATH
RUN --mount=type=cache,target=/root/.cache/pip pip3.9 install poetry==$POETRY_VERSION
COPY pyproject.toml ./
RUN poetry -vvv run pip list

'pip list' there isn't special. I was originally getting it with a different, more useful pip command there.

Running docker build . gives this:

 => ERROR [stage-0 10/10] RUN poetry -vvv run pip list                                                                                                                                                      1.5s
------                                                                                                                                                                                                           
 > [stage-0 10/10] RUN poetry -vvv run pip list:                                                                                                                                                                 
#14 0.431 /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!                                             
#14 0.431   warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "                                                                                                                              
#14 0.642 The currently activated Python version 3.9.16 is not supported by the project (~=3.10).                                                                                                                
#14 0.642 Trying to find and use a compatible version. 
#14 0.642 Trying python3
#14 0.657 Trying python3.11
#14 0.763 
#14 0.763   Stack trace:
#14 0.763 
#14 0.763   12  /usr/local/lib/python3.9/dist-packages/cleo/application.py:327 in run
#14 0.798        325│ 
#14 0.798        326│             try:
#14 0.799      → 327│                 exit_code = self._run(io)
#14 0.799        328│             except BrokenPipeError:
#14 0.799        329│                 # If we are piped to another process, it may close early and send a
#14 0.799 
#14 0.800   11  /usr/local/lib/python3.9/dist-packages/poetry/console/application.py:190 in _run
#14 0.802        188│         self._load_plugins(io)
#14 0.802        189│ 
#14 0.802      → 190│         exit_code: int = super()._run(io)
#14 0.802        191│         return exit_code
#14 0.802        192│ 
#14 0.802 
#14 0.803   10  /usr/local/lib/python3.9/dist-packages/cleo/application.py:431 in _run
#14 0.837        429│             io.input.interactive(interactive)
#14 0.837        430│ 
#14 0.837      → 431│         exit_code = self._run_command(command, io)
#14 0.837        432│         self._running_command = None
#14 0.837        433│ 
#14 0.837 
#14 0.837    9  /usr/local/lib/python3.9/dist-packages/cleo/application.py:473 in _run_command
#14 0.869        471│ 
#14 0.869        472│         if error is not None:
#14 0.869      → 473│             raise error
#14 0.869        474│ 
#14 0.869        475│         return terminate_event.exit_code
#14 0.869 
#14 0.869    8  /usr/local/lib/python3.9/dist-packages/cleo/application.py:454 in _run_command
#14 0.902        452│ 
#14 0.902        453│         try:
#14 0.902      → 454│             self._event_dispatcher.dispatch(command_event, COMMAND)
#14 0.902        455│ 
#14 0.902        456│             if command_event.command_should_run():
#14 0.902 
#14 0.902    7  /usr/local/lib/python3.9/dist-packages/cleo/events/event_dispatcher.py:26 in dispatch
#14 0.908         24│ 
#14 0.908         25│         if listeners:
#14 0.908      →  26│             self._do_dispatch(listeners, event_name, event)
#14 0.908         27│ 
#14 0.908         28│         return event
#14 0.908 
#14 0.908    6  /usr/local/lib/python3.9/dist-packages/cleo/events/event_dispatcher.py:89 in _do_dispatch
#14 0.914         87│                 break
#14 0.914         88│ 
#14 0.914      →  89│             listener(event, event_name, self)
#14 0.914         90│ 
#14 0.914         91│     def _sort_listeners(self, event_name: str) -> None:
#14 0.914 
#14 0.915    5  /usr/local/lib/python3.9/dist-packages/poetry/console/application.py:299 in configure_env
#14 0.934        297│ 
#14 0.934        298│         env_manager = EnvManager(poetry, io=io)
#14 0.934      → 299│         env = env_manager.create_venv()
#14 0.934        300│ 
#14 0.935        301│         if env.is_venv() and io.is_verbose():
#14 0.935 
#14 0.935    4  /usr/local/lib/python3.9/dist-packages/poetry/utils/env.py:977 in create_venv
#14 1.033         975│                 try:
#14 1.033         976│                     python_patch = decode(
#14 1.033      →  977│                         subprocess.check_output(
#14 1.033         978│                             [python, "-c", GET_PYTHON_VERSION_ONELINER],
#14 1.033         979│                             stderr=subprocess.STDOUT,
#14 1.033 
#14 1.034    3  /usr/lib/python3.9/subprocess.py:424 in check_output
#14 1.125         422│         kwargs['input'] = empty
#14 1.125         423│ 
#14 1.125      →  424│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#14 1.125         425│                **kwargs).stdout
#14 1.125         426│ 
#14 1.125 
#14 1.125    2  /usr/lib/python3.9/subprocess.py:505 in run
#14 1.215         503│         kwargs['stderr'] = PIPE
#14 1.215         504│ 
#14 1.215      →  505│     with Popen(*popenargs, **kwargs) as process:
#14 1.215         506│         try:
#14 1.215         507│             stdout, stderr = process.communicate(input, timeout=timeout)
#14 1.216 
#14 1.216    1  /usr/lib/python3.9/subprocess.py:951 in __init__
#14 1.311         949│                             encoding=encoding, errors=errors)
#14 1.312         950│ 
#14 1.312      →  951│             self._execute_child(args, executable, preexec_fn, close_fds,
#14 1.312         952│                                 pass_fds, cwd, env,
#14 1.312         953│                                 startupinfo, creationflags, shell,
#14 1.312 
#14 1.312   FileNotFoundError
#14 1.312 
#14 1.313   [Errno 2] No such file or directory: 'python3.11'
#14 1.313 
#14 1.313   at /usr/lib/python3.9/subprocess.py:1821 in _execute_child
#14 1.406       1817│                     else:
#14 1.406       1818│                         err_filename = orig_executable
#14 1.406       1819│                     if errno_num != 0:
#14 1.406       1820│                         err_msg = os.strerror(errno_num)
#14 1.407     → 1821│                     raise child_exception_type(errno_num, err_msg, err_filename)
#14 1.407       1822│                 raise child_exception_type(err_msg)
#14 1.407       1823│ 
#14 1.407       1824│ 
#14 1.407       1825│         def _handle_exitstatus(self, sts,
------
executor failed running [/bin/sh -c poetry -vvv run pip list]: exit code: 1

while running docker build --build-arg POETRY_VERSION=1.3.2 . completes successfully.

[tungol]

The issue is avoided by adding the line RUN poetry env use python3.10, but it seems wrong that that's needed.

[radoering]

So the issue is:

>>> subprocess.check_output(["python3.11", "-c", "print('test')"])
FileNotFoundError: ...

>>> subprocess.check_output("python3.11 -c print('test')", shell=True)
subprocess.CalledProcessError: ...

And we are catching CalledProcessError but not FileNotFoundError.

[dimbleby]

this is fixed in #7771 (by first doing shutil.which(), and only proceeding on finding something)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.