Project with path dependency cannot be installed with `pip install .` (or via requirements file) due to "RequirementParseError: Invalid URL"

[huonw]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.4
• Poetry version: 1.1.13, 1.2.0b2.dev0 (f2c365f)
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/huonw/poetry-path-deps-issue

Issue

We're working within a monorepo where we have several projects that depend on each other via path = "..." relative dependencies. Installing such a project directly with pip install fails with pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a.

Why are we trying to do this? We want to have access to pip install --target to install packages to a specific directory, when packaging for an AWS Lambda .zip file. This means we're exporting to a requirements file and installing that.

This is potentially caused by only absolute paths being turned into a URL, in https://github.com/python-poetry/poetry-core/blob/c5f4cda45402c38bb87ab73d0e32de509ba68f41/src/poetry/core/packages/directory_dependency.py#L125

Potentially related issues:

• similar error messages: Poetry install with path dependencies and entry points is unable to load the package #3930, "poetry build" generates wheel with invalid Requires-Dist URL when pyproject.toml specifies dependency by path #3899, Local path in wheel metadata seems wrong #4709, Wheel installation fails when local path dependency introduced #4868
• similar set-up: requirements.txt not installable by pip when local package depends on another poetry-plugin-export#147
• Monorepo / Monobuild support? #936 for the monorepo style with a lot of path deps

Example

https://github.com/huonw/poetry-path-deps-issue contains a full reproducible example:

cd /tmp
git clone https://github.com/huonw/poetry-path-deps-issue.git
pip install poetry-path-deps-issue/b

The core layout is:

.
├── a
│   ├── a
│   │   └── __init__.py
│   └── pyproject.toml
├── b
│   ├── b
│   │   └── __init__.py
│   └── pyproject.toml
└── c
    ├── c
    │   └── __init__.py
    ├── poetry.lock
    └── pyproject.toml

Where b/pyproject.toml contains a dependency a = { path = "../a" } (c is left over from an older version of this issue, see below).

The pip install . fails with:

Processing /poetry-path-deps-issue/b
...
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a
...

Click for older details using poetry export

c/pyproject.toml contains a dependency b = { path = "../b" }.

Commands:

cd poetry-path-deps-issue/c
poetry export -o requirements.txt
pip install -r requirements.txt

After running those commands, the requirements.txt contains something like:

a @ file:///private/tmp/poetry-path-deps-issue/a; python_version >= "3.9" and python_version < "4.0"
b @ file:///private/tmp/poetry-path-deps-issue/b; python_version >= "3.9" and python_version < "4.0"

And the pip install output is:

Processing /private/tmp/poetry-path-deps/a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing /private/tmp/poetry-path-deps/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
...
  File ".../lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

Workaround

We're preprocessing all pyproject.toml files to remove the path dependencies, after exporting the requirements, but before installing them (via find -name pyproject.toml | xargs sed -i '/path =/d'). The export should contain all of the dependencies required, so we don't them to be re-read out of each pyproject.toml.

Full output

Details

Processing /poetry-path-deps-issue/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 117, in __init__
    raise InvalidRequirement(f"Invalid URL: {req.url}")
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 246, in iter_dependencies
    requires = self.dist.iter_dependencies() if with_requires else ()
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/metadata/pkg_resources.py", line 200, in iter_dependencies
    return self._dist.requires(extras)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

[Secrus]

Hi. As with release 1.2, export is gonna be moved to plugin poetry-plugin-export, could you please check if this issue is still present with poetry 1.2 and plugin? If so, please report it in the plugin repository (linked above).

[huonw]

Thanks for checking in, @Secrus. The problem still exists with poetry 1.2 (on the master branch) plus poetry-plugin-export, however, on reflection, talking about poetry export is actually a distraction: running just pip install . in b shows the same problem, both in 1.1 and 1.2.

I'll update the issue.

[fx-kirin]

this pip issue might be related. They said poetry itself should be followed pip's way 'file://' and should not allow relative path.

[koffie]

I have the feeling this is a duplicate of #3148 , there is a workaround mentioned there in a comment #3148 (comment)

[jcosmao]

Just got same issue, i was able to install correctly with:

# relative deps from pyproject.toml are converted to absolute path
poetry export -f requirements.txt -o requirements.txt --without-hashes
pip install -r requirements.txt
# ignore dependencies as it has already been installed by requirements
pip install . --no-dependencies

[dimbleby]

Looks like this is languishing for lack of interest and indeed I don't myself really understand why it's wanted.

But it looks pretty trivial to fix this so that it always uses an absolute path, so if anyone is motivated to put together a pull request...

(cf python-poetry/poetry-core#141)