pytest fails when a path longer than 260 characters is present on Windows

[Riezebos]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

When executing pytest on windows in a directory that contains files with a path longer than 260 characters, a FileNotFoundError occurs during test collection. The full output can be seen below.

Python: 3.9.6
pip list: (new conda environment with only python and pytest specified for installation)

Package        Version
-------------- -------------------
atomicwrites   1.4.0
attrs          21.2.0
certifi        2021.5.30
colorama       0.4.4
iniconfig      1.1.1
more-itertools 8.8.0
packaging      21.0
pip            21.2.3
pluggy         0.13.1
py             1.10.0
pyparsing      2.4.7
pytest         6.2.4
setuptools     49.6.0.post20210108
toml           0.10.2
wheel          0.37.0
wincertstore   0.2

A test file was created like this:

from pathlib import Path
path = Path("/".join(["a"*50]*6))
path = Path(rf"\\?\{path.resolve()}")
path.mkdir(parents=True)
(path / "test.txt").touch()

Then the following happens:

(pytest) C:\Users\sriezebo\temptest>pytest
===================================================================================== test session starts ======================================================================================
platform win32 -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\sriezebo\temptest
collected 0 items / 1 error

============================================================================================ ERRORS ============================================================================================
________________________________________________________________________________ ERROR collecting test session _________________________________________________________________________________
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\runner.py:311: in from_call
    result: Optional[TResult] = func()
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\runner.py:341: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\main.py:690: in collect
    for direntry in visit(str(argpath), self._recurse):
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:606: in visit
    yield from visit(entry.path, recurse)
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:606: in visit
    yield from visit(entry.path, recurse)
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:606: in visit
    yield from visit(entry.path, recurse)
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:606: in visit
    yield from visit(entry.path, recurse)
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:606: in visit
    yield from visit(entry.path, recurse)
..\Miniconda3\envs\pytest\lib\site-packages\_pytest\pathlib.py:591: in visit
    for entry in os.scandir(path):
E   FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\sriezebo\\temptest\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
=================================================================================== short test summary info ====================================================================================
ERROR  - FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\sriezebo\\temptest\\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\aaaaaaaaaaaaaaaaaaaaaa...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================= 1 error in 0.18s =======================================================================================

After downgrading pytest it works in the same folder:

(pytest) C:\Users\sriezebo\temptest>pytest
===================================================================================== test session starts ======================================================================================
platform win32 -- Python 3.9.6, pytest-5.4.3, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\sriezebo\temptest
collected 0 items

==================================================================================== no tests ran in 0.02s =====================================================================================

[bluetech]

I don't have a Windows machine myself to test it, but maybe adding this also to visit will do the trick (ref #6755).

I'm surprised it works with older pytest.

[Riezebos]

Thanks for the quick response!
I had a look at the code based on your suggestion. Running ensure_extended_length_path within visit might impact performance if large directory is scanned recursively. If the first call of visit already has been checked all recursive calls should work automatically.

To test this I added a line in this place in my local installation:

pytest/src/_pytest/main.py

Line 719 in 2439729

The line is:

argpath = ensure_extended_length_path(Path(argpath))

Now it works:

(pytest) C:\Users\sriezebo\temptest>pytest
===================================================================================== test session starts ======================================================================================
platform win32 -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\sriezebo\temptest
collected 0 items

==================================================================================== no tests ran in 0.05s =====================================================================================

I've never contributed to pytest before, would it help if I made a PR based on this?

[bluetech]

I've never contributed to pytest before, would it help if I made a PR based on this?

That'd be great! I'll need to take a look if that is the best place for it, but we can discuss that on the PR.

[Zac-HD]

Closing as resolved: the underlying crash (os.scandir raising FileNotFoundError and aborting collection) was fixed generally by #13086 (implementing #13083), which makes _pytest.pathlib.scandir() catch FileNotFoundError and return [] instead of propagating it — the same WinError 3 seen here for paths >260 chars no longer crashes collection.