Limited permission directories in tmpdirs don't get removed & show warnings

[neiljp]

Context: I wanted to write a test to check behavior when permissions were not available, so turned to tmpdir with mkdir and chmod, assuming the temporary folder would be removed properly. However, admittedly at least in a rather extreme case of chmod(0), this leads to various warnings ending in:

...python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-07a6704d-839b-4882-8e94-9ae396c07a40/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'

I have tested this with pytest 6.1.0, though also 5.3.5.

The simplified test case, with error reproducible in a fresh venv with pytest 6.1.0 comes from something as simple as:

def test_1(tmpdir):
    tmpdir.mkdir("X").chmod(0)

Installed packages in the venv are:

Package            Version
------------------ -------
attrs              20.2.0 
importlib-metadata 2.0.0  
iniconfig          1.0.1  
packaging          20.4   
pip                18.1   
pkg-resources      0.0.0  
pluggy             0.13.1 
py                 1.9.0  
pyparsing          2.4.7  
pytest             6.1.0  
setuptools         40.8.0 
six                1.15.0 
toml               0.10.1 
zipp               3.2.0 

This is also confusing since it works fine 3 times, but once the cache of 3 old test runs is garbage collected, errors like the above are produced; in the simple venv case, something like as follows:

/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-neil/garbage-bd1a01c6-2aa6-488b-a1a9-8729b0a89b3c'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f/test_10
<class 'OSError'>: [Errno 39] Directory not empty: 'test_10'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
/tmp/test/lib/python3.7/site-packages/_pytest/pathlib.py:73: PytestWarning: (rm_rf) error removing /tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-neil/garbage-b3a5e41f-5d44-4f3d-a147-f5b9cb0f3b3f'
  "(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)

[The-Compiler]

Also see #5974 - FWIW I'm not sure if it should really be pytest's responsibility to re-adjust the permissions so that it can delete the files.

In my own tests I have a fixture like this:

@pytest.fixture
def unreadable_file(tmpdir):
    unreadable_file = tmpdir / 'unreadable'
    unreadable_file.ensure()
    unreadable_file.chmod(0)
    if os.access(str(unreadable_file), os.R_OK):
        # Docker container or similar
        pytest.skip("File was still readable")

    yield unreadable_file

    unreadable_file.chmod(0o755)

to handle this properly.

[neiljp]

I do agree that a workaround is feasible, but it just felt a little strange that tmpdir didn't handle this case, or that there wouldn't be a 'force' type option that would ensure this was handled - or conversely a 'keep' option with the current behavior.

Also, the error message indicates a failure from an rm_rf, which interpreted as rm -rf or similar does indeed clean up these folders just fine on the command-line.

[asottile]

fwiw, this is what pre-commit does for this

[ynikitenko]

I don't fully understand whether this issue was fixed. I have nested directories, and I change permissions for some hierarchies, then test the results of my code. For me it looks like that if my assertions fail, the results are garbage collected, which collects these bad folders. Probably I should make a dedicated try and catch block for that, but was this issue fixed, is it being cleaned for us?..
I'm using pytest 7.1.2 on Arch Linux.