gh-154137: Check for Windows handle leaks in regrtest

[vstinner]

Add handle_count() to test.support.os_helper.

• Issue: test.regrtest: detect leak of Windows handles #154137

[vstinner]

I found a leak when running tests on Windows using python -m test -R 3:3 -j0 -r:

test.test_concurrent_futures.test_process_pool leaked [24, 24, 24] handles, sum=72

I can reproduce at least one of these leaks by running test_force_shutdown_workers():

C:> python -m test test_concurrent_futures.test_process_pool -R 3:3 -m test_force_shutdown_workers   
...
test_concurrent_futures.test_process_pool leaked [4, 4, 4] handles, sum=12
...

[vstinner]

test_socket leaks 1 handle:

vstinner@WIN C:\victor\python\main>python -m test -R 3:3 test_socket     
...
test_socket leaked [1, 1, 1] handles, sum=3
...

[vstinner]

TODO: Update test_getfinalpathname_handles() of Lib/test/test_os/test_windows.py to reuse support.handle_count().

[vstinner]

test_socket leaks 1 handle

I cannot identify the leaking test by running python -m test.bisect_cmd, so I'm not sure that it's a real leak.

[vstinner]

!buildbot AMD64 Windows11 Refleaks PR

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 37c29f3 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F154140%2Fmerge

The command will test the builders whose names match following regular expression: AMD64 Windows11 Refleaks PR

The builders matched are:

• AMD64 Windows11 Refleaks PR

[vstinner]

Ah, test_pathlib fails on Windows. I will fix later.

First, I'm waiting for Windows11 Refleaks buildbot report.

[vstinner]

Oh, it seems like test_multiprocessing_spawn.test_processes also leaks:

test.test_multiprocessing_spawn.test_processes leaked [36, 36, 36] handles, sum=108

[vstinner]

Aaaand last but not least:

test_winapi leaked [8482, 8482, 8482] handles, sum=25446

[vstinner]

!buildbot AMD64 Windows11 Refleaks PR

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit e66fb14 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F154140%2Fmerge

The command will test the builders whose names match following regular expression: AMD64 Windows11 Refleaks PR

The builders matched are:

• AMD64 Windows11 Refleaks PR

[vstinner]

TODO: Update test_getfinalpathname_handles() of Lib/test/test_os/test_windows.py to reuse support.handle_count().

Done.

test_winapi leaked [8482, 8482, 8482] handles, sum=25446

Fixed by PR gh-154201.

test.test_multiprocessing_spawn.test_processes leaked [36, 36, 36] handles, sum=108

I created issue gh-154208 to investigate this leak.

[vstinner]

buildbot/AMD64 Windows11 Refleaks PR — Build done.

Oh, test_launcher also leaked. I didn't notice this one before.

test_socket leaked [1, 1, 1] handles, sum=3

test_winapi leaked [1, 1, 1] references, sum=3
test_winapi leaked [8482, 8482, 8482] handles, sum=25446

test_launcher leaked [1, 1, 1] handles, sum=3

======================================================================
FAIL: test_exists (idlelib.idle_test.test_zzdummy.ZZDummyTest.test_exists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "b:\uildarea\pull_request.ware-win11.refleak\build\Lib\idlelib\idle_test\test_zzdummy.py", line 157, in test_exists
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        conf.GetSectionList('user', 'extensions'), [])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: ['ZzDummy'] != []

First list contains 1 additional elements.
First extra element 0:
'ZzDummy'

- ['ZzDummy']
+ []

test.test_concurrent_futures.test_process_pool leaked [24, 24, 24] handles, sum=72
test.test_multiprocessing_spawn.test_processes leaked [36, 36, 36] handles, sum=108

[vstinner]

test_launcher leaked [1, 1, 1] handles, sum=3

I'm unable to reproduce this leak. But this test has just been removed in the main branch by commit bd31583! So I will just update this branch instead.

[vstinner]

I extracted the changes adding os_helper.fd_count() in the new PR gh-157637 to make this PR smaller.

[vstinner]

I modified the PR to skip check on handle count when running multiprocessing and concurrent.futures tests to work around the issue gh-154208. While there are other tests which use multiprocessing, only multiprocessing tests stop the pool before child processes have time to steal the handles from the parent process.

It's a compromise to check for handle leaks for all other tests.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 53f6c75 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F154140%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

""buildbot/ARM64 Windows Non-Debug PR" failed to build Python with "compiler is out of heap space in pass 2" error.

            Creating library C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\arm64\python316.lib and object C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\arm64\python316.exp
         Generating code
     3>C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\Python\gc.c(1452): fatal error C1002: compiler is out of heap space in pass 2 [C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pythoncore.vcxproj]
     3>C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\Objects\unicodetype_db.h(4582): fatal error C1002: compiler is out of heap space in pass 2 [C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pythoncore.vcxproj]
     3>C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\Modules\clinic\_winapi.c.h(3036): fatal error C1002: compiler is out of heap space in pass 2 [C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pythoncore.vcxproj]
     3>LINK : fatal error LNK1257: code generation failed [C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pythoncore.vcxproj]
     3>Done Building Project "C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pythoncore.vcxproj" (Build target(s)) -- FAILED.
     1>Done Building Project "C:\Users\buildworker\buildarea\pull_request.ware-win11-arm64.nondebug\build\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED.
Build FAILED.

The buildbot also fails on the main branch, so it's unrelated: https://buildbot.python.org/#/builders/730/builds/14595