Tests using the same machine-wide resource should not run in parallel

[serhiy-storchaka]

Feature or enhancement

Some resources which tests use are not per-process: the display, the sound device, the terminal. Two tests using one of them at the same time can interfere with each other, and regrtest runs test files in parallel by default.

Only tests using the same resource conflict. A GUI test can run in parallel with any non-GUI test, so pausing all the other tests would be too much.

These resources are already named on the command line: gui, audio, console and curses. A test declares which one it needs with requires(), requires_resource() or, for the memory, bigmemtest(), so those functions could lock the resource for the duration of the test or of the test file. The ~50 GUI modules, test_winsound, test_curses, test_winconsoleio and the bigmem tests would need no change.

The cost is that such tests no longer overlap each other: running test_tkinter, test_ttk and test_idle with -j4 takes as long as all three together (7.1 s here) instead of as long as the slowest (3.5 s). Tests which use no such resource keep running in parallel with them.

Linked PRs

• gh-156082: Do not run tests using the same machine-wide resource at once #156902

[terryjreedy]

~50 GUI modules: typo? or are you saying that there are 50 gui modules? If so, won't they need to be sequenced?

[serhiy-storchaka]

Claude simply counted test modules containing requires('gui') or something like this. And it mistaken, the new count returned 68 (48 only in Lib/idlelib/idle_test). None of these files need changes. GUI tests will be exclusive automatically.

Frankly, the only potential effect of this change is on bigmem and GUI tests. I have no evidence that parallel GUI tests can conflict one other, but this is plausible. Also, we might add exclusiveness for /dev/pty.

[serhiy-storchaka]

I just tried to get a conflict between parallel PTY tests, and found nothing. The BSDs and illumos handle ptys differently, so we still can find evidences.

[terryjreedy]

I was thinking of test_idle and test_tkinter, each being 1 module. I understood you as saying that they should not run in parallel, but agree that separate processes will make separate windows that can overlap ok.

Experiments: on my machine, 3.15.0rc1 test_idle and test_tkinter take 6 and 9 seconds.
py -m test -ugui test_tkinter test_idle (sequential) takes 15 seconds.
py -m test -ugui -j2 test_tkinter test_idle (parallel) takes 11.7 seconds, so some overhead beyond 9 seconds, but definitely shorter.

Note: with py -m test -ugui test _idle test_tkinter, sequential in other order, test_tkinter hangs with colorchooser window displayed. When I close the window (ok, cancel, or X), tests continue with final result

FAIL: test_askcolor (test.test_tkinter.test_colorchooser.DefaultRootTest.test_askcolor)

Traceback (most recent call last):
File "C:\Users\Terry\AppData\Local\Programs\Python\Python315\Lib\test\test_tkinter\test_colorchooser.py", line 54, in test_askcolor
self.assertRaises(ZeroDivisionError, askcolor)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ZeroDivisionError not raised by askcolor

FAIL: test_configure (test.test_tkinter.test_misc.EventTest.test_configure)

Traceback (most recent call last):
File "C:\Users\Terry\AppData\Local\Programs\Python\Python315\Lib\test\test_tkinter\test_misc.py", line 1251, in test_configure
self.assertEqual(e.x, 0)
~~~~~~~~~~~~~~~~^^^^^^^^
AssertionError: 13 != 0

I have occasionally, but I think not always, seen the color window hang before, when running the whole test suite.
Open a bug?

[serhiy-storchaka]

I got confirmation that running parallel GUI tests fails 100% on Windows. It can also fail on Linux (~50%). So indeed, this feature is needed. I can also see hangs on colorchooser window, but this is a different issue.