pytest-run-parallel

A simple pytest plugin to run tests concurrently

---

This pytest plugin takes a set of tests that would be normally be run serially and execute them in parallel.

The main goal of pytest-run-parallel is to discover thread-safety issues that could exist when using C libraries, this is of vital importance after PEP703, which provides a path for a CPython implementation without depending on the Global Interpreter Lock (GIL), thus allowing for proper parallelism in programs that make use of the CPython interpreter.

For more information about C thread-safety issues, please visit the free-threaded community guide at https://py-free-threading.github.io/

Features

• Two global CLI flags:

  • --parallel-threads to run a test suite in parallel
  • --iterations to run multiple times in each thread

• Three corresponding markers:

  • pytest.mark.parallel_threads(n) to mark a single test to run in parallel in n threads
  • pytest.mark.thread_unsafe to mark a single test to run in a single thread. It is equivalent to using pytest.mark.parallel_threads(1)
  • pytest.mark.iterations(n) to mark a single test to run n times in each thread

• And the corresponding fixtures:

  • num_parallel_threads: The number of threads the test will run in
  • num_iterations: The number of iterations the test will run in each thread

Requirements

pytest-run-parallel depends exclusively on pytest.

Installation

You can install "pytest-run-parallel" via pip from PyPI:

$ pip install pytest-run-parallel

Usage

This plugin has two modes of operation, one via the --parallel-threads and --iterations pytest CLI flags, which allows a whole test suite to be run in parallel:

pytest --parallel-threads=10 --iterations=10 tests

By default, the value for both flags will be 1, thus not modifying the usual behaviour of pytest except when the flag is set.

Note that using pytest-xdist and setting iterations to a number greater than one might cause tests to run even more times than intended.

The other mode of operation occurs at the individual test level, via the pytest.mark.parallel_threads and pytest.mark.iterations markers:

# test_file.py
import pytest

@pytest.fixture
def my_fixture():
    ...

@pytest.mark.parallel_threads(2)
@pytest.mark.iterations(10)
def test_something_1():
    # This test will be run in parallel using two concurrent threads
    # and 10 times in each thread
    ...

@pytest.mark.parametrize('arg', [1, 2, 3])
@pytest.mark.parallel_threads(3)
def test_fixture(my_fixture, arg):
    # pytest markers and fixtures are supported as well
    ...

Both modes of operations are supported simultaneously, i.e.,

# test_something_1 and test_fixture will be run using their set number of
# threads; other tests will be run using 5 threads.
pytest -x -v --parallel-threads=5 test_file.py

Additionally, pytest-run-parallel exposes the num_parallel_threads and num_iterations fixtures which enable a test to be aware of the number of threads that are being spawned and the number of iterations each test will run:

# test_file.py
import pytest

def test_skip_if_parallel(num_parallel_threads):
    if num_parallel_threads > 1:
        pytest.skip(reason='does not work in parallel')
    ...

Finally, the thread_comp fixture allows for parallel test debugging, by providing an instance of ThreadComparator, whose __call__ method allows to check if all the values produced by all threads during an specific execution step are the same:

# test_file.py
def test_same_execution_values(thread_comp):
    a = 2
    b = [3, 4, 5]
    c = None
    # Check that the values for a, b, c are the same across tests
    thread_comp(a=a, b=b, c=c)

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-run-parallel" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.