pytest-xdist is a plugin that lets you run your pytest test suite in parallel, not just across multiple CPU cores on a single machine, but even across multiple machines on your network.
Imagine you have a test suite that takes an hour to run. With pytest-xdist, you can potentially slash that time down to minutes by distributing the workload. This isn’t just about speed; it’s about getting faster feedback loops during development and making your CI/CD pipelines more efficient.
Let’s see it in action. Suppose you have a simple test file test_example.py:
import time
import pytest
def test_slow_one():
time.sleep(2)
assert True
def test_slow_two():
time.sleep(2)
assert True
def test_slow_three():
time.sleep(2)
assert True
def test_slow_four():
time.sleep(2)
assert True
If you run this with plain pytest, it will take roughly 8 seconds.
pytest test_example.py
Now, let’s introduce pytest-xdist. The primary way to use it is through the -n flag, which specifies the number of parallel workers.
pytest -n 4 test_example.py
When you run this, pytest-xdist spins up four worker processes. These workers fetch tests from the main pytest process and execute them. Because each test_slow_ function takes 2 seconds, and we have 4 workers and 4 tests, they should all complete in roughly 2 seconds (plus a tiny overhead for distribution).
The magic behind this is the Controller and Worker architecture. The main pytest process acts as the Controller. When you invoke pytest -n N, the Controller starts N Worker processes. These workers communicate with the Controller over a network socket (usually on localhost). The Controller manages the test collection and then distributes the collected tests to the available workers. When a worker finishes a test, it reports the result back to the Controller, and then it’s ready to pick up another test.
The real power comes when you need to scale beyond a single machine. pytest-xdist supports distributed execution across multiple hosts. You can start controllers on your local machine and workers on remote machines.
To do this, you’ll first start the workers on the remote machines. Let’s say you have a machine with IP 192.168.1.100. On that machine, you’d run:
# On the remote machine (192.168.1.100)
pytest-xdist --dsn=tcp://<your_controller_ip>:2345
The --dsn (Distribution Service Name) flag tells the worker where to connect. tcp://<your_controller_ip>:2345 specifies the protocol, the IP address of your controller, and a port. 2345 is a common default, but you can use any available port.
On your local machine (the controller), you’d then run:
# On your local machine
pytest -n 4 --port=2345 test_example.py
The --port flag here is crucial. It tells the controller to listen on that specific port for incoming worker connections. If you don’t specify a port for the controller, it will pick a random one, which makes it harder for the workers to find it. It’s generally best practice to explicitly define the port on both sides. The -n 4 still dictates how many workers the controller expects to manage, but in this distributed setup, those workers can be on different machines.
The ability to distribute tests across machines is invaluable for large test suites or when you have specialized hardware requirements for certain tests. You can dedicate powerful machines as workers for CPU-intensive tests, or machines with specific OS configurations.
One subtle but powerful aspect of pytest-xdist is how it handles test scheduling and distribution. By default, workers pull tests from a queue. However, you can influence this. For instance, if you have tests with varying execution times, you might want to ensure that shorter tests don’t get stuck waiting for longer ones to finish on a worker. pytest-xdist tries to balance the load dynamically, but complex scenarios might benefit from more advanced scheduling strategies or custom hooks, though these are less common for typical usage. The default behavior is usually quite effective at keeping all workers busy.
The Controller also handles test session management, including setup and teardown hooks, ensuring they are executed correctly even with distributed workers. The results are aggregated and presented by the main Controller process at the end of the test run.
When you’re running tests across multiple machines, ensuring that all necessary files and dependencies are available on the worker machines is paramount. pytest-xdist doesn’t automatically synchronize your project code. You’ll need to ensure your test environment is consistent across all machines, typically by using a shared filesystem, or by deploying your code to each worker machine consistently (e.g., via Git, Docker, or a deployment script).
The next step after mastering parallel execution on a single machine and across a small network is often exploring how to integrate pytest-xdist into your CI/CD pipelines for automated parallel test execution.