pytest’s @pytest.mark.xfail and @pytest.mark.skip are your go-to tools for managing tests that are expected to fail or are temporarily irrelevant, preventing them from cluttering your CI reports with red.
Let’s see xfail in action. Imagine you have a test for a feature that’s known to be broken in the current version of a library you depend on. You don’t want this test to fail your build, but you also don’t want to forget about it.
import pytest
@pytest.mark.xfail(reason="Bug #123: Feature X is broken in library v1.2.3")
def test_feature_x_broken():
assert 1 == 2 # This will intentionally fail
When you run pytest, this test won’t show up as a failure. Instead, pytest will report it as xfailed.
$ pytest
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0
rootdir: /path/to/your/project
collected 1 item
test_example.py x [100%]
============================== 1 xfailed in 0.01s ===============================
Now, what if that bug gets fixed? If you run the same test against a version of the library where the bug is resolved, pytest will show it as xpassed. This is crucial: it tells you that the condition you were expecting to fail has now passed, and you should likely remove the @pytest.mark.xfail decorator.
# Assume library v1.2.4 fixes the bug
# If you run the same test with the xfail decorator:
#
# $ pytest
# ============================= test session starts ==============================
# platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0
# rootdir: /path/to/your/project
# collected 1 item
#
# test_example.py X [100%]
#
# ============================= 1 xpassed in 0.01s ==============================
The reason parameter is important for documentation. It explains why the test is marked as xfail. This helps future maintainers understand the context without digging through bug trackers.
You can also control the strictness of xfail with the strict parameter. By default, strict=False. If strict=True, an xpassed test will be treated as a failure. This is useful when you want to be immediately notified if a previously failing test starts passing, indicating that a fix has been implemented and the xfail decorator should be removed.
@pytest.mark.xfail(reason="Temporary workaround for issue #456", strict=True)
def test_temporary_workaround():
assert some_flaky_operation() is not None
If some_flaky_operation() suddenly starts working reliably, this test will now fail your pytest run due to strict=True, prompting you to revisit and potentially remove the xfail mark.
@pytest.mark.skip is for tests that should not run at all, either because they are not relevant in the current environment or because they are broken and you haven’t gotten around to marking them xfail yet.
import sys
import pytest
@pytest.mark.skip(reason="This test only runs on Linux")
def test_linux_specific_feature():
assert sys.platform.startswith("linux")
@pytest.mark.skipif(sys.version_info < (3, 10), reason="Requires Python 3.10 or higher")
def test_python_310_feature():
pass # Test logic here
When you run pytest, these tests will be marked as skipped.
$ pytest
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0
rootdir: /path/to/your/project
collected 2 items
test_example.py s s [100%]
============================== 2 skipped in 0.01s ===============================
The skipif decorator is particularly powerful, allowing you to skip tests based on arbitrary conditions, such as Python version, operating system, or even the presence of specific libraries. The condition is a standard Python expression that evaluates to True if the test should be skipped.
You can also skip tests conditionally within the test function itself using pytest.skip():
def test_conditional_skip():
if not os.environ.get("RUN_SLOW_TESTS"):
pytest.skip("Skipping slow test: set RUN_SLOW_TESTS=1 to run")
# ... slow test logic ...
This gives you fine-grained control over when a test is executed.
The key difference between skip and xfail is intent. skip means "don’t run this test." xfail means "run this test, but we expect it to fail (or pass if a fix is in)." xfail provides valuable information about the state of your codebase and its dependencies, whereas skip is more about eligibility for execution.
A common pattern is to use xfail for known bugs in external dependencies and skip for tests that are platform-specific or require special setup not available in the current environment. You might also skip a test that is completely broken and you haven’t had time to properly diagnose for an xfail mark, but this should be a temporary state.
One subtle aspect is how xfail interacts with test discovery and collection. When pytest encounters an @pytest.mark.xfail decorator, it still collects the test function. The test runner then executes the test, and based on its outcome (pass or fail), it marks the test result as xfailed or xpassed. This is different from @pytest.mark.skip, where the test is not executed at all; it’s simply marked as skipped during the collection phase. This means that any setup code within an xfail-marked test function will run, whereas setup code for a skip-marked test might not, depending on where the skip is applied (e.g., a pytest.skip() call inside the test function will run the test function’s setup but then skip the test body).
If you’re dealing with tests that are flaky and sometimes pass, sometimes fail, and you want to track them without them breaking your CI, @pytest.mark.xfail is your tool. If a test is simply not applicable or you need to temporarily disable it without intending to fix it soon, @pytest.mark.skip is more appropriate.
When you fix a bug that was previously marked with @pytest.mark.xfail, the next logical step is to remove the xfail decorator. If you forget and the test now passes, pytest will report it as xpassed. If you have strict=True on your xfail marker, this xpassed will turn into a test failure, forcing you to acknowledge the change and clean up your test suite.