Cache: working with cross-testrun state¶
New in version 2.8.
Usage¶
The plugin provides two command line options to rerun failures from the
last pytest invocation:
--lf,--last-failed- to only re-run the failures.--ff,--failed-first- to run the failures first and then the rest of the tests.
For cleanup (usually not needed), a --cache-clear option allows to remove
all cross-session cache contents ahead of a test run.
Other plugins may access the config.cache object to set/get
json encodable values between pytest invocations.
Note
This plugin is enabled by default, but can be disabled if needed: see
Deactivating / unregistering a plugin by name (the internal name for this plugin is
cacheprovider).
Rerunning only failures or failures first¶
First, let’s create 50 test invocation of which only 2 fail:
# content of test_50.py
import pytest
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
pytest.fail("bad luck")
If you run this for the first time you will see two failures:
$ pytest -q
.................F.......F........................ [100%]
================================= FAILURES =================================
_______________________________ test_num[17] _______________________________
i = 17
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
_______________________________ test_num[25] _______________________________
i = 25
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
2 failed, 48 passed in 0.12 seconds
If you then run it with --lf:
$ pytest --lf
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
collected 50 items / 48 deselected
run-last-failure: rerun previous 2 failures
test_50.py FF [100%]
================================= FAILURES =================================
_______________________________ test_num[17] _______________________________
i = 17
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
_______________________________ test_num[25] _______________________________
i = 25
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
================= 2 failed, 48 deselected in 0.12 seconds ==================
You have run only the two failing test from the last run, while 48 tests have not been run (“deselected”).
Now, if you run with the --ff option, all tests will be run but the first
previous failures will be executed first (as can be seen from the series
of FF and dots):
$ pytest --ff
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
collected 50 items
run-last-failure: rerun previous 2 failures first
test_50.py FF................................................ [100%]
================================= FAILURES =================================
_______________________________ test_num[17] _______________________________
i = 17
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
_______________________________ test_num[25] _______________________________
i = 25
@pytest.mark.parametrize("i", range(50))
def test_num(i):
if i in (17, 25):
> pytest.fail("bad luck")
E Failed: bad luck
test_50.py:6: Failed
=================== 2 failed, 48 passed in 0.12 seconds ====================
New --nf, --new-first options: run new tests first followed by the rest
of the tests, in both cases tests are also sorted by the file modified time,
with more recent files coming first.
Behavior when no tests failed in the last run¶
When no tests failed in the last run, or when no cached lastfailed data was
found, pytest can be configured either to run all of the tests or no tests,
using the --last-failed-no-failures option, which takes one of the following values:
pytest --last-failed --last-failed-no-failures all # run all tests (default behavior)
pytest --last-failed --last-failed-no-failures none # run no tests and exit
The new config.cache object¶
Plugins or conftest.py support code can get a cached value using the
pytest config object. Here is a basic example plugin which
implements a pytest fixtures: explicit, modular, scalable which re-uses previously created state
across pytest invocations:
# content of test_caching.py
import pytest
import time
@pytest.fixture
def mydata(request):
val = request.config.cache.get("example/value", None)
if val is None:
time.sleep(9*0.6) # expensive computation :)
val = 42
request.config.cache.set("example/value", val)
return val
def test_function(mydata):
assert mydata == 23
If you run this command once, it will take a while because of the sleep:
$ pytest -q
F [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________
mydata = 42
def test_function(mydata):
> assert mydata == 23
E assert 42 == 23
test_caching.py:14: AssertionError
1 failed in 0.12 seconds
If you run it a second time the value will be retrieved from the cache and this will be quick:
$ pytest -q
F [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________
mydata = 42
def test_function(mydata):
> assert mydata == 23
E assert 42 == 23
test_caching.py:14: AssertionError
1 failed in 0.12 seconds
See the config.cache for more details.
Inspecting Cache content¶
You can always peek at the content of the cache using the
--cache-show command line option:
$ pytest --cache-show
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
cachedir: $REGENDOC_TMPDIR/.pytest_cache
------------------------------- cache values -------------------------------
cache/lastfailed contains:
{'test_caching.py::test_function': True}
cache/nodeids contains:
['test_caching.py::test_function']
example/value contains:
42
======================= no tests ran in 0.12 seconds =======================
Clearing Cache content¶
You can instruct pytest to clear all cache files and values
by adding the --cache-clear option like this:
pytest --cache-clear
This is recommended for invocations from Continuous Integration servers where isolation and correctness is more important than speed.