[FEATURE] usage for doctests

[FlorianGD]

Foreword: thank you for this plugin, I like the API, and it is really useful.

Is your feature request related to a problem? Please describe.
The plugin works very well for normal tests, but I'd really like to use it with doctests.
I run those with pytest --doctest-modules and/or pytest --doctest-glob="*.rst". I can define a conftest.py file, but in it I can define fixtures, and not tests, so I cannot use the pytest.mark.vcr

Describe the solution you'd like
I'd like to be able to configure the tests to use a cassette/several cassettes for the tests in (say) the README and the modules, and then the pytest --doctest-modules would record and use the cassette instead of calling the network.

Describe alternatives you've considered
I can add this in conftest.py:

from vcr import use_cassette

@pytest.fixture(autouse=True)
def with_cassette():
    with use_cassette("tests/doctests/cassette.yaml", record_mode="once"):
        yield

But this does not respect whatever will be passed to the command line arguments as it hardcodes the cassette and record mode. If it creates a cassette for each file, that would be better, instead of using one big cassette.

[Stranger6667]

Thank you for creating this issue!

This is an interesting feature! Unfortunately, I don't have the bandwidth to work on it myself in foreseeable, but would be happy to review a patch :)

[FlorianGD]

OK, thanks for your quick reply! If I come up with something usable, I'll make a PR, but not sure I can do it in the next few weeks.

[Stranger6667]

Sure! Feel free to open a draft PR - it might be easier to start the discussion there :)

[FlorianGD]

For what it's worth, I hacked something like this, I put it here for others / if I do not have the bandwidth to make a proper PR:

In conftest.py

import vcr 

@pytest.fixture(autouse=True)
def with_cassette(request):  # type: ignore
    # nodeid is like `filepath::module`
    name = request.node.nodeid.split("::")[1].replace(".", "_")
    with vcr.use_cassette(
        f"tests/doctests/cassette_{name}.yaml",
        record_mode="once",
        filter_headers=["authorization"],
    ):
        yield

[FlorianGD]

Having a look at https://github.dev/astropy/pytest-doctestplus and they define a doctest option flag for REMOTE_DATA. We could add an option flag like # doctest: +MARK_VCR to indicate that the line should be executed through a cassette maybe.

But their package entirely skips Shpinx as far as I can tell, they collect and run the tests on their own. Maybe something lighter can be done

[akaihola]

Thanks @FlorianGD I'll have to try out your approach!

I remember hitting this problem back when we were still using pytest-vcr. Seems that I gave up, didn't run doctests using Pytest, but executed them as part of the Sphinx build instead. I did that by adding this to docs/conf.py:

# Set up VCR.py automatically so we don't need to do it separately in all
# doctest test cases.
from textwrap import dedent

doctest_global_setup = dedent('''
    # imports to be available in doctests
    from datetime import datetime
    import os
    from pprint import pprint
    import mypackage
    import vcr

    # We need to match on full body so VCRpy can differentiate between
    # our POST requests:
    my_vcr = vcr.VCR(
        match_on=['method', 'path', 'query', 'body'],
        record_mode='new_episodes',
        filter_headers=['user-agent']
    )

    context = my_vcr.use_cassette(
        os.path.join(
            os.path.dirname(mypackage.__file__),
            '..',
            'docs',
            'doctest_cassette.yml'
        )
    )
    context.__enter__()
''')

doctest_global_cleanup = dedent('''
    context.__exit__()
''')