pytest-console-scripts

Pytest-console-scripts is a pytest plugin for running python scripts from
within tests. It's quite similar to subprocess.run(), but it also has an
in-process mode, where the scripts are executed by the interpreter that's
running pytest (using some amount of sandboxing).

In-process mode significantly reduces the run time of the test suites that
run many external scripts. This is speeds up development. In the CI environment
subprocess mode can be used to make sure the scripts also work (and behave the
same) when run by a fresh interpreter.

Requirements

Python 3.8+, or PyPy3,
Pytest 4.0 or newer.

Installation

You can install "pytest-console-scripts" via pip from PyPI:

$ pip install pytest-console-scripts

Normally you would add it as a test dependency in tox.ini (see tox
documentation).

Usage

This plugin will run scripts that are installed via console_scripts entry
point in setup.py, python files in current directory (or anywhere else, if
given the path), and Python scripts anywhere else in the path. It will also run
executables that are not Python scripts, but only in subprocess mode (there's
no benefit in using pytest-console-scripts for this, you should just use
subprocess.run).

Here's an example with console_scripts entry point. Imagine we have a python
package foo with the following setup.py:

setup(
    name='foo',
    version='0.0.1',
    py_modules=['foo'],
    entry_points={
        'console_scripts': ['foobar=foo:bar']
    },
)

We could use pytest-console-scripts to test the foobar script:

def test_foo_bar(script_runner):
    result = script_runner.run(['foobar', '--version'])
    assert result.returncode == 0
    assert result.stdout == '3.2.1\n'
    assert result.stderr == ''

    script_runner.run('foobar --version', shell=True, check=True)

This would use the script_runner fixture provided by the plugin to
run the script and capture its output.

The arguments of script_runner.run are the command name of the script and
any command line arguments that should be passed to it. Additionally the
following keyword arguments can be used:

cwd - set the working directory of the script under test.
env - a dictionary with environment variables to use instead of the current
environment.
stdin - a file-like object that will be piped to standard input of the
script.
check - raises an exception if returncode != 0, defaults to False.
shell - mimic shell execution, this should work well for simple cases,
defaults to False.

Type-hinting is also supported.
You may type-hint the fixture with the following code:

from pytest_console_scripts import ScriptRunner

def test_foo_bar(script_runner: ScriptRunner) -> None:
    ...

Configuring script execution mode

In the example above the foobar script would run in in-process mode (which is
the default). This is fast and good for quick iteration during development.
After we're happy with the functionality, it's time to run the script in
subprocess mode to simulate real invocation more closely. There are several
ways to do this. We can configure it via pytest configuration (for example in
tox.ini):

[pytest]
script_launch_mode = subprocess

We can give a command line option to pytest (this will override the
configuration file):

$ pytest --script-launch-mode=subprocess test_foobar.py

We can also mark individual tests to run in a specific mode:

@pytest.mark.script_launch_mode('subprocess')
def test_foobar(script_runner):
    ...

Between these three methods the marking of the tests has priority before the
command line option that in turn overrides the configuration setting. All three
can take three possible values: "inprocess", "subprocess", and "both" (which
will cause the test to be run twice: in in-process and in subprocess modes).

Interaction with mocking

It is possible to mock objects and functions inside of console scripts when
they are run using pytest-console-scripts but only in inprocess mode. When
the script is run in subprocess mode, it is executed by a separate Python
interpreter and the test can't mock anything inside of it.

Another limitation of mocking is that with simple Python scripts that are not
installed via console_scripts entry point mocking of objects inside of
the main script will not work. The reason for that is that when we run
myscript.py with $ python myscript.py the script gets imported into
__main__ namespace instead of myscript namespace. Our patching of
myscript.myfunction will have no effect on what the code in __main__
namespace sees when it's calling myfunction defined in the same file.

See this stackoverflow answer
for some ideas of how to get around this.

Suppressing the printing of script run results

When tests involving pytest-console-scripts fail, it tends to be quite
useful to see the output of the scripts that were executed in them. We try
to be helpful and print it out just before returning the result from
script_runner.run(). Normally PyTest captures all the output during a
test run and it's not shown to the user unless some tests fail. This is exactly
what we want.

However, in some cases it might be useful to disable the output capturing and
PyTest provides ways to do it. When capturing is disabled, all test run
results will be printed out and this might make it harder to inspect the other
output of the tests. To deal with this, pytest-console-scripts has an option
to disable the printing of script run results:

$ pytest --hide-run-results test_foobar.py

It's also possible to disable it just for one script run:

result = script_runner.run('foobar', print_result=False)

When printing of script run results is disabled, script output won't be
visible even when the test fails. Unfortunately there's no automatic way to
print it only if the test fails because by the time a script run completes we
don't know whether the test will fail or not. It's possible to do it manually
from the test by using:

result.print()

This, combined with --hide-run-results or print_result=False can be used to
only print interesting run results when capturing is off.

Package installation and testing during development

Since pytest-console-scripts relies on the scripts being located in the path,
it can only run the console scripts from packages that have been installed (if
you are interested in working on removing this limitation, take a look at this
ticket and in
particular this comment).
If you want to run the tests quickly during development, the additional
installation step would add a significant overhead and slow you down.

There's a way around this: install your package in development mode using
pip install -e .. If you use tox, you can take one of its
existing virtualenvs (they live in .tox/). Otherwise create a
virtualenv just for development, activate it and run python setup.py develop to install your package in development mode. You will need to
re-install every time you add a new console script, but otherwise all the
changes to your code will be immediately picked up by the tests.

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure
the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-console-scripts"
is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed
description.

Pytest-console-scripts was initially generated with Cookiecutter along
with @hackebrot's Cookiecutter-pytest-plugin template.

Dependencies of pytest-console-scripts (2).
Dependency	Optional
importlib-metadata	false
pytest	false

Package profile

pytest-console-scripts

Releases

Release table

PyPI Downloads

Dependencies

Details

Project URLs

Dependencies

Package description