option to fail fast

[jtiai]

If having configure multiple environments and running them with plain tox-command if environment in the middle fails, rest of the environments are run.

Expected that if one environment fails, rest aren't executed.

Sample tox.ini:

[tox]
envlist = lint,test,build
skipsdist = True

[testenv]
whitelist_externals = sh

[testenv:lint]
commands =
    sh -c "exit 0"

[testenv:test]
commands =
    sh -c "exit 1"

[testenv:build]
commands =
    sh -c "exit 0"

pip list:

pip (9.0.1)
pkg-resources (0.0.0)
pluggy (0.4.0)
py (1.4.34)
setuptools (36.2.7)
tox (2.7.0)
virtualenv (15.1.0)
wheel (0.29.0)

[RonnyPfannschmidt]

for tox running everything is expectedbehaviour

[The-Compiler]

That seems like expected behaviour to me. If e.g. tests with python 2 fail, I'd still like to run tests with python 3.

[obestwalter]

Hi @jtiai as already said: this is how it's supposed to be :)

You could get to the wanted behaviour by doing something like this on the command line (same semantics on POSIX and Windows):

tox -e env1 && tox -e env2 && tox -e env3

If any of those runs fail the rest of the chain will not be executed anymore.

On CI systems you usually have the concept of stages that are executed serially and if one stage fails the whole plan fails and nothing else is executed. So there you would create a stage for each of the tox environments that depend on each other.

[jtiai]

I got the impression that tox stops the execution from the docs:
http://tox.readthedocs.io/en/latest/config.html#confval-ignore_errors=True|False(default)

So maybe it's just matter of clarify in the documentation that only current environment execution is aborted, but rest of the environments defined will be executed regardless state of the others.

[The-Compiler]

@jtiai Do you have any suggestion on how this could be clarified? It already says "will abort execution of commands for that environment.".

[jtiai]

Last paragraph states:

Tox >= 2.0 will abort on an error by default, which is safer and more typical of CI and command
execution tools, as it doesn’t make sense to run tests if installing some prerequisite failed and it doesn’t >make sense to try to deploy if tests failed.

So that's where I got impression that tox stops on error.

Not sure about wording but that part should somehow emphasize the fact that only current environment execution is stopped and if there are more defined, either in envlist configuration or via -e switch from the cli rest will be executed.

[gaborbernat]

I think we only need to point out that that abort is active inside the commands 👍 not for the environments 👍

[apomelov]

Take a look at ignore_outcome's doc as well. Now it says:

If set to True a failing result of this testenv will not make tox fail, only a warning will be produced.
default: False

Really expecting tox to fail by default after some testenv fails.

[obestwalter]

o.k. that surprises me, but I get now where you are coming from, so we definitely need to make the docs clearer in those areas. Thanks!

[obestwalter]

Hi @apomelov, this also contains the term "testenv" which means that it only means that the testenv won't fail, but I get it. I think what is missing is a bit of a high level overview that is describing a tox run and everything that is involved and properly introduces the terminology.