Metadata-Version: 2.1
Name: pytest-runtime-xfail
Version: 1.0.3
Summary: Call runtime_xfail() to mark running test as xfail.
Home-page: https://github.com/okken/pytest-runtime-xfail
Author: Brian Okken
Author-email: brian+pypi@pythontest.com
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Classifier: License :: OSI Approved :: MIT License
Requires-Dist: pytest>=5.0.0

# pytest-runtime-xfail

pytest plugin, providing a `runtime_xfail` fixture, which is callable as `runtime_xfail()`, to allow runtime decisions to mark a test as `xfail`. 

## Installation

Install with pip:

    pip install pytest-runtime-xfail

## Usage

Include the fixture, then call it if you want to mark a test as `xfail` during runtime.

```python
 def test_something(runtime_xfail):
     if (runtime_condition):
        runtime_xfail()
     # ... the rest of your test
```

Can also be used in a fixture, of course.

```python
@pytest.fixture()
def foo(runtime_xfail):
  if (runtime_condition):
     runtime_xfail()
  # ... the rest of your fixture

def test_something(foo):
  # ... the rest of your test
```

## Reason this plugin is needed

pytest allows you to mark tests as expected to fail, or xfail, in two ways.

1. `@pytest.mark.xfail`. This allows you to mark tests or test parametrizations as `xfail` during test collection time.
   * pytest runs tests marked with `xfail` just like any other test.
   * If the test fails, it will result in `XFAIL`.
   * If it passes, `XPASS`. Unless you have `xfail_strict=true` or `@pytest.mark.xfail(strict=True)`, in which case, passing xfail-marked tests will result in `FAIL`.
      * This is useful to be alerted when an expected failing test starts to pass.

2. `pytest.xfail()`. If you need information only known at runtime to decide if `xfail` is appropriate, you can call `pytest.xfail()` during a test or fixture.
   * pytest runs the test as normal UNTIL `pytest.xfail()` is called.
   * When `pytest.xfail()` is called, the test execution stops and the test results in `XFAIL`.
   * The rest of the test is not run.
   * There is no way to get `XPASS` from `pytest.xfail()`.
   * `xfail_strict` has no effect.


There are times when we want a combination of these behaviors.

* We don't know until runtime if we should mark a test as `xfail`.
* We want the test run.
* We want the possibility of both `XFAIL` and `XPASS` results.
* We want to be able to use `xfail_strict=true` to alert us when the test starts passing.

This plugin fills that gap.

## Alternatives

You can get around the same limitation yourself by adding the marker through the `requests` object:

```python

def test_something(request):
     if (runtime_condition): 
        request.node.add_marker(pytest.mark.xfail(reason='some reason'))
     # ... rest of test
```

That's basically what this plugin does, just in a fixture.


## Example found in example/test_xfail.py

```python
"""
Run this with
* pytest -v
* pytest -v -o xfail_strict=true
"""

import pytest

@pytest.mark.xfail()
def test_marker_pass():
    'Can be XPASS or FAIL (if xfail_strict)'
    assert True

@pytest.mark.xfail()
def test_marker_fail():
    'Will always be XFAIL'
    assert False  # this statememt will be run

def test_old_xfail_pass():
    'Will always be XFAIL'
    pytest.xfail()
    assert True  # this statememt will NOT be run

def test_old_xfail_fail():
    'Will always be XFAIL'
    pytest.xfail()
    assert False  # this statememt will NOT be run

def test_runtime_xfail_pass(runtime_xfail):
    runtime_xfail()
    assert True  # this statement will be run

def test_runtime_xfail_fail(runtime_xfail):
    runtime_xfail()
    assert False  # this statement will be run

def test_runtime_xfail_reason(runtime_xfail):
    runtime_xfail(reason="for demo")
    assert False  # this statement will be run
```

**Output:**

```
(venv) $ pytest -v test_xfail.py 
========================= test session starts ==========================
collected 7 items                                                      

test_xfail.py::test_marker_pass XPASS                            [ 14%]
test_xfail.py::test_marker_fail XFAIL                            [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL                         [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL                         [ 57%]
test_xfail.py::test_runtime_xfail_pass XPASS                     [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL                     [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo)        [100%]

==================== 5 xfailed, 2 xpassed in 0.05s =====================
(venv) $ pytest -v test_xfail.py -o xfail_strict=true
========================= test session starts ==========================
collected 7 items                                                      

test_xfail.py::test_marker_pass FAILED                           [ 14%]
test_xfail.py::test_marker_fail XFAIL                            [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL                         [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL                         [ 57%]
test_xfail.py::test_runtime_xfail_pass FAILED                    [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL                     [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo)        [100%]

===================== 2 failed, 5 xfailed in 0.04s =====================
```
