Trouble using matchtask in pytest

[maunzCache]

Hi everyone,
i am currently trying to write tests for my custom ansible rules by utilizing the new Runner class from the ansible-lint project instead of custom yaml parsing. But i seem to have trouble asserting errors in my test as the Runner (package ansiblelint.runner) either does not execute matchtask() functions or does not respect them in a way that the result is wrapped into a MatchError.

This is irritating as the project itself has a bunch of tests using Rules which implement matchtask() and i kind of copied the implementations e.g. TestTaskHasName.py.

This is my custom rule

class ShellAltChmod(AnsibleLintRule):
    id = 'ECS008'
    shortdesc = 'Use "file" module.'
    description = 'Usage of "file" module instead of "shell" or "command" advised.'
    tags = ['shell']

    def matchtask(self, task: Dict[str, Any]) -> Union[bool, str]:
        if task['action']['__ansible_module__'] not in ['shell', 'command']:
            return False
        if 'chmod' in task['action']['__ansible_arguments__']:
            return True
        return False

The python test for the rule

from ansiblelint.rules import RulesCollection
from ansiblelint.runner import Runner

from rules.custom.ShellAltChmod import ShellAltChmod


def test_shell_alt_chmod_fails():
    collection = RulesCollection()
    collection.register(ShellAltChmod())
    playbook = 'examples/shell-alt-chmod.yml'

    bad_runner = Runner(playbook, rules=collection)
    errs = bad_runner.run()

    assert len(errs) == 1

I am not sure if it is necessary to wrap the tests into a Unitest class. It looks to me that it also works without.
This assertion always fails because errs is empty (aka [ ]). When looking into bad_runner.rules and bad_runner.checked_files it contains my custom rule and the test playbook (see below).

Of course the test playbook

---

- hosts: all
  tasks:
    - name: "[ECS008] Use file module"
      shell: chown -R hoge .
    - name: "[ECS008] Use file module"
      command: chown -R hoge .

To me it seems that everything is setup up correctly. The rule itself works when using ansible-lint -r rules/custom/ShellAltChmod.py but the test does not trigger the error or i do not know what to assert.

I took a small dive into the code to understand whether i have to user matchtask() or matchtasks() - plural - because matchtasks produces a List[MessageError] while the other only create a bool.

Can anyone help me with the concept?

Edit: Output of the pytest run

collected 1 item                                                                                                                                                                                     

test/custom_rules/TestShellRules.py::test_shell_alt_chmod_fails FAILED                                                                                                                         [100%]

============================================================================================== FAILURES ==============================================================================================
_____________________________________________________________________________________ test_shell_alt_chmod_fails _____________________________________________________________________________________

    def test_shell_alt_chmod_fails():
        collection = RulesCollection()
        collection.register(ShellAltChmod())
        playbook = 'examples/shell-alt-chmod.yml'
    
        bad_runner = Runner(playbook, rules=collection)
        errs = bad_runner.run()
    
>       assert len(errs) == 1
E       assert 0 == 1
E         +0
E         -1

bad_runner = <ansiblelint.runner.Runner object at 0x7f2768df91f0>
collection = ECS007: Use "file" module.
  Usage of "file" module instead of "shell" or "command" advised.
internal-error: Unexpecte...sible not an Ansible file.
parser-error: AnsibleParserError
  Ansible parser fails, this usually indicate invalid file.
errs       = []
playbook   = 'examples/shell-alt-chmod.yml'

test/custom_rules/TestShellRules.py:15: AssertionError
---------------------------------------------- generated xml file: ansible-lint-rules/test-results/ansible-lint.xml ----------------------------------------------
====================================================================================== short test summary info =======================================================================================
FAILED test/custom_rules/TestShellRules.py::test_shell_alt_chmod_fails - assert 0 == 1
========================================================================================= 1 failed in 0.72s ==========================================================================================

[maunzCache]

Edit: I just mismatched chmod and chown. Silly me.