Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test results reporting #774

Open
psss opened this issue Feb 7, 2019 · 3 comments
Open

Improve test results reporting #774

psss opened this issue Feb 7, 2019 · 3 comments

Comments

@psss
Copy link
Contributor

psss commented Feb 7, 2019

There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:

Let's update the pipeline according to the agreed result: The main difference is that ansible-playbook should return non-zero status only upon infrastructure errors while tests failures can be detected from the newly defined file results.yml which will have the following format:

results:
- {result: pass, test: shell/smoke}
- {result: fail, test: shell/login}
- {result: error, test: shell/func}

Extended version with links to logs:

results:
- {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log}
- {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log}
- {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}

The expected behaviour of the pipeline is this:

  • Report error when ansible returns non-zero (infrastructure error)
  • Report error if error appears in results.yml files
  • Report test failure if fail appears in results.yml files
  • Report pass otherwise

This issue is blocked by implementation of the new functionality in Standard Test Roles which is tracked here:

@johnbieren
Copy link
Contributor

@bgoncalv This is something we should address in the Jenkinsfile of the upstream-fedora-pipeline, right?

@bgoncalv
Copy link
Contributor

@johnbieren right, I think updating the Jenkinsfile on upstream-fedora-pipeline would be the easiest change. Other option would be after the playbook is executed on "package-test.sh". Where do you think makes more sense to parse the results.yml?

@johnbieren
Copy link
Contributor

@bgoncalv good point, perhaps package-test.sh is best since the error codes are right there and we don't have to rely on the proper error code being propagated through

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants