Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertions Automatic Retry w/Timeout (alternative to processing_period sleep) #14

Open
ScottFreeCode opened this issue Dec 3, 2020 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@ScottFreeCode
Copy link
Contributor

ScottFreeCode commented Dec 3, 2020

Right now, in order to give tests enough time for Rabbit messages to be processed, a hardcoded delay is introduced. Set too low and the tests become flakey, set higher and they are more reliable but take longer. It would be best if instead there was a timeout, either at the test level or the assertion level, for trying the assertions until they either pass or have continued failing till the timeout is hit.

Since Touchstone currently prints the test failure from within the assertion on failure, some refactoring will be needed to prevent printing assertion failures over and over.

I can think of a couple different possible solutions. They aren't mutually exclusive and could be used together to get the advantages of both.

Solution A: Throw

Most test frameworks I'm familiar with throw an exception on failed assertions, where the exception message describes the failure and the assertion exception class contains an expected value and an actual value.

This leaves it up to the test framework (or a reporting plugin for the framework; see #15) how to print the message and the comparison of expected vs actual.

Solution B: Retry Within Assertion

Cypress assertions do not actually fail until they've tried until a time limit is hit. Touchstone assertions could do likewise. There could be a default time limit for the test with a possibility to override it for individual assertions.

@ScottFreeCode ScottFreeCode changed the title Support Automatic Retry Assertions Automatic Retry w/Timeout (alternative to processing_period sleep) Mar 11, 2021
@shanejansen shanejansen added enhancement New feature or request and removed research needed labels Mar 26, 2021
@shanejansen shanejansen added this to the v2.0.0 milestone Mar 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants