Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a programmer, I want meaningful test failures in the systests #279

Open
epag opened this issue Aug 21, 2024 · 1 comment
Open

As a programmer, I want meaningful test failures in the systests #279

epag opened this issue Aug 21, 2024 · 1 comment

Comments

@epag
Copy link
Collaborator

epag commented Aug 21, 2024


Author Name: Jesse (Jesse)
Original Redmine Issue: 96860, https://vlab.noaa.gov/redmine/issues/96860
Original Date: 2021-09-29


Given an execution of systests
When a systest fails
Then I want the message to be meaningful
So that I can fix the code or fix the test


Redmine related issue(s): 97014, 101825


@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Jesse (Jesse)
Original Date: 2021-09-29T15:00:39Z


Example to the contrary:

junit.framework.AssertionFailedError: Comparison with benchmarks failed with code 48. expected:<0> but was:<48>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.TestCase.assertEquals(TestCase.java:401)
	at wres.systests.ScenarioHelper.assertOutputsMatchBenchmarks(ScenarioHelper.java:188)
	at wres.systests.Scenario009.testScenario(Scenario009.java:82)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

What is good about this is the "Comparison with benchmarks failed."

What is lacking is "with code 48."

The origin of the "code 48" is a transliteration of the bash assertions in the bash systests into JUnit.

Some improvements that could be made:

  • Print the names of the blobs that were different
  • Print an example difference
  • Eliminate the clunky "code" since that was a result of bash not having anything equivalent to JUnit's facilities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant