Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fails should be fails and not errors #9

Open
oxinabox opened this issue Jul 1, 2018 · 3 comments
Open

Fails should be fails and not errors #9

oxinabox opened this issue Jul 1, 2018 · 3 comments

Comments

@oxinabox
Copy link
Member

oxinabox commented Jul 1, 2018

Changing

error("new references can only be created in interactive mode ...)"

into

warn("new references can only be created in interactive mode ...)"
@test false

would do it.

But that makes testing the fail case harder.
One way to handle that is to test in a different process.
Which is what the code I was confused about in #4 did:

https://github.com/Evizero/ReferenceTests.jl/pull/4/files#diff-fce720c43af3c52c862fd7451c7374b8R26

@johnnychen94
Copy link
Member

But that makes testing the fail case harder.

If we're providing better render and by functions to capture differences, I think this won't be an issue. For example, always throw warnings to explain why the test fails.

@oxinabox
Copy link
Member Author

oxinabox commented Oct 5, 2019

Just to be clear that line is about how to test this package. not for using this package to rest others.
Though perhaps you are saying we could use the tools for checking the logger?

@johnnychen94
Copy link
Member

johnnychen94 commented Oct 11, 2019

If you ]test Test, you'll find several all-broken testsets.

Test Summary:          | Broken  Total
@test_broken with fail |      5      5
Test Summary:            | Broken  Total
@test_broken with errors |      2      2
Test Summary: | Broken  Total
@test_skip    |      4      4
@testset "@test_broken with fail" begin
    @test_broken false
    @test_broken 1 == 2
    @test_broken 1 != 1
    @test_broken strip("\t  hi   \n") != "hi"
    @test_broken strip("\t  this should fail   \n") == "hi"
end

I assume this is accepted as long as it's described clearly.

@johnnychen94 johnnychen94 mentioned this issue Jul 5, 2020
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants