opa test --bench panics on failing test

[anderseknert]

Tested with both v0.70.0 and current main. While I think --bench is meant to work with successful tests only, it seems like the error handling is borked.

b.rego

package b

import rego.v1

x := true

test_x if {
    not x
}

> opa test b.rego --bench
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x10460281c]

goroutine 28 [running]:
testing.(*common).decorate(0x1400004fbe8?, {0x140003962d0, 0x29}, 0x1054439e0?)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:775 +0x9c
testing.(*common).logDepth(0x1400037fb88, {0x140003962d0, 0x29}, 0x3)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1042 +0x150
testing.(*common).log(...)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1011
testing.(*common).Fatal(0x1400037fb88, {0x1400004fee0?, 0x105670898?, 0x140003812d0?})
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1082 +0x50
github.com/open-policy-agent/opa/tester.(*Runner).runBenchmark.func1(0x1400037fb88)
	/Users/anderseknert/git/opa/opa/tester/runner.go:588 +0x36c
testing.(*B).runN(0x1400037fb88, 0x1)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:193 +0x128
testing.(*B).run1.func1()
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:215 +0x4c
created by testing.(*B).run1 in goroutine 26
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:208 +0x90

If I change the test to succeed, the command works as expected.

[ashutosh-narkar]

Looks like it's hitting this case.

[anderseknert]

Yes, but how does that turn into a panic / nil dereference? 🤔

[ashutosh-narkar]

I haven't checked the code but are there any other goroutines running?

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.