Criterion C test framework : SIGSEGV fault only generated after all tests have passed

[nomennescio]

Not sure what is causing this behavior, might also be related to the way error messages are handled in the output pane.

During execution of code under test, which is accessing memory beyond the last element + 1 in an array (Undefined Behavior), ALL test cases pass successfully. However after the last test case is passed succesfully in a Test(_,_){} block, and an additional message is printed after the last test case, which is shown in the output, and seemingly at the point the Test block is exited, ONLY THEN an additional error message is printed. See the example below:

  sample_test("rAnDoM CaPs CrAzInEsS", "Random Caps Craziness");
  puts ("after last test");
  fflush (stdout);
}

yields

Test Passed
Log
after last test
Test Crashed
Caught unexpected signal: SIGSEGV (11). Invalid memory access.
Completed in 0.0000ms

I would expect the test framework to not pass all tests successfully, but generate the signal sooner, and fail the first test case that exhibits such undefined behavior. Else the user is very hard pressed to find an error in his code.

[FArekkusu]

The issue was found here.

To be exact, sample_test is a macro which calls a function similar this:

void do_one_test(char *input, char *expected) {
    char buffer[strlen(input)+1];
    user_solution(input, buffer);
    cr_assert_str_eq(buffer, expected);
}

user_solution is a function which copies elements from input to buffer (transforming them in the process) and mistakenly accesses elements past the terminating zero in the process. When a SIGSEGV is generated the process is not killed; in fact, at that point the buffer already contains the correct data, including the rightly copied terminating zero, so cr_assert_str_eq gets called, and the test passes.

You can have multiple tests, possibly each of them generating a SIGSEGV, but the error message about it will only appear at the end.

[nomennescio]

@FArekkusu thanks for the additional information. Weird that SIGSEGV is generated, but all tests still pass. Would like to have clarification on that. To me it seems better to have SIGSEGV terminate as soon as it is triggered, that would allow the user to pinpoint the failing testcase and/or failing inputs.

[kazk]

Can you create a minimal kumite to reproduce the issue?

[FArekkusu]

Here's a kumite based on the kata I linked.

My explanation was not entirely correct since the buffer is actually created outside do_one_test and then passed to it as an argument. If you do create the buffer inside the function as I said earlier, the SIGSEGV will correctly kill the process.