Skip to content

Criterion (C/NASM): stdout output are not synchronized #74

@Voileexperiments

Description

@Voileexperiments

Let's say I have this as the test block:

image

The stdout output by the expect and printf statements are always in a race condition. So depending on the timing one of these will happen:

image
image
image

Activity

kazk

kazk commented on Apr 22, 2018

@kazk
Member

Unfortunately, I don't think we can do anything about this. The same thing will happen with C.
We use Criterion's hooks to report in Codewars format on certain events, but outputs from test cases are not handled there (those are directly written on its own).

Criterion has logging functions like criterion_info, but we can't use them because the test needs to be invoked with -q to disable its normal output and this flag also disables logging.


The new runner added support for .description for better test case name, but it doesn't help in this case.

#include <criterion/criterion.h>
int add(int, int);
Test(add_test, should_add_integers, .description = "should add integers") {
  cr_assert_eq(2, add(1, 1));
}
changed the title [-]NASM: stdout output are not synchronized[/-] [+]Criterion (C/NASM): stdout output are not synchronized[/+] on May 8, 2018
nomennescio

nomennescio commented on Jul 3, 2019

@nomennescio

This is a big setback for C/Nasm Katas, as these usually involve pointer manipulation, which can easily lead to non-catchable exceptions being thrown by the runtime environment (segmentation faults etc.). For a user who cannot adapt the test code (except maybe for the code in the Sample Tests), this makes it very difficult to hunt down any error when trying to debug.

Any suggestion as how to improve upon this situation, given the current test runners?

nomennescio

nomennescio commented on Jul 15, 2019

@nomennescio

@kazk would it help to change the option in the Criterion framework? I could help in making a change in the framework, if it gets accepted, but only if that would get incorporated in Codewars. Or maybe the 'verbosity' flag can be used instead of the '-q' flag?

kazk

kazk commented on Jul 15, 2019

@kazk
Member

First problem is that the version we currently use doesn't have those macros (#620).
We're stuck on 2.2.x because 2.3.x was significantly slower (existing kata timeouts) with --full-stats flag which we need to report passed tests. I'll try if the latest 2.3.3 changed anything.

I'll also experiment with options.

kazk

kazk commented on Sep 4, 2019

@kazk
Member

So I tried the latest Criterion 2.3.3 and the performance issue seems to be fixed. I think the issue I ran into was Snaipe/Criterion#248.

I'll test against existing kata and see, but we can update to 2.3.3 if it is fixed. This version has macros like cr_log_info. Unfortunately, I tried running tests without -q and found that the outputs from these macros goes to stderr so it doesn't help us for this issue. I don't think there's any option to use stdout for logging and even if it did, I doubt it'll be synchronized because the Codewars output is produced using hooks and it's not available there.

One last possibility I can think of is by configuring the internal test runner. Maybe implementing custom logger and using that instead of the hooks will work.


This is a big setback for C/Nasm Katas, as these usually involve pointer manipulation, which can easily lead to non-catchable exceptions being thrown by the runtime environment (segmentation faults etc.). For a user who cannot adapt the test code (except maybe for the code in the Sample Tests), this makes it very difficult to hunt down any error when trying to debug.

I agree it can be super difficult, but I don't think having the output synchronized will help if the program crashes so badly during test. Criterion does run each test in isolated process and protects to a certain extent. I don't know how far you can go in languages like C and NASM.

nomennescio

nomennescio commented on Sep 3, 2021

@nomennescio

Encountered again a usecase where cr_log_info (or a similar custom function) would be very helpful to the Kata author/translator; the Criterion framework is not bad at all, and quite comfortable, except for its fixed tests. In some situation you want to repeat a test, calling a function with an argument (e.g. testing against a set of input integers) to execute multiple subtests per value, It would be very helpful to be able to group subtests for one argument, showing the value to the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @kazk@Voileexperiments@nomennescio

        Issue actions

          Criterion (C/NASM): stdout output are not synchronized · Issue #74 · codewars/runner