Skip to content

Reaching max buffer size in Python can abort test early without failing the tests #178

@Voileexperiments

Description

@Voileexperiments

When Python runner has too much data printed to stdout (results in Max Buffer Size Reached (1.5 MiB) printed to stderr) inside one of the it blocks, sometimes the test aborts but exit code is not set to non-zero, so the test will pass.

import codewars_test as test

@test.describe("Example")
def test_group():
    @test.it("test case 1")
    def test_case_1():
        test.pass_()
    @test.it("test case 2")
    def test_case_2():
        print('a'*2000000)
        test.fail('Should not reach here')

This passes the test:
image

I can't reproduce similar behaviours in JS.

This seems to be related to the first few blocks and hence extremely sensitive to how tests are set up. For example, this works:

import codewars_test as test

@test.describe("Example")
def test_group():
    @test.it("test case 1")
    def test_case_1():
        test.pass_()
    for _ in range(2):
        @test.it("test case 2")
        def test_case_2():
            print('a'*2000000)
            test.fail('Should not reach here')

But this does not:

import codewars_test as test

@test.describe("Example")
def test_group():
    @test.it("test case 1")
    def test_case_1():
        test.pass_()
    for _ in range(3):
        @test.it("test case 2")
        def test_case_2():
            print('a'*2000000)
            test.fail('Should not reach here')

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @kazk@Voileexperiments

        Issue actions

          Reaching max buffer size in Python can abort test early without failing the tests · Issue #178 · codewars/runner