Grafana Mimir: Unit Test Failures on IBM Power machine ppc64le

[satyabratabharati]

Hello,

I am trying to build and perform unit tests on my local dev machine (IBM Power ppc64le)

During the unit tests I am getting multiple errors specific to following modules :
Mimir Ingester Error:

--- FAIL: TestIngesterWithShippingDisabledDeletesBlocksOnlyAfterRetentionExpires (2.92s)
    ingester_test.go:4370:
                Error Trace:    /go/src/github.com/grafana/mimir/pkg/ingester/ingester_test.go:4370
                Error:          Not equal:
                                expected: 3
                                actual  : 1
                Test:           TestIngesterWithShippingDisabledDeletesBlocksOnlyAfterRetentionExpires
level=info msg="upload new block" id=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors177436268/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors177436268/002"
level=error msg="shipping failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
FAIL    github.com/grafana/mimir/pkg/ingester   185.194s

However, it got fixed After increasing retention period as below:

Mimir Alert Manager Error:

it got it fixed After increasing the context timeout period as below:
pkg/alertmanager/multitenant_test.go

Mimir Compactor Error:

I am not quite sure about the reasons for these errors that I am getting during my unit tests.
Could someone kindly help me to understand the above errors and the possible reasons for the same.

Thanks.

[satyabratabharati]

Hi @pstibrany
could you please help me to understand the above errors and the possible reasons for the same.

Thanks.

[satyabratabharati]

Hi All,
@pstibrany

Also I see "jdb2/sda2-8" consumes too much I/O on my dev machine.

Thank You.

[pstibrany]

Hi. I don't know specifics of IBM Power ppc64le, or why tests would fail on it. Perhaps this machine is just too slow and some timeouts in the test may need to be higher. But it's just a first guess without any research from my side.

[charleskorn]

I've seen one of these tests (TestIngesterWithShippingDisabledDeletesBlocksOnlyAfterRetentionExpires) fail occasionally on my machine (a M1 MacBook), so I suspect it's not an architecture-specific problem.