Skip to content

Conversation

@LucaDai
Copy link

@LucaDai LucaDai commented Nov 11, 2025

Description

Type of change

  • New feature
  • Bug fix for existing feature
  • Code quality improvement
  • Addition or Improvement of tests
  • Addition/Improvement of documentation

Summary

The flaky behavior was detected in org.zalando.riptide.compatibility.HttpOperationsTest#shouldPut when running under NonDex randomization, which revealed that the test relied on deterministic field ordering in the serialized JSON request body.

Example of the original test assertion:

verifyRequestBody(recordedRequest, "{\"name\":\"D. Fault\",\"birthday\":\"1984-09-13\"}");

However, the JSON serialization of request bodies can vary in key ordering depending on the underlying Map or serializer implementation used within Jackson.
Since JSON field order is not guaranteed by the specification, different JVM runs or environments may produce:

Run 1: {"name":"D. Fault","birthday":"1984-09-13"}
Run 2: {"birthday":"1984-09-13","name":"D. Fault"}

The helper method verifyRequestBody(...) was asserting equality on raw JSON strings:

assertEquals(expectedBody, recordedRequest.getBody().readString(UTF_8));

This made the test order-sensitive to the serialization order of JSON fields, even though the semantic data was identical.

Failure Message running NonDex:

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   HttpOperationsTest.shouldExchange:303->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldExecute:288->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldExecute:288->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldExecute:288->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldPostForLocation:143->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldPostForLocation:143->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldPostForObject:168->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldPut:190->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[ERROR]   HttpOperationsTest.shouldPut:190->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>
[INFO] 

Fix

The test verification was updated to perform semantic comparison instead of string equality:
• Both the expected and actual JSON strings are parsed into maps using Jackson’s ObjectMapper.readValue().
• The test now asserts equality on these parsed Map<String, String> objects, which ignores field order differences.
• The content type check (application/json) remains intact to preserve original behavior.

Updated code excerpt:

Map<String, String> expected = MAPPER.readValue(expectedBody, new TypeReference<>() {});
Map<String, String> actual = MAPPER.readValue(requestBody, new TypeReference<>() {});
assertEquals(expected, actual);

This ensures deterministic, logically equivalent test validation across environments and JVMs, eliminating the order-dependent flaky behavior.


Related Tests

org.zalando.riptide.compatibility.HttpOperationsTest#shouldExchange
org.zalando.riptide.compatibility.HttpOperationsTest#shouldExecute
org.zalando.riptide.compatibility.HttpOperationsTest#shouldPatchForObject
org.zalando.riptide.compatibility.HttpOperationsTest#shouldPostForLocation
org.zalando.riptide.compatibility.HttpOperationsTest#shouldPostForObject
org.zalando.riptide.compatibility.HttpOperationsTest#shouldPut

How to Reproduce

Reproduced the failure using NonDex, a tool from the University of Illinois designed to detect ID tests (Iteration-Dependent tests).

Build module and Run a single test (no shuffling)
All tests pass.

Run with NonDex (shuffling)

cd riptide-compatibility
mvn -T1 edu.illinois:nondex-maven-plugin:2.2.1:nondex \
  -Denforcer.skip=true \
  -Dtest=org.zalando.riptide.compatibility.HttpOperationsTest \
  -DnondexRuns=10 \
  -DnondexMode=FULL \
  -DfailIfNoTests=false \
  -Dmaven.test.failure.ignore=true

Expected Example

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   HttpOperationsTest.shouldExchange:303->verifyRequestBody:347 expected: <{"name":"D. Fault","birthday":"1984-09-13"}> but was: <{"birthday":"1984-09-13","name":"D. Fault"}>

Verification

  • ✅ test passes.
  • ✅ Multiple NonDex runs (-DnondexRuns=100) pass with no flakes.
  • ✅ No behavior change.

@LucaDai LucaDai changed the title Fix flaky test in HttpOperationsTest Fix nondeterministic failures in HttpOperationsTest Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant