Add Support for Storing Previous Attempts of Test Run

[alexshamrai]

Description

Currently, the CTRF schema and implementation only track the number of retries for a test but do not store detailed information about previous test runs. This makes it difficult to analyze test flakiness and understand the history of test executions(during one test run)

Proposed Changes

Update the CTRF Schema
Add a previousAttempts field to the Test schema:

"previousAttempts": {
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "status": {
        "type": "string",
        "enum": ["passed", "failed", "skipped", "pending", "other"]
      },
      "duration": { "type": "integer" },
      "start": { "type": "integer" },
      "stop": { "type": "integer" },
      "message": { "type": "string" },
      "trace": { "type": "string" }
    },
    "required": ["status", "duration"]
  }
}

Example for the test in the resulting JSON:

{
  "name": "Flaky test passed on the second run",
  "status": "passed",
  "duration": 10,
  "start": 1683540000000,
  "stop": 1683540000010,
  "retries": 1,
  "flaky": true,
  "previousAttempts": [
    {
      "status": "failed",
      "duration": 7,
      "start": 1683539999000,
      "stop": 1683539999007,
      "message": "This test is designed to fail on the first run",
      "trace": "org.opentest4j.AssertionFailedError: This test is designed to fail on the first run..."
    }
  ]
}

[alexshamrai]

@Ma11hewThomas Looks like this request is redundant. If each re-run will be stored as a separate Test, there is no need to do this change in schema. I guess the discussion can be closed.