A better way to organize the repositories of tests.

[yangby-cryptape]

Goals

• Each revision of the codes has a corresponding revision of test codes.

  • If only the branch name is recorded, the old revisions of the codes will be broken.
    e.g. Do git bisect will be difficult.

• Each PR could pass all checks, even they have to update the tests.

  • If using workflow_dispatch to run checks for PR, then only workflow scripts in default branch will be executed.

    To trigger the workflow_dispatch event, your workflow must be in the default branch.
    Ref: GitHub Docs: Configuring a workflow to run manually

    So, even you update the workflow scripts, they won't work.
    That means that the revision information of tests should not put in workflow scripts.

• Group by meaningful topics.

  • Currently, the tests are tracked in 5 different repositories.

    • The 5 different repositories are:

      • axonweb3/axon-test
      • axonweb3/v3-core
      • axonweb3/axon-hardfork-test
      • axonweb3/openzeppelin-contracts
      • axonweb3/axon:tests/e2e

    • And, each repository is split into several parts without any meaning.

      • axonweb3/v3-core: runAll0, runAll1, runAll2.
      • axonweb3/openzeppelin-contracts: OCT 1-5, OCT 6-10 OCT 11 OCT 12-15 OCT 16-19.

    • And, each repository has its own command to execute the tests:

      • npm run test:pipeline${ID}-${NUM}
      • npx hardhat test --grep "${TETS_NAME}"
      • yarn test:runAll${ID}

  • Since the groups are split with no meaning, select CI checks for PR then input /run-ci is stupid, too.
    Because there are only two choices: choose none or choose all.

[sunchengzhu]

And, each repository is split into several parts without any meaning.

We have previously discussed that the openzeppelin-contracts are divided into different parts for parallel execution, primarily because the test cases take too long to run. Just for OCT 11, it takes about an hour.

runAll0, runAll1, runAll2 are also divisions made for parallel execution. Each 'runAll' executes its test cases in parallel. Due to limitations in machine resources, it's not feasible to run all test cases simultaneously, hence the need for grouping. Grouping also has another advantage. Currently, it's difficult to pinpoint problems solely based on workflow logs. Once runAll1 fails, we only need to execute the test cases in runAll1 locally to identify the issue, rather than needing to execute runAll2.

[yangby-cryptape]

We have previously discussed that the openzeppelin-contracts are divided into different parts for parallel execution, ... ...

I didn't say I wanted to let all tests ran in one step.

But current groups are meaningless.

When a test fails, do you know which component or feature may have been affected?

When I submit a PR and select which CI checks are required, I want to select them by component or feature, not by a meaningless number.

I want to run checks for mempool, for network or for consensus, not for part 3, part 8 or part 15.

---

As an example, can you tell me, when I should select OCT 6-10 only and when I should select OCT 1-5 And 12-15 only?

Do these options have any meaning for authors of PRs?

• OCT 1-5 And 12-15
• OCT 6-10
• OCT 11
• OCT 16-19

CI Description

CI Name	Description
OCT 1-5 | 6-10 | 11 | 12-15 | 16-19	Run the compatibility tests provided by OpenZeppelin