Run e2e jobs as a single workflow

[vaidikcode]

Signed-off-by: vaidikcode [email protected]

Which problem is this PR solving?

• Task one of Implement framework to validate backwards compatibility of metrics #6278

Description of the changes

• Integrated all e2e workflows jobs into a single workflow

How was this change tested?

• CI

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.11%. Comparing base (526d36c) to head (a4b3fd2).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6310      +/-   ##
==========================================
- Coverage   96.14%   96.11%   -0.03%     
==========================================
  Files         356      356              
  Lines       20476    20476              
==========================================
- Hits        19686    19681       -5     
- Misses        603      607       +4     
- Partials      187      188       +1     

Flag	Coverage Δ
badger_v1	8.83% <ø> (ø)
badger_v2	1.62% <ø> (ø)
cassandra-4.x-v1-manual	14.73% <ø> (ø)
cassandra-4.x-v2-auto	1.56% <ø> (ø)
cassandra-4.x-v2-manual	1.56% <ø> (ø)
cassandra-5.x-v1-manual	14.73% <ø> (ø)
cassandra-5.x-v2-auto	1.56% <ø> (ø)
cassandra-5.x-v2-manual	1.56% <ø> (ø)
elasticsearch-6.x-v1	18.47% <ø> (+<0.01%)	⬆️
elasticsearch-7.x-v1	18.55% <ø> (+<0.01%)	⬆️
elasticsearch-8.x-v1	18.71% <ø> (+<0.01%)	⬆️
elasticsearch-8.x-v2	1.62% <ø> (+<0.01%)	⬆️
grpc_v1	10.29% <ø> (+<0.01%)	⬆️
grpc_v2	7.85% <ø> (ø)
kafka-v1	8.52% <ø> (ø)
kafka-v2	1.62% <ø> (ø)
memory_v2	1.62% <ø> (ø)
opensearch-1.x-v1	18.60% <ø> (ø)
opensearch-2.x-v1	18.60% <ø> (ø)
opensearch-2.x-v2	1.61% <ø> (-0.01%)	⬇️
tailsampling-processor	0.45% <ø> (ø)
unittests	95.02% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

I don't think we need crossdock and hotrod in this list. Definitely not crossdock, as it's a very special form of tests. Could go both ways on hotrod.

Is it possible to achieve this single workflow without massive copy pasting? For example, do a git move of the existing workflow files into a subfolder (e.g. ../actions/e2e/) and call them from the combined workflow?

[vaidikcode]

@yurishkuro Should I also exclude HotRod?

[yurishkuro]

Yes, let's exclude hotrod for now.

[yurishkuro]

you can check the validity of the new workflow on the Checks tab

https://github.com/jaegertracing/jaeger/actions/runs/12166601553

[yurishkuro]

try creating a dummy workflow first and referencing it like this

[vaidikcode]

i tried but the same error. It seems GitHub Actions expects workflows to be in the .github/workflows directory. How should I proceed with this now. Is the approach to add all the workflows as jobs in a single file ok or is there any other solution to it?

[yurishkuro]

the fact that it gives a specific error makes me think that the workflow inclusion is actually supported, that is why I am suggesting:

• try creating two dummy workflows A and B in the workflow dir
• make A include and run B
• check if the runner executes A successfully (with B), but standalone B is not executed

[yurishkuro]

And I would prefer you do these experiments in your own test repo, there is no need to run a battery of expensive jaeger tests while you don't have a solve for the workflows

[yurishkuro]

If including workflow in a workflow does not work, then the alternative is to define the jobs in a combined workflow and use the existing files as actions.

[vaidikcode]

I tested this in my test repository, and this approach works perfectly. I just needed to add workflow_call to the CI workflows. Thank you for the feedback!

[yurishkuro]

Q: instead of hardcoding strings like cit-badger, could we use some standard variables like workflow-name (which we already use) and job name?

[vaidikcode]

${{ github.workflow }}-${{ github.job }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }} you mean like this?

[vaidikcode]

Will this be the same for the combined CIT workflow as well?

[yurishkuro]

good question - what's the granularity of the concurrency settings? Do we need this setting on the combined workflow if individual jobs can cancel themselves as needed?

[vaidikcode]

We might not need concurrency at the combined workflow level unless there are scenarios where concurrent triggers could occur.

1. A push to the main branch and a PR to main happening simultaneously.
2. Manual trigger while a push event is in progress.
   Should we proceed with adding concurrency here, or leave it out for now?

[yurishkuro]

In those 2 scenarios what would concurrency on combine workflow achieve that won't be covered by the job-level settings? All we want is to cancel a job for a PR or a branch if a new version of the PR or branch is available, so as to not waste the runner resources.

[vaidikcode]

I was thinking along the same lines—are we agreeing that no concurrency at the combined workflow level is the approach here, relying instead on job-level concurrency to handle redundant jobs efficiently

[yurishkuro]

${{ github.workflow }}-${{ github.job }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }}

can you try & test this? If you already have a setup in a private repo, the test would be to start a workflow, open some of the runs in a tab, then push a new commit and observe those runs to be cancelled. Then repeat the same for a branch push instead of a PR.

[vaidikcode]

Tested both scenarios as discussed, with the following observations:

Scenario 1: PR with Multiple Commits
Initially, used concurrency at the combined level:
combined-cit-${{ github.workflow }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }}
along with reference-level concurrency:
${{ github.workflow }}-${{ github.job }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }}
This setup caused issues—some referenced workflows in the combined workflow were executed while others were cancelled, even on creating a simple PR.
Removed concurrency at the combined level and retained reference-level concurrency as:
cit-badger-${{ github.workflow }}-${{ github.job }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }}
With this, the workflows behaved as expected, and previous jobs were cancelled when new commits were pushed to the PR.
Scenario 2: Branch Push
Tested with no concurrency at the combined level and only the reference-level concurrency:
cit-badger-${{ github.workflow }}-${{ github.job }}-${{ (github.event.pull_request && github.event.pull_request.number) || github.ref || github.run_id }}
The previous jobs were successfully cancelled when new commits were pushed to the branch.
Observations
Keeping or removing concurrency at the combined level didn’t change the outcome, as long as the reference-level concurrency was properly configured.
For the tests to work reliably, the concurrency group needed a prefix like cit-badger or similar. Without this prefix, referenced workflows in the combined workflow might execute inconsistently (some running and others cancelling) even when just creating a PR.
Should we finalize this setup with no concurrency at the combined level and the prefix-based approach at the reference level? wdyt?

[yurishkuro]

Thanks!