Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload metrics from PR CI as artifacts in GitHub Actions #74

Open
jdno opened this issue Aug 10, 2023 · 4 comments
Open

Upload metrics from PR CI as artifacts in GitHub Actions #74

jdno opened this issue Aug 10, 2023 · 4 comments
Labels
help wanted We would appreciate help from outside contributors.

Comments

@jdno
Copy link
Member

jdno commented Aug 10, 2023

We collect a lot of metrics about each CI run. In builds run by bors within the rust-lang-ci organization, we upload these metrics to S3 and make them available at https://ci-artifacts.rust-lang.org/rustc-builds/{run.sha}/metrics-{name}.json. This provides a great tool to analyze the efficiency of our builds and continually optimize them.

For builds run on pull requests in the public rust-lang organization, we don't upload the metrics to S3 because we don't want to add a token with write permissions to S3 to the public environment.

Instead of discarding the metrics after each run, we should upload them as artifacts in GitHub Actions. This will allows us to store them somewhere without having to grant the jobs any additional permissions, and download and analyze them similar to the metrics collected by bors.

@meysam81
Copy link

Hey @jdno,

I'd like to give this a try.

I realized you mentioned "rust-lang organization public repos". Does that imply that creating a workflow file in here is enough to resolve this requirement? (and then later on, using such workflow from other repositories with uses: rust-lang/simpleinfra/github-actions/example@v1?)

@Kobzol
Copy link

Kobzol commented Sep 22, 2023

Rather than a new workflow, I think that this should be added to the existing CI code. There is code at the end of src/ci/run.sh (IIRC) that uploads the metrics to S3. If we find that S3 is not available (or that we're in a PR CI run, I think that there is now a dedicated env variable that tells us that), the metrics file should instead be uploaded as a workflow artifact.

@Kobzol
Copy link

Kobzol commented Sep 22, 2023

Sorry, I actually misunderstood your comment, I thought that you wanted to create a new workflow in rust-lang/rust. So yeah, it could be a workflow in simpleinfra, and then it would be added to our CI workflow as another step after the run step. We have to double check that the metrics file is not deleted in run.sh though and that it's available to following GHA steps.

@jdno
Copy link
Member Author

jdno commented Oct 3, 2023

Sorry that the issue description wasn't very specific. We only really care about the metrics from rust-lang/rust. I'm not even sure if other repositories produce similar metrics during their builds.

We already have a step that uploads documentation from the pull request builds:

- name: upload artifacts to github
  uses: actions/upload-artifact@v3
  with:
    # name is set in previous step
    name: ${{ env.DOC_ARTIFACT_NAME }}
    path: obj/artifacts/doc
    if-no-files-found: ignore
    retention-days: 5
  <<: *step

I think that we just need to add another step that uploads the files that upload-artifacts.sh would upload as well. Since that is only required for pull requests, it should have an if condition that limits it to pull request events.

And since we only need a single step, I would avoid creating a new GitHub Action for this.

@Kobzol Kobzol added the help wanted We would appreciate help from outside contributors. label Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted We would appreciate help from outside contributors.
Projects
Status: Ready
3 participants