? Access to CI-job results for non-VMware contributors.

[gapisback]

HI, @rosenhouse , @rtjohnso -- I am going thru the motions of what it would take to be a non-VMware contributor working in this OSS repo.

As we discussed internally earlier, non-VMware contributors do not have access to the CI-job reports, or access to the Nimbus-VMs used to run these jobs. (E.g. this link will fail to open from a personal non-VMware machine.)

I am trying to foresee the dev workflow in such a situation. If everything passes, that's good. If something fails, what are the plans / mechanisms to make those failures be known to the code authors?

Right now, there may not be much of a non-VMware volume of work to contend with. Still, just asking -- if something simple can be done to let these failure results be emailed to the contributor, so at least they have some errors / diagnostics to look at, to figure out the source for failures.

[ Cc: @ajhconway -- have you tried to do any code work in this repo in your new work situation? What's been your experience, and what are your thoughts on addressing improving this issue?

Any thoughts?

[rosenhouse]

This is a fair request, that we didn't have reason to prioritize earlier. There were good reasons (cost, security, existing expertise within the team) for choosing the setup that we did.

As I'm no longer working on SplinterDB for my day job, I don't have much to offer here in terms of a path forward.

[gapisback]

HI, @rosenhouse My recent draft-PR #616 ran into this issue, and so I feel compelled to find a solution. See below for some ideas I'm sketching out to see if those approaches are viable.

[gapisback]

Sketch of a proposed solution for this CI-infrastructure item:

1. GitHub seems to have support for Actions and Workflows. And Notifications is the way email seems to go out. Are these CI-pieces something we could leverage in our Nimbus-VM based CI runs?

2. This post shows some examples for how-to send out a brief email when a CI-job fails. The scripting involved looks very familiar to the collection of *.yml tooling that Gabe has put down in our CI. Is this an approach we could leverage and integrate into our CI ci/steps.lib.yml script?

Above post shows an example of only sending out a brief pass / fail mail, with brief contents in the body of the mail. What we need is a way to push out the output from test.sh so off-line triaging can happen to some extent.

Here is what I'm thinking we can do to achieve that:

1. Rename current test.sh to run_tests.sh.

2. Create a new wrapper script test.sh that will call run_tests.sh and output gets |tee'ed to some output file.

3. In this wrapper, if the sub-script exits with non-zero $rc hook into the emailing facility, shown by previous post, to email out the output being tee'ed to some /tmp/test.$$.out file.

4. The thing to figure out is this piece (from above example): to: [email protected],[email protected]. We need a way to tie-in the To: address to the Github notification email address that has been setup for the login that is running this CI-job .

There may be some more CI-workflows-, notifications gotchas to work through, but something like above machinery seems like it should work.

@rosenhouse -- Can you please give this a run in your head and vet this approach? thanks!

[rosenhouse]

When we first set up CI for this project, GitHub Actions were not a viable option due to budget issues. At the time, the vmware org here on GitHub that this repo belongs to was on a legacy subscription which had limited quota available for GitHub Actions runner time, and that quota was already in contention by other projects in this org. Changing the subscription model for the org was not an option at the time, and hosting our own runner was deemed a worse choice than using the managed Runway service that we ultimately chose.

Some of these factors may have changed, but I am not sure and it would take some time to run-down and re-evaluate options. I don't have the bandwidth right now to drive this kind of effort. I'm sorry.

[rtjohnso]

I would like to switch us to using github's infra for CI.

I have forked splinterdb and written a github workflow for that purpose. You can see it here:
https://github.com/rtjohnso/splinterdb

To get it running, I had to fix a few bugs in splinter and also figure out how to get enough disk space cobbled together on the github runner machines to hold our databases. I also cleaned up our testing script to not leave databases lying around, and I wrote the workflow to try to minimize the number of minutes that we use. With this workflow, we have ample space for our tests (over 100GB of disk space, whereas our tests currently use around 25GB), so I think this is a sustainable setup for the future, as well.

If you guys like it, then I propose the following:

• We switch the vmware/splinterdb repo to use this workflow for CI. I'm hopeful that, under broadcom, there may be a lot fewer open-source projects vying for the same quota of minutes.
• But, if we run into quota issues, then we use my fork as a gateway repo, i.e. we do all work on the fork and push completed PRs to the vmware/splinterdb repo when they are ready to be merged into main. My fork has unlimited minutes, so won't have any quota problems. We would essentially turn off all CI on the vmware repo, since it would all be outsourced to my fork.

We could just fork the repo and abandon the vmware/splinterdb repo, but I prefer to try to keep the vmware branding.

Thoughts?

[gapisback]

@rtjohnso -- Thanks for trying this out. I am not sure I fully understand what the proposed new workflow will be.

You said: switch the vmware/splinterdb repo to use this workflow for CI: Does this mean that my in-flight PRs that are currently active will remain as they are, but some Git-magic will be done to trigger this CI workflow? If yes, that's fine by me.

I would like to see your "CI workflow" activated in the current vmware/splinterdb repo, so I can confirm if I can see the results of CI-jobs for an in-flight open PR of mine. If these results are visible, then this workflow is most optimal.

I did go thru the commits done to your fork but am not seeing any CI-tests run or PRs here. So, if we do to the workflow described in your 2nd bullet, will I be able to see the results of CI-jobs for any PRs submitted to this forked repo? If yes, that's fine. If I am not able to see the results, then we are back to square-One.

If we do run into quota issues on existing repo, and we have to resort to the workaround you describe in the 2nd bullet, that would be ok. But not very efficient. I am afraid that it may impose an extra run-time tax on doing dev across these two repos, which ... slows things down, creates potential for code lines going out-of-sync, is an annoyance in the dev workflow etc. (That scheme would be my least preferred solution.)

[rtjohnso]

Alright, since the vmware ci infrastructure appears to be non-functional again, I went ahead and switched over to the new CI system.

To try it out, do a dummy push to your PR (e.g. pull into your branch the changes that I just made to main).

[gapisback]

HI, @rtjohnso -- I am coming back to this thread to see how this new CI system works out.

On my Linux-VM (on a Mac), I rebased this branch, , off of /main.
But am unable to push this to github remote:

agurajada-Linux-Vm:[360] $ git push origin agurajada/shmem-free-list-mgmt-Rev7
ERROR: Permission to vmware/splinterdb.git denied to gapisback.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

I played around with uploading SSH keys but it did not seem to fix this problem.
(Went thru the troubleshooting steps doc'ed in this and this notes.)

I am able to push other private branches on other repos that my login gapisback is added as a contributor in github.

Has anything changed with SplinterDB's membership? Has authorization been revoked for my login gapisback from this repo that is preventing me from pushing to github?

I do see my login listed in the Contributors page, so it's a mystery why my git push is failing.

Can you or maybe @rosenhouse help to figure out what could be going wrong, here?

[gapisback]

HI, @rtjohnso - Coming back to this Gh access issue. There are some pending issues / code-items in my queue that I'd like to either complete, or close out, with some agreement from your side.

If you have admin rights to this SplinterDB repo, can you please try to invite GH-login gapisback as a contributor to this repo?

I had to get someone else do the same thing for another repo [ certifier-framework ] so that I could git push a dev-branch from my Mac to that repo.

(I got an email stating: yelvmw invited you to ccc-certifier-framework/certifier-framework-for-confidential-computing)

Maybe the same solution needs to be applied to SplinterDB repo ???

[rtjohnso]

Sorry for the delay getting it done. Invite sent.