Skip to content

Commit 9a2d573

Browse files
InfinoidInfinoid
committed
Add test-pr.py and associated docs
1 parent ad13006 commit 9a2d573

File tree

2 files changed

+240
-0
lines changed

2 files changed

+240
-0
lines changed

‎ci/test-pr.md

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# CUDA testing
2+
3+
The script below, `test-pr.py`, is a quick and easy way to run CUDA tests on a PR.
4+
5+
This should be done after code review, see [issue #457](https://github.com/tensor-compiler/taco/issues/457) for a discussion of the overall process.
6+
7+
## System requirements
8+
9+
You will need write access to the `tensor-compiler/taco` github repo, and access to run actions.
10+
11+
You will need to have the command line `git` and `gh` tools installed, configured, and talking to github.
12+
13+
`git` needs to be set up with SSH authentication or HTTPS authentication to push to the `tensor-compiler/taco` github repo without a password prompt. If you don't have that, please see [ssh setup in the github docs](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) or [seting up an HTTPS token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token).
14+
15+
`gh` is a command-line interface to the github REST API. If you don't have `gh`, please see [their installation guide](https://github.com/cli/cli#installation).
16+
17+
`gh` needs to be set up to hit REST API endpoints without a password prompt. If you don't have that, the [gh auth login](https://cli.github.com/manual/gh_auth_login) command should help.
18+
19+
The script itself is `test-pr.py`, you can grab that below.
20+
21+
The script requires python version 3.x, and a few basic python modules that should be installed by default (like subprocess and tempfile).
22+
23+
## What it does
24+
25+
This script does the following:
26+
27+
* creates a temporary test branch for a PR
28+
* pushes that test branch to the taco github repo
29+
* kicks off the cuda test (with the default parameters) to run on that test branch
30+
* waits for the test to complete
31+
* removes the temporary test branch
32+
33+
The script will give you a link to the test run on the github website, so you can watch and inspect the results.
34+
35+
## Using it
36+
37+
`python3 test-pr.md [--protocol=(ssh|https)] <PRNUMBER> [<TRYNUMBER>]`
38+
39+
`PRNUMBER` is the PR you want to test, without the `#` prefix.
40+
41+
If specified, `TRYNUMBER` becomes a suffix for the temporary test branch, so you can have multiple test branches going at once. That can be omitted unless it is needed.
42+
43+
## what it looks like
44+
45+
This output comes from my own fork of taco as I was testing the script:
46+
47+
```
48+
% python3 test-pr.py 3
49+
50+
=== looking up ID and params of test action
51+
52+
=== creating test branch test-pr3
53+
remote:
54+
remote: Create a pull request for 'test-pr3' on GitHub by visiting:
55+
remote: https://github.com/Infinoid/taco/pull/new/test-pr3
56+
remote:
57+
58+
=== triggering test action
59+
✓ Created workflow_dispatch event for cuda-test-manual.yml at test-pr3
60+
61+
To see runs for this workflow, try: gh run list --workflow=cuda-test-manual.yml
62+
Test action is at: https://github.com/Infinoid/taco/actions/runs/882846018?check_suite_focus=true
63+
64+
=== waiting for action to complete
65+
66+
Refreshing run status every 3 seconds. Press Ctrl+C to quit.
67+
68+
X test-pr3 CUDA build and test (manual) · 882846018
69+
Triggered via workflow_dispatch about 11 minutes ago
70+
71+
JOBS
72+
X tests CUDA in 10m54s (ID 2686889157)
73+
✓ Set up job
74+
✓ Run actions/checkout@v2
75+
✓ create_build
76+
✓ cmake
77+
✓ make
78+
X test
79+
✓ Post Run actions/checkout@v2
80+
✓ Complete job
81+
82+
ANNOTATIONS
83+
X Process completed with exit code 2.
84+
tests CUDA: .github#1
85+
86+
87+
X Run CUDA build and test (manual) (882846018) completed with 'failure'
88+
Test results are at: https://github.com/Infinoid/taco/actions/runs/882846018?check_suite_focus=true
89+
90+
=== cleaning up test branch test-pr3
91+
92+
=== cleaning up temp dir
93+
94+
```
95+
96+
# Troubleshooting
97+
98+
## no access to taco repo, no access to run actions in taco repo
99+
100+
If you are a taco developer, ask Fred for access.
101+
102+
## test workflow does not exist
103+
104+
If you see output that looks like this:
105+
106+
```
107+
=== triggering test action
108+
could not create workflow dispatch event: HTTP 422: Workflow does not have 'workflow_dispatch' trigger (https://api.github.com/repos/tensor-compiler/taco/actions/workflows/someIDnumber/dispatches)
109+
```
110+
111+
This means the test branch does not have the cuda test workflow file. In other words, the file `.github/workflows/cuda-test-manual.yml` does not exist yet in the version of taco that the PR is based on.
112+
113+
To fix this, rebase or merge the PR to the current taco master branch, and then rerun the test.
114+
115+
## other stuff
116+
117+
If you have some other problem with it, at me on github (@infinoid) or email me for help figuring it out.

‎ci/test-pr.py

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env python3
2+
3+
# This script runs the TACO CUDA tests on a pull request.
4+
# Usage: python3 test-pr.py [--protocol=(https|ssh)] <PRnumber> [<Trynumber>]
5+
6+
# see test-pr.md for more details.
7+
8+
repo = "tensor-compiler/taco"
9+
action_name = "CUDA build and test (manual)"
10+
11+
import os
12+
import subprocess
13+
import sys
14+
import tempfile
15+
import time
16+
17+
def main():
18+
args=list(sys.argv[1:])
19+
repo_url = "https://github.com/" + repo
20+
try:
21+
if len(args) > 0 and args[0].startswith('--protocol='):
22+
protocol = args[0][11:]
23+
args = args[1:]
24+
if protocol == 'ssh':
25+
repo_url = "ssh://[email protected]/" + repo
26+
elif protocol == 'https':
27+
repo_url = "https://github.com/" + repo
28+
else:
29+
raise Exception("unknown protocol " + protocol)
30+
pr=args[0]
31+
pr = int(pr)
32+
attempt = None
33+
if len(args) > 1:
34+
attempt = args[1]
35+
attempt = int(attempt)
36+
except:
37+
print("Usage: {} [--protocol=(ssh|https)] <prnumber> [attemptnumber]".format(sys.argv[0]))
38+
exit(1)
39+
40+
print("\n=== looking up ID and params of test action")
41+
workflowid = find_workflow_id()
42+
43+
with tempfile.TemporaryDirectory() as tmpdir:
44+
#print("tmpdir is", tmpdir)
45+
branchname = "test-pr{}".format(pr)
46+
if attempt is not None:
47+
branchname += "-try{}".format(attempt)
48+
49+
print("\n=== creating test branch", branchname)
50+
subprocess.run(["git", "clone", "-q", repo_url, "git"], stdout=subprocess.DEVNULL, cwd=tmpdir, check=True)
51+
gitdir=os.path.join(tmpdir, "git")
52+
#print("gitdir is", gitdir)
53+
54+
subprocess.run(["git", "fetch", "-q", "origin", "pull/{}/head:{}".format(pr, branchname)], stdout=subprocess.DEVNULL, cwd=gitdir, check=True)
55+
56+
subprocess.run(["git", "checkout", "-q", branchname], stdout=subprocess.DEVNULL, cwd=gitdir, check=True)
57+
58+
subprocess.run(["git", "push", "-q", "--set-upstream", "origin", branchname], stdout=subprocess.DEVNULL, cwd=gitdir, check=True)
59+
60+
print("\n=== triggering test action")
61+
old_job_id = find_latest_workflow_run(workflowid)
62+
subprocess.run(["gh", "workflow", "run", "-R", repo, action_name, "-r", branchname], check=True)
63+
job_api_id = None
64+
while(job_api_id is None):
65+
# it takes a moment for new run requests to show up in the API.
66+
try:
67+
job_api_id, human_url = find_workflow_run(workflowid, branchname, later_than=old_job_id)
68+
except:
69+
time.sleep(1)
70+
71+
print("\nTest action is at:", human_url)
72+
73+
print("\n=== waiting for action to complete")
74+
time.sleep(10)
75+
#subprocess.run(["gh", "run", "watch", "-R", repo, str(job_api_id)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
76+
subprocess.run(["gh", "run", "watch", "-R", repo, str(job_api_id)])
77+
78+
print("\nTest results are at:", human_url)
79+
80+
print("\n=== cleaning up test branch", branchname)
81+
subprocess.run(["git", "push", "-q", "origin", "--delete", branchname], stdout=subprocess.DEVNULL, cwd=gitdir, check=True)
82+
83+
print("\n=== cleaning up temp dir")
84+
return
85+
86+
87+
def find_workflow_id():
88+
result = subprocess.run(["gh", "workflow", "view", "-R", repo, action_name], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, check=True)
89+
output = str(result.stdout, "utf-8")
90+
lines = output.split("\n")
91+
if lines[1].startswith("ID: "):
92+
workflowid = lines[1][4:]
93+
return int(workflowid)
94+
raise Exception("cannot find test workflow with name {}".format(action_name))
95+
96+
def find_latest_workflow_run(workflowid):
97+
output = subprocess.run(["gh", "run", "list", "-R", repo], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, check=True)
98+
output = str(output.stdout, "utf-8")
99+
lines = output.split("\n")
100+
line = lines[0]
101+
try:
102+
status, result, title, workflow, ref, origin, elapsed, runid = line.split("\t")
103+
return runid
104+
except:
105+
pass
106+
return None
107+
108+
def find_workflow_run(workflowid, branch, later_than=None):
109+
output = subprocess.run(["gh", "run", "list", "-R", repo, "-w", str(workflowid)], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, check=True)
110+
output = str(output.stdout, "utf-8")
111+
lines = output.split("\n")
112+
for line in lines:
113+
try:
114+
status, result, title, workflow, ref, origin, elapsed, runid = line.split("\t")
115+
if later_than is not None and later_than >= runid:
116+
continue
117+
if ref == branch:
118+
return runid, "https://github.com/{}/actions/runs/{}?check_suite_focus=true".format(repo, runid)
119+
except:
120+
pass
121+
raise Exception("could not find workflow run for branch {}".format(branch))
122+
123+
main()

0 commit comments

Comments
 (0)