syz-cluster: initial code #5620
Conversation
a-nogikh
force-pushed
the
features/syz-cluster
branch
from
dbf79ae to
0a8c2df
Compare
| var patchSubjectRe = regexp.MustCompile(`\[(?:(?:rfc|resend)\s+)*patch`) | ||
| type PatchSubject struct { | ||
| Title string | ||
| Tags []string // Sometimes there's e.g. "net" or "next-next" in the subject. |
There was a problem hiding this comment.
These tags are parsed, but lost later. What's the plan here?
Do we want to not test "RFC" patches? Then these should set Corrupted?
| Total: 1, | ||
| Version: 1, | ||
| } | ||
| if patch.Total.IsSet() { |
There was a problem hiding this comment.
std::optional has value_or method, may be handy here as well.
| Subject string | ||
| MessageID string | ||
| Version int | ||
| Total int |
There was a problem hiding this comment.
Do we need it here? It's len(Patches) unless it's corrupted, right?
| @@ -20,9 +22,29 @@ type Thread struct { | |||
| Messages []*email.Email | |||
| } | |||
|
|
|||
| type Series struct { | |||
| Subject string | |||
There was a problem hiding this comment.
Does it happen that the subject changes between versions?
If so, do we need to also extract in-reply to message-id, which may be the previous version?
| @@ -672,3 +567,142 @@ func ParseGitDiff(patch []byte) []string { | |||
| } | |||
| return files | |||
| } | |||
|
|
|||
| type GitWrapper struct { | |||
There was a problem hiding this comment.
I would call this type Git and rename git to something else. The exported type is more important.
| Dir string | ||
| Precious bool | ||
| Sandbox bool | ||
| Env []string |
There was a problem hiding this comment.
Can we unexport some of these fields if they are used only within this package?
| ignoreCC map[string]bool | ||
| } | ||
|
|
||
| func (gw GitWrapper) Git(args ...string) ([]byte, error) { |
There was a problem hiding this comment.
For exported version it may be more reasonable to call this Run or Exec.
a-nogikh
force-pushed
the
features/syz-cluster
branch
from
0a8c2df to
3ef5a48
Compare
Support filtering by the commit date.
Refactor the code to make it more reusable. Add a method to extract specifically the list of new patch series.
It will let other parts of the system to use the git-specific functionality.
The basic code of a K8S-based cluster that: * Aggregates new LKML patch series. * Determines the kernel trees to apply them to. * Builds the basic and the patched kernel. * Displays the results on a web dashboard. This is a very rudimentary version with a lot of TODOs that provides a skeleton for further work. The project makes use of Argo workflows and Spanner DB. Bootstrap is used for the web interface. Overall structure: * syz-cluster/dashboard: a web dashboard listing patch series and their test results. * syz-cluster/series-tracker: polls Lore archives and submits the new patch series to the DB. * syz-cluster/controller: schedules workflows and provides API for them. * syz-cluster/kernel-disk: a cron job that keeps a kernel checkout up to date. * syz-cluster/workflow/*: workflow steps. For the DB structure see syz-cluster/pkg/db/migrations/*.
a-nogikh
force-pushed
the
features/syz-cluster
branch
from
3ef5a48 to
2a65760
Compare
The basic code of a k8s-based cluster that:
This is a very rudimentary version with a lot of TODOs that provides a skeleton for further work.
The project makes use of Argo workflows and Spanner DB.
Bootstrap is used for the web interface.
Overall structure:
syz-cluster/dashboard: a web dashboard listing patch series and their test results.syz-cluster/series-tracker: polls Lore archives and submits the new patch series to the DB.syz-cluster/controller: schedules workflows and provides API for them.syz-cluster/kernel-disk: a cron job that keeps a kernel checkout up to date.syz-cluster/workflow/*: workflow steps.For the DB structure see
syz-cluster/pkg/db/migrations/*.TODO (for this PR):
go.modbumped to1.23.1syz-clustertests on GitHub CI (we'll need at least some local Spanner emulator binary).Questions/thoughts out loud:
1. What to do with the vendor folder
Our
vendor/folder is quite big, and this PR adds even more modules on top of that. Do we still want to keep that code in our repostory? Is it possible to only keep the modules needed by other components but notsyz-cluster(since I dogo mod downloadin the Docker containers anyway).2. Some pkg/ packages are too eager to depend on prog/ and sys/
It has made some Dockerfiles more complicated that they could have been.
Should we strive to remove these dependencies?