ci artifacts cleanup and buildlogs

[jasonbrooks]

The atomic sig artifacts directory (https://ci.centos.org/artifacts/sig-atomic/) for the centos ci could use a cleanup, most of what's in here is either no longer needed, or will be regenerated when tests run. We are, however, distributing some items from this server that'd be better distributed from a location like buildlogs.centos.org.

@cgwalters would you weigh in on the sort of sync-with-buildlogs / automated cleaning regime that'd work for the items you're responsible for, like the continuous and fedora workstation stuff?

For my part, the downstream folder is what I typically consult, to try out test builds of that release. That particular folder needn't live longer than two weeks at a time.

[cgwalters]

I'd like to look at moving it to Google Cloud Storage most likely (as opposed to S3 since it supports HTTP2).

This would be part of more of a push for moving the frontend and orchestration into the OpenShift CI cluster, we'd just use Duffy as a bare metal compute cluster.

[jasonbrooks]

What about http://buildlogs.centos.org/? In the interim, which directories should we not delete, as far as you're concerned?

[cgwalters]

Hm, it'd be a bit easier with a per-directory analysis of what's taking up space. For a non-admin this is hard to do without syncing all the data over rsync.

In the end, the most important is the https://ci.centos.org/artifacts/sig-atomic/centos-continuous/ostree/ directory. Second is https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images/{cloud,installer} (but not the others).

Most other stuff we could nuke.

[jasonbrooks]

We don't need to get down super small, it's more a matter of bstinson asking us to identify what we don't need before it gets out of hand.

And I figured this would be a good time to discuss moving things like the continuous repo and images to a better location for distribution, and buildlogs is what KB and Brian have been suggesting for that.

[bstinsonmhk]

Here are the toplevel sizes as of about 10 minutes ago:

 5.2G    sig-atomic/20160818
 7.6G    sig-atomic/atomic-ws
 4.5G    sig-atomic/build0_20150617_01
 2.9G    sig-atomic/build_090915
 5.1G    sig-atomic/build13_20150520_2306_01
 4.4G    sig-atomic/build160223
 2.4G    sig-atomic/build_20151001
 32G     sig-atomic/centos-continuous
 6.6G    sig-atomic/devel
 13G     sig-atomic/downstream
 4.8G    sig-atomic/downstream.201606
 7.7G    sig-atomic/fedora-workstation
 0       sig-atomic/rdgo
 0       sig-atomic/srv
 99G     total

We aren't particularly concerned about chasing a size target for you folks, but we want to be sure we're pruning out what we can as you and other CI projects grow.

[cgwalters]

I think an initial pruning target would be the random datestamped things. Also you can nuke fedora-workstation immediately. Or actually, I guess I can, though AFAIK it's a bit awkward to do this with rsync.
I'm executing this right now:

mkdir empty
rsync -v --delete -Hrlpt empty/ [email protected]::sig-atomic/fedora-workstation/

[jasonbrooks]

Also, I don't think sig-atomic/devel is active currently, and I'm not sure if sig-atomic/downstream builds up extra gunk over time, but anything I need from there is regenerated during the jenkins job attached to it.