Improve tar packaging

[nolange]

This introduces the normal practise of not storing personal uid/gid in the archives. tar does a different thing with them whether you unpack as root or another user.

Further the timestamp variations are useless noise, the second commit will normalize timestamps used by compiler/toolchain as well as the tar archive.

[mstorsjo]

Thanks, this is much appreciated! A couple minor detail points and questions.

For the general tar cleanup, it'd be nice to apply that on the older release build packaging as well, see release.sh and release-macos.sh. There's probably no need to go all the way with fetching the source dates and all that, but it would at least be nice to apply at least the --numeric-owner --owner=0 --group=0 parts.

Secondly, as a feature request, is it possible to do something similar for the zip files?

[mstorsjo]

Why is sparse-checkout needed here? I guess it's nice for getting a smaller checkout if just wanting to inspect the git history, but the actual files within the llvm-mingw repo are quite minimal anyway?

[nolange]

It explicitly states that the files arent needed

[mstorsjo]

Thanks; I think this looks good now. I've amended your branch (changing an instance of checkout@v3 into checkout@v4, and added a commit to release-macos.sh to check whether gtar is available), and I'll merge this.

FWIW, when linking a DLL/EXE, those files get the current time stored in a timestamp field as well. I've implemented using SOURCE_DATE_EPOCH in lld now as well, in llvm/llvm-project@0df8aed, which also is backported to the next 18.x release branch, so it will be included in the next (pre)release.

For those purposes, it would kinda be nice to have a more stable source of data for SOURCE_DATE_EPOCH than the commit date - e.g. if just tweaking the CI pipeline, by making a new commit, we'd now be affecting the contents of the output files. But anything more specific also becomes quite brittle and fiddly - e.g. if we'd have it be the latest commit date for any file touching build-llvm.sh or the relevant build-libcxx.sh or similar, plus all of wrappers, etc. (IIRC it can be important that the field actually does change, when the file contents change.)

LLD does have another flag, -Brepro on the lld-link level, which replaces the timestamp with a hash (and sets a flag indicating that the timestamp isn't a real timestamp), but that flag isn't exposed on the mingw linker level, and there's no env variable one can set to enable it.

[nolange]

For those purposes, it would kinda be nice to have a more stable source of data for SOURCE_DATE_EPOCH than the commit date

I try to clean up these sources of random noise, but fully reproducible builds are still really hard in general (if you use debinfo you need to build in the same directories, even then it sometimes fails).
It's a decent start to take out the time the build job started.

For no "productive" change your best case is ending up with identical build-ids.

[mstorsjo]

For those purposes, it would kinda be nice to have a more stable source of data for SOURCE_DATE_EPOCH than the commit date

I try to clean up these sources of random noise, but fully reproducible builds are still really hard in general (if you use debinfo you need to build in the same directories, even then it sometimes fails). It's a decent start to take out the time the build job started.

Yeah - in this case I haven't tried how it behaves if built separately elsewhere (there's a million other things that affects the output there), but having two consecutive runs of the github actions pipeline produce identical output would at least be nice. Before these patches, there were differences only in the gendef executable (one case of __DATE__, which does get handled by SOURCE_DATE_EPOCH here, but I also took it out altogether upstream, mingw-w64/mingw-w64@202e375), and the <root>/<arch>-w64-mingw32/bin/*.dll due to timestamps. With the support for SOURCE_DATE_EPOCH in upstream lld, and these changes, that shouldn't vary any longer as long as rerunning the same pipeline from the same commit. But any commit will change COMMIT_DATE_UNIX and affect that.

[mstorsjo]

Btw, I just realized, this doesn't work as intended. At this point, ${BUILD_DATE} is still not set - we've only written it into $GITHUB_OUTPUT, which is a plain file that gets added to the environment starting with the next step in the job - but does not take effect immediately within the same script. So effectively, this executes date -d "" +%s, which produces a timestamp corresponding to the start of the current day, it seems.

[mstorsjo]

See fix-build-date for a fix for this - if this seems reasonable to you, I'll push that to master.

[nolange]

Good catch, fix seems sensible

[mstorsjo]

Thanks, I'll push that then.

The failure mode here, in producing a timestamp which is quite near to the intended one, but still not quite the same, was quite surprising too :-)