Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use 7z to create more efficient ZIP archives #81

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Use 7z to create more efficient ZIP archives #81

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 23 additions & 21 deletions build-release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ export basedir=$(pwd)
# For signing keystore and password.
source ./config.sh

export ZIP="7z a -bso0 -bd -mx9"

can_sign_windows=0
if [ ! -z "${SIGN_KEYSTORE}" ] && [ ! -z "${SIGN_PASSWORD}" ] && [[ $(type -P "osslsigncode") ]]; then
can_sign_windows=1
Expand Down Expand Up @@ -47,7 +49,7 @@ sign_macos() {
codesign --force --timestamp \
--options=runtime --entitlements editor.entitlements \
-s ${OSX_KEY_ID} -v ${_appname} && \
zip -r ${_binname}_signed.zip ${_appname}"
$ZIP -r ${_binname}_signed.zip ${_appname}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this bit forces me to have 7zip installed on macOS, which is less trivial than on Linux.

It seems available on homebrew but homebrew isn't set up on HP's Mac, could likely install it.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

Copy link
Member Author

@Calinou Calinou Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

The last GUI release was made in June 2023: https://7-zip.org/
However, the CLI version was never updated past 16.02 (May 2016). This is an unfortunate artifact of 7-zip being developed around its Windows GUI first, and also being developed behind closed doors.

I don't know of any alternatives that have the same level of efficiency and performance for compressing ZIP archives. Using Zopfli to create a ZIP archive will result in slightly more efficiency, but it's more than 10× slower which makes it not worth the time. zip -9 is slightly better than the default option, but still not as good as 7z a -mx9.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last GUI release was made in June 2023: 7-zip.org

It's Windows only though, no?

I found PeaZip which seems actively maintained and properly cross-platform: https://github.com/peazip/PeaZip/releases/

Might be worth comparing with 7zip.

Copy link
Member

@bruvzg bruvzg Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

At least in macOS homebrew, there are two 7Z packages:

  • p7zip - 7z executable, built from p7zip port source from 2017 (17.04).
  • sevenzip - 7zz executable, built from official 2023 source (23.01, same as GUI version).

Seems like official source added support for Linux/macOS command line build in 2021.

Edit: Debian seems to use the same naming, so it should be standard - https://manpages.debian.org/unstable/7zip/7zz.1.en.html


_request_uuid=$(ssh "${OSX_HOST}" "xcrun notarytool submit ${_macos_tmpdir}/${_binname}_signed.zip --team-id \"${APPLE_TEAM}\" --apple-id \"${APPLE_ID}\" --password \"${APPLE_ID_PASSWORD}\" --no-progress --output-format json")
_request_uuid=$(echo ${_request_uuid} | sed -e 's/.*"id":"\([^"]*\)".*/\1/')
Expand All @@ -61,7 +63,7 @@ sign_macos() {
ssh "${OSX_HOST}" "
cd ${_macos_tmpdir} && \
xcrun stapler staple ${_appname} && \
zip -r ${_binname}_stapled.zip ${_appname}"
$ZIP -r ${_binname}_stapled.zip ${_appname}"
scp "${OSX_HOST}:${_macos_tmpdir}/${_binname}_stapled.zip" "${_reldir}/${_binname}.zip"
ssh "${OSX_HOST}" "rm -rf ${_macos_tmpdir}"
fi
Expand All @@ -82,7 +84,7 @@ sign_macos_template() {
codesign --force -s - \
--options=linker-signed \
-v macos_template.app/Contents/MacOS/* && \
zip -r macos_signed.zip macos_template.app"
$ZIP -r macos_signed.zip macos_template.app"

scp "${OSX_HOST}:${_macos_tmpdir}/macos_signed.zip" "${_reldir}/macos.zip"
ssh "${OSX_HOST}" "rm -rf ${_macos_tmpdir}"
Expand Down Expand Up @@ -240,22 +242,22 @@ if [ "${build_classical}" == "1" ]; then
# Editor
binname="${godot_basename}_linux.x86_64"
cp out/linux/x86_64/tools/godot.linuxbsd.editor.x86_64 ${binname}
zip -q -9 "${reldir}/${binname}.zip" ${binname}
$ZIP "${reldir}/${binname}.zip" ${binname}
rm ${binname}

binname="${godot_basename}_linux.x86_32"
cp out/linux/x86_32/tools/godot.linuxbsd.editor.x86_32 ${binname}
zip -q -9 "${reldir}/${binname}.zip" ${binname}
$ZIP "${reldir}/${binname}.zip" ${binname}
rm ${binname}

binname="${godot_basename}_linux.arm64"
cp out/linux/arm64/tools/godot.linuxbsd.editor.arm64 ${binname}
zip -q -9 "${reldir}/${binname}.zip" ${binname}
$ZIP "${reldir}/${binname}.zip" ${binname}
rm ${binname}

binname="${godot_basename}_linux.arm32"
cp out/linux/arm32/tools/godot.linuxbsd.editor.arm32 ${binname}
zip -q -9 "${reldir}/${binname}.zip" ${binname}
$ZIP "${reldir}/${binname}.zip" ${binname}
rm ${binname}

# Templates
Expand All @@ -279,7 +281,7 @@ if [ "${build_classical}" == "1" ]; then
cp out/windows/x86_64/tools/godot.windows.editor.x86_64.console.exe ${wrpname}
strip ${wrpname}
sign_windows ${wrpname}
zip -q -9 "${reldir}/${binname}.zip" ${binname} ${wrpname}
$ZIP "${reldir}/${binname}.zip" ${binname} ${wrpname}
rm ${binname} ${wrpname}

binname="${godot_basename}_win32.exe"
Expand All @@ -290,7 +292,7 @@ if [ "${build_classical}" == "1" ]; then
cp out/windows/x86_32/tools/godot.windows.editor.x86_32.console.exe ${wrpname}
strip ${wrpname}
sign_windows ${wrpname}
zip -q -9 "${reldir}/${binname}.zip" ${binname} ${wrpname}
$ZIP "${reldir}/${binname}.zip" ${binname} ${wrpname}
rm ${binname} ${wrpname}

# Templates
Expand All @@ -313,7 +315,7 @@ if [ "${build_classical}" == "1" ]; then
mkdir -p Godot.app/Contents/MacOS
cp out/macos/tools/godot.macos.editor.universal Godot.app/Contents/MacOS/Godot
chmod +x Godot.app/Contents/MacOS/Godot
zip -q -9 -r "${reldir}/${binname}.zip" Godot.app
$ZIP -r "${reldir}/${binname}.zip" Godot.app
rm -rf Godot.app
sign_macos ${reldir} ${binname} 0

Expand All @@ -325,7 +327,7 @@ if [ "${build_classical}" == "1" ]; then
cp out/macos/templates/godot.macos.template_release.universal macos_template.app/Contents/MacOS/godot_macos_release.universal
cp out/macos/templates/godot.macos.template_debug.universal macos_template.app/Contents/MacOS/godot_macos_debug.universal
chmod +x macos_template.app/Contents/MacOS/godot_macos*
zip -q -9 -r "${templatesdir}/macos.zip" macos_template.app
$ZIP -r "${templatesdir}/macos.zip" macos_template.app
rm -rf macos_template.app
sign_macos_template ${templatesdir} 0

Expand Down Expand Up @@ -370,15 +372,15 @@ if [ "${build_classical}" == "1" ]; then
cp -r deps/vulkansdk-macos/MoltenVK/MoltenVK.xcframework ios_xcode/
rm -rf ios_xcode/MoltenVK.xcframework/{macos,tvos}*
cd ios_xcode
zip -q -9 -r "${templatesdir}/ios.zip" *
$ZIP -r "${templatesdir}/ios.zip" *
cd ..
rm -rf ios_xcode

## Templates TPZ (Classical) ##

echo "${templates_version}" > ${templatesdir}/version.txt
pushd ${templatesdir}/..
zip -q -9 -r -D "${reldir}/${godot_basename}_export_templates.tpz" templates/*
$ZIP -r "${reldir}/${godot_basename}_export_templates.tpz" templates/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think -D was important here, so this might break the logic.

Copy link
Member Author

@Calinou Calinou Oct 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7z doesn't seem to have an equivalent of -D. According to zip's command line help:

 -D   do not add directory entries

The only difference I can notice between 2 ZIP archives created (one with -D, one without -D) is that the former will have directory entries in its 7z l listing, but the latter will specify full paths to every file.

In practice, this appears to make no concrete difference on what you see when opening the ZIP archive: both have the exact same folder structure when viewed with GUI archive managers. In short, -D does not make the ZIP's structure flat or remove the top-level folder as the option's name might suggest. The only real difference it seems to make is that it prevents empty folders from being added to the archive.

I think the -D requirement is a leftover from early Godot 3.x days where Godot behaved poorly if given a TPZ that had directory entries. I've created TPZ files without -D for a while for my custom builds and it worked just fine (before I switched to 7z).

#!/usr/bin/env bash

mkdir -p example/{folder1,folder2,empty_folder}
touch example/{folder1,folder2}/example.txt

echo -e "\n\n================ Creating archives ================"
zip -r example.zip example
zip -r example-flat.zip example/*
zip -r -D example-D.zip example
zip -r -D example-D-flat.zip example/*
7z a example-7z.zip example
7z a example-7z-flat.zip example/*

echo -e "\n\n================ File contents created by zip ================"
7z l example.zip
7z l example-flat.zip
7z l example-D.zip
7z l example-D-flat.zip

echo -e "\n\n================ File contents created by 7z ================"
7z l example-7z.zip
7z l example-7z-flat.zip

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, in the asset library I had to fix some bugs related to directory entries missing. Not sure if the export template installer handles it any better.

popd

## SHA-512 sums (Classical) ##
Expand All @@ -402,14 +404,14 @@ if [ "${build_mono}" == "1" ]; then
mkdir -p ${binbasename}_x86_64
cp out/linux/x86_64/tools-mono/godot.linuxbsd.editor.x86_64.mono ${binbasename}_x86_64/${binbasename}.x86_64
cp -rp out/linux/x86_64/tools-mono/GodotSharp ${binbasename}_x86_64/
zip -r -q -9 "${reldir_mono}/${binbasename}_x86_64.zip" ${binbasename}_x86_64
$ZIP -r "${reldir_mono}/${binbasename}_x86_64.zip" ${binbasename}_x86_64
rm -rf ${binbasename}_x86_64

binbasename="${godot_basename}_mono_linux"
mkdir -p ${binbasename}_x86_32
cp out/linux/x86_32/tools-mono/godot.linuxbsd.editor.x86_32.mono ${binbasename}_x86_32/${binbasename}.x86_32
cp -rp out/linux/x86_32/tools-mono/GodotSharp/ ${binbasename}_x86_32/
zip -r -q -9 "${reldir_mono}/${binbasename}_x86_32.zip" ${binbasename}_x86_32
$ZIP -r "${reldir_mono}/${binbasename}_x86_32.zip" ${binbasename}_x86_32
rm -rf ${binbasename}_x86_32

binbasename="${godot_basename}_mono_linux"
Expand Down Expand Up @@ -449,7 +451,7 @@ if [ "${build_mono}" == "1" ]; then
cp out/windows/x86_64/tools-mono/godot.windows.editor.x86_64.mono.console.exe ${binname}/${wrpname}.exe
strip ${binname}/${wrpname}.exe
sign_windows ${binname}/${wrpname}.exe
zip -r -q -9 "${reldir_mono}/${binname}.zip" ${binname}
$ZIP -r "${reldir_mono}/${binname}.zip" ${binname}
rm -rf ${binname}

binname="${godot_basename}_mono_win32"
Expand All @@ -462,7 +464,7 @@ if [ "${build_mono}" == "1" ]; then
cp out/windows/x86_32/tools-mono/godot.windows.editor.x86_32.mono.console.exe ${binname}/${wrpname}.exe
strip ${binname}/${wrpname}.exe
sign_windows ${binname}/${wrpname}.exe
zip -r -q -9 "${reldir_mono}/${binname}.zip" ${binname}
$ZIP -r "${reldir_mono}/${binname}.zip" ${binname}
rm -rf ${binname}

# Templates
Expand All @@ -486,7 +488,7 @@ if [ "${build_mono}" == "1" ]; then
cp out/macos/tools-mono/godot.macos.editor.universal.mono Godot_mono.app/Contents/MacOS/Godot
cp -rp out/macos/tools-mono/GodotSharp Godot_mono.app/Contents/Resources/GodotSharp
chmod +x Godot_mono.app/Contents/MacOS/Godot
zip -q -9 -r "${reldir_mono}/${binname}.zip" Godot_mono.app
$ZIP -r "${reldir_mono}/${binname}.zip" Godot_mono.app
rm -rf Godot_mono.app
sign_macos ${reldir_mono} ${binname} 1

Expand All @@ -497,7 +499,7 @@ if [ "${build_mono}" == "1" ]; then
cp out/macos/templates-mono/godot.macos.template_debug.universal.mono macos_template.app/Contents/MacOS/godot_macos_debug.universal
cp out/macos/templates-mono/godot.macos.template_release.universal.mono macos_template.app/Contents/MacOS/godot_macos_release.universal
chmod +x macos_template.app/Contents/MacOS/godot_macos*
zip -q -9 -r "${templatesdir_mono}/macos.zip" macos_template.app
$ZIP -r "${templatesdir_mono}/macos.zip" macos_template.app
rm -rf macos_template.app
sign_macos_template ${templatesdir_mono} 1

Expand All @@ -521,7 +523,7 @@ if [ "${build_mono}" == "1" ]; then
cp -r deps/vulkansdk-macos/MoltenVK/MoltenVK.xcframework ios_xcode/
rm -rf ios_xcode/MoltenVK.xcframework/{macos,tvos}*
cd ios_xcode
zip -q -9 -r "${templatesdir_mono}/ios.zip" *
$ZIP -r "${templatesdir_mono}/ios.zip" *
cd ..
rm -rf ios_xcode

Expand All @@ -541,7 +543,7 @@ if [ "${build_mono}" == "1" ]; then

echo "${templates_version}.mono" > ${templatesdir_mono}/version.txt
pushd ${templatesdir_mono}/..
zip -q -9 -r -D "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/*
$ZIP -r "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs -D equivalent.

popd

## SHA-512 sums (Mono) ##
Expand Down