Systematically benchmark compression algorithm, compression factor, block size #44
Labels
Comments
|
This test matrix includes 5 compression algorithms (zlib, lzma, zstandard, dwarfs, and libdeflate), 3 compression factors (low, mid, high), and 3 block sizes (256KB, 512KB, and 1MB), resulting in a total of 45 test cases.
|
|
Volunteers? |
|
Something that needs to be considered as well is to take into account how long it normally takes for the same application not as an appimage to start. For example we might see that on a very big application a certain algo is 30% faster, but that application even when not being an appimage due to its size takes several seconds to start anyway, and that 30% ends up being a very small percentage of the overall delay for the app. Same way for very small applications, the speed difference might not matter much, because they are very small and take no time regardless. Where problems can happen is with the mid size applications that you normally expect to start fast, those are web-browsers in other words. Right now zstd with the current default block size is actually very good, I will do some benchmarks comparing the size and startup times, however I can't measure zsync efficiency since that seems quite a bit more work. I'm also interested to know how this affects very old hardware, IE, some pre sandy bridge cpu for example, my hardware is from 2016 and not the worst I would say lol. |
|
Application startup times also depend on the hardware. On systems with slow disk but fast CPU, a highly compressed image may lead to faster application launch times than uncompressed files (iirc, I have seen this myself with a large application in the past, likely with a spinning drive). It always depends where the performance bottleneck is on a particular system. So for this to be really scientific, we'd have to execute the test matrix for typical defined machines. But then, we are not exactly writing a dissertation here ;-) |
|
I'm probably a bit late, but I think EROFS is an interesting option as well.
dwarfs is incredibly performant, but gpl3 makes it impossible to use with proprietary programs packaged in appimage. At the same time, AppImage packaging is used by large companies, so the introduction of dwarfs will cut off a significant part of the audience |
|
I got these results running on a VM with 12 GiB Memory and 4 cores:
My scriptimport itertools
import os
import pathlib
import re
import shlex
import subprocess
import time
file_ver1 = "./LibreOffice-24.8.3.2.full.help-x86_64.AppImage"
file_ver2 = "./LibreOffice-24.8.4.2.full.help-x86_64.AppImage"
block_sizes = ["16k", "64k", "256k", "512k", "1M"]
comps_squashfs = {
"gzip": {"-Xcompression-level": ["1", "6", "9"]},
"lzo": {
"-Xcompression-level": ["1", "6", "9"],
},
"lz4": {},
"xz": {
"-Xdict-size": ["100%"],
},
"zstd": {
"-Xcompression-level": ["1", "6", "11", "15", "19", "22"],
},
}
output = open("results.csv", "w")
output.write(
"Compression Method,Compression Parameters,Block Size,Compression Time,"
"FS Size,Startup Time (average of 5 runs),docx to pdf Time (average of 5 runs),zsync download size"
)
output.flush()
try:
os.mkdir("zsyncout")
except FileExistsError:
pass
if not os.path.exists("ver1.AppDir"):
subprocess.Popen([file_ver1, "--appimage-extract"]).wait()
os.rename("squashfs-root", "ver1.AppDir")
if not os.path.exists("ver2.AppDir"):
subprocess.Popen([file_ver2, "--appimage-extract"]).wait()
os.rename("squashfs-root", "ver2.AppDir")
with open("uruntime-appimage-x86_64", "rb") as f:
runtime = f.read()
for comp in comps_squashfs:
# product all possible combinations of compression parameters
comp_params = list(
itertools.product(*list(list([i, j] for j in comps_squashfs[comp][i]) for i in comps_squashfs[comp]))
)
for comp_param in comp_params:
for block_size in block_sizes:
output.write("\n")
output.write(f"SquashFS {comp},{shlex.join(sum(comp_param, []))},{block_size},")
output.flush()
print(
"Preparing AppImages for compression method %s with parameters %s and block size %s"
% (comp, sum(comp_param, []), block_size)
)
# Prepare the AppImages
command = ["mksquashfs", "ver1.AppDir", "file_ver1.squashfs", "-root-owned", "-noappend", "-comp", comp]
command.extend(["-b", block_size])
for i in comp_param:
command.extend(i)
# file_ver1 is only the base file, so do not measure efficiency
p = subprocess.Popen(command)
if p.wait() != 0:
print("Error in compression")
continue
with open("file_ver1.squashfs", "rb") as f:
squashfs = f.read()
os.remove("file_ver1.squashfs")
with open("file_ver1.AppImage", "wb") as f:
f.write(runtime)
f.write(squashfs)
del squashfs
try:
os.chmod("file_ver1.AppImage", 0o755)
except PermissionError:
pass
# Compression time
command[1] = "ver2.AppDir"
command[2] = "file_ver2.squashfs"
start = time.time()
p = subprocess.Popen(command)
if p.wait() != 0:
print("Error in compression")
continue
output.write(f"{(time.time() - start) * 1000:.1f}ms,")
output.flush()
with open("file_ver2.squashfs", "rb") as f:
squashfs = f.read()
os.remove("file_ver2.squashfs")
with open("file_ver2.AppImage", "wb") as f:
f.write(runtime)
f.write(squashfs)
output.write(f"{len(squashfs)},")
output.flush()
print("SquashFS size: %d" % len(squashfs))
del squashfs
try:
os.chmod("file_ver2.AppImage", 0o755)
except PermissionError:
pass
# Startup time
print("Measuring startup time, 10 rounds, only last 5 rounds count")
for _ in range(2):
startup_times = []
for i in range(5):
start = time.time()
p = subprocess.Popen(["./file_ver2.AppImage", "--terminate_after_init"])
p.wait()
used_time = time.time() - start
startup_times.append(used_time)
print("Startup time round %d: %f" % (i + 1, used_time))
output.write(f"{sum(startup_times) * 1000 / 5:.1f} ms,")
output.flush()
# docx to pdf time
print("Measuring docx to pdf time, 10 rounds, only last 5 rounds count")
for _ in range(2):
docx_to_pdf_times = []
for i in range(5):
start = time.time()
p = subprocess.Popen(
[
"./file_ver2.AppImage",
"--convert-to",
"pdf",
pathlib.Path("test.docx").absolute(),
"--outdir",
pathlib.Path(".").absolute(),
]
)
p.wait()
used_time = time.time() - start
docx_to_pdf_times.append(used_time)
print("docx to pdf time round %d: %f" % (i + 1, used_time))
output.write(f"{sum(docx_to_pdf_times) * 1000 / 5:.1f} ms,")
output.flush()
# zsync download size
print("Creating zsync file")
p = subprocess.Popen(["zsyncmake", "file_ver2.AppImage"])
if p.wait() != 0:
print("Error in zsync")
continue
print("Calculating zsync download size")
p = subprocess.Popen(
["zsync", "-i", "file_ver1.AppImage", "file_ver2.AppImage.zsync", "-o", "./zsyncout"],
stderr=subprocess.PIPE,
)
zsync_output = p.stderr.read().decode("utf-8")
percentage = re.search(r"(\d+\.\d+)%", zsync_output).group(1)
p.wait()
try:
os.remove("zsyncout.part")
except FileNotFoundError:
pass
output.write(f"{(1 - float(percentage) / 100) * os.path.getsize('file_ver2.AppImage'):.0f}")
output.flush()
output.close() |
|
This is amazing @CarlGao4 thank you. I modified the script to not test as many options and also dropped lzo all together, tested with the Brave AppImage: Tested on actual hardware: Xeon E5-2640 V4 + 16 GiB of mem.
I don't know how to make the table look all good like you did here. 😅 EDIT: I had this hack running in the background to kill the brave window every time it opened: |
|
Oh, I opened the csv in Excel and just copy the table and paste it here, it will automatically form a table in markdown syntax |
|
I'll try to test DwarFS later |
That's going to be a lot of block sizes btw, with dwarfs iirc it can be as high as 64MB. And the impact also varies greatly depending on the application that's tested. With this AppImage of GIMP for example |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Execute a systematic and reproducible benchmark to find the optimal combination of
in terms of
for
References:
The text was updated successfully, but these errors were encountered: