Faster time on image loading using systemd service?

[VintageDoggo]

Hi all,

I've been working with Podman for only a bit, so sorry for anything I may have overlooked.
I currently have a working service that loads an image tarball into the registry and uses podman compose to start containers from the loaded images. There is a behaviour I've noticed during the load portion of the service and it's the time it takes to actually start loading the blobs/layers.

To put into context, I have a persistent partition at which I have pointed the storage. This way, upon loading an image that already exists, the layers are not copied once again. Right as it starts loading the image, there is a consistent lapse of 6 to 8 seconds (as shown in the logs) in which I have no idea what is going on under the hood.

Jan 01 00:00:11 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:11Z" level=debug msg="Error loading /usr/share/container-images/yocto-docker-master-bullseye.tar (oci): open /usr/share/container-images/yocto-docker-master-bullseye.tar/index.json: not a directory"
Jan 01 00:00:11 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:11Z" level=debug msg="-> Attempting to load \"/usr/share/container-images/yocto-docker-master-bullseye.tar\" as an OCI archive"
Jan 01 00:00:11 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:11Z" level=debug msg="Normalized platform linux/arm64 to {arm64 linux  [] }"
Jan 01 00:00:17 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:17Z" level=debug msg="Error deleting temporary directory: <nil>"
Jan 01 00:00:17 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:17Z" level=debug msg="parsed reference into \"[overlay@/data/containers/storage+/data/containers/transient-storage:overlay.mountopt=nodev]localhost/yocto-docker:master-bullseye\""
Jan 01 00:00:17 nitrogen8mm load-containers[291]: time="1970-01-01T00:00:17Z" level=debug msg="Copying source image /usr/share/container-images/yocto-docker-master-bullseye.tar: to destination image [overlay@/data/containers/storage+/data/containers/transient-storage:overlay.mountopt=nodev]localhost/yocto-docker:master-bullseye"

However, if execute the service a second time, the hang time is no longer there and the loading is completed within 3 seconds.

I have tried testing the performance by:

• Changing the storage driver
• Changing the network stack from cni to netavark
• Changing to a different runtime from runc to crun
• Setting my service to run after boot completion (to avoid resource bottlenecks)
• Removing certain dependencies for Podman services.

While this has overall improved the boot time, I cannot seem to decrease that period of time on image loading and I haven't been able to find out the culprit for this. I've tried enabling the debugging logs for crun, but I see no output when accesing the logs. Here is an extract of how I do the loading:

while read -r NAME VERSION _; do
        echo "Loading ${NAME}-${VERSION}.tar"
        if ! podman --log-level=debug --runtime="/usr/bin/crun" --runtime-flag="--debug " load --input ${TARDIR}${NAME}-${VERSION}.tar; then
            echo "Unable to load image ${NAME}-${VERSION}"
            return 1
        else 
            echo "Successfully loaded ${NAME}-${VERSION}"
        fi
    done < ${MANIFEST}

If anyone has any hints as to what could be happening, I'd love to hear!