Nydusify generate image chunk dictionary error and use chunk dictionary without deduplication effect

[Jaycean]

Additional Information

Version of nydus being used (nydusd --version)

Version:        v2.3.0
Git Commit:     a4683baa1ea8cafecc9013bc4d6a5259b5a82678
Build Time:     2024-10-11T02:51:53.888907475Z
Profile:        release
Rustc:          rustc 1.72.1 (d5c2e9c34 2023-09-13)

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

Version:     v0.13.13
Revision:    e9d1bb738f778a2e30b8284ca4f479fa2517456c
Go version:  go1.19.6
Build time:  2024-05-15T03:57:13

Kernel information (uname -r)

4.18.0-348.el8.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

containerd-nydus-grpc command line used, if applicable (ps aux | grep containerd-nydus-grpc)

root      3067  0.0  0.3 1351116 43312 ?       Ssl  17:34   0:02 /usr/bin/containerd-nydus-grpc --config /etc/nydus/config.toml
root     17922  0.0  0.0  12136  2368 pts/0    S+   19:17   0:00 grep --color=auto containerd-nydus-grpc

client command line used, if applicable (such as: nerdctl, docker, kubectl, ctr)

kubectl
nerdctl

Screenshots (if applicable)

[root@master ~]# nydusify chunkdict generate --sources 192.168.237.131:80/library/ubuntu:20.04-nydus --sources 192.168.237.131:80/library/ubuntu:22.04-nydus --target 192.168.237.131:80/library/ubuntu:dict3 --source-insecure --target-insecure
INFO[2024-12-03T00:52:11-08:00] Parsing image 192.168.237.131:80/library/ubuntu:20.04-nydus 
INFO[2024-12-03T00:52:11-08:00] trying next host                              error="failed to do request: Head \"https://192.168.237.131:80/v2/library/ubuntu/manifests/20.04-nydus\": http: server gave HTTP response to HTTPS client" host="192.168.237.131:80"
INFO[2024-12-03T00:52:11-08:00] Parsing image 192.168.237.131:80/library/ubuntu:20.04-nydus 
INFO[2024-12-03T00:52:11-08:00] Pulling Nydus bootstrap to output/192.168.237.131:80:library:ubuntu:20.04-nydus/nydus_bootstrap 
INFO[2024-12-03T00:52:11-08:00] Parsing image 192.168.237.131:80/library/ubuntu:22.04-nydus 
INFO[2024-12-03T00:52:12-08:00] Pulling Nydus bootstrap to output/192.168.237.131:80:library:ubuntu:22.04-nydus/nydus_bootstrap 
INFO[2024-12-03T00:52:12-08:00] Invoking 'nydus-image chunkdict generate' command 
[2024-12-03 00:52:17.997449 -08:00] ERROR [/src/error.rs:18] Stack:
   0: nydus_api::error::make_error

[2024-12-03 00:52:17.997559 -08:00] ERROR [/src/error.rs:19] Error:
        "compression algorithm should be none or lz4_block"
        at utils/src/compress/mod.rs:45
Error: Invalid argument (os error 22)
ERRO[2024-12-03T00:52:17-08:00] fail to run nydus-image [chunkdict generate --log-level warn --bootstrap output/chunkdict_bootstrap --database sqlite:///root/output/database.db --output-json output/nydus_bootstrap_output.json output/192.168.237.131:80:library:ubuntu:20.04-nydus/nydus_bootstrap output/192.168.237.131:80:library:ubuntu:22.04-nydus/nydus_bootstrap]  error="exit status 1"
FATA[2024-12-03T00:52:17-08:00] invalid nydus bootstrap format: exit status 1 

Details about issue

When I was using the nydusify generate image chunk dictionary, I initially chose three nydus images to create, but it kept reporting errors. Later, I chose two images and successfully created the chunk dictionary.
However, I deduplicated the oci image based on the chunk dict. Why is the layer hash of the generated image blob the same? Does this mean that my image doesn't have the same block? My two images have many packages, and the packages in the second image as a whole include the first one. Both images have three layers, but the second one contains many more packages than the third one. Shouldn't it be normal to remove duplicate packages?

1. Why did I get an error when making chunkdit for the first question?
2. The second question is, my deduplication effect is very poor. Is there anything wrong with my method?

Here are the Dockerfiles of my two images

Thks.

[imeoer]

@cslinwang PTAL.