Preheating a 20G image is prone to failure,"error decoding response body"

[WaterAndBread]

config:
seedClient resources: requests: 2c4G ,limits 8c16G version v0.1.118
client resources: requests: 1c2G ,limits 4c8G version v0.1.118
client dowload rateLimit 20GiB upload rateLimit 20GiB

problem 1
when preheating about 20G image ,it is easy to appear "rpc error: code = Internal desc = error decoding response body" in manager console web.

preheat type is single_preheat
dragonfly installed by helm version 1.2.23 , dragonfly version v2.1.63

seedClient error log is ":download:download_partial_from_source:download_from_source:download_piece_from_source_finished:write_piece: dragonfly-client-storage/src/content.rs:317: copy "/var/lib/dragonfly/content/tasks/a7b/a7bb6d3a74392ff0e00f158f6639953f2f00ad8a65d367a0837d04ca6a98f6f8" failed:"
download_task:download:download_partial_from_source: [dragonfly-client/src/resource/task.rs:1594]
See attachment for error logs
problem1-dfdaemon.log

problem 2
When problem 1 occurs, it seems that the preheat failure will recur 。

20G image after preheat ,pod running needs about 11min, longer than no preheat , Is there any configuration that can be modified to improve the effect?

[WaterAndBread]

I opened the debug log, hoping that helps

Uploading dfdaemon-debug.zip…

[gaius-qi]

I think it is a network problem. The connection is broken when reading piece. Need to check the storage service.

[WaterAndBread]

I think it is a network problem. The connection is broken when reading piece. Need to check the storage service.

Because the image download is normal, but the preheat is abnormal, is there any command in Dragonfly that can confirm the network problem I can test it separately. thanks.

[gaius-qi]

@WaterAndBread I can't reproduce it locally. Is there an public Container Registry that I can reproduce it?