When is static dataset pipeline useful?

[shatz01]

It would be nice to showcase when static dataset saves time.

I was thinking that maybe imagenet images are always resized 224x224, and everyone always does it on the fly. Maybe we can make an example training a resnet with resizing in the static pipeline vs in the dynamic pipeline? Do you think it would make a big difference?

[itaijj]

I think that if you want to use the caching mechanism you must put the operations that creates the cached data in static,
so this is the main reason to have static pipeline.
Doing resize in static is really less recommended.
do you also ask why caching is needed?

[shatz01]

Why is resize in static less recommended?

[itaijj]

because you may want to change the size and not cache all images again

[mosheraboh]

Showcase of the static pipeline is actually a showcase of the caching mechanism.
I guess that caching will have a greater impact on large images (3D images are also a good option), in cases where we use intensive computer vision operations (including extracting radiomics features) and, as you mention, in cases that reduce the image size significantly.

Checking the time it takes to iterate over imagenet with and without caching mechanism should be very easy - then it's a good option.

[shatz01]

Added a showcase for the static pipeline / caching mechanism in PR #248 :)

[shatz01]

Results for KNIGHT: