Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use torchvision transforms instead of PIL operations for solarization and gaussian blurring #1728

Open
siemdejong opened this issue Nov 7, 2024 · 1 comment

Comments

@siemdejong
Copy link

siemdejong commented Nov 7, 2024

Problem
When using a transform that uses lightly.transforms.gaussian_blur.GaussianBlur or lightly.transforms.solarization.RandomSolarization, e.g. DINOTransform, the input must be transformed to a PIL.Image first, because the Gaussian blur and the solarization use PIL.ImageFilter.GaussianBlur and PIL.ImageOps.solarize respectively. For training, the PIL.Image has to be transformed back to a tensor.

Background
It might be beneficial to get rid of this conversion for performance purposes. Sometimes, using a different reader like pyvips/openslide/cucim is necessary to extract only a patch of an image because images are too big to fit in memory (e.g. in computational pathology or remote sensing). Here, often patches are extracted and immediately transformed to torch tensors for torchvision to transform the patches. Doing all the transforms on tensors overcomes converting to other formats, like PIL, increasing performance.

Alternative
Torchvision provides seemingly fast (jitted) implementations for solarization [1] and gaussian blurring [2].

I'm not aware of any other transforms in the lightly package that rely on PIL.

References
[1] implementation docs
[2] implementation docs

@guarin
Copy link
Contributor

guarin commented Nov 7, 2024

Thanks for raising this issue!

There is a previous discussion on supporting tensors as transform inputs #791

I think we can switch to torchvision solarization and blur implementations. When making the change we should take #1052 into account and make sure we don't re-introduce a change in the blurring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants