The difference of SSIM between pyiqa and skimage!

[Chinafsh]

I use the different implementation of SSIM to evaluate the same image pair, but the score is not the same.
So I want to know is this problem matters?
thanks for ur brilliant work, BTW.

[chaofengc]

It is expected that results might differ slightly due to variations in implementation details (such as padding, filtering, color transformation). We strive to align closely with MATLAB's implementation of this function, as it is the most widely used reference version.

The main difference is that we do not downsample the image before calculating the score, whereas MATLAB does. This choice avoids the potential influence of image quality caused by resizing. To obtain results consistent with MATLAB, you can use pyiqa.create_metric('ssim', downsample=True) in your implementation.

[lienghongky]

Thank you for maintaining this great repository! However, I’ve noticed some discrepancies in SSIM values when comparing results from MATLAB R2024b, IQA-Pytorch 0.1.13, and Skimage 0.24.0, using the same input data (denoised and ground truth images).

Could you please guide me on how to adjust IQA-Pytorch to get results more consistent with MATLAB's SSIM values? Perhaps there are specific parameters or normalization methods I should be using to align these results.

Environment:

• Torch 2.2.2
• MacOS Ventura 13.6

MATHLAB R2024b

ssim_val = ssim(denoised_patch, gt_patch);

Average SSIM: 0.959785
max SSIM: 0.999635
min SSIM: 0.744630>>

IQA-Pytorch 0.1.13

calculate_ssim = pyiqa.create_metric('ssim', downsample=True, device=device)
ssims.append(calculate_ssim( gt_patch,restored_patch))

Average SSIM: 0.9467163060615112
max SSIM: 0.9942350312089608
min SSIM: 0.5947798426732966

Skimage 0.24.0

**squeezed batch

from skimage.metrics import structural_similarity as ssim

ssims.append(ssim( gt_patch.squeeze(0).numpy(),restored_patch.squeeze(0).numpy(), multichannel=True,channel_axis=0,data_range=1.0))

Average SSIM: 0.9150699
max SSIM: 0.9910875
min SSIM: 0.59541583

[chaofengc]

Thank you for sharing this information. I will review the code and make the necessary updates to ensure the results align more closely with MATLAB. This process might take some time.

[chaofengc]

Thank you for your feedback. We understand the concern regarding the differences between our implementation and MATLAB's built-in ssim function. Here’s a detailed explanation of our approach:

1. Reference Implementations:
   Since MATLAB's built-in ssim function is not open-source, we aligned our implementation with widely recognized open-source MATLAB implementations, including:

   • Official: https://ece.uwaterloo.ca/~z70wang/research/ssim
   • utlive/ssim
   • josejuansanchez/ssim

   Our results closely match these implementations, which are consistent with the original SSIM algorithm described in the literature.

2. Discrepancy with MATLAB's Built-in Function:
   We observed that MATLAB's built-in ssim function uses replicate padding and operates in same mode, whereas the original implementation by Zhou Wang (author of the SSIM algorithm) uses valid mode. This difference in padding and convolution modes likely explains the inconsistency between our results and MATLAB's built-in function.

3. Commitment to Alignment with the Original Algorithm:
   Since MATLAB's implementation is proprietary and not open-source, we have chosen to prioritize alignment with the original SSIM script provided by Zhou Wang, which can be found here: Zhou Wang's SSIM Script. This ensures our implementation remains consistent with the algorithm as described in the original paper.

We appreciate your understanding and welcome further discussion or suggestions on how we can improve this implementation. Thank you for your contribution!