You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It was found that in pycocoevalcap/spice/spice.py, line 29 is sorting the img_ids prior the generation of SPICE metrics. The result metrics are then extracted and mapped to the corresponding image ids in pycocoevalcap/eval.py by the setImgToEvalImgs() function. However, the latter function is using an unsorted img_ids list. This incompatibility between the sorted and unsorted lists makes the SPICE metrics being mapped to incorrect image ids. This does not have any effect on the overall SPICE metric as it is only a problem of misalignment between the computed SPICE metrics and their corresponding image ids. For consistency, no sorting is needed for imgIds in pycocoevalcap/spice/spice.py.
The text was updated successfully, but these errors were encountered:
brandbir
changed the title
SPICE scores and image ids misalignment
Inconsistent SPICE scores
Apr 19, 2022
I think this problem still exists now (2022-05-14), and this causes some time-consuming trouble in my experiments. Thanks for pointing this out @brandbir
It was found that in
pycocoevalcap/spice/spice.py
, line 29 is sorting theimg_ids
prior the generation of SPICE metrics. The result metrics are then extracted and mapped to the corresponding image ids inpycocoevalcap/eval.py
by thesetImgToEvalImgs()
function. However, the latter function is using an unsortedimg_ids
list. This incompatibility between the sorted and unsorted lists makes the SPICE metrics being mapped to incorrect image ids. This does not have any effect on the overall SPICE metric as it is only a problem of misalignment between the computed SPICE metrics and their corresponding image ids. For consistency, no sorting is needed forimgIds
inpycocoevalcap/spice/spice.py
.The text was updated successfully, but these errors were encountered: