Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent SPICE scores #56

Open
brandbir opened this issue Aug 18, 2021 · 1 comment
Open

Inconsistent SPICE scores #56

brandbir opened this issue Aug 18, 2021 · 1 comment

Comments

@brandbir
Copy link

brandbir commented Aug 18, 2021

It was found that in pycocoevalcap/spice/spice.py, line 29 is sorting the img_ids prior the generation of SPICE metrics. The result metrics are then extracted and mapped to the corresponding image ids in pycocoevalcap/eval.py by the setImgToEvalImgs() function. However, the latter function is using an unsorted img_ids list. This incompatibility between the sorted and unsorted lists makes the SPICE metrics being mapped to incorrect image ids. This does not have any effect on the overall SPICE metric as it is only a problem of misalignment between the computed SPICE metrics and their corresponding image ids. For consistency, no sorting is needed for imgIds in pycocoevalcap/spice/spice.py.

@brandbir brandbir changed the title SPICE scores and image ids misalignment Inconsistent SPICE scores Apr 19, 2022
@cantabile-kwok
Copy link

I think this problem still exists now (2022-05-14), and this causes some time-consuming trouble in my experiments. Thanks for pointing this out @brandbir

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants