Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

calculate_results appears to only use the first correct answer, not all correct answers #1545

Open
kpounder opened this issue Aug 29, 2023 · 0 comments

Comments

@kpounder
Copy link

Describe the bug
In reviewing the code for calculate_result for QuestionAnsweringModel, I saw this line

truth_dict[answer["id"]] = answer["answers"][0]["text"]

which made me think that for evaluation we're only considering the first correct answer. However, according to the docs here

https://simpletransformers.ai/docs/qa-data-formats/#evaluation-data-format

it looks like there can be multiple correct answers for a given question.

{
                "id": "00001",
                "is_impossible": False,
                "question": "Where does the series take place?",
                "answers": [
                    {
                        "text": "region called the Final Empire",
                        "answer_start": 38,
                    },
                    {
                        "text": "world called Scadrial",
                        "answer_start": 74,
                    },
                ],
            }

Am I interpreting the code correctly? Is this a bug?

To Reproduce
N/A

Expected behavior
Take "maximum" across right answers -- i.e., compare given answer to each of right answers, and if any one is correct, then correct; else, if any one is similar, then similar; else incorrect.

Screenshots
N/A

Desktop (please complete the following information):
N/A

Additional context
N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant