Wrong evaluation reply running on test server

[ymxyll]

[ERROR] Evaluation failed. could not convert string to float: 'B. no' The team name in your submission is [ssl-asuka].

在test server上跑的，看之前的issue 说是已经被修复了？这里针对chatgpt的错误返回格式是不是得加一些特殊的处理

[ChonghaoSima]

Please refer to sample json https://github.com/OpenDriveLab/DriveLM/blob/main/challenge/submission.json

[ymxyll]

Please refer to sample json https://github.com/OpenDriveLab/DriveLM/blob/main/challenge/submission.json

The format of my submission.json is as follows:

{
    "method": "xxx",
    "team": "xxx",
    "authors": [
        "xxx"
    ],
    "email": "xxx",
    "institution": "xxx",
    "country": "China",
    "results": [
        {
            "id": "b789de07180846cc972118ee6d1fb027_b0e6fd5561454b2789c853e5350557a8_0",
            "question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.",
            "answer": "xxx"
        },
        {
            "id": "b789de07180846cc972118ee6d1fb027_b0e6fd5561454b2789c853e5350557a8_1",
            "question": "<image>\nWhat is the moving status of object <c1,CAM_BACK,384.2,477.5>? Please select the correct answer from the following options: A. Turn right. B. Drive backward. C. Going ahead. D. Turn left.",
            "answer": "xxx"
        },
        {
            "id": "b789de07180846cc972118ee6d1fb027_b0e6fd5561454b2789c853e5350557a8_2",
            "question": "<image>\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?",
            "answer": "xxx"
        },
        {
            "id": "b789de07180846cc972118ee6d1fb027_b0e6fd5561454b2789c853e5350557a8_3",
            "question": "<image>\nWould <c1,CAM_BACK,384.2,477.5> be in the moving direction of the ego vehicle?",
            "answer": " xxx"
        }
    ]
}

After careful checking, it is confirmed to be correct. The error message "could not convert string to float: 'B. no'" seems to be caused by the evaluation code attempting to convert ChatGPT's return value to a float. This error occurred when I submitted to the leaderboard, so it should be an error in the test server rather than related to my format.

[ChonghaoSima]

After checking your submission, it seems like your answer is trying to prompt the GPT to give the answer rather than a score between your prediction and ground truth. Please check if such thing is happening on your side?