Questions about the evaluation methodology for the generated survey papers

[ddd475657357]

Hello, first of all, thank you for your open-source code. However, I have some doubts about the evaluation part in the code. It seems that your test.py only implements some examples, rather than a general benchmark for survey papers. Has the code not been fully organized yet? I look forward to your code update.

[sky-fly97]

Hello, thank you for your attention. Currently, test.py can objectively evaluate the citations of the generated survey. You only need to use the list of citations to conduct the evaluation.

For the parts subjectively evaluated by LLM, such as outline, content and other dimensions, we are still organizing them and will update them to the repo soon.

[ddd475657357]

Thank you for your reply. We successfully ran your demo and managed to generate the survey in Markdown format. The references at the end of the document do exist, but they only include article titles, which are not in the same format as the arXiv ID numbers in ref.json. Do we need to manually organize these into ref.json?
Additionally, for new topics, do we need to manually collect the corresponding ref_bench from the total database? Would it be better if these two processes could be automated?

[sky-fly97]

Thank you for your reply. We successfully ran your demo and managed to generate the survey in Markdown format. The references at the end of the document do exist, but they only include article titles, which are not in the same format as the arXiv ID numbers in ref.json. Do we need to manually organize these into ref.json? Additionally, for new topics, do we need to manually collect the corresponding ref_bench from the total database? Would it be better if these two processes could be automated?

For the first question: You can easily obtain the citation ID from the "reference" key in the .json file.

For the second question: We only conduct the benchmark on 10 topics. Therefore, new topics can be generated, but they will not be evaluated.