Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LOTTE #1836

Open
Samoed opened this issue Jan 19, 2025 · 0 comments
Open

Add LOTTE #1836

Samoed opened this issue Jan 19, 2025 · 0 comments
Labels
new benchmark Issues related to adding a new benchmark

Comments

@Samoed
Copy link
Collaborator

Samoed commented Jan 19, 2025

LoTTE Long-Tail Topic-stratified Evaluation for IR that features 12 domain-specific search tests, spanning StackExchange communities and using queries from GooAQ. LoTTE focuses on relatively long-tail topics in its passages, unlike the Open-QA tests and many of the BEIR tasks, and evaluates models on their capacity to answer natural search queries with a practical inten

LoTTE, a new resource for out-of-domain evaluation of retrievers. LoTTE focuses on natural information-seeking queries over long-tail topics, an important yet understudied application space.

Introduced in ColBert2 paper https://arxiv.org/abs/2112.01488
Data https://github.com/stanford-futuredata/ColBERT/blob/main/LoTTE.md

@Samoed Samoed added the new benchmark Issues related to adding a new benchmark label Jan 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new benchmark Issues related to adding a new benchmark
Projects
None yet
Development

No branches or pull requests

1 participant