Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing results on new leaderboard #1840

Open
Muennighoff opened this issue Jan 20, 2025 · 2 comments
Open

Missing results on new leaderboard #1840

Muennighoff opened this issue Jan 20, 2025 · 2 comments
Labels
leaderboard issues related to the leaderboard

Comments

@Muennighoff
Copy link
Contributor

The new leaderboard only has 108 models on mteb classic, while the old has 382

@Muennighoff
Copy link
Contributor Author

I think this issue is really critical; we will get backlash if we deprive e.g. the top 10 models on the Chinese LB of their spot I think

Image Image

@Samoed Samoed added the leaderboard issues related to the leaderboard label Jan 20, 2025
@x-tabdeveloping
Copy link
Collaborator

x-tabdeveloping commented Jan 20, 2025

I have implemented metadata for most of these models, some of them don't show up because we know that they are not zero-shot on the benchmark. Others have problems with results I think, we should probably look into those (@Samoed) (I have no idea where xiabou-v2 is, but apparently we have no results on it , otherwise it should show up).
I have not included some other models, like IYUN or AGE_Hybrid because there is absolutely no information available about them. With IYUN for instance, @Muennighoff started a discussion, which asks where the model is (cause it's missing from the repo).
AGE has nothing in its readme, and I believe we should probably not have models on the leaderboard, which we (or any user) know nothing about.
This is especially a problem with the Chinese leaderboard. I went through most models by hand, and a lot of them had problems like this.

Not implementing gme-Qwen is an oversight, we'll have to fix it, sorry :D

I will take a look at the old leaderboard and see which models are missing, and if they are, is there a good reason for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
leaderboard issues related to the leaderboard
Projects
None yet
Development

No branches or pull requests

3 participants