Support for MTEB-ES (Spanish Benchmark)

[elsatch]

I am a big fan of your benchmark but I miss having information about the performance of the models in Spanish. I have checked the current codebase and some of the datasets added by Jina AI at the MTEB-ES repo. While Spanish is supported in MTEB-EU, it lacks a dedicated benchmark tab at the leaderboard.

Is there a plan to incorporate new language benchmarks before/after MMTEB v2?
Are there any technical roadblocks for Spanish integration? Would a PR adding Spanish datasets and benchmark configuration be welcome?

I'm interested in contributing to make Spanish evaluation more accessible through MTEB's leaderboard.

[isaac-chung]

Hi @elsatch !

There are currently no such new plans, but we're very close to releasing a new version of the leaderboard (under public review), which allows user to essentially create their own tab, e.g. filter by language.

We certainly welcome new additions in datasets, models, and benchmarks:

To add a dataset, follow these docs: https://github.com/embeddings-benchmark/mteb/blob/main/docs%2Fadding_a_dataset.md

Currently benchmarks are stored here: https://github.com/embeddings-benchmark/mteb/blob/main/mteb/benchmarks/benchmarks.py. These benchmarks are meant for collections of tasks with a reference, e.g. a paper.

For anyone to evaluate their model on all Spanish datasets in MTEB right now, run the following to get the tasks:

tasks = mteb.get_tasks(languages=["spa"]) # all Spanish tasks

A similar example is also shown under README: https://github.com/embeddings-benchmark/mteb?tab=readme-ov-file#task-selection