Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: v2.0.0 #1433

Draft
wants to merge 79 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
e2520df
fix: Ensure seed is based on RNG State (#1193)
KennethEnevoldsen Nov 11, 2024
9c58518
Consolidate Retrieval/Reranking/Instruction Variants (#1359)
orionw Nov 13, 2024
2a8a370
fix: Unsure TaskResults can handle runtime and version being unspecified
KennethEnevoldsen Nov 14, 2024
dea2b77
Merge branch 'v2.0.0' of https://github.com/embeddings-benchmark/mteb…
KennethEnevoldsen Nov 14, 2024
23d6cb2
fix: remove NaN handling for retrieval
KennethEnevoldsen Nov 14, 2024
8868cd4
Merge branch 'main' into v2.0.0
KennethEnevoldsen Nov 14, 2024
5470c88
fix: Merge main into v2 (#1454)
Samoed Nov 14, 2024
70a3ff2
feat: enable codecarbon by default (#1428)
Samoed Nov 15, 2024
0e9b6fd
Add decriptive stat almost to all datasets (#1466)
Samoed Nov 18, 2024
0a5bedb
fix: Fix test for empty descriptive tasks (#1413)
Samoed Nov 19, 2024
6da2a1a
fix: pin datasets version <3.0.0 (#1471)
Napuh Nov 19, 2024
a27de33
feat: Multilingual retrieval loader (#1473)
Samoed Nov 19, 2024
0df0210
fix: add citations to ModelMeta (#1477)
Samoed Nov 21, 2024
0abe1a0
Add descriptive stats to mising tasks and add number of qrels (#1476)
imenelydiaker Nov 21, 2024
a7a5214
1475 add descriptive stats to all tasks v2 (#1482)
dokato Nov 23, 2024
99247b2
fix: Fix `BrightRetrieval` calculate stats (#1484)
Samoed Nov 23, 2024
022d355
Merge main v2 (#1504)
Samoed Nov 27, 2024
6383950
Fix: retrieval stats (#1496)
Samoed Nov 27, 2024
d54fb75
fix: hatespeech filipino (#1522)
Samoed Nov 28, 2024
dec5d6a
feat: Forbid task metadata and add upload functions (#1362)
Samoed Dec 4, 2024
d0aa3a7
fix: remove `*` imports (#1569)
Samoed Dec 9, 2024
f16deb6
Merge branch 'refs/heads/main' into v2.0.0
Samoed Dec 10, 2024
06fc13f
fix: Add documentation (#1567)
KennethEnevoldsen Dec 16, 2024
6a8e188
fix: reorder argument for mteb.get_tasks (#1597)
KennethEnevoldsen Dec 18, 2024
d6130ad
fix: Make deduplication in PairClassificationEvaluator stable (#1315)
tsirif Dec 19, 2024
c9b00ac
[V2] Update v2 (#1618)
Samoed Dec 22, 2024
3f4a0da
Merge branch 'refs/heads/main' into v2.0.0
Samoed Dec 22, 2024
71c46ea
fix: [V2] Update datasets wich can't be loaded with `datasets>=3.0` (…
Samoed Dec 22, 2024
b3693fb
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 4, 2025
2519c7a
update nanobenchmark stat
Samoed Jan 4, 2025
9bc4a1a
[v2] Remove metadata dict (#1719)
Samoed Jan 8, 2025
38b9dad
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 10, 2025
cc829e5
lint
Samoed Jan 10, 2025
4247e22
[v2] Remove memory usage (#1751)
Samoed Jan 11, 2025
2b41cb4
[v2] fix contriever (add similarity_fn_name to ST wrapper) (#1749)
Samoed Jan 11, 2025
91871fe
[v2] Refactor evaluators and Abstasks (#1707)
Samoed Jan 12, 2025
997a135
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 12, 2025
f73e7ac
openai remove memory usage
Samoed Jan 12, 2025
d946ad4
fix: [v2] _run_eval() for case: co2_tracker False & add test (#1774)
sam-hey Jan 12, 2025
81a272e
Fix RepLLaMA-based models and Instructions for Cross-Encoders (#1733)
orionw Jan 13, 2025
8cf6178
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 13, 2025
296b9ea
lint
Samoed Jan 13, 2025
54018c7
[v2] Remove deprecated parameters from `MTEB` and cli (#1773)
Samoed Jan 15, 2025
3a5aa0c
[v2] remove metadata_dict (#1820)
Samoed Jan 15, 2025
ce5cb3e
[v2] add similarity_fn in ModelMeta (#1759)
sam-hey Jan 17, 2025
5d738bc
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 20, 2025
77f7c83
fix merge
Samoed Jan 20, 2025
6da8a13
[v2] ci: run bm25 and ColBERT test in ci (#1829)
sam-hey Jan 21, 2025
f1d418c
[v2] Update v2 again (#1864)
Samoed Jan 24, 2025
c26adee
Merge branch 'refs/heads/main' into v2.0.0
Samoed Jan 24, 2025
ebca6bc
[v2] Refactor retrieval (#1750)
Samoed Feb 2, 2025
904f61c
[v2] remove `save_suffix` (#1924)
Samoed Feb 3, 2025
a619ad8
[v2] Remove multilingual task (#1926)
Samoed Feb 4, 2025
3d1f80c
[v2] Merge main (#1927)
Samoed Feb 4, 2025
9c5665e
[v2] add metadata for suppressed tasks (#1995)
Samoed Feb 7, 2025
724f553
[v2] Reupload `BEIR-PL` (#1985)
Samoed Feb 7, 2025
7889200
[v2] feat: add enums for ModelMeta similarity_fn_name (#1975)
sam-hey Feb 8, 2025
ef7b788
[v2] Merge `MIEB` into v2 (#1973)
Samoed Feb 9, 2025
e74b4e7
Merge branch 'refs/heads/main' into v2.0.0
Samoed Feb 9, 2025
4fdd1a8
lint
Samoed Feb 9, 2025
cbacef1
rename test file to resolve pytest conflicts
Samoed Feb 9, 2025
ca60b82
Add LoTTE Benchmark to MTEB (#2009)
agu18dec Feb 14, 2025
9461759
Integrate Birco V2 (#2022)
AdnanElAssadi56 Feb 16, 2025
056c09a
Convert task category to indicate modality (#2107)
isaac-chung Feb 20, 2025
8131cd6
Add PUGGRetrieval task for the Polish language
Feb 20, 2025
472b527
Add PUGGRetrieval
Feb 21, 2025
ccfff84
Merge main into v2 (#2116)
KennethEnevoldsen Feb 24, 2025
cad748d
Merge branch 'refs/heads/main' into v2.0.0
Samoed Feb 24, 2025
8c27b80
[v2] reupload reranking datasets in old format (#2097)
Samoed Feb 24, 2025
4987c73
[v2] Merge from main (#2145)
KennethEnevoldsen Feb 24, 2025
9e96b06
[v2] test: Pylate version 1.1.6 (#2190)
sam-hey Feb 28, 2025
b316649
Update mteb/tasks/Retrieval/pol/PUGGRetrieval.py
Samoed Mar 1, 2025
d3afd0a
Merge branch 'v2.0.0' into v2.0.0
Samoed Mar 1, 2025
765418e
add metadatafile
Samoed Mar 1, 2025
d491800
[v2] Merge main (#2204)
Samoed Mar 4, 2025
6f4b6bb
Merge branch 'refs/heads/main' into v2.0.0
Samoed Mar 4, 2025
0fb363b
fix: Allow model to output torch.tensor (#2234)
KennethEnevoldsen Mar 4, 2025
bd33a33
[v2] Refactor text tasks to use DataLoader (#2198)
Samoed Mar 7, 2025
53b830b
Merge branch 'refs/heads/main' into v2.0.0
Samoed Mar 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Fix BrightRetrieval calculate stats (#1484)
* fix bright loader

* lint

* fix comment
Samoed authored Nov 23, 2024

Verified

This commit was signed with the committer’s verified signature.
vzaliva Vadim Zaliva
commit 99247b29e6459db1363cdc1575a5582b67c0366d
16 changes: 15 additions & 1 deletion mteb/abstasks/AbsTaskRetrieval.py
Original file line number Diff line number Diff line change
@@ -379,6 +379,13 @@ def _calculate_metrics_from_split(
top_ranked = None
instructions = None
if hf_subset and hf_subset in self.queries:
# BrightRetrieval has different splits for different subsets of the corpus.
if (
self.corpus.get(hf_subset, None) is None
or self.corpus[hf_subset].get(split, None) is None
):
return {}

queries = self.queries[hf_subset][split]
corpus = self.corpus[hf_subset][split]
relevant_docs = self.relevant_docs[hf_subset][split]
@@ -393,6 +400,12 @@ def _calculate_metrics_from_split(
instructions = {}
top_ranked = {}
for hf_subset in self.metadata.eval_langs:
# BrightRetrieval has different splits for different subsets of the corpus.
if (
self.corpus.get(hf_subset, None) is None
or self.corpus[hf_subset].get(split, None) is None
):
continue
queries.update(process_docs(self.queries, hf_subset, split))
corpus.update(process_docs(self.corpus, hf_subset, split))
relevant_docs.update(
@@ -506,7 +519,8 @@ def calculate_length(
queries_lens.append(len(query))
else:
queries_lens.extend([len(turn) for turn in query])

if corpus is None:
return None, queries_lens
for doc in corpus.values():
if isinstance(doc, dict):
doc_lens.append(len(doc["text"]))
Loading