Preparing Embeddings Benchmarks (LangTest)

[ArshaanNazir]

No description provided.

[JustHeroo]

@ArshaanNazir please add your self as an assignee.

[JustHeroo]

@ArshaanNazir could you share your updates?

[JustHeroo]

@chakravarthik27 please share the latest updates.

[chakravarthik27]

Hi @JustHeroo,

We have finished benchmarking the paul_graham dataset and now I am working on creating curated datasets for retrieval evaluation to do the benchmarking embedding models. I plan to generate question-answer pairs for each dataset and implement metrics to evaluate embedding models.

[JustHeroo]

@chakravarthik27 please update the latest status here.

[Cabir40]

@ArshaanNazir could you share your update

[ArshaanNazir]

@Cabir40 Kalyan is working on it. He will be enhancing embedding benchmarks for other retrieval tasks. He is working on FinBERT-QA right now.

[Cabir40]

@chakravarthik27 is there any update?

[chakravarthik27]

Hi @Cabir40,

Still, it is in progress, I am currently working alone on the langtest project and implementing a high priority feature similar to Open LLM Leaderboard by Hugging Face(Eleuther). It may take some time to complete the embedding benchmarks.