New leaderboard filtering cross-encoders

[Muennighoff]

Old LB has the ability to filter out CEs which is mostly relevant for w/ Instructions

[x-tabdeveloping]

We should probably annotate this somehow in the model metadata. Perhaps add it as a framework?
Btw I might be using the filters wrong, but it doesn't seem to me that the cross-encoders button works as intended on the old leaderboard. If I only select that one, nothing shows up, if I select that and Open a lot of models show up that are not cross-encoders, and even some non-open ones.

[Samoed]

I think we can add an is_cross_encoder field to the model meta. The approach with the framework wouldn’t work, as most of them are using Sentence Transformers.

[x-tabdeveloping]

Yeah but I imagine you could add frameworks=["Sentence Transformers", "Cross Encoders"] and then you would know that they are sentence-transformer compatible.
is_cross_encoder is not a bad idea either, we could totally do that.

[Samoed]

We have some reranking models that don't use Sentence Transformers, and since there's no framework specifically called CrossEncoders, it can be a bit confusing.

mteb/mteb/models/rerankers_custom.py

Line 93 in 2fac8ba

class MonoBERTReranker(RerankerWrapper):

[x-tabdeveloping]

Alright, we do it that way then

[sam-hey]

As mentioned in #1869 (comment), I suggest developing a more general method that is not limited to Cross-Encoders but can also be applied to other model categories, such as Late Interaction.