mxbai-rerank-large-v2

[qdrddr]

Model description

Currently mixedbread-ai/mxbai-rerank-large-v2 does not support rerank. Reason: the loaded model cannot fullfill rerank. Options are 'embed'.

infinity_emb v2 --port 7997 \
  --model-id mixedbread-ai/mxbai-rerank-large-v2 --batch-size 8 --revision "refs/heads/main"

curl --location 'http://localhost:7997/v1/rerank' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
    "model": "mixedbread-ai/mxbai-rerank-large-v2",
    "query": "What is the capital of the United States?",
    "documents": [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 3
  }'

{"error":{"message":"ModelNotDeployedError: model=`mixedbread-ai/mxbai-rerank-large-v2` does not support `rerank`. Reason: the loaded moded cannot fullyfill `rerank`. Options are {'embed'}.","type":null,"param":null,"code":400}}

Open source status & huggingface transformers.

• The model implementation is available on transformers
• The model weights are available on huggingface-hub
• I verified that the model is currently not running in the latest version pip install infinity_emb[all] --upgrade
• I made the authors of the model aware that I want to use it with infinity_emb & check if they are aware of the issue.

[michaelfeil]

https://huggingface.co/michaelfeil/mxbai-rerank-large-v2-seq
https://huggingface.co/michaelfeil/mxbai-rerank-base-v2-seq

The prompt for classification and background in this script: https://github.com/michaelfeil/infinity/blob/main/docs/lm_head_to_classifier/convert_lm.py

I'd recommend running this model as a "classifier" and applying the chat template similar to the mixedbread v2 reference implementation locally (client side!)

[qdrddr]

Something like this?
https://github.com/qdrddr/infinity-mxbai-rerank-seq-v2

[michaelfeil]

Your prompt is incorrect, use this once, together with chat template.
/rerank is not going to cut it, it needs to be a sequence classification / classify endpoint model.

https://github.com/mixedbread-ai/mxbai-rerank/blob/ca0c55d03770d9bb183ca759850bf7cdfbcc9f50/mxbai_rerank/mxbai_rerank_v2.py#L34

Edit: I added the template here: https://huggingface.co/michaelfeil/mxbai-rerank-large-v2-seq

[qdrddr]

Thanks, that worked!

[rawsh-rubrik]

@michaelfeil dumb question, where did you find the instruct prompt? feel like they had a list of example instructions for different tasks on release but I can't find the list

[rawsh-rubrik]

Also - seeing Cannot handle batch sizes > 1 if no padding token is defined out of the box with the wrapped model https://huggingface.co/michaelfeil/mxbai-rerank-large-v2-seq

{
  "input": [
    "<|endoftext|><|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n<|im_end|>\n<|im_start|>user\nquery: symptoms of flu \ndocument: Common flu symptoms include fever, chills, muscle aches, cough, congestion, runny nose, headaches, and fatigue. Most people recover from flu within a week, but some may develop complications such as pneumonia. \nYou are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:<|im_end|>\n<|im_start|>assistant\n",
    
    "<|endoftext|><|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n<|im_end|>\n<|im_start|>user\nquery: symptoms of flu \ndocument: The best way to prevent seasonal flu is to get vaccinated every year. Flu vaccines cause antibodies to develop in the body about two weeks after vaccination, providing protection against infection. \nYou are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:<|im_end|>\n<|im_start|>assistant\n",
    
    "<|endoftext|><|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n<|im_end|>\n<|im_start|>user\nquery: symptoms of flu \ndocument: Diabetes is a chronic medical condition that affects how your body turns food into energy. If you have diabetes, your body either doesn't make enough insulin or can't use the insulin it makes as well as it should. \nYou are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:<|im_end|>\n<|im_start|>assistant\n",
    
    "<|endoftext|><|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n<|im_end|>\n<|im_start|>user\nquery: how to code in Python \ndocument: Python is a high-level, interpreted programming language known for its readability and simple syntax. To start coding in Python, install Python from python.org, set up an IDE like PyCharm or VS Code, and begin with basic syntax like variables, loops, and functions. \nYou are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:<|im_end|>\n<|im_start|>assistant\n",
    
    "<|endoftext|><|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n<|im_end|>\n<|im_start|>user\nquery: how to code in Python \ndocument: Java is a class-based, object-oriented programming language developed by Sun Microsystems. It is designed to have as few implementation dependencies as possible and follows the WORA (Write Once, Run Anywhere) principle. \nYou are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:<|im_end|>\n<|im_start|>assistant\n"
  ],
  "raw_scores": false
}

[rawsh-rubrik]

@michaelfeil opened PR on base and large. tested with:

infinity_emb v2 --model-id michaelfeil/mxbai-rerank-large-v2-seq --device mps --engine torch --no-model-warmup --revision refs/pr/1