Does Dify have plans to support rerank and embedding models launched by vLLM?

[massif-01]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Does Dify have plans to support rerank and embedding models launched by vLLM?
When deploying large models and Dify locally on performance-limited devices (such as Orin X or a Mac mini with 64GB of RAM), using Xinference to launch the models can consume precious memory. Using VLLM to launch various models together is a better choice.

2. Additional context or comments

No response

3. Can you help us with this feature?

• I am interested in contributing to this feature.

[HiddenPeak]

+1

[crazywoola]

Because of this #11588, we no longer accept the prs related to model runtimes.
We are going to launch v1.0 in the near future. After that, I think the community could help with that.

[HiddenPeak]

vLLM rerank task api :

• Route: /score, Methods: POST
• Route: /v1/score, Methods: POST
  I try to modify the route /v1/score to /v1/rerank
  but dify show me the notice paramate error

waiting for your v1.0 😭

[dosubot]

Hi, @massif-01. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

• You inquired about Dify's plans to support rerank and embedding models from vLLM for better memory efficiency.
• HiddenPeak supported your inquiry and provided details about vLLM's rerank task API, noting a parameter error when modifying the route.
• Crazywoola mentioned that due to a related issue, pull requests for model runtimes are not being accepted, but community help might be possible post Dify v1.0 launch.
• HiddenPeak expressed anticipation for the Dify v1.0 release.

Next Steps

• Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

[hundredwz]

vLLM rerank task api :

• Route: /score, Methods: POST
• Route: /v1/score, Methods: POST
  I try to modify the route /v1/score to /v1/rerank
  but dify show me the notice paramate error

waiting for your v1.0 😭

vLLM 0.7.2 now support /v1/rerank.

However, we will face 404 error when trying to add rerank model to dify, since there is a bug in rerank.py

we could modify the code in L67 to following

 data = {"model": model_name, "query": query, "documents": docs, "return_documents": True}
  if top_n is not None:
      data["top_n"] = top_n

If premitted, I could submit pr to fix it.