Early stop with Mistral-7B

[Utanapishtim31]

I have followed the Quickstart documentation to test the Mistral-7B model. It works fine, but when I call app.query() with the proposed config.yaml file, the answer contains the same repeated sentence. I know that some models have an 'early_stop' parameter to stop the token generation when an EOS token occurs.
Is it possible to do the same with embedchain + Mistral-7B?

[deven298]

Hey @Utanapishtim31 Thanks for trying out Embedchain!

One way to control the length of response is defining the max_tokens config param. I think defining this should solve the problem of repeated answers. Please refer - https://docs.embedchain.ai/api-reference/advanced/configuration

You can also try the Mistral model with Mistral API we recently added to the Embedchain! Refer - https://docs.embedchain.ai/components/llms#mistral-ai

Please feel free to reach out to us if you are having trouble developing your RAG application with Embedchain.

[Utanapishtim31]

Thank you for your help. I'm currently using HuggingFace LLMs because the interface of Embedchain makes it really easy. However, I had to customize the prompt template in the config.yaml file to get a correct result depending on the LLM used (for the moment "mistralai/Mistral-7B-Instruct-v0.2").
I have seen that the HuggingFace API gives access to a method tokenizer.apply_chat_template() to build a query according to the model specifics. It would be nice to give the possibility to use this method instead of setting the prompt in the configuration file to avoid having to tune the prompt each time we try another LLM.
To make the link with the subject of my question, it seems that the problem of the answer never ending with Mistral-7B-Instruct was solved once I started the prompt with the token "<s>", which is not obvious unless you dive into the documentation of this model. Using apply_chat_template() would have automatically added this token at the beginning of the prompt.

[deven298]

@Utanapishtim31 That is a great insight and it would be good to apply tokenizer when using HuggingFace LLMs. Would you be interested in contributing by adding this?

[vineeth-balusani]

Hi @Utanapishtim31,
I have been trying to use huggingface llm and embedder but running into some issues. Can you share your config file, please?

Here is the error that I am getting -
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like sentence-transformers/all-mpnet-base-v2 is not the path to a directory containing a file named config.json.