watt-ai tool calling

[bigrobinson]

I am curious about watt-ai 8B model, as it appears near the top of the leaderboard. It is a fine tune of meta-llama/Llama-3.1-8B-Instruct and uses the same added_tokens_decoder; however, it uses a much simpler chat template. Also, it is supposed to be a function calling model, but BFCL is using LlamaHandler (for chat) instead of LlamaFCHander (for tool calling). I find this confusing. Can you please explain the choice?

[HuanzhiMao]

Hey @bigrobinson ,

1. It’s ultimately up to the model provider which chat template they use. In the extreme case, they could rely on one template during fine-tuning and list a different one publicly, and that’s totally valid. The main requirement is that the chat template referenced in _format_prompt aligns with the one in their HuggingFace model card (so the general public sees consistent instructions).
2. For locally hosted models, the handler mostly dictates which chat template logic to apply to assemble the chat history and the parsing logic to extract tool calls from model response, rather than which mode the model is in. The watt-ai model’s chat template happens to match the llama model’s prompting approach, so they just reuse the LlamaHandler. I agree it can be confusing—maybe we should rename it or create a dedicated watt.py to help distinguish the two.

Let me know if you have more questions!

[bigrobinson]

Thanks for the reply @HuanzhiMao. You mention the chat template on the HuggingFace model card. For watt-tool, it is a very simplistic chat template that contains no handling for tool calls or tool responses. This is the origin of my confusion. From your answer and taking another look at the code, it appears the watt-tool model should work with the Llama 3.1 chat template, like ToolACE. I'm just not sure why watt-tool model card does not apparently have the right chat template.