Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Supporting opensource multimodal LLMs such as Llama3.2 11B or Llama3.2 90B #406

Merged
merged 3 commits into from
Nov 19, 2024
Merged

Supporting opensource multimodal LLMs such as Llama3.2 11B or Llama3.2 90B #406

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
061aa98
Supporting opensource multimodal LLMs such as Llama3.1 11B or Llama3.…
hiamitabha Nov 13, 2024
3178a52
Fixing code review on adding condition to check for custom provider
hiamitabha Nov 16, 2024
011a01f
Alligning braces
hiamitabha Nov 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions chipper/pkg/wirepod/ttr/kgsim_cmds.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ var ValidLLMCommands []LLMCommand = []LLMCommand{
// not impl yet
ParamChoices: "front, lookingUp",
Action: ActionGetImage,
SupportedModels: []string{openai.GPT4o, openai.GPT4oMini},
SupportedModels: []string{"all"},
},
{
Command: "newVoiceRequest",
Expand Down Expand Up @@ -152,7 +152,7 @@ func ModelIsSupported(cmd LLMCommand, model string) bool {
func CreatePrompt(origPrompt string, model string, isKG bool) string {
prompt := origPrompt + "\n\n" + "Keep in mind, user input comes from speech-to-text software, so respond accordingly. No special characters, especially these: & ^ * # @ - . No lists. No formatting."
if vars.APIConfig.Knowledge.CommandsEnable {
prompt = prompt + "\n\n" + "You are running ON an Anki Vector robot. You have a set of commands. If you include an emoji, I will make you start over. If you want to use a command but it doesn't exist or your desired parameter isn't in the list, avoid using the command. The format is {{command||parameter}}. You can embed these in sentences. Example: \"User: How are you feeling? | Response: \"{{playAnimationWI||sad}} I'm feeling sad...\". Square brackets ([]) are not valid.\n\nUse the playAnimation or playAnimationWI commands if you want to express emotion! You are very animated and good at following instructions. Animation takes precendence over words. You are to include many animations in your response.\n\nHere is every valid command:"
prompt = prompt + "\n\n" + "You are running ON an Anki Vector robot. You have a set of commands. If you include an emoji, I will make you start over. If you want to use a command but it doesn't exist or your desired parameter isn't in the list, avoid using the command. The format is {{command||parameter}}. You can embed these in sentences. Example: \"User: How are you feeling? | Response: \"{{playAnimationWI||sad}} I'm feeling sad...\". Square brackets ([]) are not valid.\n\nUse the playAnimation or playAnimationWI commands if you want to express emotion! You are very animated and good at following instructions. Animation takes precendence over words. You are to include many animations in your response.\n\nHere is every valid command:"
for _, cmd := range ValidLLMCommands {
if ModelIsSupported(cmd, model) {
promptAppendage := "\n\nCommand Name: " + cmd.Command + "\nDescription: " + cmd.Description + "\nParameter choices: " + cmd.ParamChoices
Expand Down Expand Up @@ -476,7 +476,11 @@ func DoGetImage(msgs []openai.ChatCompletionMessage, param string, robot *vector
c = openai.NewClientWithConfig(conf)
} else if vars.APIConfig.Knowledge.Provider == "openai" {
c = openai.NewClient(vars.APIConfig.Knowledge.Key)
}
} else {
hiamitabha marked this conversation as resolved.
Show resolved Hide resolved
conf := openai.DefaultConfig(vars.APIConfig.Knowledge.Key)
conf.BaseURL = vars.APIConfig.Knowledge.Endpoint
c = openai.NewClientWithConfig(conf)
}
ctx := context.Background()
speakReady := make(chan string)

Expand Down
Loading