Is there anyway to save context for the model ? #2110
-
Hi im new to LLM and machine learning in general, i was wondering if its possible to save context of the prompts for example like chatgpt chats it remembers previous questions and i was wondering what saving the state does for llama.cpp. I want to build a local chat bot and save prompts in db. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
Yes, check out the If you want to save prompts to speed up starting from a certain point later on, you'll probably want to use Since the project is under rapid development you probably shouldn't assume cached prompt files are a reliable archive of LLM state though. |
Beta Was this translation helpful? Give feedback.
-
Thanks for the help! appreciate it |
Beta Was this translation helpful? Give feedback.
Well, not directly:
llama.cpp
(currently) only loads the cached prompts from binary files.My advice is to start out playing around with the commandline options I mentioned so you understand how the prompt cache works. After that, you can start thinking about how to interface with a DB.
Probably the approach that makes the most sense is to leave the cached prompts as files and use the DB to index rather than trying to actually the data inside the DB. Note also that the files can get quite large. The size is basically proportional to the context size: with 16bit memory (the default) I believe 2,048 tokens will end up with a 1GB file. Most of the ti…