Ongoing ramblings?

[HenryNewcomer]

I've tested several different Llama 2 models (and adjusted the prompts accordingly), but I keep getting weird outputs. It's just a simple attempt at prompting, but the responses tend to go on and on, and rarely answer my question. Testing this with a standalone Llama.cpp instance, it works as expected, but when I use llama-cpp-python with langchain, this odd behavior starts back up.

Thoughts on what I'm doing wrong? I'm thinking the prompt may need tweaked, but whatever I change it to, it doesn't help.

from langchain import PromptTemplate, LLMChain
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

template_llama2_uncensored = """### HUMAN:
{prompt}

### RESPONSE:"""

prompt = PromptTemplate(template=template_llama2_uncensored, input_variables=["prompt"])

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

n_gpu_layers = 1
n_batch = 512

llm = LlamaCpp(
    model_path="../models/llama-2-7b.ggmlv3.q2_K.bin",
    n_gpu_layers=n_gpu_layers,
    n_batch=n_batch,
    f16_kv=True,
    callback_manager=callback_manager,
    verbose=True,
)

running = True

while running:
    prompt = input("")
    if prompt == "exit":
        running = False
    else:
        print(llm(prompt))

[rlleshi]

Can confirm this is the case

[MadsRC]

Can confirm that running the above code and attempting to have a conversation leads to gibberish...

At one point it started responding to me in German... Now I did play around with this model and langchain and some instructions to respond to me in German earlier today (from here: https://python.langchain.com/docs/use_cases/question_answering/how_to/chat_vector_db) in the same venv, so I'm not sure if there's some caching at some level somewhere?

Still pretty new to llama-cpp-python and langchain so I might be missing something.

I also tried tweaking the prompt using the structure from here but that didn't help at all.