Can GPTcache to refer the previous cache record?

[terryweijian]

Hi ,

I have a script contains 4 questions and loop those 4 questions 4 times. I run this script several times, so the cache database have the questions record. But as the time record it looks there is no hit cache record.
So want to understand the GPTcache behavior for this scenario, If it only can check the cache record within the session and will not refer the old records that generated in previous session? Or there is any switch or setting can change it?

here is the script result ( local answer means result from LLM model directly, and cache answer means go through GPTcache

First Loop
Question: what is TV ?
local answer: TV is a television channel that broadcasts live television.
Local Time Spent = 0.2
Cache answer: TV is a television channel that broadcasts live television.
Cache Hit Time Spent = 0.39 # the time indicates cache miss

---

Question: can you explain what function of TV is ?
local answer: a television channel
Local Time Spent = 0.09
Cache answer: TV is a television channel that broadcasts live television.
Cache Hit Time Spent = 0.04

---

Question: can you tell me more about TV ?
local answer: a tv show
Local Time Spent = 0.11
Cache answer: TV is a television channel that broadcasts live television.
Cache Hit Time Spent = 0.03

---

Question: what is the function of money ?
local answer: money is a currency
Local Time Spent = 0.1
Cache answer: money is a currency
Cache Hit Time Spent = 0.11

---

Second Round in the same session

Question: what is TV ?
local answer: TV is a television channel that broadcasts live television.
Local Time Spent = 0.17
Cache answer: TV is a television channel that broadcasts live television.
Cache Hit Time Spent = 0.03 # the time indicates the cache hit

---

Question: can you explain what function of TV is ?
local answer: a television channel
Local Time Spent = 0.1
Cache answer: money is a currency
Cache Hit Time Spent = 0.04

---

Question: can you tell me more about TV ?
local answer: a tv show
Local Time Spent = 0.12
Cache answer: TV is a television channel that broadcasts live television.
Cache Hit Time Spent = 0.03

---

Question: what is the function of money ?
local answer: money is a currency
Local Time Spent = 0.11
Cache answer: money is a currency
Cache Hit Time Spent = 0.03

---

[SimFG]

@terryweijian can you show me your code snippet for the cache initialization? if possible, you can give me a demo code.

[terryweijian]

from langchain.llms import HuggingFacePipeline
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM
from langchain import PromptTemplate, HuggingFaceHub, LLMChain
from langchain import PromptTemplate

from gptcache.adapter.langchain_models import LangChainLLMs
from gptcache.manager import get_data_manager, CacheBase, VectorBase
from gptcache import Cache
from gptcache.embedding import Onnx
from gptcache.processor.pre import get_prompt
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

import time
import os
from transformers import AutoModel, AutoTokenizer
from gptcache.embedding import Huggingface

os.environ['HUGGINGFACEHUB_API_TOKEN'] = ''
path = "*************************************"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForSeq2SeqLM.from_pretrained(path)
pipe = pipeline(
"text2text-generation",
model=model,
tokenizer=tokenizer,
max_length=100
)
huggingface = Huggingface(model='******')
local_llm = HuggingFacePipeline(pipeline=pipe)
llm_cache = Cache()
cache_base = CacheBase('mysql', sql_url='mysql+pymysql://root:*************@127.0.0.1:3306/justtest')
vector_base = VectorBase('faiss', dimension=huggingface.dimension)
data_manager = get_data_manager(cache_base, vector_base, max_size=100, clean_size=2)
llm_cache.init(
pre_embedding_func=get_prompt,
embedding_func=huggingface.to_embeddings,
data_manager=data_manager,
similarity_evaluation=SearchDistanceEvaluation(),
)

questions = [
'what is TV ?',
'can you explain what function of TV is ?',
'can you tell me more about TV ?',
'what is the function of money ?',
'what is TV ?',
'can you explain what function of TV is ?',
'can you tell me more about TV ?',
'what is the function of money ?',
]
for question in questions:
print(f'Question: {question}')
before = time.time()
answer = local_llm(prompt=question)
print(f'local answer: {answer}')
print("Local Time Spent =", round((time.time() - before), 2))
before = time.time()
cached_llm = LangChainLLMs(llm=local_llm)
answer = cached_llm(prompt=question, cache_obj=llm_cache)
print(f'Cache answer: {answer}')
print("Cache Hit Time Spent =", round((time.time() - before), 2))
print("_______________________________________________________________")

[terryweijian]

And here is the questions record in the database

[terryweijian]

So you can see the question record from 18~23, every time I run the above script, GPTcache records two new questions. But it didn't checked the old records in DB, and actually there are the same questions already be there. So the cache hit only happen within this session. I believe the best case is that can refer the old records in database. Is that possible?

Thanks

[SimFG]

Thanks your feedback, i will checkout it and then reply you.

[SimFG]

@terryweijian yes, we don't check the existed data when inserting the data to cache, which is a good idea. I will improve about this part in a few days.

For better solving your problem, I simply adjusted the program, but I don't seem to have encountered the problem you mentioned. I used the openai model, the default onnx embedding model and the traditional database uses sqlite. I guess the problem is caused by the unstable embedding model. And you can remove all history data and retry it. Below is my sample code:

import time

from langchain.llms import OpenAI

from gptcache import Cache, cache
from gptcache.adapter.api import init_similar_cache
from gptcache.adapter.langchain_models import LangChainLLMs

cache.set_openai_key()
local_llm = OpenAI()

llm_cache = Cache()
init_similar_cache(data_dir="llm_cache", cache_obj=llm_cache)

questions = [
    'what is TV ?',
    'can you explain what function of TV is ?',
    'can you tell me more about TV ?',
    'what is the function of money ?',
]
for question in questions:
    before = time.time()
    cached_llm = LangChainLLMs(llm=local_llm)
    answer = cached_llm(prompt=question, cache_obj=llm_cache)
    print(f'Question: {question}')
    print(f'Cache answer: {answer}')
    print("Cache Hit Time Spent =", round((time.time() - before), 2))
    print("_______________________________________________________________")

first running output

second running output

the db data

[terryweijian]

@SimFG ， Thanks a lot for your checking!
It looks this problem didn't happen in your environment.
I noticed there are two timestamps in the database record, "create_on" and "last_access", I suppose that old records can be referred by the other new running cycle, so the last access time should be newer than the create_on time. But in your screen shoot, that two time is very close. What is the actual design at here? Suppose you run your program first time and then wait for 5 minutes and run it second time, the last access time should be newer 5 minutes than the create on time, and means all cache hits in second time, is that correct?

And I tried to use the Onnx() and SQLite in my program, but the problem is still there

Do you mind share all your program? I didn't see the CacheBase, VectorBase and get_data_manager functions setting. Not sure if have relationship with the current program.

Thanks

[SimFG]

I have actually given my test code above, you can try it locally, pay attention to changing the cache data directory. The initialization of the cache is wrapped in the init_similar_cache method, which you can also access.

[terryweijian]

ok, I see. the init_similar_cache method defines the detault CacheBase, VectorBase .
I used the init_similar_cache method in my program, but the data base records are the same as before, It will add two new questions after I run the script every time. Not sure where is the problem. Now the difference of the two scripts are that I used the Hugging face model, and your is OpenAI.

[SimFG]

yes, so i guess the embedding model that you choose isn't stable. you can try to run my demo code in your computer.

[terryweijian]

I used Llama CPP and ggml-model-q4_0.bin model, then cannot find the issue again. I agree with you that embedding model probably the root cause .
Very interesting thing is after I used the new script of LlamaCPP and ggml model to generate the cache records, and rerun the previous script, then issue also didn't happen again. The result shows all cache hits with the new cache records. See the following my cache records

[SimFG]

Cool, when adding the cache data, there isn't a check, which is improved in the future.