Would gptCache only work for non memory Chains?

[jasan-s]

I'm integrating gptCache with langchain in hopes of improving user experience. I'm using ConversationalRetrievalChain which adds history to memory and appends conversation to the new question thus the input always changes.

Would this lib be helpful in my case?

[SimFG]

I don't quite understand, can you give some examples of scenarios?

[jasan-s]

Scenario:
Chat with a AI answering questions conversationally. Each new user input and AI response i.e chat history is fed to llm model to improve new ai response.

In this scenario with the following conversations:

User: hi
AI: hello, what can I help you with? (LlM)
User: what's the summary of this document?
AI: the document is about effects of coding on the brain.(LLM)
User: I see.

User 2: Give me a doc summary
AI: the document is about effects of coding on the brain.(cache using semantic cache)

So user 2 asked a similar question, in this the LLM doesn't need calling. Since the response can be served from cache.

[SimFG]

Thanks for your example, I understand that you want to apply GPTCache in multiple rounds of conversations.

At present, we actually have some methods to deal with this scenario, such as using context process in the preprocessing stage, such as summarization_context, selective_context, concat_context + rwkv embedding (api reference: https://gptcache.readthedocs.io/en/latest/references/processor.html), and using rerank similarity evaluation to further confirm whether the cache is valid, such as cohere_rerank, sbert_crossencoder (api reference: https://gptcache.readthedocs.io/en/latest/references/similarity_evaluation.html).

However, these parts are all for openai requests. If you want to use it in langchain, you may need to make some adjustments yourself