You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I want chonkie to support Contextual Retrieval that Anthropic talks about in their blog out of the box.
We can add the contextual retrieval from Anthropic via a ContextualRefinery class as a part of the Refinery module, such that given a particular generative module and original text, the ContextualRefinery can add appropriate context to the chunks, which can then be used for embeddings for vector search or in a bm25 index.
The API should look something like:
fromchonkieimportContextualRefinery, TokenChunker, AnthropicGeniegenie=AnthropicGenie()
refinery=ContextualRefinery(genie)
chunker=TokenChunker()
text= ...
chunks=chunker(text)
contextual_chunks=refinery(chunks)
# use the chunks normally!
To get to this point, we need to complete the following:
Add support for the Genie classes, specifically AnthropicGenie
Plan for saving structured prompts for each genie inside the genie module
Create the ContextualRefinery that can take in a genie module
The text was updated successfully, but these errors were encountered:
Hey there! 🦛
I want chonkie to support Contextual Retrieval that Anthropic talks about in their blog out of the box.
We can add the contextual retrieval from Anthropic via a
ContextualRefinery
class as a part of theRefinery
module, such that given a particular generative module and original text, theContextualRefinery
can add appropriate context to the chunks, which can then be used for embeddings for vector search or in a bm25 index.The API should look something like:
To get to this point, we need to complete the following:
Genie
classes, specificallyAnthropicGenie
The text was updated successfully, but these errors were encountered: