v0.4.0

Highlights

Added the new RecursiveChunker that uses complex recursive rules to create structurally meaningful chunks, maintaining natural separations as much as possible. Try it out~

from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=512)
chunks = chunker("Woah! Chonkie has it's own recursive chunker now~ so cooool!")

What's Changed

• Add initial support for Recursive Chunking (RecursiveChunker) by @bhavnicksm in #107
• [FEAT] Add support for RecursiveChunking + minor fixes by @bhavnicksm in #108
• [fix] Correct the start and end indices for TokenChunker in Batch mode (#84) by @bhavnicksm in #109
• [fix] Correct the start and end indices for TokenChunker in Batch mode (#84) by @bhavnicksm in #110
• [fix] #106: Missing last sentence in the SemanticChunker by @bhavnicksm in #112
• [fix] Add fix for #106: Reconstruction tests for SemanticChunker failing, missing last sentence by @bhavnicksm in #113
• [chore] Bump version to "v0.4.0" + minor change by @bhavnicksm in #114

Full Changelog: v0.3.0...v0.4.0