Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Add support for code file chunking #117

Open
3 tasks done
bvqbao opened this issue Dec 31, 2024 · 1 comment
Open
3 tasks done

[FEAT] Add support for code file chunking #117

bvqbao opened this issue Dec 31, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@bvqbao
Copy link

bvqbao commented Dec 31, 2024

Checklist
Before you begin, please make sure of the following points:

  • This feature is not implemented yet in the library in any form
  • This feature has not been suggested previously
  • This feature is relevant to the library

Describe the solution you'd like

Something like this code chunking algo?
https://docs.sweep.dev/blogs/chunking-2m-files

This is useful for code-related RAG implementation.

@bvqbao bvqbao added the enhancement New feature or request label Dec 31, 2024
@bhavnicksm
Copy link
Collaborator

Hey @bvqbao!

Thanks for opening an issue 😊

Yes, we actually had this in mind for Chonkie's future releases! Though at this may take some time to add as it's not an immediate priority at the moment~ We want to focus on getting the core chunkers of Chonkie to be stable, adding additional chunking methods, and offering support to greater input modalities first (e.g., Markdown, PDF, JSON etc.). We definitely plan for code chunking before the v1 stable release, however, and possibly even in Q1/Q2 2025.

I hope you continue to support Chonkie as it grows. :))

Thanks~ ☺️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants