[FEAT] Support Cohere Embeddings for SemanticChunker and SDPMChunker #118

[Udayk02]

• added every model support for Cohere embeddings.
• added the same to the registry to support AutoEmbeddings.
• added tests for the same.

key things to notice:

1. some models do not have tokenizers available. used the default "embed-english-light-v3.0" for those. (please review and conclude)
2. there is a limit of 96 on the number of documents in a single API call.
3. the model context length mentioned in the Cohere documentation is 512. but, they did not mention anywhere that it will be truncated in the case of larger texts, rather mentioned that it is not optimal. reflected the same in the code.
4. used tokenizers for the complete tokenizing via Cohere official urls as I faced an issue while loading the same through hf via autotiktoken.
5. there isn't a way to check whether the client is initialized with api_key or not. check_api_key is a deprecated function. along with that, there isn't a retry functionality provided by Cohere. handled it in chonkie end.

[bhavnicksm]

Hey @Udayk02!

Thanks for opening a PR 🚀

I am curious about point 4, you mentioned,

used tokenizers for the complete tokenizing via Cohere official urls as I faced an issue while loading the same through hf via autotiktoken.

Could you describe the issue you faced with autotiktokenizer?

Thanks ☺️

[Udayk02]

As I delved through autotiktokenizer, I found the bug. It is a very minor bug. vocab is always assumed to be a dict. But, in the cohere tokenizer.json, it is presented as a list of lists. I raised an issue there. Please check!