HybridChunker with tiktoken tokenizer

[ruizguille]

Hi,

What would be the recommended way to use HybridChunker with the tiktoken tokenizer for use with OpenAI's embedding models? Reading the code, it seems to accept only HuggingFace tokenizers.

Thank you for the amazing library!

[vagenas]

Hi @ruizguille 👋

At the moment, HybridChunker itself indeed only supports HF tokenizers (transformers.PreTrainedTokenizerBase).

That said, the actual text splitting library used in parts of the workflow —semchunk— already supports tiktoken.

So one could expand the HybridChunker such that it can operate with both, by allowing self._tokenizer be of that Union.

👉 Based on the usage of self._tokenizer, one would need to resolve the following for tiktoken (equivalently to HF):

• the max tokens allowed for the model
• the number of tokens a given piece of text would correspond to
• additionally, as string input is still supported, one would have to decide which of HF / tiktoken that would get mapped to (I'd say HF for backwards compatibility — but one could also pursue both on a trial-and-error basis)

Would you be interested in submitting a PR yourself? 🙌

[ruizguille]

Hi @vagenas,

Thanks a lot for the details!
Sounds great, it would be very useful to have tiktoken support. I will take a look at the code in depth and submit the PR.