v1.23.0

What's Changed

• Update HuggingFace tokenizers dependency to v0.22.0 by @Nav31 in #44
• catch and return panics in encoders by @steved in #46
• bump tiktoken-rs/fancy-regex to 0.16 by @steved in #45

New Contributors

• @Nav31 made their first contribution in #44
• @steved made their first contribution in #46

Full Changelog: v1.22.2...v1.23.0

v1.22.2 fix: handle decoding partial UTF-8 characters

Full Changelog: v1.22.1...v1.22.2

v1.22.1 Link time version check

Full Changelog: v1.22.0...v1.22.1

v1.22.0 Better errors on Tokenizer init

Full Changelog: v1.21.1...v1.22.0

v1.21.1 Tiktoken stability

Full Changelog: v1.21.0...v1.21.1

v1.21.0 Tiktoken support

Full Changelog: v2.20.2...v1.21.0

v1.20.2

What's Changed

• feat: better error message when tokenizers lib mismatch by @daulet in #28
• feat: FromPretrained to load tokenizer directly from HF by @berkayersoyy in #27

New Contributors

• @berkayersoyy made their first contribution in #27

Full Changelog: v0.9.0...v1.20.2

v0.9.0

What's Changed

• feat: add option to retrieve offsets from tokenizer by @riccardopinosio in #21
• Update to huggingface/tokenizers v0.20.0 by @daulet in #23

New Contributors

• @riccardopinosio made their first contribution in #21

Full Changelog: v0.8.0...v0.9.0

v0.8.0

Breaking change:

Path to compiled rust library needs to be specified via -ldflags. I found it most convenient to use CGO_LDFLAGS env variable to avoid always setting it. See #18 for more details.

What's Changed

• Update to allow for platform dependent libs in CGO by @jmoney in #18

New Contributors

• @jmoney made their first contribution in #18

Full Changelog: v0.7.1...v0.8.0

v0.7.1

• Update core tokenizers library to latest: v0.15.2;
• Expose init time parameter to encode special tokens (or not);

Full Changelog: v0.7.0...v0.7.1