support tiktoken

[chengchingwen]

Support the tokenizers provided by tiktoken. The tokenizer files are re-host on my gist due to Artifacts.jl lacking support for lazy artifact without unpacking.

cc @ztangent

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (ad9d33b) 80.20% compared to head (e2efa4e) 83.63%.

❗ Current head e2efa4e differs from pull request most recent head 3c2b05e. Consider uploading reports for the commit 3c2b05e to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master       #7      +/-   ##
==========================================
+ Coverage   80.20%   83.63%   +3.43%     
==========================================
  Files           8        9       +1     
  Lines         389      495     +106     
==========================================
+ Hits          312      414     +102     
- Misses         77       81       +4     

Files	Coverage Δ
src/BytePairEncoding.jl	100.00% <ø> (ø)
src/bpe.jl	91.97% <100.00%> (+0.92%)	⬆️
src/bytefallback.jl	0.00% <ø> (ø)
src/gpt2_utils.jl	100.00% <100.00%> (ø)
src/tokenization.jl	77.77% <100.00%> (+6.34%)	⬆️
src/tiktoken.jl	94.62% <94.62%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chengchingwen]

julia> using BytePairEncoding, TextEncodeBase

julia> tkr = BytePairEncoding.load_tiktoken("cl100k_base")
FlatTokenizer(MatchTokenization(BPETokenization(Cl100kBaseTokenization, bpe = TikTokenBPE(100256 merges)), 5 patterns))

julia> tkr(TextEncodeBase.Sentence("hello world aaaaaaaaaaaa"))
5-element Vector{TextEncodeBase.TokenStage}:
 Token("hello", (ismatch = false,))
 Token(" world", (ismatch = false,))
 Token(" a", (ismatch = false,))
 Token("aaaaaaaa", (ismatch = false,))
 Token("aaa", (ismatch = false,))