Skip to content

Commit

Permalink
Fix typo
Browse files Browse the repository at this point in the history
Co-authored-by: Alexander Neubeck <[email protected]>
  • Loading branch information
hendrikvanantwerpen and aneubeck authored Oct 14, 2024
1 parent 3f1b056 commit 9b7f311
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion crates/bpe-openai/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ pub use bpe::*;
/// A byte-pair encoding tokenizer that supports a pre-tokenization regex.
/// The direct methods on this type pre-tokenize the input text and should
/// produce the same output as the tiktoken tokenizers. The type gives access
/// to the regex and underlying bye-pair encoding if needed. Note that using
/// to the regex and underlying byte-pair encoding if needed. Note that using
/// the byte-pair encoding directly does not take the regex into account and
/// may result in output that differs from tiktoken.
pub struct Tokenizer {
Expand Down

0 comments on commit 9b7f311

Please sign in to comment.