Skip to content

Bindings to copy of SentencePiece esaxx library (fast suffix array and frequent substrings).

License

Notifications You must be signed in to change notification settings

oxy-hq/esaxx-rs

This branch is up to date with thewh1teagle/esaxx-rs:main.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

5a573fa · Oct 6, 2023

History

37 Commits
Jul 1, 2020
Jul 31, 2020
Jul 1, 2020
Jul 31, 2020
Oct 6, 2023
Jun 6, 2020
Oct 5, 2023
Oct 5, 2023
Jun 6, 2020
Jul 1, 2020
Jul 1, 2020
Oct 5, 2023

Repository files navigation

esaxx-rs

This code implements a fast suffix tree / suffix array.

This code is taken from sentencepiece and to be used by hugging face.

Small wrapper around sentencepiece's esaxx suffix array C++ library. Usage

let string = "abracadabra";
let suffix = esaxx_rs::suffix(string).unwrap();
let chars: Vec<_> = string.chars().collect();
let mut iter = suffix.iter();
assert_eq!(iter.next().unwrap(), (&chars[..4], 2)); // abra
assert_eq!(iter.next(), Some((&chars[..1], 5))); // a
assert_eq!(iter.next(), Some((&chars[1..4], 2))); // bra
assert_eq!(iter.next(), Some((&chars[2..4], 2))); // ra
assert_eq!(iter.next(), Some((&chars[..0], 11))); // ''
assert_eq!(iter.next(), None);

About

Bindings to copy of SentencePiece esaxx library (fast suffix array and frequent substrings).

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C++ 52.6%
  • Rust 47.4%