Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Eliminate look-ahead in regex #32

Closed
wants to merge 2 commits into from
Closed

Eliminate look-ahead in regex #32

wants to merge 2 commits into from

Conversation

hendrikvanantwerpen
Copy link
Contributor

@hendrikvanantwerpen hendrikvanantwerpen commented Oct 17, 2024 •
edited
Loading

Eliminates look-ahead in the pre-tokenization regexes by supporting manually implemented trim functions in the pretokenizer.

The pretokenizer still accepts fancy regex, but since we don't use these features, it'll fall back to the regular regex crate.

Tasks

  • Run benchmark, update figures and text.

@hendrikvanantwerpen hendrikvanantwerpen self-assigned this Oct 17, 2024
@hendrikvanantwerpen
Copy link
Contributor Author

Closed in favor of #33.

@hendrikvanantwerpen hendrikvanantwerpen deleted the eliminate-look-ahead branch October 18, 2024 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@hendrikvanantwerpen hendrikvanantwerpen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@hendrikvanantwerpen