Skip to content

Fix CoreBPE::new type inference failure#558

Open
fallintoplace wants to merge 1 commit into
openai:mainfrom
fallintoplace:fix-corebpe-new-nse
Open

Fix CoreBPE::new type inference failure#558
fallintoplace wants to merge 1 commit into
openai:mainfrom
fallintoplace:fix-corebpe-new-nse

Conversation

@fallintoplace
Copy link
Copy Markdown

Summary

  • remove the unused NSE type parameter from CoreBPE::new
  • add a Rust regression test that exercises the public constructor through a normal call site

Why

CoreBPE::new exposed an unconstrained generic parameter that never appeared in the arguments or return type. That made normal downstream calls fail type inference with E0283, even though the crate publishes an rlib and the constructor is part of the public Rust API.

Fixes #557.

Validation

  • cargo test --manifest-path Cargo.toml
  • cargo check --manifest-path /tmp/tiktoken-downstream/Cargo.toml

@fallintoplace fallintoplace marked this pull request as ready for review May 24, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CoreBPE::new has an unconstrained NSE type parameter, making normal downstream calls fail type inference

1 participant