Performance fix MistralTokenizer: cache special ids and tokens

[juliendenize]

Purpose

The PR #26358 to refactor the MistralTokenizer brought a performance issue due to the lack of caching of special tokens: https://github.com/vllm-project/vllm/pull/26358/files#diff-7b4368a1d7d34fa1071838db6d63fea4911f32e60ac065829979b327ad539649R410

The problem of performance makes it about 1000 times slower than expected.

Test Plan

from vllm.transformers_utils.tokenizers.mistral import MistralTokenizer
tokenizer = MistralTokenizer.from_pretrained("mistralai/Mistral-Small-3.2-24B-Instruct-2506")

# timeit -n 100 -r 10
tokenizer.convert_tokens_to_string([
    "<s>",
    "yolo"
] * 10)

Test Result

from main branch:

22.7 ms ± 113 μs per loop (mean ± std. dev. of 10 runs, 100 loops each)

to this PR:

2.29 μs ± 54.2 ns per loop (mean ± std. dev. of 10 runs, 100 loops each)

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request effectively addresses a significant performance issue in the MistralTokenizer by caching special tokens and their corresponding IDs during initialization. The changes correctly replace expensive list lookups with fast set lookups in performance-critical methods. I've included a couple of suggestions to improve code maintainability by making an implicit dependency explicit, which will make the code more robust against future refactoring.

[gemini-code-assist]

Code Review

This pull request effectively addresses a performance bottleneck in MistralTokenizer by caching special tokens and their IDs. The changes are well-structured and the use of sets for lookups is a good optimization. My review includes one high-severity comment regarding a potential correctness issue due to inconsistent logic for identifying special tokens in tekken tokenizers. While the inconsistency existed before, this PR touches related code and provides a good opportunity to fix it to ensure robust tokenization.