Support gemma3 HF tokenizer.json

[larryliu0820]

This PR beef'ed up HF tokenizer.

HF tokenizer changes

• Added Normalization Module: Introduced the Normalizer base class and its derived classes (ReplaceNormalizer and SequenceNormalizer) to support customizable string normalization. A factory class, NormalizerConfig, was added to simplify normalizer creation and configuration.
• HF-Specific BPE Logic: Added the HFWord structure and implemented HF-specific token merging logic in _byte_pair_merge. Overrode byte_pair_encode_ to integrate normalization and pre-tokenization. [1] [2]
• Normalizer Integration: Integrated the normalization module into HFTokenizer, allowing it to load and use normalizers from JSON configuration during tokenizer initialization.

BPE Improvements:

• Merge Map and Utility Function: Added a hash-based MergeMap type and the buildMergeRanksMap utility function to handle BPE merge rules efficiently. This ensures proper handling of token merging based on ranks.
• Virtual Methods for BPE Logic: Added virtual methods (_byte_pair_merge and byte_pair_encode_) to allow derived classes to customize BPE merging logic. Updated BPETokenizerBase to use pre-computed merge ranks for token merging. [1] [2]

---

Tested with Gemma3 tokenizer.json manually

TEST(HFTokenizerTest, TestEncodePresidentQuestion) {
  HFTokenizer tokenizer;
  auto path = _get_resource_path("test_hf_tokenizer.json");
  auto error = tokenizer.load(path);
  EXPECT_EQ(error, Error::Ok);
  std::string text = "Who is the president of the US?";
  auto result = tokenizer.encode(text, /*bos*/ 1, /*eos*/ 0);
  EXPECT_TRUE(result.ok());
  std::vector<uint64_t> expected = {
      2, 15938, 563, 506, 6207, 529, 506, 2590, 236881};
  EXPECT_EQ(result.get(), expected);
}

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[jackzhxng]

• Remove std::cout
• Add gemma tokenizer.json for E2E tests
• Add tests for encode merging (rank is respected, merge is made), can use small merge list

[larryliu0820]

Add gemma tokenizer.json for E2E tests

Too big for internal. I'll add test infra in next PRs.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.

[facebook-github-bot]

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this in D77761574.