Adding EOS Tokens to Qwen Models

[ariG23498]

Fix #2481

Current status 👇

from torchtune.models.qwen2 import Qwen2Tokenizer
from torchtune.data import Message

messages = [
    Message(role="user", content="Hello world!", masked=True),
    Message(role="assistant", content="How are you?", masked=False),
]

tokenizer = Qwen2Tokenizer(
    path="Qwen2-0.5B-Instruct/vocab.json",
    merges_file="Qwen2-0.5B-Instruct/merges.txt",
)
tokenized_text = tokenizer.tokenize_messages(messages)
print(tokenized_text)


# (
# [None, 151644, 872, 198, 9707, 1879, 0, 151645, 198, 151644, 77091, 198, 4340, 525, 498, 30, 151645, 198, 151645],
# [True, True, True, True, True, True, True, True, True, False, False, False, False, False, False, False, False, False, True]
# )

Not very sure why None comes as a token. Am I missing something obvious here? Any help would be great.

After the initial review, I will add tests.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2512

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 4 Cancelled Jobs

As of commit 914567a with merge base 1be43b6 ():

NEW FAILURES - The following jobs have failed:

• GPU tests / gpu_test (3.10, stable) (gh)
  tests/torchtune/models/qwen2/test_qwen2_tokenizer.py::TestQwenTokenizer::test_tokenize_message_drop_eos
• Lint / lint (3.10) (gh)
  Process completed with exit code 1.
• Unit Test / unit_tests (3.9) (gh)
  tests/torchtune/models/qwen2/test_qwen2_tokenizer.py::TestQwenTokenizer::test_tokenize_message_drop_eos

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• GPU tests / gpu_test (3.11, stable) (gh)
  tests/torchtune/models/qwen2/test_qwen2_tokenizer.py::TestQwenTokenizer::test_tokenize_message_drop_eos
• GPU tests / gpu_test (3.9, stable) (gh)
  tests/torchtune/models/qwen2/test_qwen2_tokenizer.py::TestQwenTokenizer::test_tokenize_message_drop_eos
• Unit Test / unit_tests (3.10) (gh)
  tests/torchtune/models/qwen2/test_qwen2_tokenizer.py::TestQwenTokenizer::test_tokenize_message_drop_eos
• Unit Test / unit_tests (3.11) (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ariG23498]

A gentle ping on this.

[joecummings]

Thanks for the ping, sorry this fell through the cracks!

The none is coming from the mis-use of the bos_id here. bos_id is not actually defined for Qwen2 so it's showing up as none.

[ariG23498]

Interesting.

Can I get a quick review on the changes I have made so far, just to be sure about my implementation.