Skip to content

fix: stop the trainer's tokenizer from padding - #379

Merged
stephantul merged 2 commits into
mainfrom
fix-trainer-padding-2
Sep 28, 2026
Merged

stephantul merged 2 commits into
mainfrom
fix-trainer-padding-2

Conversation

@stephantul

Copy link
Copy Markdown
Contributor

The trainable model's copy of the tokenizer kept its padding setting, so tokenizers with padding enabled padded every text to the longest text in its tokenization chunk. Share the padding-disabling helper from StaticModel, which keeps the pad token but never pads.

The trainable model's copy of the tokenizer kept its padding setting, so
tokenizers with padding enabled padded every text to the longest text in
its tokenization chunk. Share the padding-disabling helper from
StaticModel, which keeps the pad token but never pads.
@stephantul
stephantul marked this pull request as ready for review September 27, 2026 17:44
@stephantul
stephantul requested a review from Pringled September 27, 2026 17:44
@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Refactors tokenizer padding behavior in model initialization.

The PR appears safe to merge; no actionable issue was identified.

Reviews (1) · Last reviewed commit: "fix(train): stop the trainer's tokenizer..."

@codecov

codecov Bot commented Sep 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
model2vec/model.py 97.56% <100.00%> (+0.02%) ⬆️
model2vec/train/base.py 99.54% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stephantul
stephantul merged commit 2618240 into main Sep 28, 2026
11 checks passed
@stephantul
stephantul deleted the fix-trainer-padding-2 branch September 28, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants