Adding evaluation checks to prevent Transformer ValueError

[stsfaroz]

When neither eval_dataset nor evaluator is provided, Transformers raises a ValueError stating that an eval_dataset must be passed or eval_strategy. However, this error does not account for the evaluator parameter.

Error raised by Transformers:

ValueError: You have set `args.eval_strategy` to "steps" but you didn't pass an `eval_dataset` to `Trainer`. 
Either set `args.eval_strategy` to "no" or pass an `eval_dataset`.

Instead, we now raise a more specific error:

ValueError: Either `eval_dataset` or `evaluator` must be provided for evaluation, 
or set `eval_strategy='no'` to skip evaluation.

[tomaarsen]

Hello!

Thanks for tackling this, I think it's indeed quite smart to "get ahead" of the error that transformers will give and give our own Sentence Transformers-specific error. I updated the phrasing a bit because I quite like how the transformers error flowed ("you did X, but not Y. Please do Y or avoid X").

The edge case of no eval_strategy/no evaluator was already tackled a bit in #3035 to get Transformers v4.46 compatibility, but the "no eval_strategy and no evaluator" case was left as-is:

If evaluator is not set, then the ValueError is acceptable I think

So I also updated the test that I made back then with more details about what the expected ValueError should be.

What do you think? @stsfaroz

• Tom Aarsen

[tomaarsen]

Looks like the tests failed for Python 3.9 and 3.10 because Python 3.11 changed how Enums are formatted by default: https://docs.python.org/3/whatsnew/3.11.html#enum

I.e. they now get printed as IntervalStrategy.STEPS instead of "steps" (the string they represent).

• Tom Aarsen

Copilot reviewed 2 out of 2 changed files in this pull request and generated no suggestions.