[feat] Add math eval to CI nightly run

[XiaotongJiang]

Fix forward for #2652

How did i test:

• tested by temp moving the nightly test suites to per-commit

[XiaotongJiang]

took a look of the recent flaky nightly tests, theres some flakiness around gsm8k threshold for certain models, but the main contributor is human eval timing out from evalplus.evaluate, it seems they are using sync client in strict sequential manner link and its slow. I didnt see a good way to speed it up, should we swap to other human eval lib? @Ying1123 @zhaochenyang20

[zhaochenyang20]

took a look of the recent flaky nightly tests, theres some flakiness around gsm8k threshold for certain models, but the main contributor is human eval timing out from evalplus.evaluate, it seems they are using sync client in strict sequential manner link and its slow. I didnt see a good way to speed it up, should we swap to other human eval lib? @Ying1123 @zhaochenyang20

As we discussed, use is_in_ci to detect wether it's in the CI and only run default_samll_model_name on human eval.

Thanks, great job!

[zhaochenyang20]

Why disable human eval rather than add more time out?

[zhaochenyang20]

@XiaotongJiang LGTM