Support weka evaluation oe eval

[vwxyzjn]

Tested with the following.

# weka eval
python scripts/submit_eval_jobs.py \
    --model_name ppo_ray_3.9_1111_β_0.1_epoch_1__weka_eval \
    --location /weka/oe-adapt-default/costah/models/rlvr/1111/ppo_ray_3.9_1111_β_0.1_epoch_{1} \
    --cluster ai2/saturn-cirrascale ai2/neptune-cirrascale ai2/jupiter-cirrascale-2	 --evaluate_on_weka \
    --is_tuned \
    --workspace "tulu-3-results" \
    --priority high \
    --preemptible \
    --use_hf_tokenizer_template \
    --beaker_image "nathanl/open_instruct_auto" \
    --upload_to_hf allenai/tulu-3-evals \
    --run_oe_eval_experiments \
    --run_safety_evaluations \
    --skip_oi_evals

# nfs eval
python scripts/submit_eval_jobs.py \
    --model_name ppo_ray_3.9_1111_β_0.1_epoch_1__nfs_eval \
    --location 01JCE2PYZW1E96994GMW7MDB4R \
    --is_tuned \
    --workspace "tulu-3-results" \
    --priority high \
    --preemptible \
    --use_hf_tokenizer_template \
    --beaker_image "nathanl/open_instruct_auto" \
    --upload_to_hf allenai/tulu-3-evals \
    --run_oe_eval_experiments \
    --run_safety_evaluations \
    --skip_oi_evals

Note the only difference is that

    --location /weka/oe-adapt-default/costah/models/rlvr/1111/ppo_ray_3.9_1111_β_0.1_epoch_{1} \
    --cluster ai2/saturn-cirrascale ai2/neptune-cirrascale ai2/jupiter-cirrascale-2	 --evaluate_on_weka \

vs

    --location 01JCE2PYZW1E96994GMW7MDB4R \

[vwxyzjn]

Seems to work both for beaker dataset and weka

[jacob-morrison]

LGTM!

[natolambert]

This looks good thx