Could vf-eval forward matching env args to load_environment?

[ob1-s]

hey,

right now if my load_environment function needs to know something like num_examples (say i want to pre-process just a slice of the dataset, or the env needs some other vf-eval arg for some reason), i have to pass it twice.

example:

uv run vf-eval my-env -n 10 -a '{"num_examples": 10}'

idea / question

i was wondering if it could make sense for vf-eval to auto-pass its own args to load_environment when they match the function signature (and only when they match).

so in this case, since my function takes num_examples, vf-eval could forward -n automatically. then i would only need:

uv run vf-eval my-env -n 10

the same could apply to args like model, temperature, etc.

trade-offs / concerns

possible naming collisions (e.g. model could mean different things in env vs eval), it could break some existing envs. also makes the boundary between env config and eval params fuzzier.

curious if the current explicit design is intentional to keep that separation. would love to hear your reasoning.

---

i thought of this while building a multi-turn env that pre-generates the initial user messages, and i noticed i have to repeat params like this so it doesn't take too long.

I'd be glad to open a PR if this seems reasonable

thanks!

[ob1-s]

if anyone wonders, i figured out a better way to solve my original setup.

instead of passing num_examples to my env, i added a cache to the pre-generation step inside load_environment.

this makes the generation a one-time cost. after the first run, it's instant, so the env doesn't need to know how many examples are being run anymore. it completely removes the need for the num_examples env_arg. after all, i'm not evaluating the user model. caching the initial turn decreases any possible friction on that end.

would still be nice to know everyone's thoughts on forward matching env args though.

[Sohailm25]

Aight so

I think the separation between load_environment() args and vf-eval args are intentional:

1. Environment Construction (load_environment(**env_args)):

   • dataset loading/filtering
   • setup of rubrics, parsers, tools
   • config that affects the environment's BEHAVIOR

2. Evaluation Execution (env.evaluate(num_examples=...)):

   • how many examples to run
   • sampling parameters
   • concurrency settings
   • things that affect HOW you USE the environment, not what it IS

# wordle/wordle.py:47-50
def load_environment(
    num_train_examples: int = 2000,  # **HOW BIG** to make the dataset
    num_eval_examples: int = 20,
    ...
)

# eval.py:276-280
--num-examples, -n  # **HOW MANY** examples to evaluate from the dataset

These are 2 different concepts:

• num_train_examples => "construct a dataset with 2000 training examples"
• num_examples => "evaluate 10 examples from whatever dataset exists"

So yes, if we auto-forwarded we'd need to guess which one the user meant (like you said)

Also:

1. It creates ambiguity (auto-forward would need precedence rules)
2. It would break existing environments (bc many already use num_*_examples, model, temp

honestly, your caching solution is clean

i'd highly suggest you document the caching pattern you discovered, maybe add a helper for common caching scenarios