Skip to main content
To evaluate any taskset, use the eval entrypoint:
You can also use .toml files for configuration:
Validate the config by using uv run vf-eval @ config.toml --dry-run. To run the evaluation, use uv run vf-eval @ config.toml. Use dotted arguments to set values using the CLI, e.g. --sampling.temperature 0.5. CLI arguments overwrite toml arguments when both are present. The output from evaluations are written into outputs/<env>--<model>--<harness>/<uuid>/ by default, where <env> is the taskset, prefixed by the paired env id when --env.id sets one (use output_dir to overwrite the folder). The folder contains the used config.toml, all the episodes in traces.jsonl, as well as logs of the run and workers in logs/attempt_<n>/eval.log — one directory per launch attempt (a resume starts a new one), with logs/latest pointing at the current attempt.

​
Common config values

  • model — the model id to evaluate, e.g. nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B
  • sampling — generation params passed to the model, e.g. sampling.temperature
  • env.taskset.id — pick the taskset (or the positional eval <taskset-id>)
  • env.agent.harness.id — pick the agent’s harness ([env.agent.harness] in TOML)
  • select — which tasks to evaluate (see Selecting tasks). -n sets select.limit: not setting it means all tasks, and an infinite taskset (a procedural generator, e.g. wordle) requires it. -s sets select.shuffle (fixed seed)
  • num_rollouts — rollouts per task
  • verbose — log at debug instead of info
  • rich — the live dashboard (default); --no-rich streams logs to the console and prints each trace as JSON at the end
  • rich.show_logs — replace the dashboard’s per-rollout rows with a live tail of the attempt’s logs (logs/latest/eval.log), the env workers’ lines included
  • push — upload the run to the Prime Intellect platform as it goes (default; --no-push keeps it local). run.attach <evaluation-id> streams into a run the platform already created instead of opening a new one — this is how a hosted evaluation’s sandbox runs the same command; it is not something a local run sets by hand

​
Resuming evaluations

--resume <output-dir> re-runs only the rollouts a previous run left missing or errored, appending to that run’s own traces.jsonl. It reloads the run’s saved config.toml verbatim, so it takes no other arguments. Good rollouts are kept, while errored ones are dropped and redone.

​
Disabling tools

Almost every harness comes with a disabled_tools list, which can be used to disable one or multiple tools:
The names of these tools are set by the respective harness. Consult the relevant documentation for the given harness for the relevant name(s). Some harnesses do not offer support to disable tools.

​
Skills

Harnesses whose program supports SKILL.md skills natively (e.g. Claude Code, Codex) take a skills list. A local skill folder is uploaded to <skills dir>/<folder name>. A {runtime = "..."} entry copies the contents of a directory already inside the runtime into the program’s skill discovery directory:
Tasks can supply the same sources through TaskData.skills, for example skills=[{"runtime": "/opt/skills"}]. Task sources are installed first, followed by harness sources; later files override matching earlier files. Each run gets its own installed skills. Sources are resolved after task setup, and missing source directories fail the run. Setting skills on a task or harness without native harness skill support fails up front.