Tasksets

A taskset defines the work to be done, which will be solved by the agent in a harness running in a runtime. You can scaffold a new taskset with the following:

uv run vf-init addition-v1

The generated package has two important files:

environments/addition_v1/addition_v1/
├── __init__.py  # exports the taskset entry point
└── taskset.py   # defines the data, tasks, and taskset

The command also supports:

-p, --path <dir> — parent directory, default: ./environments
-T, --add-tool — also scaffold a vf.Toolset tool server at servers/tool.py
- Use this to create custom tools which are installed into supported harnesses via MCP.
-H, --add-harness — also scaffold a custom vf.Harness at harness.py, selectable via --env.agent.harness.id <name>
- Prefer a built-in harness unless the model needs to run inside a custom program.

For a production-scale catalog of tasksets, see the companion research-environments repository.

An example taskset

Tasksets are made of the following components:

The Taskset loads the actual Tasks from a dataset using the load() function. It can be configured with the TasksetConfig, to e.g. load a certain split. Configs are exposed to the user and thus should only contain configurable values.
A Task defines the scoring, stop conditions, setup, judging etc. of the task to solve. It also gets the tools or user config. It gets configured by a TaskConfig, e.g., to set a specific judge model.
The TaskData is the immutable object that holds the actual data, i.e., the prompts, images, expected outputs etc., as well as other information such as timeouts (if set).

The following taskset generates addition questions and checks whether the model returned the exact answer.

import verifiers.v1 as vf


class AdditionData(vf.TaskData):
    # One immutable row in the dataset, including its reference answer.
    answer: int


class AdditionTask(vf.Task[AdditionData]):
    # @vf.reward denotes the scoring function for the task.
    # It needs the trace, which contains the whole message graph, including function calls, user messages etc.
    # It returns the reward for the single task based on this function.
    @vf.reward
    async def exact_match(self, trace: vf.Trace) -> float:
        return float(trace.last_reply == str(self.data.answer))


class AdditionConfig(vf.TasksetConfig):
    # Values users can configure for the whole taskset.
    num_tasks: int = 100


# The Taskset itself
class AdditionTaskset(vf.Taskset[AdditionTask, AdditionConfig]):
    # The loading function for the actual tasks
    def load(self) -> list[AdditionTask]:
        return [
            AdditionTask(
                AdditionData(prompt=f"What is {i} + {i}?", answer=2 * i),
                self.config.task,
            )
            for i in range(self.config.num_tasks)
        ]

If a config class is not explicitly created, it means that no configurable, custom values are exposed to the user. In this example, there is no vf.TaskConfig, so no task values (like judge models) are configurable. The scaffold also exports the taskset from addition_v1/__init__.py:

from addition_v1.taskset import AdditionTaskset

__all__ = ["AdditionTaskset"]

The exported AdditionTaskset is what verifiers loads and makes discoverable for evaluation.

Data and configuration

Keep values on the narrowest object that needs them:

Put load-time values shared across the dataset, such as its split, name, seed, or size, on TasksetConfig.
Put values used by every task during execution or scoring under TasksetConfig.task.

class AdditionTaskConfig(vf.TaskConfig):
    tolerance: float = 0.0


class AdditionTask(vf.Task[AdditionData, vf.State, AdditionTaskConfig]):
    @vf.reward
    async def exact_match(self, trace: vf.Trace) -> float:
        error = abs(float(trace.last_reply) - self.data.answer)
        return float(error <= self.config.tolerance)


class AdditionConfig(vf.TasksetConfig):
    num_tasks: int = 100
    task: AdditionTaskConfig = AdditionTaskConfig()

These values can be overridden with --env.taskset.num-tasks and --env.taskset.task.tolerance, or with the equivalent TOML fields ([env.taskset]).

Task identity

A task’s idx is its index in the load() stream; it is set automatically. Set TaskData.id to a durable ID from the source (e.g. a dataset’s instance ID) and TaskData.name to a readable name; both are optional. A task’s hash identifies its exact serialized data. Its key provides stable identity across runs and defaults to the hash. Override Task.key with a durable source ID when task data contains run-local fields such as idx; keys must be unique within a taskset. Both values are recorded on traces.

Selecting tasks

Views on a taskset pick which tasks an iteration yields. They are lazy and chain in any order:

taskset.include(idx=["0:100"]).exclude(names=["broken"]).shuffle(seed=0).take(5)

include keeps and exclude drops the tasks named by idx (ints and Python slices start:stop:step, e.g. "100:", ":50", "::2"; no negative positions), ids, keys or names; a task matches when any list names it. An entry that matches no task logs a warning once the stream was read. The eval, debug, validate and GEPA entrypoints take a select block and apply it with taskset.select(config.select), which chains the views in a fixed order: include, exclude, shuffle, skip, limit:

uv run vf-eval gsm8k --select.limit 50 --select.shuffle         # 50 random tasks (short: -n 50 -s)
uv run vf-eval gsm8k --select.include.idx 0:100                 # the first 100 tasks
uv run vf-eval gsm8k --select.shuffle --select.skip 100 --select.limit 100   # a random split disjoint from the first

[select]
include = { idx = ["0:500"] }
exclude = { names = ["broken-task"] }
shuffle = true
limit = 100

Because the shuffle comes before skip and limit, a larger limit extends the same selection instead of drawing a new one.

Lazy and infinite tasksets

load() may be a generator instead of returning a list: yield each task as it’s built. Consumers iterate the taskset lazily (eval -n 5 builds 5 tasks, not the whole set, unless shuffle needs all of them) — so a generator pays off whenever building a task is expensive. A procedural taskset can keep yielding forever. Declare INFINITE = True so consumers know the stream never ends — infinity is inherent to the taskset, not a config knob; how many tasks a run takes is the run’s choice (-n), not the taskset’s:

import itertools
from collections.abc import Iterator


class AdditionTaskset(vf.Taskset[AdditionTask, vf.TasksetConfig]):
    INFINITE = True

    def load(self) -> Iterator[AdditionTask]:
        for i in itertools.count():
            yield AdditionTask(
                AdditionData(prompt=f"What is {i} + {i}?", answer=2 * i),
                self.config.task,
            )

Two rules follow from infinity: a run over an infinite taskset must be bounded with take (-n on the CLI) or an include of only closed idx ranges — omitting both is an error — and shuffle needs a bound before it: there is no whole set to sample from otherwise. select applies limit after the shuffle, so on the CLI only closed include.idx ranges can bound it (--select.include.idx 0:1000 -s -n 50); in Python, take(n).shuffle() works too. The generator runs once, client-side (the eval entrypoint or the prime-rl orchestrator pulls tasks off it and ships each task’s data to the env server), so nothing needs to re-produce the same sequence across processes; keep load() deterministic only if you want --resume to regenerate the same first n tasks (see alphabet_sort, color_codeword, or the built-in textarena taskset).

Adding Tools

Some tasksets require custom tools, which are bundled as a vf.Toolset (similar to how a vf.Taskset bundles vf.Task). Tools are exposed as MCP servers to the given harness and thus need a harness which exposes MCP support (via SUPPORTS_MCP). You can create them like this (remember the bootstrapping with uv run vf-init MY_ENV -T):

DATABASE = None


class SearchToolset(vf.Toolset[vf.SharedToolsetConfig]):
    TOOL_PREFIX = "search"

    @vf.tool
    async def query(self, text: str) -> list[str]:
        """Search the task corpus."""
        return DATABASE.search(text)


# User-configurable knobs
class SearchConfig(vf.TasksetConfig):
    tools: vf.SharedToolsetConfig = vf.SharedToolsetConfig()


class SearchTaskset(vf.Taskset[vf.Task, SearchConfig]):
    @classmethod
    def toolsets(cls, config: SearchConfig) -> list[vf.Toolset]:
        return [SearchToolset(config.tools)]

Taskset tools are shared by a worker’s rollouts. Tools can also be set per task.

Using Judges

If your reward is semantic, use an LLM judge.

import verifiers.v1 as vf
from functools import cached_property


class Task(vf.Task):
    answer: str


class CorrectnessJudge(vf.Judge[bool]):
    # The rubric for the judge
    prompt = """Question: {question}
    Answer: {answer}
    Response: {response}
    Correct? Reply yes or no."""

    # Parse the response from the judge
    def parse(self, response: vf.JudgeResponse[bool]) -> bool:
        return "yes" in response.text


class JudgedData(vf.TaskData):
    answer: str


class JudgedTaskConfig(vf.TaskConfig):
    # The judge inherits base_url and api keys from the client config
    judge: vf.JudgeConfig = vf.JudgeConfig(model="openai/gpt-5-mini")


class JudgedTask(vf.Task[JudgedData, vf.State, JudgedTaskConfig]):
    @vf.reward()
    async def correct(self, trace: vf.Trace) -> float:
        # Keeping judge configuration on TaskConfig makes it overridable from CLI/TOML.
        judge = CorrectnessJudge(self.config.judge)
        result = await judge.evaluate(
            trace=trace,
            question=self.data.prompt_text,
            answer=self.data.answer,
            # give the last assistant message to the judge
            response=trace.last_reply,
        )
        return float(result.parsed)


class SetConfig(vf.TasksetConfig):
    task: JudgedTaskConfig = JudgedTaskConfig()


class JudgeTraceTaskset(vf.Taskset[JudgedTask, SetConfig]):
    def load(self) -> list[JudgedTask]:
        return [
            JudgedTask(
                JudgedData(prompt="What is 2+2?", answer="4"),
                self.config.task,
            )
        ]

To override the judge model, set env.taskset.task.judge.model in your config (it is a string).

Plugging hooks via config

Stop conditions, metrics, and rewards can also be plugged into any task from config, without touching the taskset’s code. Each entry names an async function by import path (pkg.module.function, pkg.module:function, or path/to/file.py:function); like decorated hooks, functions declare what they need by parameter name (task, trace, runtime — stops receive the trace only):

[env.taskset.task.stops]
single_turn = { fn = "my_hooks.py:two_turns" }

[env.taskset.task.metrics]
reply_length = { fn = "my_hooks.py:reply_length", priority = 10 }

[env.taskset.task.rewards]
exact_match = { fn = "my_hooks.py:exact_match", weight = 0.5 }

Plugged hooks merge with the task’s decorated @vf.stop / @vf.metric / @vf.reward methods; a plugged hook replaces a decorated one with the same name, so an existing signal can be swapped out (like single_turn above). Rewards take a weight, and every entry takes a priority (higher runs first); unset fields keep the replaced method’s values. Leaving fn out keeps the task’s own decorated method and only overrides its metadata — for example, turning an existing zero-weight reward into a training signal:

[env.taskset.task.rewards]
exact_match = { weight = 1.0 }

Beyond one agent

One episode doesn’t have to be one agent run: agents, the control flow between agents, and cross-agent rewards are the environment’s job — see The Env.

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

An example taskset

Data and configuration

Task identity

Selecting tasks

Lazy and infinite tasksets

Adding Tools

Using Judges

Plugging hooks via config

Beyond one agent

​An example taskset

​Data and configuration

​Task identity

​Selecting tasks

​Lazy and infinite tasksets

​Adding Tools

​Using Judges

​Plugging hooks via config

​Beyond one agent

An example taskset

Data and configuration

Task identity

Selecting tasks

Lazy and infinite tasksets

Adding Tools

Using Judges

Plugging hooks via config

Beyond one agent