Skip to main content

GitHub Repository

Verifiers Repo
verifiers is a framework for defining tasks, running agents and harnesses, scoring them on set tasks, and using those for evaluations and reinforcement learning. The following concepts are important when creating or running tasksets, be it for evals or training:

​
Environment Hub

The Environment Hub is Prime Intellect’s collection of user-created tasksets which are installable and ready to use with verifiers.

​
Taskset

A taskset is the collection and loader for the work to evaluate or train on. Each task combines a serializable TaskData row (prompt, files, references, resource requirements) with its task class’s behavior (lifecycle hooks, tools, metrics, and rewards). The taskset’s load() method constructs those objects and declares their task/config types through Taskset[TaskT, ConfigT].

​
Harness

A harness is the program the model is run in, e.g. Claude Code, Codex, or mini-swe-agent.

​
Agent

An agent is the combination of harness × model × runtime policy, which produces a Trace for each rollout.

​
Environment

An environment has one or multiple agents and defines the control flow among them across an episode (for example, running a single agent, retrying across multiple attempts, or pairing an agent with a judge or simulated user).

​
Toolset

A set of tools defined by the taskset that are installed as MCP servers into the harnesses that support them.

​
Trace

A trace records the message graph, rewards, metrics, errors, and one per-call record (ModelCall) per provider exchange (its model, sampling, finish reason, usage, timing, and any error). When using verifiers for training with prime-rl, it stores additional information such as tokens and logprobs, built incrementally using renderers.

​
Documentation

  • Getting started — Installation, CLI commands, and skills
  • Architecture — How verifiers works behind-the-scenes
  • Building environments — Build a package and separate agent execution from grading
  • Tasksets — How to create tasksets
  • Runtimes — Containers, files, resources, deployments, and network access
  • Debugging — Check setup, diagnose failures, and re-score saved runs
  • Harnesses — How to build custom harnesses
  • Agent — How to run standalone agents
  • Env — How to build multi-agent environments
  • GEPA — Optimize a taskset’s system prompt