> ## Documentation Index
> Fetch the complete documentation index at: https://docs.primeintellect.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runtimes

A `Runtime` provides command and file access inside a sandbox with immutable user, `HOME`, and environment bindings. Task and harness hooks receive it as `runtime: vf.Runtime`, while configuration fields such as `agent.runtime` (and `--env.agent.runtime.type`, which defaults to `prime`) describe the runtime policy used to provision the sandbox. Because Python task hooks like `setup`, `finalize`, and `@vf.reward` always execute in the evaluator process, use their `runtime` parameter whenever you want to read files or run commands inside the sandbox; calling `Path(...).read_text()` directly reads from the evaluator host instead.

## Capabilities

Every runtime runs commands, reads and writes files, and starts background services. The runtimes differ in where they run and which features they support:

* **`subprocess`** runs commands directly on the evaluator host, with full access to local files, processes, and networking. Use it for fast local debugging with lightweight harnesses such as `bash` or `null`.
* **`docker`** runs each rollout in a local container. Use it for local testing of tasks that need an image or a container-only harness. CPU, memory, and GPU limits depend on the host; GPU pass-through requires the NVIDIA Container Toolkit.
* **`prime`** runs each rollout in a remote Prime sandbox, built for high-concurrency evaluation and RL training. It uses your Prime Intellect credentials, and it is the only runtime that enforces hard `disk` limits.
* **`custom`** loads a runtime from your own package. See [runtime plugins](#runtime-plugins).

Each runtime declares its features through its backend's class attributes, and network policy support through its config. Before it provisions anything, the framework refuses a run that needs a feature the runtime does not declare. The error names the runtime and the missing feature.

| Feature | Declared by | Built-in runtimes |
| - | - | - |
| Task image, container-only harnesses, isolated scoring | `SUPPORTS_TASK_IMAGES` | `docker`, `prime` |
| Task network policy | config inherits `NetworkPolicyConfig` | `docker`, `prime` |
| Task `agent_user` | `SUPPORTS_USER_SWITCHING` | `docker`, `prime` |
| Tool server in its own runtime | `SUPPORTS_PORT_EXPOSURE` | `subprocess`, `docker`, `prime` |
| Compose services | `SUPPORTS_COMPOSE` | `docker`, `prime` (subset below) |

Runtime capabilities are orthogonal to agent harness capabilities like MCP tool support, image inputs, or conversation resumption; see [harnesses](https://docs.primeintellect.ai/verifiers/harnesses.md) for harness features and [building environments](https://docs.primeintellect.ai/verifiers/building-environments.md#separate-agent-and-grading-sandboxes) for running separate solver and grading sandboxes.

## Images, resources, and timeouts

Tasks can declare their own container image, working directory, hardware resources (CPU cores, memory in GB, and disk in GB), phase timeouts (in seconds), and network rules directly on `TaskData`:

```python theme={null}
data = vf.TaskData(
    prompt="Write the result to /workspace/answer.txt.",
    image="registry.example.com/my-task:1",
    workdir="/workspace",
    resources=vf.TaskResources(cpu=2, memory=4, disk=10),
    timeout=vf.TaskTimeout(setup=300, agent=1800, scoring=120),
    network=vf.NetworkPolicyConfig(allow=[]),
)
```

When resolving a sandbox configuration, `TaskData.image` takes precedence over `runtime.image` so individual tasks in a dataset can bring their own containers. By contrast, `TaskData.workdir` and `TaskData.resources` act as task-specific defaults: they apply when the runtime config leaves those fields at their default values, while explicit runtime overrides (such as `--env.agent.runtime.cpu 4`) take priority. On `prime`, setting `runtime.checkpoint` boots the sandbox from an existing filesystem checkpoint instead of a container image, replacing both `runtime.image` and `TaskData.image` and preserving the checkpoint's disk size while still applying your CPU and memory settings.

Timeouts and token budgets can be tuned at evaluation time under `env.agent`:

* **Phase timeouts**: set `setup`, `rollout`, `finalize`, or `scoring` under `env.agent.timeout` to override the corresponding `TaskData.timeout` values (any phase left unset falls back to the task's timeout, or runs without a limit if neither is set).
* **Turn and token budgets**: set `max_turns`, `max_input_tokens`, `max_output_tokens`, or `max_total_tokens` on `env.agent` to cap an entire agent rollout, and `sampling.max_tokens` to cap each individual model response. Rollout token budgets are checked between turns, so the turn that crosses a threshold completes before the rollout stops. `max_turn_tokens` on `env.agent` also caps each response (it lowers `sampling.max_tokens` to its value), and stops the rollout after a response that hits it.

## Execution users

On `docker` and `prime`, you can set `TaskData.agent_user` to run the agent under an existing non-root user account in the container image (with a writable home directory). Task `setup` still runs as the container's default user, after which harness setup, the agent process itself, and any colocated task tools execute as `agent_user`. Once the agent finishes, `finalize`, scoring, and artifact collection switch back to the default user.

Calling `runtime.with_user(...)` or `runtime.with_env(...)` returns an independent view over the same underlying resource without mutating existing views. Under the hood, a `RuntimeBackend` (such as `DockerBackend` or `PrimeBackend`) owns the provider lifecycle, including container startup, network policy, and teardown; `backend.bind()` creates a `Runtime` view without provisioning a new resource, and `runtime.backend` exposes the backend when explicit infrastructure operations are needed. `Agent.provision()` lends a runtime whose cleanup remains owned by the caller's context.

Because setting `agent_user` does not create the Unix account or change existing file ownership, install system packages and create directories during image build or task `setup`, then grant `agent_user` permission to read its inputs and write to its working directory and output paths before `setup` returns. Note that a view is an execution handle rather than a security boundary: user permissions are enforced by the container OS, and if a task uses private tests or sensitive grading assets, you should stage them in a separate [grading sandbox](https://docs.primeintellect.ai/verifiers/building-environments.md#separate-agent-and-grading-sandboxes).

## Runtime operations

Task lifecycle hooks (`setup`, `finalize`, `stage_verifier`) and `@vf.reward` / `@vf.metric` methods can access the active sandbox by declaring a `runtime: vf.Runtime` parameter:

```python theme={null}
async def setup(self, trace: vf.Trace, runtime: vf.Runtime) -> None:
    await runtime.write("input.txt", b"task input\n")
    result = await runtime.run(["sh", "-c", "mkdir -p results"], {})
    if result.exit_code:
        raise RuntimeError(result.stderr)


async def finalize(self, trace: vf.Trace, runtime: vf.Runtime) -> None:
    trace.info["answer"] = (
        await runtime.read("results/answer.txt", max_bytes=1_000_000)
    ).decode()
```

The `vf.Runtime` interface provides a small set of methods for interacting with the sandbox:

* `run(argv, env)` executes a command and returns a `ProgramResult` containing `exit_code`, `stdout`, and `stderr`. Nonzero exit codes do not raise automatically, so check `result.exit_code` when a command must succeed.
* `read(path, max_bytes=...)` and `write(path, data)` transfer raw `bytes` between the evaluator and the sandbox filesystem.
* `prepare_uv_script(script)` and `run_uv_script(script, args, env)` prepare and execute self-contained Python scripts that declare PEP 723 inline dependencies.
* `run_background(argv, env, log)` launches a background service inside the sandbox. Because standard container runtimes replace the image entrypoint with a keep-alive process (unlike Docker Compose tasks, which preserve service entrypoints), use `run_background` in `setup` to start any daemons the image would normally launch and verify they are ready before returning.

To pass environment variables into sandbox commands without serializing them into `TaskData`, override `Task.runtime_env()` (or set `TaskData.scoring.env` / `agent.scoring.env` for variables used only during scoring). For credentials needed by an agent harness, set `harness.forward_env = ["TOKEN_NAME"]` so saved configs record only the variable name rather than the secret value. Processes inside the sandbox can read forwarded variables, so keep judge-only credentials on the evaluator host.

### Packaged grader scripts

Dependencies added to your environment package's `pyproject.toml` (via `uv add --project environments/<package> ...`) are installed on the evaluator for dataset loading and Python hooks, not inside the agent or grading containers. Libraries and CLI tools that the agent needs belong in the task image or `setup`, while dependencies for a sandboxed Python grader can either be baked into the verifier image or declared via PEP 723 inline script metadata.

If you bundle a self-contained grading script alongside your taskset and load its bytes with `importlib.resources`, you can pre-install its inline dependencies with `runtime.prepare_uv_script` and run it with `runtime.run_uv_script`:

```python theme={null}
import json
from importlib.resources import files

GRADER = files("my_task").joinpath("check.py").read_bytes()


async def stage_verifier(self, runtime: vf.Runtime) -> None:
    await runtime.prepare_uv_script(GRADER)


@vf.reward
async def correct(self, runtime: vf.Runtime) -> float:
    result = await runtime.run_uv_script(GRADER, args=["/workspace/answer.py"])
    if result.exit_code:
        raise RuntimeError(result.stderr)
    return float(json.loads(result.stdout)["score"])
```

Calling `prepare_uv_script` during `stage_verifier` (or during `setup` for a public checker that runs inside the solver sandbox) installs the script's PEP 723 dependencies before execution-time network restrictions lock down the container. Passing the identical script bytes to `run_uv_script` during scoring then reuses the prepared interpreter offline. Whenever your grader imports or executes code written by the agent, run it inside a container (`NEEDS_CONTAINER = True`) rather than in a Python hook on the evaluator. Have the grader script exit `0` and emit a `0.0` score when the submission is wrong or crashes, and reserve nonzero exit codes (raising in `@vf.reward`) for failures that prevented a valid grade from being computed.

## Runtime plugins

verifiers ships only the `prime` sandbox runtime. Run other providers, such as Modal or
E2B, as plugins. Select `type = "custom"` to use a `RuntimeBackend` subclass from your
own package:

```toml theme={null}
[env.agent.runtime]
type = "custom"
import_path = "my_package.runtime.MyBackend"
region = "us-east-1"
```

The backend sets `config_class` to its own `CustomRuntimeConfig` subclass. That class
validates every field next to `import_path` when the config loads. The framework
constructs the backend as `MyBackend(config, name)` with the validated config, like
the built-in backends. A backend with `SUPPORTS_TASK_IMAGES` declares `image` and `workdir`
fields, and a config that inherits `NetworkPolicyConfig` enforces task network
policies:

```python theme={null}
from typing import ClassVar

from verifiers.v1.runtimes import CustomRuntimeConfig, CustomRuntimeInfo, RuntimeBackend


class MyConfig(CustomRuntimeConfig):
    region: str
    image: str = "python:3.11-slim"
    workdir: str | None = None


class MyBackend(RuntimeBackend):
    config_class = MyConfig
    is_local: ClassVar[bool] = False
    SUPPORTS_TASK_IMAGES: ClassVar[bool] = True

    def __init__(self, config: MyConfig, name: str | None = None) -> None:
        super().__init__(name)
        self.config = config
        self.info = CustomRuntimeInfo(**config.model_dump())
```

A backend implements this contract; `Runtime` views add user and environment
bindings and the uv-script helpers on top of it:

| Method | Required | Purpose |
| - | - | - |
| `start()` | yes | Provision the resource. |
| `cleanup()` or `teardown()` | yes | Free it. `cleanup` is synchronous and also runs at interpreter exit. |
| `exec(runtime, argv, env)` | yes | Start a process and return a handle that streams its stdio. |
| `write(runtime, path, data)` | yes | Write a file, creating parent directories. |
| `run(runtime, argv, env)` | no | Run to completion. The default drains an `exec`. |
| `run_background(runtime, argv, env, log)` | no | Start a long-lived process with its output in `log`. The default keeps an `exec` open for the process's life; override it when the provider runs detached jobs. |
| `read(runtime, path, max_bytes)` | no | Read a file. The default reads through `run`. |
| `expose(port)` | with `SUPPORTS_PORT_EXPOSURE` | Publish a port to a URL outside the runtime. |
| `prepare_execution(routes, network)` | with `NetworkPolicyConfig` | Apply the network policy, or `network` in its place, before the agent starts; `None` routes reopen the network. |
| `host_url(url)` | no | Rewrite a host URL for programs inside the runtime. |

Install the plugin's package in every process that loads the config, because
validation imports the backend. The capability flags and `is_local` default to off.
`tests/unit/test_runtimes.py` has a complete example.

### Checking a runtime

`tests/unit/test_runtimes.py` tests the backend contract and every runtime feature. A
feature the runtime declares must work, and a feature it does not declare must be
refused before anything is provisioned. Pass your plugin's config with `--runtime` to
run the tests against it. Run them before you use the plugin, and after you change it:

```bash theme={null}
uv run pytest tests/unit/test_runtimes.py --runtime \
    '{"type": "custom", "import_path": "my_package.runtime.MyBackend", "region": "us-east-1"}'
```

`--runtime` also takes a built-in type, for example `--runtime docker`. The tests
expect `sh` and Python 3 in the runtime's default image.

## Deployment and services

A `Deployment` coordinates all execution resources for a single rollout attempt: provisioning, placement, connectivity, and cleanup of the harness environment, the task workspace, supporting sidecar services, and fresh verification targets.

The **workspace** (`deployment.workspace`) is the `Runtime` where task commands run and task files live; setting `agent.deployment.mode = "colocated"` places the harness and colocated tools inside that same workspace.

On Prime, return a `ServicesSpec` from `deployment_spec()` to provision one native VM per named service beside the task's `main` workspace. Each service has its own image, resources, filesystem, and egress policy. The VMs join a private LAN for the rollout; they can call each other and the workspace through the names in `deployment.hosts`. Services bind to `0.0.0.0`, since each VM has its own loopback interface. No Docker daemon runs inside these VMs.

Task setup starts processes with `deployment.target(name).run_background()` and waits for readiness before returning. Image entrypoints do not start services automatically. Pass process environment variables to `run_background`; the workspace's environment is not copied into service VMs. For example:

```python theme={null}
import asyncio

import verifiers.v1 as vf


class WebTask(vf.Task):
    def deployment_spec(self):
        return vf.ServicesSpec(
            {"web": vf.PrimeConfig(image="python:3.11-slim", cpu=1, memory=2, allow=[])}
        )

    async def setup(self, runtime, deployment):
        web = deployment.target("web")
        await web.write("/srv/index.html", b"Hello from a native VM")
        await web.run_background(
            [
                "python",
                "-m",
                "http.server",
                "8000",
                "--bind",
                "0.0.0.0",
                "--directory",
                "/srv",
            ],
            {},
            log="/tmp/web.log",
        )
        url = f"http://{deployment.hosts['web']}:8000"
        async with asyncio.timeout(30):
            while (
                await runtime.run(
                    [
                        "python",
                        "-c",
                        "import sys, urllib.request; urllib.request.urlopen(sys.argv[1], timeout=2)",
                        url,
                    ],
                    {},
                )
            ).exit_code:
                await asyncio.sleep(1)
        await runtime.write("/app/web-url", url.encode())
```

Use a Prime agent runtime for this task. Setup controls process startup order; there are no automatic dependency or restart rules. LAN access remains available when each VM's external egress policy is applied after setup. Before isolated grading, the deployment deletes the workspace, calls `finalize_services()` to collect service artifacts, then releases the remaining VMs and LAN. Each service adds its own VM resource allocation. Processes that can share the workspace's filesystem and policy can instead start directly with `runtime.run_background()`.

A task tool server can run in a named service using `ToolsetConfig.service`; the harness reaches it at the service VM's LAN hostname. `deployment.target(name)` gives task hooks access to any named service.

Return a `ComposeSpec` to resolve a task's Compose services. Local Docker starts the authored containers and selects the workspace service. Compose files can mount host paths and request Docker privileges, so local Compose requires `agent.deployment.trust_compose = true` (`--env.agent.deployment.trust-compose`). Only enable it for task packages you trust. Local Compose does not enforce network policies.

On Prime, Compose prepares stopped `linux/amd64` containers on the evaluator's Docker host, imports their filesystems as cached private VM images, and runs their entrypoint and command as native processes. This requires Docker Compose on the evaluator, including for image inspection; workload VMs do not run Docker. Each service keeps its environment, named user and working directory. Prime LAN rules follow Compose network membership; service names and aliases resolve only on shared project networks. Services attached exclusively to `internal` networks have no internet access, including during setup; the workspace retains framework routes during execution. Startup honors `service_started`, `service_healthy`, and `service_completed_successfully`; health checks gate initial readiness. Service logs are at `/tmp/vf-compose.log`. Shutdown uses the image's stop signal, allows ten seconds, then releases the VM.

The Prime subset requires a workspace named `main` and project-scoped networks, with globally unambiguous aliases. It supports task-local Dockerfile builds, per-service CPU/memory/disk VM allocations, `shm_size`, and root `cap_add` capabilities when the VM provides them. Environment files must stay inside the task. A named volume that one service mounts, like an image-declared volume, remains an ordinary directory on that service's ephemeral VM disk. Bind mounts of task files are copied to the service's VM with each rollout, up to 256 MiB, and mounted at their targets, read-only when declared; files that nested mounts hide stay behind. Volumes mounted over bound files, including anonymous volumes, start from the image's directory, as on Docker. A `container_name` is one more name for the service on its project networks, and `hostname` sets the VM's hostname. Services in `network_mode: bridge` or `default` share a separate Prime LAN group within the rollout, with IP connectivity but no Compose service-name aliases. Published ports, `tty` and `stdin_open` have no effect: peers reach every port, and processes run without a terminal. Shared network namespaces, volumes shared between services, binds from outside the task, overlapping writable binds, tmpfs mounts, restart policies, replicas, network drivers other than `bridge`, IPAM, numeric non-root users and user/group overrides are rejected. Compose CPU and memory limits size each service's VM, not container cgroups, and never shrink it below the default VM; for `main`, an explicit runtime `cpu` or `memory` takes precedence, as over task resources. Cold image preparation counts toward the setup timeout; use `ServicesSpec` when task code should manage preparation and startup itself.

Task and runtime disk settings apply only to `main`. Set a service's `storage_opt: {size: "20G"}` to request its own VM disk size; other storage driver options are rejected. Sizes use Docker's binary units (`G`, `GB`, and `GiB` all mean 1024³ bytes); a number without a suffix is bytes. A non-default disk setting on `main` takes precedence over its Compose size. Sidecars otherwise use the 5 GB default, and Compose sizes never reduce that default. Every disk is also large enough for its image plus room to write. These are total disk sizes per VM, not a shared allocation or extra writable space.

A tool server can also run on a Prime VM of its own, under an egress policy of its own: set its `ToolsetConfig.runtime` to a Prime runtime, such as `vf.PrimeConfig(image="example/tools:1", allow=["*"])` for a server that calls public APIs beside an offline workspace, or `allow=[]` for one that runs agent-submitted code with no egress at all. Beside a Prime workspace without a task service spec, the server's VM boots with the workspace and joins it over a LAN network of the rollout's own, on which the workspace's VM may open connections to the server and nothing may open one to the workspace; the harness reaches the server at its VM's LAN name (`<sandbox-id>.sandbox.internal`), and `deployment.target(name)` gives task hooks the server's VM. The VM's policy applies when the workspace's does, after setup. Elsewhere such a server keeps its runtime's published port, which takes no network policy. On Prime, the published port is a public HTTPS URL (`https://<port>-<sandbox-id>.sandbox-v2-prod.pinfra.io`) without authentication.

To validate an isolated task without calling a model, finalize the submission, collect its artifacts, and call `vf.grade_submission(task, trace, runtime)`. This runs the task's normal grader, verifier image, timeouts, and retries in a fresh verification target without taking ownership of the supplied solver runtime. As in a rollout, the verifier derives from the agent's runtime config on the trace (`trace.agent.config.runtime`), not from the solver's resolved config, so the solver task's requirements, such as its network policy, don't carry over. Inside `Task.validate`, request `trace` by name for vf-validate's trace, which records its runtime config.

## Network policies

A task's policy (`TaskData.network`, a `vf.NetworkPolicyConfig` with `allow` and `block`) combines with the runtime's own rules (`runtime.allow` and `runtime.block`) to control what the sandbox can reach once the agent starts:

* `["*"]` adds no restriction of its own, deferring to whichever policy is set on the other side.
* `[]` blocks external egress and permits only connections required by verifiers (such as model API interception and MCP tool traffic).
* When both the task and the runtime specify allowlists, a destination must match both lists; when both specify blocklists, the blocked destinations are merged. (A specific destination allowlist cannot be combined with a blocklist.)

For example, to give the agent general internet access while blocking Hugging Face domains:

```toml theme={null}
[env.agent.runtime]
type = "prime"
allow = ["*"]
block = ["huggingface.co", "*.huggingface.co"]
```

Network policies take effect after task and harness `setup` complete (so `setup` can still download packages or assets) and remain active through `finalize` and scoring (except in Harbor's shared verifier mode, which [reopens network access during grading](https://docs.primeintellect.ai/verifiers/harbor.md#network-policies)).

Scoring can have a policy of its own. `ScoringConfig.network` is the task's policy while scoring runs, in place of `TaskData.network`, and combines with the runtime's rules the same way. Verifiers applies it right before the scoring hooks run, in the workspace (`shared`) or the fresh verifier (`isolated`), so a grader that installs packages under a no-egress task needs no network code:

```python theme={null}
vf.TaskData(
    ...,
    network=vf.NetworkPolicyConfig(allow=[]),
    scoring=vf.ScoringConfig(
        network=vf.NetworkPolicyConfig(allow=["pypi.org", "files.pythonhosted.org"])
    ),
)
```

Unset, isolated scoring runs under `TaskData.network` and shared scoring keeps the agent phase's network. Shared scoring can't restrict a runtime that started without a policy; use isolated scoring for that. A hook that needs a different policy at another point can call `runtime.set_network(policy)`, which enforces the policy as given on the whole runtime.

These egress rules govern processes inside the sandbox (including task tools configured with `colocated = true`), not Python hooks running on the evaluator or tools running in separate containers. For provider-hosted server tools that execute outside the sandbox, verifiers forwards supported domain restrictions to web search tools and disables any provider tools it cannot constrain (see [Harbor network policies](https://docs.primeintellect.ai/verifiers/harbor.md#network-policies)). By default, restricted runs also append a short notice to the agent's system prompt (or first user message when the harness cannot append a system prompt) explaining the network policy; pass `--no-env.agent.network-notice` if a benchmark requires the unmodified prompt verbatim.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.