# API keys
Source: https://docs.primeintellect.ai/api-reference/api-keys

How to generate and use authentication keys with our API

## Overview

This guide explains how to generate and manage API keys for authenticating requests to our API. Proper use of API keys is critical for securing your application and controlling access.

## Generating an API Key

To generate API Key, navigate to [Settings -> API Keys](https://app.primeintellect.ai/dashboard/tokens), and click the `Generate New Key +` button.

![create key](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/api/keys/create-key.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=47b043c74405eb713304b4db281ede9e)

You can assign fine-grained permissions to each key, allowing you to create multiple keys with specific scopes or roles. This enhances the security of your application by limiting each key's responsibilities. Additionally, you can set an expiration date for each key. We strongly recommend **always** setting an expiration date to avoid using indefinitely valid keys.

> [!WARNING]
>
> Ensure that all API keys are stored securely and never shared in untrusted environments or with third-party applications.

Once a key is generated, you will be able to copy it for immediate use.

![copy key](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/api/keys/new-key.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=b8a3ccc3cdc7d9bc72456634ab801588)

> [!WARNING]
>
> The key value is displayed only once. If you lose it, you will not be able to retrieve it again.

## Revoking an API Key

If you no longer need a particular API key go to [Settings -> API Keys](https://app.primeintellect.ai/dashboard/tokens), find your key on the list and click on `Remove` button on the right side.

## Prime Traces permissions

For Traces SDK and CLI operations, create a key with the required Prime Traces read/write permissions. Confirm the personal or team workspace as well as the key permissions. See [Traces authentication and workspace selection](/traces/uploading#authentication-and-workspace).


# Delete Token
Source: https://docs.primeintellect.ai/api-reference/api-keys/delete-token

https://api.primeintellect.ai/openapi.json delete /api/v1/tokens/{token_id}



# Disable Token
Source: https://docs.primeintellect.ai/api-reference/api-keys/disable-token

https://api.primeintellect.ai/openapi.json post /api/v1/tokens/{token_id}/disable
Disable an API key. Re-enabling is dashboard-only, so a leaked key cannot
turn back on keys its owner disabled.



# List Tokens
Source: https://docs.primeintellect.ai/api-reference/api-keys/list-tokens

https://api.primeintellect.ai/openapi.json get /api/v1/tokens/



# Get Disks Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-disks-availability

https://api.primeintellect.ai/openapi.json get /api/v1/availability/disks



# Get Gpu Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-gpu-availability

https://api.primeintellect.ai/openapi.json get /api/v1/availability/gpus



# Get Gpu Summary
Source: https://docs.primeintellect.ai/api-reference/availability/get-gpu-summary

https://api.primeintellect.ai/openapi.json get /api/v1/availability/gpu-summary
Get GPU pricing summary data grouped by GPU type and instance count.



# Get Legacy Cluster Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-legacy-cluster-availability

https://api.primeintellect.ai/openapi.json get /api/v1/availability/clusters



# Get Legacygpu Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-legacygpu-availability

https://api.primeintellect.ai/openapi.json get /api/v1/availability/



# Get Multinode Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-multinode-availability

https://api.primeintellect.ai/openapi.json get /api/v1/availability/multi-node



# Get Multinode Summary
Source: https://docs.primeintellect.ai/api-reference/availability/get-multinode-summary

https://api.primeintellect.ai/openapi.json get /api/v1/availability/multi-node-summary
Get multi-node (cluster) pricing summary data grouped by GPU type and instance count.



# Token usage and cost for a single RFT run
Source: https://docs.primeintellect.ai/api-reference/billing/token-usage-and-cost-for-a-single-rft-run

https://api.primeintellect.ai/openapi.json get /api/v1/billing/runs/{run_id}/usage
Return the running token + cost totals for one RFT run.

Mirrors the per-row training data shown on the billing page (Training
tokens, Inference input/output tokens, price per million tokens, and
total cost) so an agent can poll this endpoint to monitor a live run.



# Wallet balance + most recent billing rows
Source: https://docs.primeintellect.ai/api-reference/billing/wallet-balance-+-most-recent-billing-rows

https://api.primeintellect.ai/openapi.json get /api/v1/billing/wallet
Return the wallet's current balance and most recent billing rows.

All resource types are included (compute, training, inference, traces,
disks, sandboxes, images) — same source-of-truth `Billing` table the
dashboard's Billing History tab reads. Sorted by `lastBilledAt` desc.



# Get Availability Information
Source: https://docs.primeintellect.ai/api-reference/check-gpu-availability

How to check GPU, cluster, and disk availability and pricing

> [!NOTE]
>
> Before you start, ensure that you have [API Key](./api-keys) with `Availability -> Read` permission

## Available Endpoints

The availability API provides two endpoints to query different types of resources:

* **`/api/v1/availability/gpus`** - Get GPU instance availability
* **`/api/v1/availability/disks`** - Get standalone disk availability

## Retrieving GPU Availability Data

Suppose you want to check pricing options for a single **H100 GPU**, with the location restricted to the **United States** or **Canada**. To do this, send a request to our availability endpoint as shown below:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/gpus?regions=united_states&regions=canada&gpu_count=1&gpu_type=H100_80GB' \
  --header 'Authorization: Bearer your_api_key'
```

> [!TIP]
>
> You can generate request samples using our interactive [`Availability API`](./availability/get-gpu-availability)
> documentation

This is a **GET** request, and it requires an **Authorization: Bearer** token with your API key. The request accepts query parameters, allowing you to filter GPUs by region, GPU count, GPU type, and other criteria. In this case, the filters include **regions**, **gpu\_count**, and **gpu\_type**.

### Query Parameters

All availability endpoints support pagination and the following filters:

| Parameter | Type | Description |
| - | - | - |
| `regions` | List\[string] | Filter by region(s) (e.g., `united_states`, `canada`, `europe_west`) |
| `gpu_count` | integer | Desired number of GPUs |
| `gpu_type` | string | GPU model (e.g., `H100_80GB`, `A100_80GB`) |
| `socket` | string | GPU socket type (e.g., `PCIe`, `SXM`) |
| `security` | string | Security type: `secure_cloud` |
| `data_center_id` | string | Filter by specific data center ID |
| `cloud_id` | string | Filter by provider's cloud ID |
| `disks` | List\[string] | Filter by disk IDs (see [Disk Filtering](#filtering-by-user-disks) section) |
| `page` | integer | Page number (default: 1, min: 1) |
| `page_size` | integer | Results per page (default: 100, max: 100) |

## Understanding the GPU Response Object

The GPU availability endpoint returns a paginated response with a list of available GPU configurations and the total count. Here is an example of the JSON response you might receive:

```json Example response theme={null}
{
  "items": [
    {
      "cloudId": "NVIDIA H100 PCIe",
      "gpuType": "H100_80GB",
      "socket": "PCIe",
      "provider": "runpod",
      "region": "united_states",
      "dataCenter": "US-KS-2",
      "country": "US",
      "gpuCount": 1,
      "gpuMemory": 80,
      "disk": {
        "minCount": 80,
        "defaultCount": 80,
        "maxCount": 1000,
        "pricePerUnit": 0.00014,
        "step": 1,
        "defaultIncludedInPrice": false,
        "additionalInfo": null
      },
      "vcpu": {
        "defaultCount": 16
      },
      "memory": {
        "defaultCount": 251
      },
      "internetSpeed": null,
      "interconnect": null,
      "interconnectType": null,
      "provisioningTime": null,
      "stockStatus": "Low",
      "security": "secure_cloud",
      "prices": {
        "onDemand": 2.69,
        "isVariable": false,
        "currency": "USD"
      },
      "images": [
        "ubuntu_22_cuda_12",
        "cuda_12_1_pytorch_2_2",
        "cuda_11_8_pytorch_2_1",
        "stable_diffusion",
        "flux",
        "axolotl",
        "bittensor",
        "vllm_llama_8b",
        "vllm_llama_70b",
        "vllm_llama_405b"
      ],
      "isSpot": null,
      "prepaidTime": null
    }
  ],
  "totalCount": 247
}
```

> [!NOTE]
>
> For a full breakdown of the response schema, see the [`Get Gpu Availability
>     Endpoint`](./availability/get-gpu-availability) documentation

### Response Structure

#### `items`

An array containing the GPU availability records for the current page. Each item represents a unique GPU configuration from a provider.

#### `totalCount`

The total number of GPU configurations matching your filters across all pages.

For example, if `totalCount` is 247 and `page_size` is 100, you know there are 3 pages total.

Let's walk through the most important fields in each GPU availability item:

#### `provider`

Indicates the company or platform providing the GPU. You may encounter multiple offers from the same provider.

#### `cloudId`

A unique identifier provided by the GPU provider. This value is required if you plan to provision the GPU through our provisioning API.

#### `region`

The geographic region where the GPU is located (e.g., `united_states`, `europe_west`). This helps you select resources closer to your users.

#### `dataCenter`

Optional but necessary if present in the response. If the provider operates multiple datacenters with the same `cloudId`, you must specify which data center to use when provisioning the GPU.

#### Resource Specifications: `disk`, `sharedDisk`, `vcpu`, `memory`

Each resource field contains detailed specification information with the following attributes:

* **`minCount`**: Minimum allowed value for this resource
* **`maxCount`**: Maximum allowed value for this resource
* **`defaultCount`**: Default value provided by the provider
* **`pricePerUnit`**: Cost per unit per hour (if customizable)
* **`step`**: Increment step for adjusting the value
* **`defaultIncludedInPrice`**: Whether the default value is included in the base price
* **`additionalInfo`**: Extra information about pricing or constraints

**`disk`**: Instance-local disk storage in GB. This storage is attached to your GPU instance.

**`vcpu`**: Number of virtual CPUs allocated to the instance.

**`memory`**: RAM size available in GB.

#### Cost Calculation Example

In the example above, your server will include `16 vCPUs` and `251GB of RAM`, with customizable `disk` space. If the disk space is adjustable, its cost is calculated separately from the **prices** property. Here's an example breakdown of total costs:

```
GPU cost: $2.69
vcpu cost: $0.00
memory cost: $0.00
disk cost: $0.0112 (80 units * $0.00014)

Total cost: $2.7012
```

The total cost varies based on the disk space that you send to the provisioning API. The value can range between `minCount` and `maxCount` adjustable by `step` and you'll pay `pricePerUnit/hr` for every unit you want to use.

Some providers offer predefined server configurations, with the `defaultIncludedInPrice` set to `true`. This means the `defaultCount` is included in the base price, but any changes (even reductions) will incur additional charges.

#### `interconnect` and `interconnectType`

For multi-GPU setups, these fields specify the interconnect speed (in Gbps) and technology type (e.g., `InfiniBand`, `NVLink`) used for GPU-to-GPU communication.

#### `provisioningTime`

Estimated time in minutes required to provision and start the GPU instance.

#### `prices` and `security`

GPUs are provided by providers that maintain security standards and are hosted in secured datacenters. The price is defined in `prices.onDemand`.

Some providers may also use dynamic pricing for their instances. If the field `isVariable` is set to `true`, the price may fluctuate based on demand or currency conversion.

#### `prepaidTime`

If set, you will be pre-charged for the total number of hours specified in this field at the time of ordering, regardless of actual usage. After the prepaid time expires, the standard hourly rate will apply.

## Filtering by User Disks

The `disks` query parameter allows you to filter GPU availability based on your existing disks with persisted data. This is particularly useful when you want to provision a GPU instance that can attach to disks where you've already stored your datasets, models, or other important data.

### How It Works

When you provide disk IDs via the `disks` parameter, the API:

1. Fetches the disk information for the provided disk IDs
2. Filters GPU availability to show only instances that can be provisioned in locations where your disks are available for attachment
3. Applies rest of the filters to narrow down the results.

### Example Request

Suppose you have two disks with IDs `clhxy6aw80000j8080gdf8kqv` and `clhxy9bz50001j8080hdf9lrw` containing your training datasets in different datacenters. You can find GPUs that can attach to either disk:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/gpus?disks=clhxy6aw80000j8080gdf8kqv&disks=clhxy9bz50001j8080hdf9lrw&gpu_type=H100_80GB' \
  --header 'Authorization: Bearer your_api_key'
```

This will return H100 GPUs available in the same locations where your disks are stored, making it easy to reuse your persisted data.

### Use Cases

* **Data Persistence**: Reuse datasets, models, and checkpoints stored on existing disks
* **Quick Provisioning**: Attach pre-loaded data disks to new GPU instances without re-uploading

## Checking Disk Availability

The disk availability endpoint allows you to query standalone network-attached storage options across different providers and datacenters.

### Endpoint

```bash theme={null}
GET https://api.primeintellect.ai/api/v1/availability/disks
```

### Example Request

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/disks?regions=united_states&page=1&page_size=50' \
  --header 'Authorization: Bearer your_api_key'
```

### Query Parameters for Disk Availability

| Parameter | Type | Description |
| - | - | - |
| `regions` | List\[string] | Filter by region(s) |
| `data_center_id` | string | Filter by specific data center ID |
| `cloud_id` | string | Filter by provider's cloud ID |
| `page` | integer | Page number (default: 1, min: 1) |
| `page_size` | integer | Results per page (default: 100, max: 100) |

### Disk Availability Response

The disk endpoint returns a paginated response with disk availability objects and the total count:

```json Example disk response theme={null}
{
  "items": [
    {
      "cloudId": null,
      "provider": "hyperstack",
      "dataCenter": "US-1",
      "country": "US",
      "region": "united_states",
      "spec": {
        "minCount": 0,
        "defaultCount": 0,
        "maxCount": 100000,
        "pricePerUnit": 0.00015,
        "step": 1,
        "defaultIncludedInPrice": false,
        "additionalInfo": null
      },
      "stockStatus": "Available",
      "security": "secure_cloud",
      "isMultinode": false
    }
  ],
  "totalCount": 15
}
```

### Key Fields in Disk Response

#### `spec`

Contains the disk size specification with the same structure as resource specifications in GPU responses:

* **`minCount`**: Minimum disk size in GB
* **`maxCount`**: Maximum disk size in GB
* **`pricePerUnit`**: Cost per GB per hour
* **`step`**: Size increment (usually 1 GB)

#### `isMultinode`

Indicates whether this disk can be attached to multiple instances simultaneously. This is useful for shared data scenarios where multiple GPU instances need access to the same dataset.

#### `security`

Disks are hosted on secure cloud infrastructure.

### Disk Cost Calculation

Disk costs are straightforward to calculate:

```
Disk cost per hour = disk_size_gb * pricePerUnit
```

For example, a 500GB disk with `pricePerUnit: 0.00015`:

```
500 GB * $0.00015/GB/hr = $0.075/hr
```

## Pagination

All availability endpoints support pagination to handle large result sets efficiently:

* Use `page` to specify which page of results to retrieve (1-indexed)
* Use `page_size` to control how many results per page (max 100)
* Results are returned in consistent order for predictable pagination

Example with pagination:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/gpus?gpu_type=A100_80GB&page=2&page_size=50' \
  --header 'Authorization: Bearer your_api_key'
```


# Create Disk
Source: https://docs.primeintellect.ai/api-reference/disks/create-disk

https://api.primeintellect.ai/openapi.json post /api/v1/disks/



# Delete Disk
Source: https://docs.primeintellect.ai/api-reference/disks/delete-disk

https://api.primeintellect.ai/openapi.json delete /api/v1/disks/{disk_id}



# Get Disk
Source: https://docs.primeintellect.ai/api-reference/disks/get-disk

https://api.primeintellect.ai/openapi.json get /api/v1/disks/{disk_id}



# List Disks
Source: https://docs.primeintellect.ai/api-reference/disks/list-disks

https://api.primeintellect.ai/openapi.json get /api/v1/disks/



# Update Disk
Source: https://docs.primeintellect.ai/api-reference/disks/update-disk

https://api.primeintellect.ai/openapi.json patch /api/v1/disks/{disk_id}



# Bulk Delete Evaluations
Source: https://docs.primeintellect.ai/api-reference/evals/bulk-delete-evaluations

https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/bulk-delete



# Create Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/create-evaluation

https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/
Create a new evaluation

This endpoint supports:
- Environment evaluations: Provide environments
- Prime RL evaluations: Provide run_id
- Suite evaluations: Provide suite_id

Ownership:
- If team_id is provided in request, the evaluation will be owned by the team
- Otherwise, the evaluation will be owned by the authenticated user



# Delete Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/delete-evaluation

https://api.primeintellect.ai/openapi.json delete /api/v1/evaluations/{evaluation_id}



# Finalize Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/finalize-evaluation

https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/{evaluation_id}/finalize
Mark an evaluation as complete and compute final statistics.



# Get Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/get-evaluation

https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/{evaluation_id}
Get detailed information about a specific evaluation.



# Get Samples
Source: https://docs.primeintellect.ai/api-reference/evals/get-samples

https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/{evaluation_id}/samples
Get samples for a specific evaluation



# List Evaluations
Source: https://docs.primeintellect.ai/api-reference/evals/list-evaluations

https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/
Get a list of evaluations owned by the authenticated user or a team.

Without team_id, returns the user's personal evaluations only. With team_id,
returns that team's evaluations; the user must be a member.



# Push Samples
Source: https://docs.primeintellect.ai/api-reference/evals/push-samples

https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/{evaluation_id}/samples
Push evaluation samples

This endpoint can be called multiple times to stream samples as they're generated.



# Update Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/update-evaluation

https://api.primeintellect.ai/openapi.json put /api/v1/evaluations/{evaluation_id}
Update an existing evaluation



# Get Factory Nodes
Source: https://docs.primeintellect.ai/api-reference/factory/get-factory-nodes

https://api.primeintellect.ai/openapi.json get /api/v1/factory/nodes
Per-cluster GPU node tables for the caller's team (the sinfo):
node name, coarse state (ready / cordoned / offline / unknown),
GPU type and total/used counts, and which allocation the node is
assigned to ("slurm" for a claimed node, null otherwise).

Node names are the team's own hardware and are shown to the team
only; data of other teams is never returned. Unusable capacity
snapshots surface as a per-cluster capacity source with an error
status, never as a silent empty node list.



# Get Factory Status
Source: https://docs.primeintellect.ai/api-reference/factory/get-factory-status

https://api.primeintellect.ai/openapi.json get /api/v1/factory/status
Fleet status for the caller's team: cluster capacity split into
training / inference / slurm allocations with reserved, in-use,
idle-inside, unassigned and unknown GPU counts plus per-source
freshness.

The selected `team_id` must be a team the token's user is a member
of; data of other teams is never returned.



# Get Factory Utilization
Source: https://docs.primeintellect.ai/api-reference/factory/get-factory-utilization

https://api.primeintellect.ai/openapi.json get /api/v1/factory/utilization
The team's Model Factory utilization curve: Slurm node
allocated/idle counts per bucket, derived from the slurmdbd
accounting record with the live jobs read as the truth for the
current bucket. Per-source freshness travels in `sources`; gaps
are gaps, never zero-filled.



# Get Factory Workloads
Source: https://docs.primeintellect.ai/api-reference/factory/get-factory-workloads

https://api.primeintellect.ai/openapi.json get /api/v1/factory/workloads
The team's Model Factory workloads feed (the squeue): leaf
training runs, inference deployments and native Slurm jobs with
owner, state, requested/allocated GPUs, per-source freshness and
— for terminal states (completed/failed/all) — the completion
stamp of the terminal record (`ended_at`), bounded and newest
first.

The selected `team_id` must be a team the token's user is a
member of; data of other teams is never returned. The default
state window is running + queued. Windows that include terminal
states read the terminal records (RFTRun/Job rows; the slurmdbd
accounting read for native Slurm jobs) and are bounded to
`limit` rows (default 50, max 200), newest first; `since` bounds
them further by timestamp.



# Submit Feedback
Source: https://docs.primeintellect.ai/api-reference/feedback/submit-feedback

https://api.primeintellect.ai/openapi.json post /api/v1/feedback



# Cancel Hosted Evaluation Route
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/cancel-hosted-evaluation-route

https://api.primeintellect.ai/openapi.json patch /api/v1/hosted-evaluations/{evaluation_id}/cancel
Cancel a running hosted evaluation.



# Create Hosted Evaluation
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/create-hosted-evaluation

https://api.primeintellect.ai/openapi.json post /api/v1/hosted-evaluations
Create and start a hosted evaluation.



# Get Hosted Evaluation Logs Route
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/get-hosted-evaluation-logs-route

https://api.primeintellect.ai/openapi.json get /api/v1/hosted-evaluations/{evaluation_id}/logs
Get real-time logs from the sandbox running a hosted evaluation.



# Get Inference Models
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/get-inference-models

https://api.primeintellect.ai/openapi.json get /api/v1/hosted-evaluations/models
Get available models from Prime Inference API for hosted evaluations



# Upload Hosted Evaluation Logs Route
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/upload-hosted-evaluation-logs-route

https://api.primeintellect.ai/openapi.json put /api/v1/hosted-evaluations/{evaluation_id}/logs
Store the runner's final log. Called by the hosted-eval sandbox itself as it
exits, with the run's own key; a VM sandbox is gone before the exit webhook.



# Delete User Image
Source: https://docs.primeintellect.ai/api-reference/images/delete-user-image

https://api.primeintellect.ai/openapi.json delete /api/v1/images/{image_name}/{image_tag}
Delete a user image from the database and Google Artifact Registry.

For team images, only the image creator or team admins can delete.

## Rate Limit

100 requests per 60 seconds per IP and token.


# Get Build Status
Source: https://docs.primeintellect.ai/api-reference/images/get-build-status

https://api.primeintellect.ai/openapi.json get /api/v1/images/build/{build_id}
Get the status of an image build.



# Initiate Image Build
Source: https://docs.primeintellect.ai/api-reference/images/initiate-image-build

https://api.primeintellect.ai/openapi.json post /api/v1/images/build
Build a VM image from a public source or an uploaded Dockerfile context.

## Rate Limit

150 requests per 60 seconds per IP and token. Source builds also have per-image account limits.


# List Image Builds
Source: https://docs.primeintellect.ai/api-reference/images/list-image-builds

https://api.primeintellect.ai/openapi.json get /api/v1/images/builds
List all image builds for the current user.



# List User Images
Source: https://docs.primeintellect.ai/api-reference/images/list-user-images

https://api.primeintellect.ai/openapi.json get /api/v1/images
List images and builds for the current user, scoped by team context. Returns personal images by default, or team images if teamId is provided.

Pagination note: `limit`, `offset`, and `totalCount` count logical images
(unique `owner/imageName:imageTag`). A VM image may produce multiple
`data[]` rows (the registered artifact plus any active builds), so
`len(data)` can exceed `limit` and `totalCount` reflects images, not rows.

The optional `search` parameter narrows results (case-insensitively) to
images whose name, tag, or `name:tag` reference contains the given
substring.

Setting `ownerScope=platform` lists only org-less platform images and
builds.

## Rate Limit

300 requests per 60 seconds per IP and token.


# Start Image Build
Source: https://docs.primeintellect.ai/api-reference/images/start-image-build

https://api.primeintellect.ai/openapi.json post /api/v1/images/build/{build_id}/start
Start the image build process after uploading the build context.

Runs img2vm in a VM sandbox to build and convert the image locally.



# Update Images
Source: https://docs.primeintellect.ai/api-reference/images/update-images

https://api.primeintellect.ai/openapi.json patch /api/v1/images
Update one or many logical images: visibility, name/tag, and owner.

Applies up to 100 independent patches; each item names its source image
exactly and may change `visibility`, rename `name`/`tag`, move a personal
image into one of the caller's teams, or promote a personal/team image to
an org-less platform image (platform admins only; platform images are
always PUBLIC).

Renames and owner moves update the whole logical group (container + VM
artifacts and their linked builds) atomically without moving the backing
artifacts: the old reference stops resolving and the new reference
resolves to the same content. An owner move settles storage billing to
the source wallet at the cutover instant and bills the destination
afterwards. Destinations are never overwritten (`destination_exists`).

`dryRun` performs resolution, authorization, collision detection, and
quota projection without writing; a valid request with item-specific
failures still returns 200 with per-item errors.

## Rate Limit

50 requests per 60 seconds per IP and token.


# Chat Completions API
Source: https://docs.primeintellect.ai/api-reference/inference-chat-completions

Generate responses with Prime Inference models

Send OpenAI-compatible requests to `POST https://api.pinference.ai/api/v1/chat/completions`. Pass an exact model ID from the [models API](/api-reference/inference-models).

```bash theme={null}
export PRIME_API_KEY="your-api-key"

curl https://api.pinference.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $PRIME_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "z-ai/glm-5.3",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

To charge a team, add `-H "X-Prime-Team-ID: your-team-id"` to each request. See [team billing](/inference/overview#team-billing) for SDK setup and account details.

## Stream a response

Set `"stream": true` in the request body to receive server-sent events as tokens are generated. In the OpenAI Python SDK:

```python theme={null}
stream = client.chat.completions.create(
    model="z-ai/glm-5.3",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

## Usage details

If you need token counts and cost in the response, request usage details with the Prime Inference extension:

```python theme={null}
response = client.chat.completions.create(
    model="z-ai/glm-5.3",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"usage": {"include": True}},
)
print(response.usage)
```

Supported parameters and limits depend on the selected model. Check the [model catalog](https://app.primeintellect.ai/dashboard/inference?tab=models) for current capabilities rather than assuming every model accepts the same options.


# Evaluations API
Source: https://docs.primeintellect.ai/api-reference/inference-evaluations

Score, classify, and answer yes/no questions with evaluation models such as Clef

Evaluation models such as [Clef](/inference/models/clef) do not generate text. They read a `state` and answer a set of typed `questions` about it, returning a probability, a choice, or a score for each. Send these requests to `POST https://api.pinference.ai/api/v1/evaluations`; evaluation models are rejected on the [Chat Completions API](/api-reference/inference-chat-completions).

```bash theme={null}
export PRIME_API_KEY="your-api-key"

curl https://api.pinference.ai/api/v1/evaluations \
  -H "Authorization: Bearer $PRIME_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cloudflare/clef-flash",
    "state": "User: the checkout page returns a 500 every time I pay.",
    "questions": {
      "incident": {
        "type": "boolean",
        "instructions": "Is the user reporting an outage?"
      },
      "team": {
        "type": "choice",
        "criteria": {
          "engineering": "Bugs, errors, and outages",
          "billing": "Charges, refunds, and invoices"
        }
      },
      "severity": {
        "type": "score",
        "criteria": ["minor", "major", "critical"]
      }
    }
  }'
```

```json theme={null}
{
  "answers": {
    "incident": {"type": "boolean", "probability": 0.95},
    "team": {"type": "choice", "choice": "engineering", "probabilities": {"engineering": 0.97, "billing": 0.03}},
    "severity": {"type": "score", "score": 1.5}
  },
  "usage": {"inputTokens": 123, "outputTokens": 0},
  "warnings": []
}
```

All questions in a request are answered together against the same `state`. To charge a team, add `-H "X-Prime-Team-ID: your-team-id"` to each request. See [team billing](/inference/overview#team-billing).

## Request body

| Field | Type | Description |
| - | - | - |
| `model` | string | An evaluation model ID from the [models API](/api-reference/inference-models), such as `cloudflare/clef` or `cloudflare/clef-flash`. |
| `state` | string, object, or array | The content to evaluate: a conversation, a document, an agent trace, or any JSON value. A state too long for the model's context window (65,536 tokens for Clef, 24,576 for Clef Flash) is truncated to fit. |
| `questions` | object | 1 to 64 questions, keyed by an ID you choose. IDs may contain letters, digits, `_`, `.`, and `-`, up to 100 characters. Each answer is returned under the same ID. |
| `images` | array of strings | Optional. Up to 4 images as base64 `data:` URLs (PNG, JPEG, or WebP), at most 4 MiB each and 8 MiB in total. Remote image URLs are not accepted. |

The whole request body can be at most 16 MiB.

## Question types

Every question has a `type` and an optional `instructions` field that tells the model what to judge. `instructions` and the `criteria` descriptions can be strings or JSON values.

| Type | `criteria` | Answer fields |
| - | - | - |
| `boolean` | Optional. An object with `"true"` and/or `"false"` keys that describe what each outcome means. | `probability`: the probability that the answer is true, from 0 to 1. |
| `choice` | Required. An object of 1 to 255 options, mapping each option name to a description (or `null`). | `choice`: the selected option name. May include `probabilities` per option. |
| `score` | Required. An array of 2 to 10 descriptions, ordered from lowest to highest. | `score`: a number on the scale defined by `criteria`. |

Each answer also echoes its `type`. Answers can include additional fields such as `confidence` depending on the model, so read the fields you need and ignore the rest.

## Usage and billing

`usage.inputTokens` and `usage.outputTokens` report the tokens processed for the request. Evaluation models are billed on input tokens at the price listed in the [model catalog](https://app.primeintellect.ai/dashboard/inference?tab=models). The response includes an `X-Inference-Id` header you can share with support to trace a request.

## Vercel AI SDK compatibility

The endpoint accepts the same request and response format as the Vercel AI Gateway evaluation model protocol, so clients written for that protocol can point at Prime Inference. Prime Inference also serves the gateway's protocol path, `POST https://api.pinference.ai/v4/ai/evaluation-model`, which reads the model ID from the `ai-model-id` request header when the body has no `model` field.


# Models API
Source: https://docs.primeintellect.ai/api-reference/inference-models

List available inference models and inspect their metadata

The model catalog is available at `https://api.pinference.ai/api/v1/models`. It includes model IDs, serving types, pricing, and specifications when available. Availability and prices can change; read the current response rather than relying on a fixed list in this guide.

```bash theme={null}
export PRIME_API_KEY="your-api-key"

curl https://api.pinference.ai/api/v1/models \
  -H "Authorization: Bearer $PRIME_API_KEY"
```

To select a team billing account for authenticated requests, add `-H "X-Prime-Team-ID: your-team-id"` (see [team billing](/inference/overview#team-billing)). You can also list models with `prime inference models` or browse the [model catalog](https://app.primeintellect.ai/dashboard/inference?tab=models).

Use the returned `id` as the `model` in a [chat completion](/api-reference/inference-chat-completions). For example, the hosted GLM-5.3 model uses `z-ai/glm-5.3`; see the [GLM-5.3 guide](/inference/models/glm-5-3). Evaluation models such as `cloudflare/clef` list `decisions` as their output modality; call them with the [Evaluations API](/api-reference/inference-evaluations) instead.


# API Overview
Source: https://docs.primeintellect.ai/api-reference/introduction



The Prime Intellect API provides programmatic access to our full platform. Get started by [setting up an API Key](./api-keys).

## Inference API

Access frontier models via an OpenAI-compatible API.

> **[List models](/api-reference/inference-models)**
>
> View available models and their capabilities

> **[Chat completions](/api-reference/inference-chat-completions)**
>
> Generate text with frontier models

## Compute API

Provision and manage GPU instances across providers.

> **[Check GPU availability](/api-reference/check-gpu-availability)**
>
> View GPU resources and availability across providers

> **[Provision GPUs](/api-reference/provision-gpu)**
>
> Deploy GPU instances with customizable configurations

> **[Manage instances](/api-reference/managing-pods)**
>
> Monitor and control your active GPU instances

> **[Manage storage](/api-reference/managing-disks)**
>
> Create persistent network-attached disks

## Traces API

Upload, query and search training, evaluation and inference traces.

> **[List traces](/api-reference/traces/traces/list-traces)**
>
> Query trace summaries with filters and cursor pagination

> **[Upload traces](/api-reference/traces/ingestion/upload-traces)**
>
> Upload traces as JSONL in a single multipart request

## Sandboxes API

Run code securely in isolated containers.

> **[Sandboxes overview](/sandboxes/overview)**
>
> Learn about sandboxes and their use cases

> **[Sandboxes SDK](/sandboxes/sdk)**
>
> Create and manage sandboxes programmatically


# Managing Disks
Source: https://docs.primeintellect.ai/api-reference/managing-disks

How to create and manage network-attached storage disks

> [!NOTE]
>
> Before you start, ensure that you have [API Key](./api-keys) with `Disks -> Read and write` permission

## Overview

Network-attached disks provide persistent storage that can be shared across multiple GPU instances. Disks persist independently from instances, making them ideal for storing datasets, model checkpoints, and other data that needs to survive instance termination.

## Finding Available Disk Options

Before creating a disk, use the disk availability endpoint to find available storage options across different providers and datacenters.

### Query Disk Availability

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/disks?regions=united_states&page=1&page_size=50' \
  --header 'Authorization: Bearer your_api_key'
```

### Understanding the Response

The response returns available disk configurations:

```json theme={null}
[
    {
        "cloudId": null,
        "provider": "hyperstack",
        "dataCenter": "US-1",
        "country": "US",
        "region": "united_states",
        "spec": {
            "minCount": 0,
            "defaultCount": 0,
            "maxCount": 100000,
            "pricePerUnit": 0.00015,
            "step": 1,
            "defaultIncludedInPrice": false,
            "additionalInfo": null
        },
        "stockStatus": "Available",
        "security": "secure_cloud",
        "isMultinode": false
    }
]
```

### Key Fields

* **`cloudId`**: Identifies the storage configuration (if applicable)
* **`provider`**: The provider offering the storage (e.g., `hyperstack`, `runpod`)
* **`dataCenter`**: Location where the disk will be created
* **`spec.minCount`**: Minimum disk size in GB
* **`spec.maxCount`**: Maximum disk size in GB
* **`spec.pricePerUnit`**: Cost per GB per hour
* **`isMultinode`**: Whether the disk can be attached to multiple instances

> [!TIP]
>
> Filter by `regions` or `data_center_id` to find disks in specific locations where you plan to run your GPU instances.

## Creating a Disk

Once you've identified a suitable disk configuration from the availability API, create a disk using the information from the availability response.

### Request Body Structure

```json theme={null}
{
  "disk": {
    "size": 500,
    "name": "training-dataset",
    "dataCenterId": "US-1"
  },
  "provider": {
    "type": "hyperstack"
  }
}
```

### Parameters

| Parameter | Required | Description |
| - | - | - |
| `disk.size` | Yes | Disk size in GB (must be between `minCount` and `maxCount` from availability) |
| `disk.name` | No | Human-friendly name for the disk (auto-generated if not provided) |
| `disk.cloudId` | Yes\* | Cloud ID from the availability response (\*if provided in availability response) |
| `disk.dataCenterId` | Yes\* | Data center ID from availability (\*if provided in availability response) |
| `provider.type` | Yes | Provider type from availability response (e.g., `hyperstack`, `runpod`) |

### Example: Creating a 500GB Disk

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/disks/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "disk": {
    "size": 500,
    "name": "ml-training-data",
    "dataCenterId": "US-1"
  },
  "provider": {
    "type": "hyperstack"
  }
}'
```

### Response

On successful creation, you'll receive a `201 Created` response with the disk details:

```json theme={null}
{
  "id": "fdb5205fd9b14c9d804d3f70b4c96da0",
  "name": "ml-training-data",
  "createdAt": "2025-11-07T21:07:15.183000",
  "updatedAt": "2025-11-07T21:07:15.183000",
  "terminatedAt": null,
  "status": "PROVISIONING",
  "providerType": "hyperstack",
  "size": 500,
  "info": {
    "cloudId": null,
    "country": "US",
    "isMultinode": false,
    "dataCenterId": "US-1"
  },
  "priceHr": 0.075,
  "stoppedPriceHr": 0.075,
  "provisioningPriceHr": 0.0,
  "userId": "clvb0itli0000oxevorvgrpfn",
  "teamId": null,
  "walletId": "clvb0iv0c0004oxev5mnt1xiv",
  "pods": [],
  "clusters": []
}
```

### Cost Calculation

The hourly cost is calculated as:

```
Hourly Cost = size (GB) × pricePerUnit (from availability)
```

For a 500GB disk at \$0.00015/GB/hr:

```
500 GB × $0.00015/GB/hr = $0.075/hr
```

> [!WARNING]
>
> Disks are billed continuously from creation until termination, regardless of whether they're attached to an instance.

## Listing Your Disks

Retrieve all disks associated with your account:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/disks/?limit=50&offset=0' \
  --header 'Authorization: Bearer your_api_key'
```

### Query Parameters

| Parameter | Default | Description |
| - | - | - |
| `limit` | 50 | Number of results per page (max 200) |
| `offset` | 0 | Number of results to skip for pagination |

### Response

```json theme={null}
{
  "totalCount": 1,
  "data": [
    {
      "id": "fdb5205fd9b14c9d804d3f70b4c96da0",
      "name": "ml-training-data",
      "createdAt": "2025-11-07T21:07:15.183000",
      "updatedAt": "2025-11-07T21:08:01.934000",
      "terminatedAt": null,
      "status": "ACTIVE",
      "providerType": "hyperstack",
      "size": 500,
      "info": {
        "cloudId": null,
        "country": "US",
        "isMultinode": false,
        "dataCenterId": "US-1"
      },
      "priceHr": 0.075,
      "stoppedPriceHr": 0.075,
      "provisioningPriceHr": 0.0,
      "userId": "clvb0itli0000oxevorvgrpfn",
      "teamId": null,
      "walletId": "clvb0iv0c0004oxev5mnt1xiv",
      "pods": [],
      "clusters": []
    }
  ],
  "limit": 50,
  "offset": 0
}
```

## Getting a Single Disk

Retrieve detailed information about a specific disk:

```bash theme={null}
curl --request GET \
  --url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
  --header 'Authorization: Bearer your_api_key'
```

### Response

```json theme={null}
{
  "id": "fdb5205fd9b14c9d804d3f70b4c96da0",
  "name": "ml-training-data",
  "createdAt": "2025-11-07T21:07:15.183000",
  "updatedAt": "2025-11-07T21:08:01.934000",
  "terminatedAt": null,
  "status": "ACTIVE",
  "providerType": "hyperstack",
  "size": 500,
  "info": {
    "cloudId": null,
    "country": "US",
    "isMultinode": false,
    "dataCenterId": "US-1"
  },
  "priceHr": 0.075,
  "stoppedPriceHr": 0.075,
  "provisioningPriceHr": 0.0,
  "userId": "clvb0itli0000oxevorvgrpfn",
  "teamId": null,
  "walletId": "clvb0iv0c0004oxev5mnt1xiv",
  "pods": [],
  "clusters": []
}
```

The response includes:

* **`pods`**: Array of pod IDs currently using this disk
* **`clusters`**: Array of cluster IDs currently using this disk
* **`info`**: Additional metadata about disk location and capabilities

## Updating a Disk

Currently, you can update the disk name:

```bash theme={null}
curl --request PATCH \
  --url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "updated-training-dataset"
}'
```

### Response

```json theme={null}
{
  "id": "fdb5205fd9b14c9d804d3f70b4c96da0",
  "name": "updated-training-dataset"
}
```

## Deleting a Disk

Terminate a disk when you no longer need it. This will permanently delete the disk and all its data.

```bash theme={null}
curl --request DELETE \
  --url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
  --header 'Authorization: Bearer your_api_key'
```

### Response

```json theme={null}
{
  "status": "TERMINATED"
}
```

> [!WARNING]
>
> **Disk deletion is permanent and irreversible**. All data stored on the disk will be lost. Make sure to backup any important data before deleting a disk.

> [!NOTE]
>
> **Detachment Timing**: The disk detachment process can take longer than instance termination. After terminating an instance, you may need to wait a few moments before the disk becomes available for deletion or reuse.

## Disk Status Values

Disks go through different statuses during their lifecycle:

| Status | Description |
| - | - |
| `PROVISIONING` | Disk is being created |
| `PENDING` | Disk status is changing |
| `ACTIVE` | Disk is ready and can be attached to instances or deleted |
| `STOPPED` | Disk is stopped |
| `ERROR` | An error occurred during disk operations |
| `DELETING` | Disk is being deleted |
| `TERMINATED` | Disk has been deleted |
| `UNKNOWN` | Disk status is unknown (cannot access the current status) |

> [!NOTE]
>
> After terminating an instance, the disk detachment process may take a few moments to complete. You may need to wait briefly before the disk becomes available for deletion or reattachment.

## Common Use Cases

### Shared Training Data

Create large disks with your training datasets that can be reused across multiple training runs. This is ideal for storing preprocessed data, large image datasets, or any training data that needs to be accessed by multiple GPU instances. By keeping your data on a persistent disk, you avoid having to re-upload or regenerate datasets for each new training session.

### Model Checkpoints and Artifacts

Use disks to persist model checkpoints, trained weights, and training artifacts. This is especially valuable when running spot instances or interruptible workloads, as your progress remains safe even if the instance is terminated. You can resume training from the last checkpoint by simply attaching the disk to a new instance.

### Multi-Node Distributed Training

For distributed training across multiple GPUs or nodes, use multinode-compatible disks that can be accessed simultaneously by multiple instances. This enables shared access to training data and synchronized checkpointing across your distributed training cluster. Check the `isMultinode` field in the disk availability response to verify support.

## Best Practices

### Location Planning

* Create disks in the same datacenter where you plan to run GPU instances
* Use the [availability endpoint](./check-gpu-availability) with `disks` filter to find compatible GPUs

### Size Planning

* Start with the size needed - you cannot resize disks after creation
* Consider future growth when selecting disk size
* Remember that larger disks cost more per hour

### Data Management

* Regularly backup important data from disks
* Use descriptive names to identify disk contents

## Related Documentation

* [Check GPU Availability](./check-gpu-availability) - Filter GPUs by disk location
* [Provision GPU](./provision-gpu) - Attach disks when creating instances
* [Managing Instances](./managing-pods) - Manage instances with attached disks


# Managing Pods
Source: https://docs.primeintellect.ai/api-reference/managing-pods

How to get pods, statuses and delete instances

> [!NOTE]
>
> Before you start, ensure that you have [API Key](./api-keys) with `Instances -> Read` permission (or `Read and
>     Write` if you want to `Delete` instances)

## Retrieving Existing Pods

To view a list of active instances, use the [Get Pods](./pods/get-pods) endpoint.

```bash theme={null}
curl --request GET \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key'
```

This is a paginated request, with a default `limit` of **100** results. To adjust the number of results per page, use query parameters:

```bash theme={null}
curl --request GET \
  --url https://api.primeintellect.ai/api/v1/pods/?offset=0&limit=10 \
  --header 'Authorization: Bearer your_api_key'
```

This will return a JSON object containing the total number of instances (`total_count`), the specified `limit` and `offset` and the list of pods under `data` (check out [Pod schema](./pods/get-pod) for details)

```json theme={null}
{
  "total_count": 15,
  "offset": 0,
  "limit": 10,
  "data": [...]
}
```

### Retrieving a Specific Pod

If you know the `podId`, you can retrieve details for that specific pod:

```bash theme={null}
curl --request GET \
  --url https://api.primeintellect.ai/api/v1/pods/my_pod_id \
  --header 'Authorization: Bearer your_api_key'
```

## Checking Pod Status

After provisioning an instance, retrieve its status to obtain port mappings and connection details using the [Get Pod Status](./pods/status) endpoint. This request also supports checking multiple instance statuses simultaneously:

```bash theme={null}
curl --request GET \
  --url https://api.primeintellect.ai/api/v1/pods/status/?pod_ids=my_first_pod_id&pod_ids=my_second_pod_id \
  --header 'Authorization: Bearer your_api_key'
```

It returns a `data` with the list of object like:

```json theme={null}
{
  "podId": "my_first_pod_id",
  "providerType": "primeconpute",
  "status": "ACTIVE",
  "sshConnection": "root@135.23.125.123 -p 22",
  "costPerHr": 3.52,
  "primePortMapping": [
    {
      "internal": "22",
      "external": "22",
      "protocol": "TCP",
      "usedBy": "SSH",
      "description": ""
    }
  ],
  "ip": "135.23.125.123",
  "installationFailure": null,
  "installationProgress": 100
}
```

If the instance has additional open ports, these will be listed under `primePortMapping`.

> [!NOTE]
>
> In case of an error, `installationFailure` will be set. Our system will attempt automatic retries, but issues related to the image or configuration may require further action.

## Deleting a Pod

To delete an instance you no longer need, use the [Delete Pod](./pods/delete-pod) endpoint.

```bash theme={null}
curl --request DELETE \
  --url https://api.primeintellect.ai/api/v1/pods/my_pod_id \
  --header 'Authorization: Bearer your_api_key'
```

Once a pod is terminated, it is moved to history and becomes inaccessible via the standard [Get Pod](./pods/get-pod) or [Get Pods](./pods/get-pods) endpoints. To view historical data, use the [Get Pods History](./pods/get-pods-history) endpoint.


# Create Pod
Source: https://docs.primeintellect.ai/api-reference/pods/create-pod

https://api.primeintellect.ai/openapi.json post /api/v1/pods/



# Delete Pod
Source: https://docs.primeintellect.ai/api-reference/pods/delete-pod

https://api.primeintellect.ai/openapi.json delete /api/v1/pods/{pod_id}



# Get Pod
Source: https://docs.primeintellect.ai/api-reference/pods/get-pod

https://api.primeintellect.ai/openapi.json get /api/v1/pods/{pod_id}



# Get Pod Logs Api
Source: https://docs.primeintellect.ai/api-reference/pods/get-pod-logs-api

https://api.primeintellect.ai/openapi.json get /api/v1/pods/{pod_id}/log
Get logs for a pod.

Args:
    tail: Number of most recent logs to return (default: 100)



# Get Pods
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods

https://api.primeintellect.ai/openapi.json get /api/v1/pods/



# Get Pods History
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods-history

https://api.primeintellect.ai/openapi.json get /api/v1/pods/history



# Get Pods Status
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods-status

https://api.primeintellect.ai/openapi.json get /api/v1/pods/status



# Provision Instance
Source: https://docs.primeintellect.ai/api-reference/provision-gpu

How to provision an instance using availability data

> [!NOTE]
>
> Before you start, ensure that you have [API Key](./api-keys) with `Instances -> Read and write` permission

## Retrieving offers from the availability API

> [!NOTE]
>
> For an in-depth guide to using the availability endpoint, refer to [This Guide](./check-gpu-availability)

Our goal is to provision H100 GPU using availability data. First, we need to call the availability endpoint to get the current offers:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/gpus?gpu_type=H100_80GB&regions=united_states&regions=canada&gpu_count=1' \
  --header 'Authorization: Bearer your_api_key'
```

We will use the Hyperstack provider and the following offer:

```json theme={null}
{
  "cloudId": "n3-H100x1",
  "gpuType": "H100_80GB",
  "socket": "PCIe",
  "provider": "hyperstack",
  "dataCenter": "CANADA-1",
  "country": "CA",
  "gpuCount": 1,
  "gpuMemory": 80,
  "disk": {
    "minCount": null,
    "defaultCount": 100,
    "maxCount": null,
    "pricePerUnit": null,
    "step": null,
    "defaultIncludedInPrice": null,
    "additionalInfo": null
  },
  "vcpu": {
    "minCount": null,
    "defaultCount": 180,
    "maxCount": null,
    "pricePerUnit": null,
    "step": null,
    "defaultIncludedInPrice": null,
    "additionalInfo": null
  },
  "memory": {
    "minCount": null,
    "defaultCount": 180,
    "maxCount": null,
    "pricePerUnit": null,
    "step": null,
    "defaultIncludedInPrice": null,
    "additionalInfo": null
  },
  "internetSpeed": null,
  "interconnect": null,
  "interconnectType": null,
  "provisioningTime": null,
  "stockStatus": "Available",
  "security": "secure_cloud",
  "prices": {
    "onDemand": 1.9,
    "isVariable": null,
    "currency": "USD"
  },
  "images": [
    "ubuntu_22_cuda_12",
    "cuda_12_1_pytorch_2_2",
    "cuda_11_8_pytorch_2_1",
    "stable_diffusion",
    "flux",
    "axolotl",
    "bittensor",
    "vllm_llama_8b",
    "vllm_llama_70b",
    "vllm_llama_405b"
  ],
  "isSpot": null,
  "prepaidTime": null
}
```

This GPU configuration has fixed resources, so we don't have to worry about those and just use the default ones.

## Creating the Instance Request Body

First, lets go through the [**Create Pod**](./pods/create-pod) endpoint and explain how it works. The request requires a `body` with `pod`, `provider` and optional `team` definitions.

### `pod`

The `pod` object defines the instance’s characteristics:

```json theme={null}
"pod": {
    "name": "My first pod",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud"
  }
```

We can choose any `name`, but the rest of the parameters is copied from the availability offer. Since this offer includes a `dataCenterId` and `country`, we're going to pass those values during provisioning, as it indicates that the provider has GPUs with the same `cloudId` available in different locations. We also copy the rest of the GPU definition data:

* `gpuType -> gpuType`
* `socket -> socket`
* `gpuCount -> gpuCount`
* `security -> security`

Last thing to do is to select an `image`. Available values are stored within the `images` property of the availability offer. We're going to select the default `ubuntu_22_cuda_12` image.

### `provider`

```json theme={null}
"provider": {
    "type": "hyperstack"
  },
```

The `provider` object is straightforward, we only need to specify the `type`, which in our case is `hyperstack`.

### `team`

```json theme={null}
"team": {
    "teamId": "my_team_id"
  }
```

If you want to assign the pod to a specific `team`, include the team object.

> [!TIP]
>
> You can find the `teamId` on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile)

### `sharedWithTeam`

```json theme={null}
"sharedWithTeam": true
```

Set to `true` to share the instance with **all members** of the team. When enabled, every team member's SSH keys are added to the instance, giving them direct access. Requires `team.teamId` to be set.

### `teamMemberIds`

```json theme={null}
"teamMemberIds": ["clvb1jtmk0001pxfv1abc2def", "clvb2kumr0002qygw3bcd4efg"]
```

Alternatively, provide a list of specific user IDs to share the instance with. Only those members' SSH keys will be added. Use the [List Team Members](./teams) endpoint to get user IDs.

> [!NOTE]
>
> You can use either `sharedWithTeam` for all members or `teamMemberIds` for specific members. If both are provided, `sharedWithTeam` takes precedence.

## Sending the Create Request

With all parts configured, the final request looks like this:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "My first pod",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud"
  },
  "provider": {
    "type": "hyperstack"
  }
}'
```

On successful completion, you should receive a `200 OK` response with the pod details in the [Response Body](./pods/create-pod).

## Modifying instance resources

If the availability offer allows resource customization, you can adjust the default resources during provisioning. Below is an example of a modified resource configuration:

```json theme={null}
  "disk": {
    "minCount": 50,
    "defaultCount": 100,
    "maxCount": 1000,
    "pricePerUnit": 0.0003,
    "step": 10,
    "defaultIncludedInPrice": false,
    "additionalInfo": null
  },
  "vcpu": {
    "minCount": 4,
    "defaultCount": 16,
    "maxCount": 32,
    "pricePerUnit": 0.004,
    "step": 2,
    "defaultIncludedInPrice": true,
    "additionalInfo": null
  },
```

This configuration enables changes to `disk` and `vcpu` specifications.

> [!NOTE]
>
> Be aware that default `vcpu` is included in price (`defaultIncludedInPrice` set to `true`) the default disk is not, which results in an **additional cost of \$0.03** when using the default disk size.

### Increase disk size

To increase the disk size, set a new value in the `pod` property when sending the create request. For this example, since the provider supports increments of **10** with a minimum of **50**, we'll set the disk size to **200**. This adjustment will affect the total hourly cost as follows:

```
GPU cost: $2.69
vcpu cost: $0.00 (we're not paying for vcpu because `defaultIncludedInPrice == true`)
memory cost: $0.00
disk cost: $0.06 (200 units * $0.0003)

Total cost: $2.75
```

So the final request will look like:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "My first pod",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud",
    "diskSize": 200
  },
  "provider": {
    "type": "hyperstack"
  }
}'
```

### Modifying vcpu

This case is a little more complicated. Because `defaultIncludedInPrice` allows us to use default of **16 vcpus for free**, there are 2 options in which we're going to pay additional amount for vcpus

#### Increasing vcpu

Raising `vcpu` to **20** will increase the cost beyond the base instance price:

```
GPU cost: $2.69
vcpu cost: $0.08 (20 units * $0.004)
memory cost: $0.00
disk cost: $0.03 (100 units[default] * $0.0003)

Total cost: $2.80
```

#### Decreasing vcpu

Reducing `vcpu` to **10** can also increase costs compared to the default configuration. This is because some servers use predefined containers, and altering configurations may incur additional fees, making it more economical to use the default setup:

```
GPU cost: $2.69
vcpu cost: $0.04 (10 units * $0.004)
memory cost: $0.00
disk cost: $0.03 (100 units[default] * $0.0003)

Total cost: $2.76
```

### Example request with adjusted disk and vcpu

With both `disk` and `vcpu` increased our request will look like:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "My first pod",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud",
    "diskSize": 200,
    "vcpus": 20
  },
  "provider": {
    "type": "hyperstack"
  }
}'
```

and the total cost breakdown is as follows:

```
GPU cost: $2.69
vcpu cost: $0.08 (20 units * $0.004)
memory cost: $0.00
disk cost: $0.06 (200 units[default] * $0.0003)

Total cost: $2.83
```

## Dynamic pricing

Certain offers feature dynamic pricing, meaning that rates may fluctuate throughout the instance’s lifetime due to factors such as:

* Price being pegged to a foreign currency and fluctuating with exchange rates
* Payment in alternative currencies or tokens
* Market-based pricing

If an offer supports dynamic pricing, `prices -> isVariable` will be set to `true`. In this case, it’s recommended to specify a `maxPrice` when provisioning the instance to set a cap:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "My first pod",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud",
    "maxPrice": 2.70,
  },
  "provider": {
    "type": "hyperstack"
  }
}'
```

This configuration limits provisioning to instances at or below the specified price. However, **prices may still vary during the instance’s lifetime**.

## Attaching Existing Disks

You can attach existing disks with persisted data when provisioning a new GPU instance. This is particularly useful when you want to reuse datasets, models, or checkpoints stored on network-attached disks without re-uploading them.

### Finding Compatible GPUs

Before provisioning, use the [availability endpoint](./check-gpu-availability) with the `disks` filter to find GPU instances that can attach to your existing disks:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/availability/gpus?disks=clhxy6aw80000j8080gdf8kqv&gpu_type=H100_80GB' \
  --header 'Authorization: Bearer your_api_key'
```

This returns only GPU offers available in the same location as your disk.

### Provisioning with Disks

To attach disks when creating an instance, add the `disks` array to your request body:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "Training instance with data",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "security": "secure_cloud"
  },
  "provider": {
    "type": "hyperstack"
  },
  "disks": ["clhxy6aw80000j8080gdf8kqv"]
}'
```

### Response

On successful provisioning, you'll receive a response with the instance details and attached disk information:

```json theme={null}
{
  "id": "11de9260fb52451085f3936c20b0ffdb",
  "userId": "clvb0itli0000oxevorvgrpfn",
  "teamId": null,
  "walletId": "clvb0iv0c0004oxev5mnt1xiv",
  "name": "Training instance with data",
  "providerType": "hyperstack",
  "status": "PROVISIONING",
  "installationStatus": "PENDING",
  "installationFailure": null,
  "installationProgress": null,
  "createdAt": "2025-11-07T23:36:36.391000",
  "updatedAt": "2025-11-07T23:36:36.418000",
  "terminatedAt": null,
  "gpuName": "H100_80GB",
  "gpuCount": 1,
  "socket": "PCIe",
  "priceHr": 1.9,
  "stoppedPriceHr": null,
  "provisioningPriceHr": 0.0,
  "environmentType": "ubuntu_22_cuda_12",
  "customTemplateId": null,
  "clusterId": null,
  "primePortMapping": [
    {
      "internal": "22",
      "external": "22",
      "protocol": "TCP",
      "usedBy": "SSH",
      "description": "SSH access"
    },
    {
      "internal": "*",
      "external": "*",
      "protocol": "TCP",
      "usedBy": null,
      "description": "All other ports"
    }
  ],
  "sshConnection": null,
  "ip": null,
  "resources": {
    "memory": 180,
    "disk": 1250,
    "sharedDisk": null,
    "vcpus": 28
  },
  "attachedResources": [
    {
      "resourceType": "DISK",
      "id": "clhxy6aw80000j8080gdf8kqv",
      "status": "UNATTACHED",
      "isDetachable": true,
      "mountPath": "/data",
      "resourcePath": "/dev/vdc",
      "size": 500,
      "isShared": false
    },
    {
      "resourceType": "DISK",
      "id": "ephemeral",
      "status": "ATTACHED",
      "isDetachable": false,
      "mountPath": "/ephemeral",
      "resourcePath": "/dev/vdb",
      "size": 750,
      "isShared": false
    }
  ],
  "isSpot": null,
  "autoRestart": null
}
```

The response includes:

* **`attachedResources`**: Array showing both your persistent disk (`clhxy6aw80000j8080gdf8kqv`) and the ephemeral instance disk
* **`resources.disk`**: Total disk space including both persistent and ephemeral storage (500GB + 750GB = 1250GB)
* **Disk status**: `UNATTACHED` initially, will become `ATTACHED` once the instance is fully provisioned, and then `MOUNTED` when disk it mounted to the working directory.

> [!NOTE]
>
> The persistent disk will be automatically attached and mounted at `/data` once the instance completes provisioning. The ephemeral disk is immediately available at `/ephemeral`.

### Multiple Disks

You can attach multiple disks to a single instance:

```bash theme={null}
curl --request POST \
  --url https://api.primeintellect.ai/api/v1/pods/ \
  --header 'Authorization: Bearer your_api_key' \
  --header 'Content-Type: application/json' \
  --data '{
  "pod": {
    "name": "Multi-disk training",
    "cloudId": "n3-H100x1",
    "gpuType": "H100_80GB",
    "socket": "PCIe",
    "gpuCount": 1,
    "image": "ubuntu_22_cuda_12",
    "dataCenterId": "CANADA-1",
    "country": "CA",
    "security": "secure_cloud"
  },
  "provider": {
    "type": "hyperstack"
  },
  "disks": [
    "clhxy6aw80000j8080gdf8kqv",
    "clhxy9bz50001j8080hdf9lrw"
  ]
}'
```

### Use Cases

* **Training with Large Datasets**: Attach disks containing preprocessed training data
* **Model Checkpointing**: Resume training from checkpoints stored on persistent disks
* **Shared Data Access**: Use multi-node compatible disks across multiple instances
* **Spot Instances**: Keeping your data when running spot instances

> [!WARNING]
>
> * Only disks you own can be attached to your instances
> * Disks must be in an `ACTIVE` state
> * Some providers may have limitations on the number of disks that can be attached


# Create Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/create-secret

https://api.primeintellect.ai/openapi.json post /api/v1/secrets/
Create a new global secret.

The secret will be encrypted and stored securely. If teamId is provided,
the secret will be associated with that team (requires team membership).



# Delete Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/delete-secret

https://api.primeintellect.ai/openapi.json delete /api/v1/secrets/{secret_id}
Delete a secret.

This operation cannot be undone. Secrets that are linked to environments
will be automatically unlinked.



# Get Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/get-secret

https://api.primeintellect.ai/openapi.json get /api/v1/secrets/{secret_id}
Get a specific secret by ID.

Note: The secret value is not returned, only metadata.



# List Secrets
Source: https://docs.primeintellect.ai/api-reference/secrets/list-secrets

https://api.primeintellect.ai/openapi.json get /api/v1/secrets/
List global secrets for the authenticated user or a team.

If teamId is provided, returns secrets for that team (requires team membership).
Otherwise, returns the user's personal secrets.



# Update Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/update-secret

https://api.primeintellect.ai/openapi.json patch /api/v1/secrets/{secret_id}
Update an existing secret.

You can update the name, value, and/or description.



# Add Member
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/add-member

https://api.primeintellect.ai/openapi.json post /api/v1/slurm-clusters/{team_id}/{job_id}/members



# Deploy Cluster
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/deploy-cluster

https://api.primeintellect.ai/openapi.json post /api/v1/slurm-clusters/{team_id}/deploy
Deploy a Slurm cluster onto one of the team's dedicated clusters.

Team-admin gated like resize and add/remove member. The cluster must
belong to `team_id` and have self-service enabled (checked by the
service, which also pins the new cluster to this team). Node planning
and claiming run before this returns, so "not enough nodes" or a port
conflict is an immediate error; the Helm install continues in the
background and the returned cluster starts out PENDING.



# Get Connection Info
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/get-connection-info

https://api.primeintellect.ai/openapi.json get /api/v1/slurm-clusters/{cluster_id}/connection
SSH coordinates for one RUNNING Slurm cluster deployment.

Team-member gated like the roster reads; ownership follows the same
visibility boundary (`assert_job_visible_to_team` normalizes every
cross-team/wrong-type/missing probe to a generic 404, so probing
other teams' job ids distinguishes nothing). The payload carries
only the connect flags and coordinates — no release names or
namespaces.



# List Clusters
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/list-clusters

https://api.primeintellect.ai/openapi.json get /api/v1/slurm-clusters/{team_id}



# List Members
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/list-members

https://api.primeintellect.ai/openapi.json get /api/v1/slurm-clusters/{team_id}/{job_id}/members



# Remove Member
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/remove-member

https://api.primeintellect.ai/openapi.json delete /api/v1/slurm-clusters/{team_id}/{job_id}/members/{username}



# Resize Cluster
Source: https://docs.primeintellect.ai/api-reference/slurm-clusters/resize-cluster

https://api.primeintellect.ai/openapi.json post /api/v1/slurm-clusters/{team_id}/{job_id}/resize
Resize a Slurm cluster by changing its worker node count.

Requires `factory:write` (see module docstring); `slurm_clusters`
scopes alone are rejected. Team-admin gated like add/remove member.
The cluster must have
self-service enabled (checked by the resize service). Grows by
claiming eligible idle nodes (same filters as deploy); shrinks by
releasing only idle nodes, never touching running jobs.

Returns the verified membership after the mutation settles — the
actual final size may differ from the request if node release was
best-effort and fell short.



# Delete Ssh Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/delete-ssh-key

https://api.primeintellect.ai/openapi.json delete /api/v1/ssh_keys/{key_id}



# Get Ssh Keys
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/get-ssh-keys

https://api.primeintellect.ai/openapi.json get /api/v1/ssh_keys/



# Set Primary Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/set-primary-key

https://api.primeintellect.ai/openapi.json patch /api/v1/ssh_keys/{key_id}



# Upload Ssh Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/upload-ssh-key

https://api.primeintellect.ai/openapi.json post /api/v1/ssh_keys/



# Teams
Source: https://docs.primeintellect.ai/api-reference/teams

How to list teams and team members using the API

> [!NOTE]
>
> Before you start, ensure that you have an [API Key](./api-keys) with `Teams -> Read` permission

## Listing Your Teams

Retrieve the teams you belong to using the `GET /user/teams` endpoint.

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/user/teams' \
  --header 'Authorization: Bearer your_api_key'
```

This is a paginated request with a default `limit` of **100** results. Use query parameters to adjust:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/user/teams?offset=0&limit=50' \
  --header 'Authorization: Bearer your_api_key'
```

### Response

```json theme={null}
{
  "total_count": 2,
  "offset": 0,
  "limit": 50,
  "data": [
    {
      "teamId": "clvb0itli0000oxevorvgrpfn",
      "name": "My Team",
      "slug": "my-team",
      "role": "ADMIN",
      "createdAt": "2025-01-15T10:30:00.000000"
    },
    {
      "teamId": "clvb1jtmk0001pxfv1abc2def",
      "name": "Research Lab",
      "slug": "research-lab",
      "role": "MEMBER",
      "createdAt": "2025-02-01T14:20:00.000000"
    }
  ]
}
```

### Response Fields

| Field | Type | Description |
| - | - | - |
| `teamId` | string | Unique team identifier |
| `name` | string | Team display name |
| `slug` | string | Team URL slug |
| `role` | string | Your role in the team (`ADMIN` or `MEMBER`) |
| `createdAt` | string | ISO 8601 timestamp of when you joined the team |

## Listing Team Members

Retrieve the members of a team using the `GET /teams/{team_id}/members` endpoint. You must be a member of the team to access this endpoint.

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/teams/{team_id}/members' \
  --header 'Authorization: Bearer your_api_key'
```

This is a paginated request with a default `limit` of **100** results. Use query parameters to adjust:

```bash theme={null}
curl --request GET \
  --url 'https://api.primeintellect.ai/api/v1/teams/{team_id}/members?offset=0&limit=50' \
  --header 'Authorization: Bearer your_api_key'
```

### Response

```json theme={null}
{
  "total_count": 2,
  "offset": 0,
  "limit": 50,
  "data": [
    {
      "userId": "clvb0itli0000oxevorvgrpfn",
      "userName": "Alice",
      "userEmail": "alice@example.com",
      "role": "ADMIN",
      "joinedAt": "2025-01-15T10:30:00.000000"
    },
    {
      "userId": "clvb1jtmk0001pxfv1abc2def",
      "userName": "Bob",
      "userEmail": "bob@example.com",
      "role": "MEMBER",
      "joinedAt": "2025-02-01T14:20:00.000000"
    }
  ]
}
```

### Response Fields

| Field | Type | Description |
| - | - | - |
| `userId` | string | Unique user identifier |
| `userName` | string | User's display name |
| `userEmail` | string | User's email address |
| `role` | string | Team role (`ADMIN` or `MEMBER`) |
| `joinedAt` | string | ISO 8601 timestamp of when the user joined the team |

> [!TIP]
>
> You can find your `teamId` on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile) or by calling `GET /user/teams`.


# List Team Members
Source: https://docs.primeintellect.ai/api-reference/teams/list-team-members

https://api.primeintellect.ai/openapi.json get /api/v1/teams/{team_id}/members



# Get an episode
Source: https://docs.primeintellect.ai/api-reference/traces/episodes/get-an-episode

https://prime-traces.pintel.dev/openapi.json get /api/v1/episodes/{episode_id}
An episode's own fields plus totals computed from its traces: tokens, agent time, participating agents and whether any trace failed.

`has_error` reports an error on the episode itself, and `any_trace_error` reports one on any member trace, so an episode-level failure stays visible even when every trace succeeded. An episode with no traces is valid and returns empty totals.

With `raw=true`, returns the stored episode document, with `traces` listing the member trace IDs.



# List episode traces
Source: https://docs.primeintellect.ai/api-reference/traces/episodes/list-episode-traces

https://prime-traces.pintel.dev/openapi.json get /api/v1/episodes/{episode_id}/traces
Summaries of the traces in an episode, cursor-paginated.

An episode that does not exist returns `404`, so an empty page always means an existing episode with no matching traces.



# List episodes
Source: https://docs.primeintellect.ai/api-reference/traces/episodes/list-episodes

https://prime-traces.pintel.dev/openapi.json get /api/v1/episodes
Episode summaries for the authenticated owner, newest first. Cursor-paginated like the trace list. `upload_id` is the ID in an upload receipt, so it returns the episodes one upload stored.

`context.<key>=<value>` matches the context map supplied at upload. Episodes store no context of their own, so it is matched on member traces: an episode with no traces never matches.



# Upload traces
Source: https://docs.primeintellect.ai/api-reference/traces/ingestion/upload-traces

https://prime-traces.pintel.dev/openapi.json post /api/v1/traces
Upload traces as a `multipart/form-data` request with two parts:

* `metadata` (`application/json`): `{"schema_version": 1, "context": {...}}`. `context` is an optional flat string map stored on every trace in the upload.
* `traces` (`application/x-ndjson`): one complete trace per line, one complete episode per line with `X-Prime-Line-Format: episode`, or one Prime Agent session per line with `X-Prime-Line-Format: prime_agent` (a JSON array of the session file's lines, header first). The part may be sent with `Content-Encoding: gzip`.

`Idempotency-Key` is `sha256:<64 lowercase hex>` of the uncompressed `traces` bytes, newlines included, and is also the upload ID. A request can be up to 256 MiB and a single line up to 64 MiB, both measured uncompressed.

`201` means every line was stored; there is no partial success. `400` means the content was rejected and nothing was stored. `409` means the same bytes were already uploaded with a different line format. `429` and `503` are safe to retry with the same bytes after `Retry-After`.



# Get a run
Source: https://docs.primeintellect.ai/api-reference/traces/runs/get-a-run

https://prime-traces.pintel.dev/openapi.json get /api/v1/runs/{run_id}
One run's totals over its traces. A run with no traces returns `404`.



# List run steps
Source: https://docs.primeintellect.ai/api-reference/traces/runs/list-run-steps

https://prime-traces.pintel.dev/openapi.json get /api/v1/runs/{run_id}/steps
The distinct `run_step` values recorded in a run, with each step's trace count, episode count and average reward.



# List runs
Source: https://docs.primeintellect.ai/api-reference/traces/runs/list-runs

https://prime-traces.pintel.dev/openapi.json get /api/v1/runs
Runs for the authenticated owner, with totals over their traces: trace and episode counts, average reward, step range, and the environments and models used. Cursor-paginated.



# Search a run
Source: https://docs.primeintellect.ai/api-reference/traces/runs/search-a-run

https://prime-traces.pintel.dev/openapi.json get /api/v1/runs/{run_id}/search
Case-sensitive literal search over the content of a run's traces. The query needs at least three characters.

Returns one match per node, with Unicode offsets and an excerpt. Choose what to search with `field` (`content`, `reasoning_content` or `tool_calls`) and narrow with the other filters. Continue with `next_cursor` and the same filters until it is null.

Only indexed traces are searched. The first page's `coverage` reports traces that were not yet searchable. A search that reads more than the server allows returns `400` with `search_limit_exceeded`; retrying unchanged will not help, so make the query more specific or add filters.



# Delete a run's traces
Source: https://docs.primeintellect.ai/api-reference/traces/traces/delete-a-runs-traces

https://prime-traces.pintel.dev/openapi.json delete /api/v1/traces
Delete every trace in a run. `run_id` is required.

Only traces are deleted; episodes in the run are kept. The delete covers the traces stored when it looks up the run: an upload that finishes while the request is in progress may or may not be deleted, and one that finishes after it returns is not affected.

Returns `202` once the traces are no longer readable. A run with no traces returns `404`, including a repeat of a delete that already succeeded.



# Delete a trace
Source: https://docs.primeintellect.ai/api-reference/traces/traces/delete-a-trace

https://prime-traces.pintel.dev/openapi.json delete /api/v1/traces/{trace_id}
Delete every stored copy of a trace.

Returns `202` once the trace is no longer readable; storage is reclaimed in the background. `created_at` is an optional hint used only to check that the trace exists: a hint that matches no stored copy returns `404`. Deleting a trace that is already gone also returns `404`.



# Get a trace
Source: https://docs.primeintellect.ai/api-reference/traces/traces/get-a-trace

https://prime-traces.pintel.dev/openapi.json get /api/v1/traces/{trace_id}
A trace's summary, or with `raw=true` the stored trace document as uploaded, with recognized credentials replaced by `<REDACTED BY PRIME TRACES>`.

The raw document can be large, up to the 64 MiB line limit, so fetch the summary first when you only need its fields. `size_bytes` on the summary tells you how large the raw document is.



# List trace model calls
Source: https://docs.primeintellect.ai/api-reference/traces/traces/list-trace-model-calls

https://prime-traces.pintel.dev/openapi.json get /api/v1/traces/{trace_id}/calls
A trace's model calls in order, for timelines: spans, models and finish reasons.

Indexing, `409 trace_not_indexed`, cursor rules and `partial_index` work as for trace nodes. A call's start and end times are omitted if either is missing or the end precedes the start. `node_idx` is null when the call recorded no valid node reference.



# List trace nodes
Source: https://docs.primeintellect.ai/api-reference/traces/traces/list-trace-nodes

https://prime-traces.pintel.dev/openapi.json get /api/v1/traces/{trace_id}/nodes
A trace's message nodes in step order, without downloading the whole trace. Use this to render a trace page by page.

Nodes come from an index built shortly after upload. Until it is ready the request returns `409` with `trace_not_indexed`; retry, or read the trace with `raw=true`. For very large traces the index stops at a per-trace cap, and `partial_index: true` on the page means the rest is only available with `raw=true`.

If the trace is re-uploaded while you are paging, the cursor is rejected rather than mixing the two versions.



# List traces
Source: https://docs.primeintellect.ai/api-reference/traces/traces/list-traces

https://prime-traces.pintel.dev/openapi.json get /api/v1/traces
Trace summaries for the authenticated owner, newest first by default.

Narrow the list with the query parameters below; an unknown parameter is rejected with `invalid_filter` rather than ignored. `created_after` and `created_before` are the cheapest filters. `sort` is one of `created_at`, `reward`, `duration_ms` or `run_step`.

`upload_id` is the ID in an upload receipt, so it returns the traces one upload stored.

If a trace was uploaded more than once, the newest copy is returned. Copies with different `created_at` values can still appear on two different pages.

Pages are cursor-paginated: pass `next_cursor` back as `cursor` with the same filters.



# Create Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/create-dedicated-run

https://api.primeintellect.ai/openapi.json post /api/v1/training/runs
Dispatch a dedicated full-FT prime-rl run on a registered PrimeCluster.

Access is gated by ClusterAllocation - the picker in
`training_service.create_dedicated_run` rejects callers whose
team/user has no matching allocation on a live PrimeCluster. When
`team_id` is set the caller must additionally be a member of that
team (mirrors the LoRA/shared dispatch path).

Admission checks (runtime pins, hub env ids, HF repo names, per-run
GPU cap) run sync so bad configs return 400 immediately. Local clients
supply normalized configs; legacy clients use the validator schema.
Cluster-side probes (GPU capacity, model cache) and the helm
install run async in a Cloud Task; callers poll
`GET /api/v1/rft/runs/{run_id}` for status transitions PENDING ->
CREATING -> RUNNING (or FAILED with errorMessage on a cluster-side
rejection).



# Create Volume
Source: https://docs.primeintellect.ai/api-reference/training/create-volume

https://api.primeintellect.ai/openapi.json post /api/v1/training/volumes



# Create Volume Session
Source: https://docs.primeintellect.ai/api-reference/training/create-volume-session

https://api.primeintellect.ai/openapi.json post /api/v1/training/volumes/{name}/sessions



# Delete Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/delete-dedicated-run

https://api.primeintellect.ai/openapi.json delete /api/v1/training/runs/{run_id}
Delete a dedicated run: helm uninstall + namespace delete + DB cleanup.

Idempotent. Safe to call repeatedly. Cleans up the linked Job row too
(which uninstalls the helm release on the customer cluster). Soft-
deletes the RFTRun row (matches the existing run-delete UX - Billing
refs are preserved). Access is per-row: `verify_run_ownership`
accepts personal runs the caller owns and team runs the caller is
a member of.



# Delete Volume
Source: https://docs.primeintellect.ai/api-reference/training/delete-volume

https://api.primeintellect.ai/openapi.json delete /api/v1/training/volumes/{name}



# Expand Volume
Source: https://docs.primeintellect.ai/api-reference/training/expand-volume

https://api.primeintellect.ai/openapi.json patch /api/v1/training/volumes/{name}
Raise a volume's size cap; volumes can't shrink.



# Get Available Fft Models
Source: https://docs.primeintellect.ai/api-reference/training/get-available-fft-models

https://api.primeintellect.ai/openapi.json get /api/v1/training/available-fft-models
Models pre-cached on the caller's eligible PrimeClusters for FFT dispatch.

Same principal collapse as the dispatch picker (team wins when
present, otherwise personal) and the same cluster filter (uncordoned
+ heartbeat fresh + non-zero ClusterAllocation) so what surfaces
here is what dispatch would actually land on. Only PRESENT-cache
clusters contribute — ABSENT / ERROR would 400 the dispatch and
NULL (unprobed) hasn't reported a manifest yet.

Each model carries the list of clusters it's warm on with `gpu_type`
+ `cache_synced_at` so callers can pin the follow-up dispatch to a
specific GPU family without hitting the "picker landed on a cluster
that has the GPU but not this model" reject path.



# Get Available Gpu Types
Source: https://docs.primeintellect.ai/api-reference/training/get-available-gpu-types

https://api.primeintellect.ai/openapi.json get /api/v1/training/available-gpu-types
Distinct GPU types the caller could dispatch a dedicated FFT run on.

Same principal collapse as the dispatch picker (team wins when
present, otherwise personal) so what the dropdown offers matches
what dispatch would land on. When `team_id` is set the caller must
be a team member, matching the create-run gate.



# Get Volume Session
Source: https://docs.primeintellect.ai/api-reference/training/get-volume-session

https://api.primeintellect.ai/openapi.json get /api/v1/training/volumes/{name}/sessions/{session_id}



# List Volume Sessions
Source: https://docs.primeintellect.ai/api-reference/training/list-volume-sessions

https://api.primeintellect.ai/openapi.json get /api/v1/training/volumes/{name}/sessions
The caller's sessions that still block deleting the volume.



# List Volumes
Source: https://docs.primeintellect.ai/api-reference/training/list-volumes

https://api.primeintellect.ai/openapi.json get /api/v1/training/volumes



# Pause Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/pause-dedicated-run

https://api.primeintellect.ai/openapi.json post /api/v1/training/runs/{run_id}/pause
Pause a running dedicated run for good at its next checkpoint.

The run turns PAUSING, then PAUSED once its checkpoint is back on its
volume in R2 and its GPUs are released. Resume it with a new run from
that volume: `resumeFrom` on POST /runs.



# Prepare Local Validation
Source: https://docs.primeintellect.ai/api-reference/training/prepare-local-validation

https://api.primeintellect.ai/openapi.json post /api/v1/training/runs/prepare
Resolve the default/selected runtime for local CLI config validation.

No run, run token, GPU allocation, volume or deployment is created.
The result is metadata, not authorization to submit a compiled artifact.



# Resume Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/resume-dedicated-run

https://api.primeintellect.ai/openapi.json post /api/v1/training/runs/{run_id}/resume
Resume a PAUSED dedicated run as a new run from its checkpoint.

The new run reuses the paused run's config, team, name, mode, image and
GPU type, attaches its volume and starts from its last checkpoint.
`gpuType` and `imageTag` override the stored ones; secrets are sent
again because they are dropped once a run deploys.



# Stop Volume Session
Source: https://docs.primeintellect.ai/api-reference/training/stop-volume-session

https://api.primeintellect.ai/openapi.json delete /api/v1/training/volumes/{name}/sessions/{session_id}



# Volume Transfer
Source: https://docs.primeintellect.ai/api-reference/training/volume-transfer

https://api.primeintellect.ai/openapi.json post /api/v1/training/volumes/{name}/transfer
Where `prime volumes get/put` should transfer (ENG-6585). A put
needs rft:write, like every other volume write.



# Volume Transfer Release
Source: https://docs.primeintellect.ai/api-reference/training/volume-transfer-release

https://api.primeintellect.ai/openapi.json post /api/v1/training/volumes/{name}/transfer/release
A finished direct put releases its lease (`leaseId` from transfer),
so runs and deletes stop waiting on it. Idempotent; needs what a put
needs (rft:write).



# Bulk Delete Tunnels Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/bulk-delete-tunnels-endpoint

https://api.primeintellect.ai/openapi.json delete /api/v1/tunnel
Bulk soft-delete multiple tunnels by IDs or labels.

## Rate Limit

100 requests per 60 seconds per user.


# Create Tunnel Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/create-tunnel-endpoint

https://api.primeintellect.ai/openapi.json post /api/v1/tunnel
Create a new tunnel for exposing a local service.

## Rate Limit

600 requests per 60 seconds per user.


# Delete Tunnel Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/delete-tunnel-endpoint

https://api.primeintellect.ai/openapi.json delete /api/v1/tunnel/{tunnel_id}
Delete a tunnel and disconnect any active clients.

## Rate Limit

100 requests per 60 seconds per user.


# Get Tunnel Status Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/get-tunnel-status-endpoint

https://api.primeintellect.ai/openapi.json get /api/v1/tunnel/{tunnel_id}
Get status of a specific tunnel.

## Rate Limit

300 requests per 60 seconds per tunnel.


# List Tunnels Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/list-tunnels-endpoint

https://api.primeintellect.ai/openapi.json get /api/v1/tunnel
List all active tunnels for the current user.

## Rate Limit

300 requests per 60 seconds per user.


# Get Whoami
Source: https://docs.primeintellect.ai/api-reference/user/get-whoami

https://api.primeintellect.ai/openapi.json get /api/v1/user/whoami



# List My Teams
Source: https://docs.primeintellect.ai/api-reference/user/list-my-teams

https://api.primeintellect.ai/openapi.json get /api/v1/user/teams



# Set Username Slug
Source: https://docs.primeintellect.ai/api-reference/user/set-username-slug

https://api.primeintellect.ai/openapi.json patch /api/v1/user/slug



# Get Availability Information
Source: https://docs.primeintellect.ai/cli-reference/check-gpu-availability

How to check GPU and disk availability and pricing

> [!NOTE]
>
> Before you start, ensure you have set your API keys and SSH key path as outlined in the [CLI introduction](/cli-reference/introduction).

## Retrieving GPU Availability Data

To see all available GPU types and their current pricing across all regions, simply run:

```bash theme={null}
prime availability list
```

This command will display a table showing all available GPU configurations, including details like GPU type, count, location, price per hour, and stock status.

## Retrieving Disk Availability Data

To see all available disk storage options across providers and datacenters:

```bash theme={null}
prime availability disks
```

This displays available persistent storage configurations with pricing, maximum capacity, and multinode support information.

## Retrieving Filtered GPU Availability Data

You can then use filters like `--gpu-type`, `--regions`, or `--gpu-count` to narrow down the results. Here are all available filters:

### GPU Availability Filters

* `--gpu-type`: Filter by specific GPU model
  * Example: `--gpu-type H100_80GB`
  * Optional string value

* `--gpu-count`: Filter by number of GPUs
  * Example: `--gpu-count 2`
  * Optional integer value

* `--regions`: Filter by geographic regions
  * Example: `--regions united_states,canada` or `--regions united_states --regions canada`
  * Optional, can specify multiple regions

* `--socket`: Filter by socket type
  * Example: `--socket PCIe`
  * Accepts values: `PCIe`, `SXM2`, `SXM3`, `SXM4`, `SXM5`
  * Optional string value

* `--disks`: Filter GPUs by disk IDs (shows GPUs in same location as disks)
  * Example: `--disks disk-id-1 --disks disk-id-2`
  * Optional, can specify multiple disk IDs
  * Useful for finding GPUs compatible with your existing disks

* `--group-similar`: Group similar configurations from same provider
  * Example: `--no-group-similar`
  * Default: true
  * Optional boolean flag

### Example with Multiple Filters

```bash theme={null}
prime availability list --gpu-type H100_80GB --regions united_states --socket PCIe --no-group-similar
```

### Finding GPUs Compatible with Your Disks

If you have existing disks and want to find GPUs in the same datacenter/region:

```bash theme={null}
# First, list your disks to get their IDs
prime disks list

# Then filter GPUs by disk IDs
prime availability list --disks disk-id-1 --disks disk-id-2
```

This ensures you see only GPU configurations that can access your existing disks.

## Retrieving Filtered Disk Availability Data

### Disk Availability Filters

* `--regions`: Filter by geographic regions
  * Example: `--regions united_states,eu_north`
  * Optional, can specify multiple regions

* `--data-center-id`: Filter by specific datacenter
  * Example: `--data-center-id US-1`
  * Optional string value

* `--cloud-id`: Filter by cloud configuration ID
  * Example: `--cloud-id "Provider Storage A"`
  * Optional string value

### Example with Filters

```bash theme={null}
prime availability disks --regions united_states --data-center-id US-1
```

## Understanding the Terminal Output

### GPU Availability Output

Here's an example of what the command output looks like:

| ID | GPU Type | GPUs | Socket | Provider | Location | Stock | Price/Hr | Memory (GB) | Security | vCPUs | RAM (GB) |
| - | - | - | - | - | - | - | - | - | - | - | - |
| 346663 | H100\_80GB | 2 | PCIe | runpod | CA | Low | \$5.40 | 160 | secure\_cloud | 32 | 502 |
| 551ffd | H100\_80GB | 2 | PCIe | runpod | US | Low | \$5.40 | 160 | secure\_cloud | 32 | 502 |

Each row represents a unique GPU configuration available for deployment. The output includes:

* A unique ID for the configuration
* GPU specifications (type, count, socket)
* Provider and location information
* Current stock status
* Pricing and hardware details

### Disk Availability Output

For disk availability, the output includes:

| ID | Provider | Location | Stock | Price/Hr/GB | Max Size (GB) | Is Multinode |
| - | - | - | - | - | - | - |
| 7d2232 | runpod | US (US-WA-1) | Available | \$0.00011111 | 8192 | Yes |
| 472b26 | hyperstack | NO (NORWAY-1) | Available | \$0.00009700 | 100000 | No |

Each row represents available disk storage with:

* A unique ID for the configuration
* Provider offering the storage
* Location (country and datacenter)
* Stock availability status
* Price per GB per hour
* Maximum disk size supported
* Whether the disk supports multinode (can be attached to multiple instances)


# Configuration
Source: https://docs.primeintellect.ai/cli-reference/config-cli

Configure your Prime CLI settings

The Prime CLI provides several commands to manage your configuration settings. All configuration commands are accessed through the `prime config` command group.

## View Configuration

View your current configuration settings:

```bash theme={null}
prime config view
```

This displays a table with your current settings, including:

* API Key (partially masked)
* Team ID
* Base URL
* SSH Key Path

## API Key Management

Set your Prime Intellect API key:

```bash theme={null}
prime config set-api-key
```

## Team Settings

> **Set Team ID**
>
> Configure your team ID for team-based access:
>
> ```bash theme={null}
> prime config set-team-id
> ```

> **Remove Team ID**
>
> Switch back to your personal account:
>
> ```bash theme={null}
> prime config remove-team-id
> ```

## Sharing Settings

Configure whether new instances are automatically shared with all team members:

```bash theme={null}
prime config set-share-resources-with-team true
```

When enabled, `prime pods create` will automatically set `sharedWithTeam` on new instances without needing to pass `--share-with-team` each time. Set to `false` to disable:

```bash theme={null}
prime config set-share-resources-with-team false
```

## Prime Traces uploads

CLI v0.9.4 documents a machine-level opt-out for runs using the Prime Runs SDK:

```bash theme={null}
prime config set-traces-opt-out true
```

Set it to `false` to allow uploads again, subject to workspace settings and higher-precedence client overrides. Opted-out runs use legacy sample storage only while that service remains available. See [upload controls and precedence](/traces/pricing-and-settings#opt-out-for-a-run-or-client) and the [November 6 legacy retirement](/traces/legacy-storage).

## Connection Settings

> **Base URL**
>
> Set the API endpoint URL:
>
> ```bash theme={null}
> prime config set-base-url
> ```

> **SSH Key**
>
> Configure SSH key path:
>
> ```bash theme={null}
> prime config set-ssh-key-path
> ```

## Reset Configuration

Reset all settings to their default values:

```bash theme={null}
prime config reset
```

> [!WARNING]
>
> This will remove all your custom settings, including your API key. You'll need to reconfigure the CLI after resetting.

## Configuration Options

| Command | Description | Default |
| - | - | - |
| `view` | Display current configuration | - |
| `set-api-key` | Set your API key | - |
| `set-team-id` | Set team ID for team access | - |
| `remove-team-id` | Switch to personal account | - |
| `set-base-url` | Set API base URL | `https://api.primeintellect.ai` |
| `set-ssh-key-path` | Set SSH private key path | `~/.ssh/id_rsa` |
| `set-share-resources-with-team` | Auto-share new instances with team | `false` |
| `reset` | Reset to default settings | - |

> [!TIP]
>
> All configuration commands will prompt for values if not provided as command-line arguments.


# Environments Hub
Source: https://docs.primeintellect.ai/cli-reference/environments

Create, install and manage environments from the Environments Hub

> [!NOTE]
>
> For documentation on how to use the Prime CLI with the Environments Hub, refer to the [Environments Hub Tutorial](/tutorials-environments/getting-started).


# Overview
Source: https://docs.primeintellect.ai/cli-reference/introduction

Command line interface for managing Prime Intellect compute, RL environments and code sandboxes.

The Prime Intellect CLI provides a powerful command-line interface for managing all of our Prime Intellect offerings.
Including:

* Managing compute resources on our platform
* Creating, publishing and installing RL environments from our [environment hub](https://app.primeintellect.ai/dashboard/environments)
* Managing code sandboxes for secure code execution

> [!NOTE]
>
> Check out our [open source repository](https://github.com/PrimeIntellect-ai/prime) on GitHub.

## Quick Start

1. Install uv (if not already installed):

```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2. Install the CLI with uv:

```bash theme={null}
uv tool install prime
```

3. Authenticate:

```bash theme={null}
prime login
```

Alternatively you can also manually set your [API key](/api-reference/api-keys) using `prime config set-api-key`

4. Configure SSH key for pod access:

```bash theme={null}
prime config set-ssh-key-path
```

\[!TIP] You can generate your ssh keys on [app.primeintellect.ai](https://app.primeintellect.ai/dashboard/profile) under the /dashboard/profile section. We recommend creating a dedicated directory for your Prime Intellect keys in your home directory, like `/Users/<username>/pi-keys`.
5\. Verify your configuration:

```bash theme={null}
prime config view
```

![View CLI Configuration](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/cli/cli-config-view.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=565c51879381e689fb53a1f2b807c6c4)

### Alternative Installation: Using pip

If you prefer not to use uv:

```bash theme={null}
pip install prime
```

## Key Features

* **GPU Resource Management**: Query and filter available GPU resources
* **Pod Management**: Create, monitor, and terminate compute pods
* **SSH Access**: Direct SSH access to running pods
* **Team Support**: Manage resources across team environments

## Basic Commands

```bash theme={null}
# List available GPUs
prime availability list

# Filter by GPU type
prime availability list --gpu-type H100_80GB

# Create a pod
prime pods create

# List your pods
prime pods list

# SSH into a pod
prime pods ssh <pod-id>
```

For detailed documentation on each command and feature, explore the sections above.

> **[Check GPU availability](/cli-reference/check-gpu-availability)**
>
> View and filter available GPU resources across providers using simple commands.

> **[Provision Instances](/cli-reference/provision-gpu)**
>
> Create, monitor, and manage compute pods
> directly from your terminal.

> **[Config Settings](/cli-reference/config-cli)**
>
> Configure connection settings and team accounts.


# Managing Disks
Source: https://docs.primeintellect.ai/cli-reference/managing-disks

How to create and manage network-attached storage using the CLI

> [!NOTE]
>
> Before you start, ensure that you have [API Key](/api-reference/api-keys) with `Disks -> Read and write` permission

## Overview

Network-attached disks provide persistent storage that can be shared across multiple GPU instances. Disks persist independently from instances, making them ideal for storing datasets, model checkpoints, and other data that needs to survive instance termination.

## Checking Disk Availability

Before creating a disk, check available storage options:

```bash theme={null}
prime availability disks
```

> [!TIP]
>
> Filter results using these options:
>
> * `--regions`: Geographic locations (e.g., united\_states,eu\_north)
> * `--data-center-id`: Specific datacenter (e.g., US-1)
> * `--cloud-id`: Cloud configuration ID

Example with filters:

```bash theme={null}
prime availability disks --regions united_states
```

### Example Output

| ID | Provider | Location | Stock | Price/Hr/GB | Max Size (GB) | Is Multinode |
| - | - | - | - | - | - | - |
| c008ad | runpod | US (US-CA-2) | Available | \$0.00011111 | 8,192 | Yes |
| 4e50ab | runpod | US (US-IL-1) | Available | \$0.00011111 | 8,192 | Yes |
| 4eb3b5 | runpod | US (US-KS-2) | Available | \$0.00011111 | 8,192 | Yes |
| 334eaa | runpod | US (US-MO-1) | Available | \$0.00011111 | 8,192 | Yes |
| d9c8e5 | runpod | US (US-NC-1) | Available | \$0.00011111 | 8,192 | Yes |
| 241b41 | runpod | US (US-TX-3) | Available | \$0.00011111 | 8,192 | Yes |
| 7d2232 | runpod | US (US-WA-1) | Available | \$0.00011111 | 8,192 | Yes |
| a869c3 | hyperstack | US (US-1) | Available | \$0.00009700 | 100,000 | No |
| 8121df | crusoecloud | US (us-southcentral1-a) | Available | \$0.00011546 | 8,192 | No |
| 81cdd2 | crusoecloud | US (us-east1-a) | Available | \$0.00011546 | 8,192 | No |
| 2f139f | dc\_roan | US | Available | \$0.00007000 | 8,192 | Yes |

**Column Descriptions:**

* **ID**: Short identifier for easy disk creation (use with `--id` parameter)
* **Provider**: Storage provider offering the disk (runpod, hyperstack, crusoecloud, dc\_roan)
* **Location**: Country code and datacenter identifier
* **Stock**: Current availability status
* **Price/Hr/GB**: Cost per gigabyte per hour (hourly rate × disk size = total hourly cost)
* **Max Size (GB)**: Maximum disk capacity supported by the provider
* **Is Multinode**: Whether the disk can be attached to multiple instances simultaneously

## Creating a Disk

Create a new persistent disk:

```bash theme={null}
prime disks create --id c008ad --size 500
```

### Available Options

**--id** `string` *required*

Short ID from availability disks list

**--size** `integer` *required*

Disk size in GB (must be within provider's min/max limits)

**--name** `string`

Human-friendly name for the disk (auto-generated if not provided)

**--team-id** `string`

Team identifier for the disk

**--yes** `boolean`

Skip confirmation prompt

### Creation

```bash theme={null}
prime disks create \
  --id 7d2232
  --size 500 \
  --name ml-training-data
```

> [!NOTE]
>
> Disks are billed continuously from creation until termination. Make sure to terminate disks when no longer needed.

## Listing Your Disks

View all your disks:

```bash theme={null}
prime disks list
```

### Output Options

Display as JSON:

```bash theme={null}
prime disks list --output json
```

With pagination:

```bash theme={null}
prime disks list --limit 50 --offset 0
```

The output shows:

* Disk ID, name, and size
* Current status
* Provider and location
* Creation date and age
* Hourly cost
* Attached pods and clusters

## Getting Disk Details

View detailed information about a specific disk:

```bash theme={null}
prime disks get <disk-id>
```

Display as JSON:

```bash theme={null}
prime disks get <disk-id> --output json
```

The detailed view includes:

* Complete disk configuration
* Current status and pricing
* List of attached pods
* List of attached clusters
* Creation and update timestamps
* Location information

## Updating a Disk

Update the disk name:

```bash theme={null}
prime disks update <disk-id> --name new-disk-name
```

> [!NOTE]
>
> Currently, only the disk name can be updated. Disk size cannot be changed after creation.

## Terminating a Disk

Terminate a disk when you no longer need it:

```bash theme={null}
prime disks terminate <disk-id>
```

Skip confirmation prompt:

```bash theme={null}
prime disks terminate <disk-id> --yes
```

> [!WARNING]
>
> **Disk termination is permanent and irreversible**. All data stored on the disk will be lost. Make sure to backup any important data before terminating a disk.

> [!NOTE]
>
> **Detachment Timing**: After terminating an instance with attached disks, you may need to wait a few moments for the disk to detach before you can terminate it or attach it to another instance.

## Disk Status Values

Disks go through different statuses during their lifecycle:

| Status | Description |
| - | - |
| `PROVISIONING` | Disk is being created |
| `PENDING` | Disk status is changing |
| `ACTIVE` | Disk is ready and can be attached to instances or terminated |
| `STOPPED` | Disk is stopped |
| `ERROR` | An error occurred during disk operations |
| `DELETING` | Disk is being terminated |
| `TERMINATED` | Disk has been terminated |
| `UNKNOWN` | Disk status is unknown (cannot access the current status) |

## Using Disks with Instances

### Attach Disks During Instance Creation

```bash theme={null}
prime pods create \
  --id 346663 \
  --disks disk-id-1 \
  --disks disk-id-2
```

Learn more in [Provision Instance](/cli-reference/provision-gpu).

### Best Practices

**Location Planning**

* Create disks in the same datacenter where you plan to run GPU instances
* Use `prime availability disks` to find compatible storage options

**Size Planning**

* Start with the size you need - you cannot resize disks after creation
* Consider future growth when selecting disk size
* Remember that larger disks cost more per hour

**Data Management**

* Use descriptive names to identify disk contents
* Regularly backup important data from disks
* Terminate unused disks to avoid unnecessary costs

**Multinode Disks**

* Use multinode-capable disks for distributed training
* Check the `isMultinode` field in availability output
* Ensure your provider supports simultaneous access if needed

## Common Use Cases

### Shared Training Data

Create large disks with your training datasets that can be reused across multiple training runs:

```bash theme={null}
# Create a 1TB disk for training data
prime disks create --size 1000 --name training-datasets --id 7d2232

# Attach to multiple training instances
prime pods create --id gpu-id --disks disk-id-1
```

### Model Checkpoints

Use disks to persist model checkpoints and training artifacts:

```bash theme={null}
# Create disk for checkpoints
prime disks create --size 200 --name model-checkpoints --id 7d2232

# Use with training instance
prime pods create --id gpu-id --disks checkpoint-disk-id
```

### Multi-Node Distributed Training

Use multinode-compatible disks for distributed training:

```bash theme={null}
# Find multinode-capable disk storage
prime availability disks --regions united_states

# Create multinode disk (look for "Yes" in Is Multinode column)
prime disks create --size 500 --name shared-data --id multinode-disk-id

# Attach to multiple nodes
prime pods create --id gpu-id-1 --disks shared-disk-id
prime pods create --id gpu-id-2 --disks shared-disk-id
```

## Related Documentation

* [Check Availability](/cli-reference/check-gpu-availability) - Find available disk options
* [Provision Instance](/cli-reference/provision-gpu) - Attach disks when creating instances
* [API Disk Management](/api-reference/managing-disks) - API reference for disk operations


# Provision Instance
Source: https://docs.primeintellect.ai/cli-reference/provision-gpu

How to provision an instance using availability data

> [!NOTE]
>
> Before you start, ensure that you have [API Key](/api-reference/api-keys) with `Instances -> Read and write` permission

## Checking GPU Availability

First, check available GPU configurations:

```bash theme={null}
prime availability list
```

> [!TIP]
>
> Filter results using these options:
>
> * `--gpu-type`: Specific GPU model (e.g., H100\_80GB)
> * `--regions`: Geographic locations (e.g., united\_states,canada)
> * `--gpu-count`: Number of GPUs needed
> * `--socket`: Socket type (PCIe, SXM4, etc.)
> * `--provider`: Name of a provider

## Creating an Instance

Create a new instance interactively:

```bash theme={null}
prime pods create
```

### Available Options

**--id** `string`

Short ID from availability list

**--cloud-id** `string`

Cloud ID from provider

**--gpu-type** `string`

GPU model (e.g., H100\_80GB, A100)

**--gpu-count** `integer`

Number of GPUs required

**--name** `string`

Instance name (alphanumeric and dashes only)

**--disk-size** `integer`

Storage size in GB

**--vcpus** `integer`

Number of virtual CPUs

**--memory** `integer`

RAM in GB

**--image** `string`

Operating system image

**--team-id** `string`

Team identifier for the instance

**--env** `string`

Environment Variable in form `KEY=value` - this can be used multiple times

**--disks** `string`

Attach existing disk IDs to the instance. Can be specified multiple times for multiple disks.

* Example: `--disks disk-id-1 --disks disk-id-2`

**--share-with-team** `boolean`

Share the instance with all team members. Requires a team to be set via `--team-id` or `prime config set-team-id`. Team members' SSH keys will be added to the instance.

**--add-members** `boolean`

Interactively select specific team members to share the instance with. Displays a numbered list of team members and prompts for selection. Enter comma-separated numbers or `all` for everyone.

### Interactive Creation Process

If options are not provided, the CLI will guide you through:

1. GPU Selection
2. Configuration Options
3. Resource Allocation
4. Image Selection
5. Team Assignment

> [!NOTE]
>
> The CLI will show default values and valid ranges for resources like disk size, vCPUs, and memory based on the selected GPU configuration.

### Attaching Persistent Disks

To attach existing persistent disks to your instance:

```bash theme={null}
prime pods create \
  --id 346663 \
  --disks disk-id-1 \
  --disks disk-id-2
```

This is useful for:

* Accessing shared datasets across multiple instances
* Persisting model checkpoints and training artifacts
* Resuming training from previous sessions

> [!TIP]
>
> Use `prime disks list` to see your available disks and their IDs. Learn more about creating and managing disks in [Managing Disks](/cli-reference/managing-disks).

## Sharing with Team Members

When working in a team, you can share instances so that other members' SSH keys are added to the instance, giving them direct access.

### Share with all team members

```bash theme={null}
prime pods create --share-with-team
```

### Interactively select members

```bash theme={null}
prime pods create --add-members
```

This displays a numbered list of team members (excluding yourself) and prompts you to select which members to share with. Enter comma-separated numbers (e.g. `1,3,5`) or type `all` to share with everyone.

### Auto-share by default

To always share new instances with your team without passing a flag each time:

```bash theme={null}
prime config set-share-resources-with-team true
```

> [!NOTE]
>
> Sharing flags require a team to be configured. Set one with `prime config set-team-id` or pass `--team-id` when creating the instance.

## Managing Your Instance

Monitor instance status:

```bash theme={null}
prime pods status <pod-id>
```

SSH into your instance:

```bash theme={null}
prime pods ssh <pod-id>
```

> [!WARNING]
>
> Configure your SSH key path in CLI settings and run chmod 400 on your private\_key.pm before using SSH.

Terminate instance:

```bash theme={null}
prime pods terminate <pod-id>
```

> [!NOTE]
>
> You'll be prompted to confirm before termination.


# Teams
Source: https://docs.primeintellect.ai/cli-reference/teams

Manage and view your teams and team members

## List Your Teams

View the teams you belong to:

```bash theme={null}
prime teams list
```

### Available Options

**--limit** `integer`

Maximum number of teams to return (default: 100)

**--offset** `integer`

Number of teams to skip for pagination (default: 0)

**--output** `string`

Output format: `table` (default) or `json`

### Example Output

| ID | Name | Slug | Role | Created |
| - | - | - | - | - |
| clvb0itli0000oxevorvgrpfn | My Team | my-team | ADMIN | 2025-01-15T10:30:00 |
| clvb1jtmk0001pxfv1abc2def | Research Lab | research-lab | MEMBER | 2025-02-01T14:20:00 |

## List Team Members

View the members of a specific team:

```bash theme={null}
prime teams members
```

This uses the team from your config. To specify a different team:

```bash theme={null}
prime teams members --team-id clvb0itli0000oxevorvgrpfn
```

### Available Options

**--team-id** `string`

Team ID to list members for. Uses the configured team ID if not specified.

**--output** `string`

Output format: `table` (default) or `json`

### Example Output

| User ID | Name | Email | Role | Joined |
| - | - | - | - | - |
| clvb0itli0000oxevorvgrpfn | Alice | [alice@example.com](mailto:alice@example.com) | ADMIN | 2025-01-15T10:30:00 |
| clvb1jtmk0001pxfv1abc2def | Bob | [bob@example.com](mailto:bob@example.com) | MEMBER | 2025-02-01T14:20:00 |

> [!TIP]
>
> Set a default team with `prime config set-team-id` so you don't need to pass `--team-id` every time.


# FAQ
Source: https://docs.primeintellect.ai/faq

Frequently Asked Questions

## Instance Types and Pricing

> **What is an On-Demand instance?**
>
> An On-Demand instance is a non-interruptible virtual machine that you can deploy and terminate at any time, paying only for the compute time you use.

> **What is a Spot instance?**
>
> A Spot instance is a cost-effective virtual machine that uses unused cloud capacity. It offers discounts of up to 90% compared to On-Demand instances, but it can be interrupted if the capacity is needed by other users. Spot instances are ideal for workloads that are flexible and can tolerate interruptions, such as batch jobs, data analysis, or fault-tolerant applications.

> **How will I be charged for GPU instances?**
>
> Each Pod has an hourly cost depending on the provider and GPU type. Your Prime Intellect credits are deducted every minute while the Pod is active. If your credits run out, your Pods will be automatically deleted. We highly recommend setting up our auto-top-up feature in your [billing settings](https://app.primeintellect.ai/dashboard/billing) to ensure balances are automatically topped up as needed.

## Cloud Providers and Infrastructure

> **Which cloud provider do you support?**
>
> We are constantly adding new providers! You can find the current list of providers in our app.

> **How long does it take for instances to launch?**
>
> Instances typically launch within a few minutes, but the exact time may vary depending on the provider.
> More detailed information on spin-up time is shown on your instance card.

> **Is there an SLA or guaranteed uptime for my instances?**
>
> We don't provide formal SLAs at this time, as the underlying infrastructure relies on various providers. If you experience repeated downtime or unreliable performance, contact support. We'll work with the provider to resolve the issue or offer refunds/credits when disruptions stem from provider-side instability. For more reliability, choose Secure Cloud providers or verify uptime and past performance before deploying long-term workloads.

## Data Management and Storage

> **How can I pause or stop my running instance without terminating it?**
>
> You can pause and resume instances on supported providers (currently Runpod), allowing you to save state and continue later at a lower cost than keeping the instance running. When resuming, note that GPU resources may not be immediately available, leading to wait times. Look for the "Pause" button on the instance details page. ⚠️ Important: If you terminate (rather than pause) your instance, you will lose all data.

> **How do I ensure my data persists after pausing or restarting an instance?**
>
> For paused instances only, data can be persisted in a special directory that is shown in your instance details in the dashboard. For example, Runpod stores persistent data in the `/workspace` directory. Note that this only works when pausing an instance - if you terminate the instance, all data will be lost. Always back up critical data to external storage (e.g., `S3`) before terminating or pausing.

> **Can my data be recovered once I've terminated my instance?**
>
> No, once an instance is terminated, all data stored on it is permanently deleted and cannot be recovered.

> **Can I get very large storage volumes (multi-terabyte)?**
>
> While most providers let you configure storage during instance creation, very large volumes may not always be honored. If your requested storage isn't fully available or doesn't match what's provisioned, contact support. We'll verify the provider's limitations and may offer guidance or alternative providers that can meet larger volume requirements.

## Configuration and Connectivity

> **Can I specify custom hardware configurations such as CPU, RAM, or disk size?**
>
> Yes. Many providers allow you to configure CPU, RAM, and storage during instance creation. However, some have limits. Hover over the pricing or configuration fields on the instance creation page to see what's available. We're also adding filters to search by CPU, RAM, and storage soon.

> **I'm having trouble connecting via SSH or Jupyter. How do I fix connection issues?**
>
> First, ensure you have the correct SSH key from your instance's "SSH Connection" button and set permissions on your local machine correctly with `chmod 400 private_key.pem`. For Jupyter notebooks, simply click the provided link on your instance page to access the notebook interface directly in your browser. If you still face issues, contact support with error details.

> **How can I see which ports are open on my instance?**
>
> You can click the Port Information button on your instance to see which ports are open:
>
> ![Click Port Information Button](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/instance-running-click-port-info.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=6c996bd5a6d8cb689511e72f4e32843b)

> **Can I open custom ports or host a service accessible over HTTP/HTTPS on my instance?**
>
> Yes. Certain providers (e.g., Latitude, Cudo, Scaleway, Hyperstack, Oblivus, Massedcompute, Datacrunch) already have open ports. We're also adding the ability to specify custom port mappings. Check provider details or the FAQ to see if port configuration is supported. If not, you may need to wait for the upcoming port-exposure feature rollout.

> **Can I run Docker or other system services on my instance?**
>
> Some providers support `docker-in-docker` configurations and system-level services, while others do not. Check the provider's documentation or contact support if you encounter issues. We're working on better clarifications and tutorials to help you understand what's supported in your chosen environment.

## Billing and Payments

> **Is there a way to get a refund if my instance fails or I'm overcharged?**
>
> If your instance fails due to provider issues (such as failing to start, crashing early, or being impossible to terminate from your dashboard), contact support with the instance details. We typically provide refunds or credits when the issue stems from our system or provider instability. Please contact us in the support chat by typing REFUND as your first word, then describe your issue in detail and include screenshots.

> **How do I apply a promotional code to get credits?**
>
> When you sign up or top up credits, look for the promo code input field. If you've already signed up, the promo code field is typically found in the billing/top-up section. If you have trouble finding it, contact support and we'll manually credit your account.
>
> ![Redeem Coupon](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/redeem-coupon.jpg?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=bd43dd52fcc402f4d295440bff6ed904)

> **How do I set up team accounts or custom billing emails for invoicing?**
>
> We've introduced a "Teams" feature where you can invite multiple users and set a dedicated billing email. Create a team from your dashboard's team settings, add members, and specify the billing email (e.g., [billing@company.com](mailto:billing@company.com)) so that all invoices are sent there. This helps consolidate usage and manage costs for multiple users under one account.

> **Can I pay in other currencies or use alternative payment methods?**
>
> Currently, most billing occurs in USD and via standard payment methods. If you need invoicing in other currencies or want to explore alternative digital payment options, please contact support. We're actively considering these features and may arrange manual invoicing or special payment methods in the future.

## Troubleshooting

> **My instance won't start or is stuck in 'pending.' What should I do?**
>
> If your instance fails to launch or remains "pending" for an extended period, first try terminating it and creating a new one. If the problem persists, contact support with the instance details. In cases where the issue is on the provider's side, we can often provide refunds or credits.

## GPU Providers

> **How do I become a GPU provider?**
>
> We are actively expanding our network of GPU providers, including both individual compute providers and larger cloud platforms. For more information, please contact us at [contact@primeintellect.ai](mailto:contact@primeintellect.ai).


# Browser Environments
Source: https://docs.primeintellect.ai/guides/browser-environments

Guide to building and training browser-based environments with Prime Intellect and Browserbase.

# Multimodal Browser Environments in verifiers

**BrowserEnv now supports vision-based web browsing via Computer Use Agent (CUA) mode.**

Both modes are modes in verifiers and we provide examples for trying both. This guide covers the two interaction modes, how to get started, and walks through two real environments: a lightweight eval (`bb_demo.py`) and a full benchmark for training (`webvoyager_no_anti_bot.py`). It also covers BrowserEnv-specific pieces of RL training, including DOM and CUA training configurations and how to integrate with Hosted Training.

***

## Two Modes of Browser Interaction

`BrowserEnv` is a unified `StatefulToolEnv` subclass that supports two operating modes, selected via the `mode` parameter. Both modes use [Browserbase](https://browserbase.com) as the browser provider.

### DOM Mode: Natural Language Control

DOM mode uses the [Stagehand](https://github.com/browserbase/stagehand) SDK to translate natural language instructions into browser actions. Stagehand runs its own LLM internally (configured via `stagehand_model`, defaults to `openai/gpt-4o-mini`) to interpret the page DOM and execute the appropriate operations.

The agent's tool surface is high-level and semantic:

* `navigate(url)`: go to a URL
* `observe(instruction)`: find possible actions matching a natural language description
* `act(instruction)`: execute an action described in natural language (click a button, fill a form)
* `extract(instruction, schema_json)`: extract structured data from the page

The agent never sees the rendered page. It works through Stagehand's abstraction of the DOM, which means it requires no coordinates, and no screenshots. This is fast and effective when pages have reliable semantic HTML.

DOM mode works best when actionable page state is exposed semantically through Stagehand. Visually ambiguous cases, especially overlays or elements that are easier to disambiguate by pixels, can still be easier in CUA.

**Stagehand routing:** When `proxy_model_to_stagehand=False` (default), Stagehand uses its own `stagehand_model` and `MODEL_API_KEY`. When `proxy_model_to_stagehand=True`, BrowserEnv injects the rollout client's model name, base URL, and API key into `observe`, `act`, and `extract`, so those Stagehand calls run through the same client/model endpoint as the rollout model.

### CUA Mode: Vision-Based Control (Multimodal)

CUA mode gives the agent a live screenshot of the rendered page after every action. The agent sees pixels and acts via screen coordinates, the same way a human would interact with a screen.

The agent's tool surface is low-level and coordinate-based:

* `click(x, y, button)`: click at screen coordinates
* `double_click(x, y)`: double-click at coordinates
* `type_text(text)`: type text into the focused element
* `keypress(keys)`: press keyboard keys
* `scroll(x, y, scroll_x, scroll_y)`: scroll at a position
* `goto(url)`: navigate to a URL
* `back()` / `forward()`: browser history navigation
* `wait(time_ms)`: wait for a specified duration
* `screenshot()`: capture the current page state

Each action returns a multimodal response: a text status block (URL, viewport dimensions, success/error) and a base64 PNG screenshot encoded as an `image_url` content block. The model processes both. After the tool call returns, `BrowserEnv` moves screenshot parts out of tool messages into a trailing user message. When `keep_recent_screenshots` is set, older screenshots are replaced with `[Screenshot removed to save context]` placeholders.

**When to use it:** Tasks that require visual understanding, such as navigating unfamiliar UIs, interacting with canvas-rendered apps, clicking elements that lack semantic markup, or any workflow where a human would need to *look* at the screen to proceed.

### Side-by-Side Comparison

| Aspect | DOM Mode | CUA Mode |
| - | - | - |
| **Control paradigm** | Natural language via Stagehand | Vision-based screen coordinates |
| **Observation space** | Text (DOM abstractions) | Multimodal (screenshots + text) |
| **Server requirement** | None (Stagehand SDK direct) | CUA server (auto-deployed or manual) |
| **Extra API key** | `MODEL_API_KEY` by default, or the rollout client when proxied | The CUA server template can forward `OPENAI_API_KEY` internally if it is present |
| **Best for** | Structured web interactions | Visual/complex UIs |
| **Speed** | Faster (direct DOM manipulation) | Slower (screenshot round-trips) |

***

## Getting Started

### Installation

Both example environments require the `browser` extra:

```bash theme={null}
# Install verifiers with browser support
uv pip install "verifiers[browser]"
```

### Environment Variables

```bash theme={null}
# Required for both modes
export BROWSERBASE_API_KEY="your-api-key"
export BROWSERBASE_PROJECT_ID="your-project-id"

# Required for LLM judge model - pick your favorite model/your custom judge model, here we use OpenAI as an example
export OPENAI_API_KEY="your-openai-key"

# Required for DOM mode only (Stagehand's internal LLM) - pick your favorite model provider
export MODEL_API_KEY="your-llm-api-key"
```

### Running the Examples

Both examples ship with the same task: navigate to the Prime Intellect homepage and read the headline. Note to do the below, you need to have a prime account.

```bash theme={null}
# Install both examples from the Environments Hub
prime env install primeintellect/browser-dom-example primeintellect/browser-cua-example

# DOM mode
uv run vf-eval browser-dom-example -m openai/gpt-4.1-mini

# CUA mode (pre-built image, fastest and recommended)
uv run vf-eval browser-cua-example -m openai/gpt-4.1-mini
```

***

## CUA Mode Execution Backends

CUA mode requires a **CUA server** — a lightweight TypeScript HTTP service that bridges the Python browser agent and the browser. Under the hood, Browserbase uses [Understudy](https://github.com/browserbase/stagehand/tree/main/packages/core/lib/v3/understudy), a custom and powerful Chrome DevTools Protocol (CDP) driver developed at Browserbase as part of [Stagehand](https://github.com/browserbase/stagehand), the AI browser automation framework. Understudy was built to overcome limitations in Playwright's built-in automation capabilities (e.g. more granular input control and richer session introspection). Because Understudy's CDP engine is written in TypeScript, a thin server layer is needed to expose it as a REST API that the Python environment can call. The CUA server is that layer — it receives coordinate-based action requests over HTTP, translates  into CDP commands via Understudy, and returns screenshots back to the agent.

Three execution modes are available, from fastest to most flexible:

### 1. Pre-built Docker Image (Default)

Uses `deepdream19/cua-server:latest` with everything pre-installed. Startup takes \~5–10 seconds. This is the recommended approach.

```bash theme={null}
uv run vf-eval browser-cua-example -m openai/gpt-4.1-mini -b https://api.openai.com/v1 -k OPENAI_API_KEY
```

### 2. Binary Upload

Builds and uploads a custom CUA server binary to a sandbox at runtime. Startup takes \~30–60 seconds. Use this when you need a modified server.

```bash theme={null}
uv run vf-eval browser-cua-example -m openai/gpt-4.1-mini -a '{"use_prebuilt_image": false}'
```

### 3. Manual Server (Local Development)

Connect to a locally running CUA server. Start the server yourself, then point the environment at it:

```bash theme={null}
# Terminal 1: start the CUA server
cd assets/templates/browserbase/cua && pnpm dev

# Terminal 2: run with sandbox disabled
uv run vf-eval browser-cua-example -m openai/gpt-4.1-mini -a '{"use_sandbox": false}'
```

### Screenshot Management

CUA mode captures a screenshot after every action. Two parameters control how screenshots flow:

* `save_screenshots` (bool): persist every screenshot to disk as timestamped PNGs in `screenshot_dir` (defaults to `./screenshots`). The CUA example defaults this to `False`; the `BrowserEnv` class itself defaults to `True`.
* `keep_recent_screenshots` (int | None): how many recent screenshots to retain in the conversation context window sent to the model. Defaults to `2`. Set to `None` to keep all (higher token cost).

***

## Example Environments

### [`bb_demo`](https://app.primeintellect.ai/dashboard/environments/prime/bb-demo): Quick Eval with Click Visualization

`bb_demo.py` is a lightweight single-task environment designed for quick CUA mode evaluation and debugging. It serves as a hello world for multimodal browser environments and pairs a simple browsing task with a custom `CUAMode` subclass that annotates saved screenshots with click markers.

![bb\_demo eval rollout](https://mintcdn.com/primeintellect/14whN4o7A3u5_UrF/guides/images/bb_demo_eval.png?fit=max\&auto=format\&n=14whN4o7A3u5_UrF\&q=85\&s=d510e01b28abb6f3304a982232d8cf84)

**Task.** The agent is asked to navigate to the Prime Intellect website, find the blog page, and summarize the latest post. A `JudgeRubric` backed by `gpt-4.1-mini` evaluates whether the agent's response adequately describes the blog content, returning `1.0` for "yes" and `0.0` otherwise.

**Click marker overlay.** The environment subclasses `CUAMode` as `ClickMarkerCUAMode`, which overrides `click()` and `double_click()` to store the target (x, y) coordinates before each action. When the resulting screenshot is saved to disk, `_save_screenshot_with_marker()` uses Pillow to composite a visual marker (concentric circles, crosshairs, coordinate label) onto the PNG. The marker is drawn on the *saved* file only. The screenshot returned to the model in conversation context is unmodified, keeping the agent's visual input clean while giving developers an annotated record of every click.

**Subclassing pattern.** `ClickMarkerBrowserEnv` intercepts `BrowserEnv.__init__` when `mode="cua"`. It manually extracts all CUA-specific parameters from `kwargs`, bypasses the parent's mode setup by calling `StatefulToolEnv.__init__` directly, then constructs and registers the custom `ClickMarkerCUAMode`. For DOM mode, it delegates entirely to the parent. This pattern is useful any time you need to swap in a custom mode implementation without forking `BrowserEnv`.

**Usage:**

```python theme={null}
from bb_demo import load_environment

env = load_environment(
    save_screenshots=True,
    screenshot_dir="./my_screenshots",
    mark_clicks=True,
    click_marker_radius=15,
)
```

Requires `uv pip install Pillow`. Without it, screenshots still save without markers and the environment shows a warning at startup.

### [`webvoyager_no_anti_bot`](https://app.primeintellect.ai/dashboard/environments/browserbase/webvoyager-no-anti-bot): Training-Scale Benchmark

WebVoyager is a 600-task web navigation benchmark spanning real websites (Allrecipes, Amazon, Apple, ArXiv, GitHub, Google Flights/Maps/Search, ESPN, and more). This version filters out 43 tasks from `dictionary.cambridge.org` that are blocked by Cloudflare anti-bot protection, leaving 93.3% of the original dataset intact. It supports both DOM and CUA modes, making it a good candidate for RL training runs. Here is an [example training run](https://app.primeintellect.ai/training/shared/v8uz1wu40av5bssrqz3mi203) with Qwen3-VL-8B-Instruct:

![WebVoyager training run](https://mintcdn.com/primeintellect/14whN4o7A3u5_UrF/guides/images/webvoyager_training.png?fit=max\&auto=format\&n=14whN4o7A3u5_UrF\&q=85\&s=da6307484ec0a917aa8ca746d322a588)

**Dataset.** Tasks are loaded from a local JSONL file (`WebVoyager_data_clean.jsonl`). Each row has a natural language task (`ques`), a starting URL (`web`), a website name (`web_name`), and a task ID. There are no ground-truth answers because WebVoyager is task-completion-based rather than answer-matching. You can filter by website with `web_filter` (e.g., `web_filter="Amazon"`) and limit the number of examples with `num_examples`.

**Evaluation.** Since there are no explicit answers, the environment uses a task-completion judge. The agent's entire multi-turn trajectory is rendered into a structured text transcript by `WebVoyagerTrajectoryParser`, a custom `vf.Parser` subclass. This transcript includes assistant messages, tool calls with normalized arguments, and truncated tool results, while images are excluded. The transcript is capped at 12,000 characters to keep judge context manageable. The judge prompt instructs the LLM to verify that the agent navigated to the correct site, performed the required actions, and reached the requested end state. If the agent made no tool calls at all, the reward is automatically `0.0` without consulting the judge.

**Transcript rendering.** `render_webvoyager_transcript()` walks the completion messages and emits a line-by-line log: `ASSISTANT:` for text, `TOOL_CALL name({args})` for tool invocations, and `TOOL_RESULT:` for tool responses (truncated to 500 characters each). This keeps the judge grounded in what actually happened rather than what the agent claims happened. The judge prompt explicitly says to treat unsupported assertions as insufficient evidence.

**Mode flexibility.** `load_environment()` accepts `mode="dom"` or `mode="cua"` and passes all relevant configuration through to `BrowserEnv`. Both modes work against the same dataset and rubric.

**Usage:**

```bash theme={null}
# Install the environment
prime env install browserbase/webvoyager-no-anti-bot

# All 600 tasks, DOM mode
uv run vf-eval webvoyager-no-anti-bot -m openai/gpt-4.1-mini

# CUA mode, filtered to Amazon tasks
uv run vf-eval webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"mode": "cua", "web_filter": "Amazon"}'

# 10 examples for a quick test
uv run vf-eval webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"num_examples": 10}'
```

Note: the full 600-task suite takes a while to run. For initial testing, use `num_examples` or `web_filter` to scope it down.

***

## Building Your Own Browser Environment

Both examples follow the standard verifiers environment contract: a Python module exposing `load_environment(**kwargs) -> vf.Environment`. The pattern is:

1. **Define a dataset**: a HuggingFace `Dataset` with `question`, `answer`, and optionally `start_url` and `task_id` columns. For task-completion benchmarks where there's no ground-truth answer (like WebVoyager), set `answer` to an empty string and rely on a task-completion judge.
2. **Define a rubric**: typically a `JudgeRubric` with an LLM judge. For answer-matching tasks, the judge compares agent output to the expected answer. For task-completion tasks, subclass `vf.Parser` to render the trajectory into a judge-friendly transcript and evaluate whether the task was actually done.
3. **Construct a `BrowserEnv`**: pass `mode`, dataset, rubric, system prompt, and any mode-specific configuration.
4. **Package it**: add a `pyproject.toml` with `verifiers[browser]>=0.1.8` as a dependency.

```python theme={null}
import verifiers as vf
from verifiers.envs.integrations.browser_env import BrowserEnv
from datasets import Dataset

def load_environment(max_turns: int = 15, **kwargs) -> vf.Environment:
    dataset = Dataset.from_dict({
        "question": ["What is the title of the first blog post?"],
        "answer": ["Expected answer here"],
        "start_url": ["https://example.com/blog"],
    })

    rubric = vf.JudgeRubric(judge_model="gpt-4o-mini", judge_prompt="...")
    rubric.add_reward_func(your_judge_func, weight=1.0)

    return BrowserEnv(
        mode="cua",  # or "dom"
        dataset=dataset,
        rubric=rubric,
        max_turns=max_turns,
        system_prompt="...",
        **kwargs,
    )
```

Then install and evaluate:

```bash theme={null}
uv pip install -e ./environments/your_env
uv run vf-eval your-env -m openai/gpt-4.1-mini
```

***

## RL Training with Browser Environments

Browserbase integrates with Prime Intellect through `BrowserEnv`. Browserbase handles the cloud browsers, `BrowserEnv` wraps those sessions as a verifiers environment, and Prime runs the evaluation or RL training loop.

In practice, that means the same BrowserEnv configuration can be used in both evaluation and training. The training-specific questions are usually:

* whether you want `mode="dom"` or `mode="cua"`
* whether Stagehand should keep its own model or be routed through the rollout model
* which browser/session settings you want in the `[[env]]` config

### Before You Train

1. Validate the environment with evals first. Reward quality matters more than running a large training job quickly.
2. Check reward distribution. If everything is `0.0` or everything is `1.0`, the run will not teach you much.
3. Start with a small training run before scaling up browser minutes and GPU time.

### Training Target in This Repo

`webvoyager_no_anti_bot` is the training-scale BrowserEnv example in this repo.

* 600 tasks in the filtered WebVoyager dataset
* supports both `mode="dom"` and `mode="cua"`
* uses task-completion reward rather than answer matching
* renders assistant text, tool calls, and truncated tool results into the judge transcript
* returns `0.0` reward immediately if a rollout makes no tool calls

That makes it the cleanest local reference for comparing DOM and CUA training without changing the benchmark itself.

Example Prime DOM training run: [WebVoyager RL run](https://app.primeintellect.ai/training/shared/aixtf0gfgstmasfsugb9x3bg).
Example Prime CUA training run: [WebVoyager RL run](https://app.primeintellect.ai/training/shared/v8uz1wu40av5bssrqz3mi203).

### DOM Training

A DOM sample run can be done with the following configuration:

* model: `Qwen/Qwen3-4B-Instruct-2507`
* `batch_size = 64`
* `rollouts_per_example = 8`
* environment id: `browserbase/webvoyager-no-anti-bot`

By default, Stagehand keeps using its own `stagehand_model` and `MODEL_API_KEY`. If you want the trained rollout model to also handle Stagehand's DOM operations, enable `proxy_model_to_stagehand=true`. In that configuration, `observe`, `act`, and `extract` are routed through the same client/model endpoint as the rollout model.

The checked-in sample does not add extra BrowserEnv args. To enable shared Stagehand routing, pass `mode = "dom"` and `proxy_model_to_stagehand = true` as taskset arguments for `browserbase/webvoyager-no-anti-bot`, and supply `MODEL_API_KEY` at dispatch with `-e`.

**Suggested starting point for DOM mode:** a 4B-class instruct model, `max_steps = 100`, `batch_size = 64`, `group_size = 8`, `max_completion_tokens = 1024`. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

### CUA Training

A vision-language sample run can be done with the following configuration:

* model: `Qwen/Qwen3-VL-4B-Instruct`
* `batch_size = 32`
* `rollouts_per_example = 4`
* environment id: `prime/webvoyager-no-anti-bot`
* BrowserEnv args: `mode = "cua"`, `max_turns = 10`, `viewport_width = 800`, `viewport_height = 600`, `keep_recent_screenshots = 2`, `memory_gb = 6`

**Suggested starting point for CUA mode:** a 4B-class VL model, `max_steps = 100`, `batch_size = 32`, `group_size = 4`, `max_completion_tokens = 512`, with the BrowserEnv args listed above passed as taskset arguments. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

### Credentials and Workflow Notes

* Always provide `BROWSERBASE_API_KEY` and `BROWSERBASE_PROJECT_ID` when BrowserEnv is creating Browserbase sessions.
* DOM mode needs `MODEL_API_KEY` unless you route Stagehand through the rollout model with `proxy_model_to_stagehand=true`.
* CUA mode uses the same BrowserEnv backend choices as evaluation. The current bundled CUA server template forwards `OPENAI_API_KEY` into Stagehand if it is present, but that is a server-template detail rather than a separate BrowserEnv training argument.
* Browserbase credentials belong in environment variables or training secrets, not in the TOML file.
* The same BrowserEnv args used with `vf-eval -a '{...}'` belong in the training config under `[[env]]` `args = { ... }`.
* BrowserEnv does not expose sandbox authentication as a constructor argument.

Both Hosted Training and self-managed `prime-rl` use the same BrowserEnv-side mode choice and environment args. The difference is in how the training job is orchestrated, not in how BrowserEnv behaves.

### Picking DOM vs CUA for Training

| Question | DOM Mode | CUA Mode |
| - | - | - |
| What does the model observe? | Semantic browser actions through Stagehand | Screenshots plus tool status text |
| What kind of model fits best? | Text models | Vision-language models |
| How do Stagehand calls behave during training? | Separate `stagehand_model` by default, or shared rollout routing with `proxy_model_to_stagehand=true` | Not applicable to the agent-side tool surface |
| What usually matters most? | Semantic reliability and cheaper context | Visual grounding and UI disambiguation |

### Performance Notes

* Browser training rollouts are slower than text-only environments because each rollout step interacts with a real browser session.
* CUA mode is heavier than DOM mode because screenshots must be rendered, carried in context, and consumed by the model.
* If your task does not need visual grounding, DOM mode is usually the faster and cheaper place to start.

***

## Quick Reference

### DOM Mode Arguments

| Argument | Default | Description |
| - | - | - |
| `project_id` | Required | Browserbase project ID |
| `browserbase_api_key_var` | `"BROWSERBASE_API_KEY"` | Env var for Browserbase API key |
| `stagehand_model` | `"openai/gpt-4o-mini"` | Model for Stagehand DOM operations |
| `model_api_key_var` | `"MODEL_API_KEY"` | Env var for Stagehand's API key |
| `max_turns` | `10` | Max conversation turns |
| `proxy_model_to_stagehand` | `False` | Route Stagehand LLM calls through eval model |

### CUA Mode Arguments

| Argument | Default | Description |
| - | - | - |
| `use_sandbox` | `True` | Auto-deploy CUA server to sandbox |
| `use_prebuilt_image` | `True` | Use pre-built Docker image (fastest) |
| `prebuilt_image` | `"deepdream19/cua-server:latest"` | Docker image for sandbox |
| `server_url` | `"http://localhost:3000"` | CUA server URL (when `use_sandbox=False`) |
| `viewport_width` | `1024` | Browser viewport width |
| `viewport_height` | `768` | Browser viewport height |
| `save_screenshots` | `True` | Persist screenshots to disk |
| `keep_recent_screenshots` | `2` | Screenshots in model context (`None` = all) |
| `max_turns` | `15` | Max conversation turns |
| `env` | `"BROWSERBASE"` | Browser provider (`"LOCAL"` or `"BROWSERBASE"`) |
| `proxies` | `False` | Enable Browserbase proxies |
| `advanced_stealth` | `False` | Enable anti-bot detection stealth mode |
| `cpu_cores` | `2` | Sandbox CPU cores |
| `memory_gb` | `4` | Sandbox memory |

### CUA Execution Modes

| Mode | Flag | Startup | Use Case |
| - | - | - | - |
| Pre-built image | *(default)* | \~5-10s | Production |
| Binary upload | `use_prebuilt_image=false` | \~30-60s | Custom server |
| Manual server | `use_sandbox=false` | Instant | Local dev |


# Overview
Source: https://docs.primeintellect.ai/guides/index

Step-by-step guides for training runs, environments, and agent workflows.

Practical walkthroughs for common workflows. Start with training recipes for reusable patterns, or follow a full build-and-train guide for a concrete end-to-end project.

> **[Your First Model Training](/guides/rl-training)**
>
> End-to-end walkthrough from workspace setup to Hosted Training and deployment.

> **[Training Recipes](/guides/recipes)**
>
> Reusable RL patterns for math, code generation, tool use, and more.

> **[Search Agents](/guides/search-agents)**
>
> Build document-search environments and train them end to end.


# Training Recipes
Source: https://docs.primeintellect.ai/guides/recipes

Practical RL training recipes for math reasoning, code generation, tool use, and more.

Each example here covers a common RL use case: what kind of environment to build, a minimal working implementation, a training config, and practical tips. Use the environment code as a starting point for your own runs on [Prime Intellect](/platform-overview).

If you haven't launched a training run yet, start with the [Getting Started](/hosted-training/getting-started) guide first.

***

## Math Reasoning

Train models to solve mathematical problems step-by-step, using symbolic verification to reward correct answers.

**Environment type:** `SingleTurnEnv` with `MathRubric`

**Why RL works here:** Models learn to produce correct final answers through trial and error. The reward signal is binary and cheap to compute — symbolic math verification checks whether the model's `\boxed{}` answer matches the ground truth, without needing an LLM judge.

**Example environment:**

```python theme={null}
import verifiers as vf
from datasets import load_dataset

def load_environment(split: str = "train", num_examples: int = -1) -> vf.Environment:
    ds = load_dataset("openai/gsm8k", split=split)
    dataset = vf.Dataset.from_hf(ds, question_col="question", answer_col="answer")
    if num_examples > 0:
        dataset = dataset.select(range(num_examples))

    rubric = vf.MathRubric()
    return vf.SingleTurnEnv(
        dataset=dataset,
        rubric=rubric,
        system_prompt="Solve the problem step by step. Put your final answer in \\boxed{}.",
    )
```

**Suggested starting point:** a 4B-class instruct model, `max_steps = 200`, `batch_size = 256`, `group_size = 8`, `max_completion_tokens = 1024`. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

**Tips:**

* Start with GSM8K for validation — baseline models typically score 40–70%, leaving room for improvement.
* For harder tasks (AIME, competition math), use a larger model like `Qwen/Qwen3-235B-A22B-Thinking-2507` and increase `max_tokens`.

***

## Code Generation with Sandboxes

Train models to write correct code by executing their solutions in sandboxed environments and verifying outputs against test cases.

**Environment type:** `PythonEnv` or `SandboxEnv`

**Why RL works here:** The model gets a concrete pass/fail signal from running code. Unlike static checking, execution-based verification catches subtle bugs and rewards solutions that actually work. Multi-turn interaction lets the model iteratively debug when tests fail.

**Example environment:**

```python theme={null}
import verifiers as vf
from datasets import Dataset

def load_environment() -> vf.Environment:
    dataset = Dataset.from_list([
        {
            "question": "Write a function `fibonacci(n)` that returns the nth Fibonacci number.",
            "info": '{"test_code": "assert fibonacci(0) == 0\\nassert fibonacci(1) == 1\\nassert fibonacci(10) == 55"}'
        },
        # ... more examples
    ])

    async def tests_pass(completion, info, state) -> float:
        code = completion[-1]["content"]
        test_code = info["test_code"]
        try:
            exec_result = state.get("exec_result", "")
            return 1.0 if "PASSED" in exec_result else 0.0
        except Exception:
            return 0.0

    rubric = vf.Rubric(funcs=[tests_pass])
    return vf.PythonEnv(
        dataset=dataset,
        rubric=rubric,
        max_turns=5,
    )
```

**Suggested starting point:** a 30B-A3B-class model, `max_steps = 300`, `batch_size = 256`, `group_size = 16`, `max_completion_tokens = 2048`. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

**Tips:**

* Use `PythonEnv` for Python-specific tasks — it provides a persistent REPL that the model can use across turns.
* Use `SandboxEnv` for multi-language tasks or when you need shell access.
* Set `max_turns` to 3–5 to let the model iterate on failing test cases.
* Consider a partial reward for passing some but not all tests, rather than all-or-nothing scoring.

***

## Multi-Turn Games and Puzzles

Train models on interactive tasks where they must take actions over multiple turns, receiving feedback after each move.

**Environment type:** Custom `MultiTurnEnv` subclass

**Why RL works here:** Games provide dense, structured reward signals. The model learns strategies through repeated play — each rollout is a complete game, and the final score becomes the reward. Multi-turn structure naturally teaches planning and sequential decision-making.

**Example environment (word guessing game):**

```python theme={null}
import verifiers as vf
from datasets import Dataset
import random

class WordGameEnv(vf.MultiTurnEnv):
    async def setup_state(self, state, **kwargs):
        state["target"] = state["info"]["target_word"]
        state["guesses"] = []
        return await super().setup_state(state, **kwargs)

    async def env_response(self, messages, state):
        guess = messages[-1]["content"].strip().lower()
        target = state["target"]
        state["guesses"].append(guess)

        if guess == target:
            state["won"] = True
            return [{"role": "user", "content": "Correct! You found the word."}]

        # Give hints: which letters are in the right position
        hints = []
        for i, (g, t) in enumerate(zip(guess, target)):
            if g == t:
                hints.append(f"Position {i+1}: correct")
            elif g in target:
                hints.append(f"Position {i+1}: wrong position, letter is in the word")
            else:
                hints.append(f"Position {i+1}: letter not in word")

        return [{"role": "user", "content": "\n".join(hints) + "\nGuess again."}]

    @vf.stop
    async def game_won(self, state):
        return state.get("won", False)


def load_environment() -> vf.Environment:
    words = ["apple", "brain", "cloud", "dance", "eagle"]
    dataset = Dataset.from_list([
        {"question": "Guess the 5-letter word. I'll give you hints after each guess.",
         "info": f'{{"target_word": "{w}"}}'} for w in words
    ])

    async def win_reward(state) -> float:
        if state.get("won"):
            return max(0.2, 1.0 - 0.15 * len(state["guesses"]))
        return 0.0

    rubric = vf.Rubric(funcs=[win_reward])
    return WordGameEnv(dataset=dataset, rubric=rubric, max_turns=8)
```

**Suggested starting point:** a 4B-class instruct model, `max_steps = 100`, `batch_size = 128`, `group_size = 8`, `max_completion_tokens = 256`. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

**Tips:**

* Games are excellent for validating your setup since they tend to show clear reward improvements within a small number of steps.
* Shape rewards to be gradient-rich — instead of just 0/1 for win/loss, give partial credit (e.g., reward based on number of turns taken to win).
* The built-in `alphabet-sort` environment is a great starting point — install it with `prime env install primeintellect/alphabet-sort`.

***

## Tool Use and Agentic Tasks

Train models to use tools effectively — calling the right tool with the right arguments to accomplish a goal.

**Environment type:** `ToolEnv` or `MCPEnv`

**Why RL works here:** Tool use requires the model to reason about which tool to call, compose correct arguments, interpret results, and decide on next steps. RL training lets the model learn this decision-making loop through practice, improving both tool selection and argument construction.

**Example environment (research assistant with search):**

```python theme={null}
import verifiers as vf
from datasets import Dataset

async def web_search(query: str) -> str:
    """Search the web for information.

    Args:
        query: The search query to look up.

    Returns:
        Search results as text.
    """
    # your search implementation
    return await do_search(query)

async def calculate(expression: str) -> str:
    """Evaluate a mathematical expression.

    Args:
        expression: A math expression to evaluate (e.g. "2 + 2 * 3").

    Returns:
        The result of the evaluation.
    """
    try:
        return str(eval(expression))
    except Exception as e:
        return f"Error: {e}"

def load_environment() -> vf.Environment:
    dataset = Dataset.from_list([
        {
            "question": "What is the population of France divided by the population of Switzerland?",
            "answer": "approximately 8.3"
        },
        # ... more examples requiring tool use
    ])

    async def answer_quality(completion, answer, judge) -> float:
        verdict = await judge(completion, answer)
        return 1.0 if "correct" in verdict.lower() else 0.0

    rubric = vf.JudgeRubric(judge_model="gpt-4.1-mini")
    rubric.add_reward_func(answer_quality)

    return vf.ToolEnv(
        dataset=dataset,
        tools=[web_search, calculate],
        rubric=rubric,
        max_turns=10,
    )
```

**Suggested starting point:** a 30B-A3B-class model, `max_steps = 200`, `batch_size = 256`, `group_size = 16`, `max_completion_tokens = 1024`. Pass API keys at dispatch with `-e` / `--env-file`. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

**Tips:**

* Use `JudgeRubric` with an LLM judge for open-ended tasks where exact matching isn't feasible.
* Store API keys for external services (judge models, search APIs) in a `secrets.env` file and reference it with `env_file`.
* Monitor tool call counts via the automatic metrics — if the model isn't calling tools, the task may need a clearer prompt.
* `MCPEnv` is useful when your tools are already implemented as MCP servers.

***

## Multi-Environment Training

Train a single model on multiple tasks simultaneously to improve generalization.

**Why RL works here:** Training on diverse tasks prevents the model from overfitting to a single task's reward surface. The model learns transferable skills (reasoning, tool use, instruction following) that improve performance across all tasks.

**Suggested starting point:** a flagship MoE model, `max_steps = 500`, `batch_size = 512`, `group_size = 16`, `max_completion_tokens = 2048`, one `[[orchestrator.train.source]]` block per environment, plus `[wandb]` and `[eval]`. Set `ratio` on each source to weight the mix. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

**Tips:**

* Run baseline evaluations on each environment before training to understand starting performance.
* Use W\&B logging to compare per-environment reward curves during training.

***

## Workflow Summary

Regardless of the use case, the typical Hosted Training workflow is:

**1. Build your environment**

Create an environment with a dataset, harness, and rubric using the [verifiers](/verifiers/overview) library.

**2. Evaluate baseline**

Run `vf-eval` against your environment (or `prime eval run --hosted` on the platform) to measure where the model starts.

**3. Configure and launch training**

Write a `.toml` config and launch with `prime train`.

**4. Monitor and iterate**

Watch reward curves on the dashboard, adjust your environment or config, and re-run.

**5. Deploy**

Download the trained weights from your [volume](/hosted-training/volumes) or the dashboard.

> **[Getting Started](/hosted-training/getting-started)**
>
> Launch your first Hosted Training run in minutes.

> **[End-to-End Run](/hosted-training/end-to-end-run)**
>
> Detailed walkthrough of a complete training run.

> **[Environments](/verifiers/legacy/environments)**
>
> Learn how to build custom environments with verifiers.

> **[Config Reference](/prime-rl/configuration)**
>
> Multi-environment training, evals, and more.


# Your First Model Training
Source: https://docs.primeintellect.ai/guides/rl-training

Go from idea to training run to deployed agent. No programming or RL experience required.

In this guide, we’ll walk you through setting up your workspace, creating your first agent environment, using it to evaluate baseline performance, launching a Hosted Training run, and deploying your model for inference.

These instructions are intended for use on a Mac or Linux CPU development environment. No previous experience with RL is required. In fact, experience with *coding* isn’t even required — we’ll use agents for everything. We’ll just assume that you have Claude Code, Codex, Cursor, OpenCode, Amp, or some other similar coding agent installed on your computer.

## Setting Up Your Lab Workspace

Ensure you have `uv` installed for managing Python packages:

```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Install the `prime` CLI:

```bash theme={null}
uv tool install prime
```

Choose a folder on your machine as your Lab workspace (e.g. `~/dev/my-lab`) and do:

```bash theme={null}
prime lab setup
```

This command prepares your workspace with:

> **Python project bootstrap**
>
> Creates a Python project and installs `verifiers` for environment development.

> **Coding-agent setup**
>
> Configures your workspace for coding-agent workflows.

> **Instruction files**
>
> Downloads agent instruction files like `AGENTS.md` and Agent Skills.

> **Starter configs**
>
> Downloads example training and evaluation configs.

```text theme={null}
~/dev/demo prime lab setup
Supported coding agents: codex, claude, cursor, opencode, amp
Primary coding agent [codex]:
Using multiple coding agents? [y/N]:
No pyproject.toml found, initializing uv project...
Running: uv init
Initialized project `demo`
Running: uv add verifiers
...
... # install + download outputs omitted
...
[................................................................................] 1371 / 1371
Downloaded configs/rl/wordle.toml from https://github.com/primeintellect-ai/verifiers

+------------------------------------------ get started -------------------------------------------+
|                                                                                                  |
|  idea -> environment -> eval -> training                                                         |
|                                                                                                  |
|  +-------------------------------- ask codex ---------------------------------+                  |
|  |                                                                            |                  |
|  | I want to train a model for <my task domain>. Propose an initial           |                  |
|  | environment scaffold including relevant tools, generate a small            |                  |
|  | synthetic dataset, run a quick eval baseline, inspect the results,         |                  |
|  | and decide how to iterate on refining the implementation.                  |                  |
|  |                                                                            |                  |
|  +----------------------------------------------------------------------------+                  |
|                                                                                                  |
|  +-------------------- quick commands --------------------+                                      |
|  |                                                        |                                      |
|  | $ uv run vf-init my-env                                |                                      |
|  | $ uv run vf-eval my-env -m openai/gpt-5.4-nano -n 5    |                                      |
|  | $ prime train configs/rl/qwen.toml                     |                                      |
|  | $ uv run vf-gepa my-env -m openai/gpt-5.4-nano         |                                      |
|  |                                                        |                                      |
|  +--------------------------------------------------------+                                      |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+
```

> [!TIP]
>
> Use one Lab workspace per research project and version it with Git. A workspace can contain multiple environments, configs, scripts, data, and eval outputs.

## Prompting Your Coding Agent

For many low-to-medium complexity environments, we find that the latest coding agents are often capable of “one-shotting” them, when equipped with the provided context from `prime lab setup` and given a sufficiently detailed prompt.

Providing the prompt below to a frontier coding agent (OpenCode + Codex 5.3) resulted in a fully functional environment for a calendar scheduling agent:

> [!NOTE]
>
> Save this as `prompt.md` and pass it directly to your coding agent as your initial task prompt.

```md prompt.md wrap theme={null}
Make an environment for a calendar scheduling agent.
In each task, there should be a set of people with busy calendars, and individual + global constraints for scheduling the meeting.
Some constraints can be "hard" (not allowed to violate), others can be "soft", where violating a constraint incurs some utility cost for certain attendees.
Each attendee has a utility for the proposed meeting time between 0 and 1, and the task score will be the weighted average of attendee scores if an acceptable meeting time is found, and 0 otherwise.
Attendee importance weights should be normalized to 1 for each task.

We should be able to programmatically generate task problems, and deterministically validate that satisfying solutions exist (and what their best possible score would be).
We should have fine-grained controls for key degrees of freedom in task generation, with higher-level parameters ("easy" / "medium" / "hard") for the full task set, which then map into setting ranges for the more fine-grained controls.
Be creative, and use your judgment to design clean composition rules for converting meeting choices and conflicts into scores. Avoid complex branching/conditional logic where possible.
Think carefully about designing your system in a way which discourages "backdoor" strategies or reward hacks.
The best approach for an agent should be to make a good-faith effort to satisfy constraints as best as possible.
Experiment with sampling strategies to ensure that tasks are solvable most of the time (so that we can pre-filter any unsolvable tasks cheaply), and that they aren't too easy -- there shouldn't be an abundance of valid solutions, random proposal times should be a bad strategy.

Types of constraints we want to potentially account for:

- Conflicting schedules
- Time zones + early/late/day preferences
- Meeting length
- Room availability
- Back-to-back meeting preferences
- Desired-but-optional attendees
- Other related constraints which reflect real-world calendar challenges

Degrees of freedom:

- Number of attendees
- Window of consideration
- Types of constraints
- Tightness of constraints

Use the StatefulToolEnv pattern, and in-memory data structures for the calendar + attendee information. The agent should have tools for things like:

- Checking attendee calendars
- Viewing attendee constraints
- Checking score of a proposed window
- Submitting a window

The environment should have a max_turns parameter, and tool results should show the remaining turns to the agent.
Default limit should be enough to allow reasonable exploration, but not so high that the agent can brute-force search all times.

We should also have a nice standalone script in the environment which creates a TUI to visualize a "calendar problem" similar to typical meeting apps, including attendees, timeblocks, and constraints, but fully in the terminal, using Rich styling.

Create a detailed design doc and plan for testing (PLAN.md), implement in full, revise PLAN.md after major milestones to reflect accomplishments and updated TODOs, and run basic small evals throughout as needed.
You are welcome to use the PRIME_API_KEY set in my environment for inference tests (see configs/endpoints.toml for models).
Let me know when you're happy with your implementation.
```

You can view the created environment in the Environments Hub:

> **[prime/calendar-scheduling](https://app.primeintellect.ai/dashboard/environments/prime/calendar-scheduling)**
>
> Environments Hub

We can use the visualizer script we asked our agent to make for viewing the environment task structure more directly:

```bash theme={null}
~/dev/demo (main*) uv run --project environments/calendar_scheduling calendar-scheduling-tui --show-oracle --difficulty medium --seed 5

Calendar Scheduling TUI

╭───────────────────────────────────────────────────── Calendar Scheduling Problem ─────────────────────────────────────────────────────╮
│ Task ID                   medium-209463                                                                                               │
│ Difficulty                medium                                                                                                      │
│ Seed                      209463                                                                                                      │
│ Window                    5 days                                                                                                      │
│ Candidate UTC hours       09:00 - 19:00                                                                                               │
│ Meeting duration          90 minutes                                                                                                  │
│ Score-check budget        6                                                                                                           │
│ Total candidates          90                                                                                                          │
│ Valid candidates          8                                                                                                           │
│ Oracle best score         0.7970                                                                                                      │
│ Random baseline           0.0630                                                                                                      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                                               Attendees
┏━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ ID         ┃ Name   ┃ Type     ┃ TZ    ┃ Weight ┃ Preferred Day      ┃ Preferred Local ┃ Hard Local ┃
┡━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ attendee_1 │ Dakota │ optional │ UTC-2 │ 0.073  │ Day 1 (Tue Jan 06) │ 11.7-17.5       │ none       │
│ attendee_2 │ Morgan │ optional │ UTC-7 │ 0.155  │ Day 1 (Tue Jan 06) │ 8.8-14.4        │ none       │
│ attendee_3 │ Elliot │ required │ UTC+5 │ 0.072  │ Day 4 (Fri Jan 09) │ 10.9-16.6       │ none       │
│ attendee_4 │ Avery  │ required │ UTC+0 │ 0.146  │ Day 0 (Mon Jan 05) │ 12.5-17.0       │ none       │
│ attendee_5 │ Parker │ required │ UTC-7 │ 0.180  │ Day 0 (Mon Jan 05) │ 10.7-16.4       │ 8.3-18.8   │
│ attendee_6 │ Robin  │ required │ UTC-6 │ 0.185  │ Day 3 (Thu Jan 08) │ 10.2-14.7       │ none       │
│ attendee_7 │ Reese  │ optional │ UTC-1 │ 0.189  │ Day 4 (Fri Jan 09) │ 9.3-14.2        │ none       │
└────────────┴────────┴──────────┴───────┴────────┴────────────────────┴─────────────────┴────────────┘
╭───────────────────────────────────────────────────────────── Day Labels ──────────────────────────────────────────────────────────────╮
│ Day 0 (Mon Jan 05) | Day 1 (Tue Jan 06) | Day 2 (Wed Jan 07) | Day 3 (Thu Jan 08) | Day 4 (Fri Jan 09)                                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                              Availability Timeline (X busy, . free, =/# best window overlay)
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Lane          ┃ Timeline                                                                                                 ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ attendee_1    │ ........X...........|X..XXXXX..XXX.===...|....................|..XXX...............|.........XXXXX...... │
│ attendee_2    │ ............XXXXX...|..............===...|....................|..XXX...............|XXXXX............... │
│ attendee_3    │ ....................|.X............===...|.........XXXX.......|......X.............|XXX..X.............. │
│ attendee_4    │ ....................|..............===...|.....X.........X....|...............XXXXX|.................... │
│ attendee_5    │ ................XX..|..............===...|.XXXX...XXXX........|....................|.....X.............. │
│ attendee_6    │ .XXX............XXXX|..............===...|....................|.........XXXXX......|.....XX............X │
│ attendee_7    │ .XX.................|..XX.......XXX###...|...............XXXX.|....................|............XX...... │
│ room_1 (room) │ ....................|..........XXX.===X..|....................|....................|.................... │
└───────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘
╭────────────────────────────────────────────────────────── Constraint Model ───────────────────────────────────────────────────────────╮
│ Hard constraints                                                                                                                      │
│ - Required attendees must be able to attend                                                                                           │
│ - Hard local-time bounds cannot be violated                                                                                           │
│ - Chosen room must be available                                                                                                       │
│ - Duration must match task duration exactly                                                                                           │
│                                                                                                                                       │
│ Soft utility penalties                                                                                                                │
│ - Early/late local-time penalties                                                                                                     │
│ - Day-preference distance penalties                                                                                                   │
│ - Back-to-back penalty near busy blocks                                                                                               │
│ - Optional attendee absence penalty                                                                                                   │
│                                                                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                  Oracle Best Windows
┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Rank ┃ Window                                       ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1    │ Day 1 (Tue Jan 06) 16:00-17:30 UTC in room_1 │
└──────┴──────────────────────────────────────────────┘
╭─────────────────────────────────────────────────────────── Oracle Summary ────────────────────────────────────────────────────────────╮
│ Best score: 0.7970                                                                                                                    │
│ Valid windows: 8 / 90                                                                                                                 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Legend: X busy, . free, = highlighted best free slot, # highlighted busy slot

```

## Hosted Training

After the environment is created, we prompt our agent to test performance more exhaustively, then to start a Hosted Training run using `Qwen/Qwen3-30B-A3B-Instruct-2507`, which is available via Hosted Training.

```text wrap=true theme={null}
test with GPT-4.1 models, make sure we're seeing proper rollouts that succeed with non-zero scores. those models should be able to solve; if they can't we have a bug somewhere or need clearer instructions in the env. Once you feel good about the env, make a RL config and start a training run with bs 128, 8 rollouts per group, for the 30b instruct qwen model.
```

Available models can be viewed with:

```bash theme={null}
prime train models
```

A config sets the model, the step and batch sizes, the environment to train on and its arguments, and optional `[wandb]` and `[eval]` blocks. For the current format and a worked example, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

Keys your environment needs go in a file you pass at dispatch rather than in the config:

```bash theme={null}
# secrets.env
OPENAI_API_KEY="${OPENAI_API_KEY}"
WANDB_API_KEY="${WANDB_API_KEY}"
```

Runs can be started by running:

```bash theme={null}
prime train configs/rl/calendar-scheduling.toml --env-file secrets.env --volume research
```

From the platform, you can view training curves, rollouts, configs, logs, and checkpoints.

![RL training curves](https://mintcdn.com/primeintellect/CIB0NGon5jBp6ynT/guides/images/rl-curves.png?fit=max\&auto=format\&n=CIB0NGon5jBp6ynT\&q=85\&s=2698a7c6b5ba477dc97e9c5abf1bdd8b)

![Rollout viewer](https://mintcdn.com/primeintellect/CIB0NGon5jBp6ynT/guides/images/rollout-viewer.png?fit=max\&auto=format\&n=CIB0NGon5jBp6ynT\&q=85\&s=1c8acfb3532aa01b1836e960b331fe39)

Training runs are shareable. You can view this example run here:

> **[calendar-sch--qwen3-30b-a3b-instru--wxhl0r](https://app.primeintellect.ai/training/shared/wxhl0r3gqmj6qxgpcg35m605)**
>
> Hosted Training

## Getting Your Model

This walkthrough was recorded against the shared LoRA service, which produced a deployable adapter at the end of every run. That service has been [retired](/hosted-training/shared-lora-retired), and adapter deployment on Prime Inference has since been removed.

For a Hosted Training run, add `[ckpt]`, dispatch with `--volume`, and retrieve the checkpoint with `prime volumes get` after the volume sync finishes. See [Checkpoint settings](/hosted-training/volumes#checkpoint-settings) for retention and cluster-specific requirements, and [Step 7 of the End-to-End Run](/hosted-training/end-to-end-run#step-7-review-results) for exporting weights to evaluate or serve.

## Trace storage and existing data

Review [Prime Traces pricing and workspace upload settings](/traces/pricing-and-settings) before launching runs. For viewing and deleting stored traces, see [Managing traces](/traces/managing-traces). Export older evaluation and training samples using the [legacy archive workflow](/traces/legacy-storage) before November 6, 2026.


# Training Search Agents
Source: https://docs.primeintellect.ai/guides/search-agents

Build three progressive RL environments for document search and run Hosted Training.

Step-by-step walkthrough for training a model to search and reason over documents using RL. The example uses patents, but the same architecture applies to any document domain: legal filings, SEC documents, medical literature, enterprise knowledge bases.

**Stack:** verifiers, ChromaDB, BigQuery, Hosted Training, Qwen3-4B, Llama-3.2-3B

***

## Environments Overview

| Level | Name | Reward type | What the agent does |
| - | - | - | - |
| Level 1 | Metadata Retrieval | Exact match | Single tool call, parse structured response |
| Level 2 | Multi-step Computation | Binary on deterministic answers | 2+ tool calls, arithmetic, cross-patent comparison |
| Level 3 | Open-ended Analysis | LLM judge, normalized | Full patent text reading, synthesis, cross-patent reasoning |

***

## Prerequisites

* Google Cloud account with BigQuery access. Free tier is sufficient; with that, you get 1 TB/month of queries. Downloading \~1,500 patents with full text uses \~3 GB.
* A Prime Intellect account. You'll push environments to the Environments Hub and launch training runs from the CLI.
* OpenAI API key, which is used for generating Level 3 Q\&A ground truth and the LLM judge at training time.
* Python 3.10+ with `verifiers`, `chromadb`, `openai`, and the `prime` CLI. Install the Prime CLI with `uv tool install prime`.

***

## Step 1 - Get Patent Data

Use Google's `patents-public-data` BigQuery dataset. It has full patent text (abstract, claims, description) and supports SQL filtering by company.

### BigQuery query

```sql theme={null}
-- fetch_patents.sql
SELECT
  publication_number,
  title,
  abstract,
  claims,
  description,
  filing_date,
  grant_date,
  ARRAY_LENGTH(claims_localized) AS claim_count
FROM
  `patents-public-data.patents.publications`
WHERE
  assignee_harmonized.name IN (
    'QUALCOMM INCORPORATED',
    'ERICSSON',
    'NOKIA', 
    'SAMSUNG',
    'HUAWEI',
    'SONY',
  )
  AND country_code = 'US'
  AND grant_date IS NOT NULL
LIMIT 1500
```

Export the results to a GCS bucket or download directly as JSON. You can start with just \~1,500 patents; it produces meaningful training signal without excessive embedding costs.

### Format each patent

Structure each patent as a markdown document and store metadata separately. Preserving document structure matters for Level 3 because the agent needs to navigate named sections.

```python theme={null}
# format_patents.py
def format_patent(row):
    doc = f"""# {row['title']}

## Abstract
{row['abstract']}

## Claims
{row['claims']}

## Description
{row['description']}
"""
    metadata = {
        "patent_id": row["publication_number"],
        "title": row["title"],
        "filing_date": row["filing_date"],
        "grant_date": row["grant_date"],
        "claim_count": row["claim_count"],
    }
    return doc, metadata
```

***

## Step 2 - Design Three RL Environments

The three levels impose a curriculum. For example, an agent that can't reliably call `get_metadata()` and parse a date string shouldn't attempt multi-patent technical comparisons. Each level introduces qualitatively harder capabilities.

### Level 1 - Single-tool metadata retrieval

Questions that require exactly one tool call with no computation or reasoning

| Question type | Example |
| - | - |
| Filing date | "When was patent X filed?" |
| Grant date | "When was patent X granted?" |
| Claim count | "How many claims does patent X have?" |
| Title | "What is the title of patent X?" |

**Dataset size:** 6,000 Q\&A pairs. Every answer is deterministically verifiable from the metadata.
**Reward:** Binary - 1.0 for exact string match, 0.0 otherwise.
**Tools available:** `search_patents(query)`, `get_metadata(patent_id)`

Level 1 is a pipeline validation step. If reward doesn't climb toward 1.0 within 15 steps, something is wrong with your tool definitions, reward function, or data formatting. It isn't a model problem.

### Level 2 - Multi-step computation and comparison

Questions requiring 2 or more tool calls followed by arithmetic or string comparison.

| Question type | Example | What the agent does |
| - | - | - |
| Days to grant | "How many days between filing and grant for patent X?" | `get_metadata()` then parse two dates and subtract |
| Filed first | "Which was filed first: X or Y?" | `get_metadata()` twice, then compare |
| Claim difference | "How many more claims does X have than Y?" | `get_metadata()` twice, then subtract |
| Search + retrieve | "Find a patent about SSB and tell me its filing date" | `search_patents()` then `get_metadata()` |
| Abstract content | "Does the abstract of patent X mention '5G'?" | `get_abstract()` then string check |

**Dataset size:** 500 Q\&A pairs.
**Reward:** Binary on deterministic answers. For search questions with multiple valid answers, the reward function checks against a precomputed set of valid answers.
**New tool:** `get_abstract(patent_id)` returns abstract text.

The environment does not do the math. The agent receives raw dates and counts from tool calls and must compute the answer itself. This is intentional to train the reasoning.

### Level 3 - Open-ended technical analysis

Questions that require reading patent content, understanding technical concepts, and synthesizing answers that can't be verified by pure string matching.

| Question type | Example |
| - | - |
| Technical summary | "Summarize the key technical innovation in patent X in 2-3 sentences" |
| Problem identification | "What problem does patent X solve?" |
| Cross-patent comparison | "What is the key technical difference between patent X and Y?" |
| Standards identification | "What wireless communication standards are referenced in patent X?" |
| Claim feature extraction | "What are the key limiting features in claim 1 of patent X?" |
| Abstract-claim mapping | "Which claims are most directly described in the abstract?" |

**Dataset size:** 500 Q\&A pairs (LLM-generated with ground truth).
**Reward:** LLM judge, normalized.
**New tools:** `view_sections(patent_id)`, `read_section(patent_id, section_name)`

***

## Step 3 - Generate Verifiable Q\&A Pairs

### Levels 1 and 2

Iterate over the patent dataset, select random patents (or pairs for comparison questions), compute the answer directly from structured data, and generate the pair. Every answer is verified against the source data.

For search-type questions, precompute all valid answers across the entire dataset. For example, if and question asks to identify "SSB" patents and three patents mention "SSB" in their title or abstract, all three filing dates are valid answers. Store this set, and the reward function will check against it at training time.

```python theme={null}
# generate_qa.py
import random
from datetime import datetime

def generate_date_question(patents: list[dict]) -> dict:
    patent = random.choice(patents)
    return {
        "question": f"When was patent {patent['patent_id']} filed?",
        "answer": patent["filing_date"],  # e.g. "2019-03-15"
        "patent_ids": [patent["patent_id"]],
        "level": 1,
        "type": "filing_date",
    }

def generate_days_to_grant(patents: list[dict]) -> dict:
    patent = random.choice([p for p in patents if p["grant_date"]])
    filed = datetime.strptime(patent["filing_date"], "%Y-%m-%d")
    granted = datetime.strptime(patent["grant_date"], "%Y-%m-%d")
    days = (granted - filed).days
    return {
        "question": f"How many days elapsed between the filing and grant of patent {patent['patent_id']}?",
        "answer": str(days),
        "patent_ids": [patent["patent_id"]],
        "level": 2,
        "type": "days_to_grant",
    }
```

### Level 3 - LLM-generated ground truth

Each Level 3 entry needs a structured reference that the judge can use at training time. Use the LLM to generate three components per question:

* **answer** - the reference answer in writing
* **key\_points** - specific factual claims the answer must contain
* **source\_quotes** - direct quotes from the patent text supporting each key point

```python theme={null}
# generate_l3_qa.py
import json
from openai import OpenAI

client = OpenAI()

GENERATION_PROMPT = """You are generating ground truth for a patent analysis training dataset.

Patent text:
{patent_text}

Question: {question}

Return ONLY valid JSON with no preamble:
{{
    "answer": "prose answer to the question",
    "key_points": ["specific factual claim 1", "specific factual claim 2"],
    "source_quotes": ["exact quote from patent supporting point 1", "exact quote supporting point 2"]
}}"""

def generate_l3_ground_truth(patent_text: str, question: str) -> dict:
    response = client.chat.completions.create(
        model="model",
        messages=[{
            "role": "user",
            "content": GENERATION_PROMPT.format(
                patent_text=patent_text,
                question=question
            )
        }],
        temperature=0
    )
    return json.loads(response.choices[0].message.content)
```

### Validate your dataset

Run these checks on every generated pair before training:

* Every patent ID referenced in a question exists in the dataset
* Every Level 3 ground truth has at least one key point
* Source quotes from Level 3 ground truth actually appear in the patent text (substring match)

Manually review a stratified sample across all question types. This surfaces systematic prompt issues that automated checks miss such as questions conflating problem and solution, or rubric with contradictions, or answers with technical inaccuracies.

***

## Step 4 - Reward Design

### Levels 1 and 2

Level 1 and 2 reward designs were straightforward.

```python theme={null}
# rewards.py
def reward_exact(agent_answer: str, ground_truth: str) -> float:
    """Binary reward for single-answer questions."""
    return 1.0 if normalize(agent_answer) == normalize(ground_truth) else 0.0

def reward_set(agent_answer: str, valid_set: set[str]) -> float:
    """For search questions with multiple valid answers."""
    return 1.0 if normalize(agent_answer) in valid_set else 0.0

def normalize(s: str) -> str:
    return s.strip().lower().replace(",", "").replace(".", "")
```

### Level 3 - LLM judge

Getting the judge right required three iterations. Here's what failed and what ended up working.

**Iteration 1 - didn't work:** Multi-dimensional weighted scoring (accuracy, completeness, reasoning, conciseness, 0-10 each). Failed because regex parsing of scores was fragile, and per-category weight tuning is a second optimization problem on top of the first.

**Iteration 2 - didn't work:** Per-question custom rubrics (5 criteria x 2 points each, LLM-generated per question). More principled, but LLM-generated rubrics introduced contradictions like asking for content that would actually weaken an otherwise correct answer.

**Iteration 3 - works:** Universal rubric with content-specific ground truth. The ground truth (answer, key\_points, source\_quotes) already provides all content specificity needed.

| Criterion | Points | What it catches |
| - | - | - |
| Factually accurate relative to patent | 3 | Wrong technical details |
| Free of hallucinated information | 3 | Made-up claims, features, standards |
| Covers key points from ground truth | 2 | Missing important content |
| Directly answers the question asked | 1 | Off-topic or evasive responses |
| References specific patent content | 1 | Unsupported assertions |

The judge receives the question, the agent's response, the reference answer, key points, and source quotes. It returns structured JSON, normalized to \[0, 1]:

```json theme={null}
// judge_output_schema.json
{
    "key_points": [
        {"point": "SSB spatial overloading concept", "covered": true},
        {"point": "Spatially separated beams", "covered": true},
        {"point": "Reduced bandwidth overhead", "covered": false}
    ],
    "hallucination": false,
    "factual_error": false,
    "final_score": 7
}
```

```python theme={null}
# rewards.py
def reward_l3(judge_output: dict) -> float:
    score = judge_output["final_score"] / 10.0  # normalize to [0, 1]
    if judge_output.get("hallucination"):
        score *= 0.2  # hallucination is catastrophic in patent context
    elif judge_output.get("factual_error"):
        score *= 0.5
    return score
```

Note on the 0.2 multiplier: zeroing out hallucinated responses entirely creates a sharp gradient that can cause instability. A 0.2 multiplier still strongly penalizes hallucination while providing a small gradient signal. In a patent context, hallucinated technical claims have direct commercial and legal consequences so we err on the side of undertrained vs. positively trained on hallucinations.

***

## Step 5 - Build the Tool Environment

The environment is implemented using the [verifiers](https://github.com/willccbb/verifiers) library as a `ToolEnv`. The agent receives a system prompt describing its role as a patent analyst with access to a dataset of N patents, a question, and a set of tools.

### Tool definitions

```python theme={null}
# patent_env.py
import chromadb
from verifiers import ToolEnv

class PatentEnv(ToolEnv):
    def __init__(self, patents: list[dict], level: int):
        self.corpus = {p["patent_id"]: p for p in patents}
        self.level = level
        self._init_vector_store(patents)
        super().__init__(tools=self._get_tools(level))

    def _init_vector_store(self, patents):
        self.chroma = chromadb.Client()
        collection = self.chroma.create_collection("patents")
        collection.add(
            ids=[p["patent_id"] for p in patents],
            documents=[f"{p['title']} {p['abstract']}" for p in patents],
        )
        self.collection = collection

    def search_patents(self, query: str) -> list[dict]:
        """Returns patent IDs and titles matching a semantic query."""
        results = self.collection.query(query_texts=[query], n_results=5)
        return [
            {"patent_id": id_, "title": self.corpus[id_]["title"]}
            for id_ in results["ids"][0]
        ]

    def get_metadata(self, patent_id: str) -> dict:
        """Returns title, filing_date, grant_date, claim_count."""
        p = self.corpus[patent_id]
        return {
            "title": p["title"],
            "filing_date": p["filing_date"],
            "grant_date": p["grant_date"],
            "claim_count": p["claim_count"],
        }

    def get_abstract(self, patent_id: str) -> str:
        return self.corpus[patent_id]["abstract"]

    def view_sections(self, patent_id: str) -> list[str]:
        """Lists available section names in the full patent text."""
        doc = self.corpus[patent_id]["full_text"]
        return [
            line.lstrip("# ").strip()
            for line in doc.split("\n")
            if line.startswith("## ")
        ]

    def read_section(self, patent_id: str, section_name: str) -> str:
        """Returns the full text of a named section."""
        doc = self.corpus[patent_id]["full_text"]
        start = doc.find(f"## {section_name}")
        if start == -1:
            return f"Section '{section_name}' not found."
        end = doc.find("\n## ", start + 1)
        return doc[start:end if end != -1 else len(doc)]

    def _get_tools(self, level: int):
        tools = [self.search_patents, self.get_metadata]
        if level >= 2:
            tools.append(self.get_abstract)
        if level >= 3:
            tools += [self.view_sections, self.read_section]
        return tools
```

Keep tools stateless. The agent can call `get_metadata()` on the same patent multiple times with no side effects.

### Schema to tools generalization

Patents have standardized sections (Abstract, Claims, Description) and coded metadata. This makes them well-suited for tool-based environments. The same structure applies to any document domain with a known schema such as legal filings map to `get_case_metadata()`, SEC filings to `read_section("Risk Factors")`, and so on.

***

## Step 6 - Parsing Strategy

Avoid regex in the pipeline because when not implemented well, it can introduce reward hacking. Use simple string methods and structured JSON instead:

```python theme={null}
# parsing.py
import json

def extract_claims_text(full_text: str) -> str:
    """Extract the Claims section using string slicing, not regex."""
    start = full_text.find("## Claims")
    if start == -1:
        return ""
    end = full_text.find("\n## ", start + 1)
    return full_text[start:end if end != -1 else len(full_text)]

def parse_patent_id(full_code: str) -> str:
    """'US10123456B2' -> split and handle deterministically."""
    return full_code.split("-")[0] if "-" in full_code else full_code

def parse_judge_output(raw: str) -> dict:
    """Parse judge JSON with a robust fallback."""
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        # Fallback: find final_score by character iteration
        key = '"final_score":'
        idx = raw.find(key)
        if idx == -1:
            return {"final_score": 0, "hallucination": True, "key_points": []}
        score_start = idx + len(key)
        digits = ""
        for ch in raw[score_start:].lstrip():
            if ch.isdigit():
                digits += ch
            elif digits:
                break
        return {
            "final_score": int(digits) if digits else 0,
            "hallucination": "hallucin" in raw.lower(),
            "key_points": [],
        }
```

***

## Step 7 - Train on Prime Intellect

[Hosted Training](/hosted-training/getting-started) handles GPU orchestration for you. You push your environment to the Environments Hub, define a training config, and launch the run from the CLI.

### Step 7.1 - Push environment to the Hub

```bash theme={null}
# Push each level as a separate named environment
prime env push --name basic-patent-q-and-a
prime env push --name advanced-patent-q-and-a
prime env push --name patent-technical-analysis
```

### Step 7.2 - Define a training config

**Suggested starting point:** a 3B-class instruct model, `max_steps = 100`, `batch_size = 128`, `group_size = 8`, `max_completion_tokens = 1024`, with one `[[orchestrator.train.source]]` pointing at your pushed environment. Pass `OPENAI_API_KEY` at dispatch with `-e` so the Level 3 judge can run. For the config format, see [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config).

### Step 7.3 - Calibrate model to environment difficulty

Before committing to a full training run, verify the base model finds the task challenging but not impossible. A model that starts too high has nothing to learn; one that starts too low can't generate useful gradient signal. Target starting reward is around 0.15 to 0.35.

Do a short 10-step run first and read the reward curve. It plots mean reward per step across all rollouts in the batch. If starting reward is outside that range, adjust difficulty or model size before launching the full run.

```bash theme={null}
# Quick calibration: 10 steps to measure starting reward
prime train config.toml \
  --env-var OPENAI_API_KEY=sk-... \
  --override max_steps=10
```

If starting reward is above 0.7, make the questions harder or use a smaller base model. If it's below 0.1, the questions may be too hard or the tools too opaque.

### Step 7.4 - Launch the full run

```bash theme={null}
prime train config.toml --env-var OPENAI_API_KEY=sk-...
```

You can pass secrets at runtime with `--env-var`, or set them in the environment settings — both work.

Once the run is live, the dashboard shows per-step metrics including mean reward, reward standard deviation, response length, and tool call count. Tool call count is worth checking early. If the agent isn't calling tools in the first few steps, the system prompt isn't clear enough about what tools are available and when to use them. Below are some screenshots of what the Prime Intellect dashboard looks like.

![Reward and metrics dashboard](https://mintcdn.com/primeintellect/WdmRRTmBxbiqzYNi/guides/reward_metrics.png?fit=max\&auto=format\&n=WdmRRTmBxbiqzYNi\&q=85\&s=ed734ceaecb64f3bad4b979a460b19e0)

![Rollout viewer](https://mintcdn.com/primeintellect/WdmRRTmBxbiqzYNi/guides/image__1_.png?fit=max\&auto=format\&n=WdmRRTmBxbiqzYNi\&q=85\&s=464adcfa91f15d4c6730b5c709bfda3d)

### Config parameters to tune

| Parameter | What it controls | Starting point |
| - | - | - |
| `max_steps` | Training duration | 50 for L1, 100 for L2/L3 |
| `batch_size` | Examples per step | 128 |
| `rollouts_per_example` | Trajectories sampled per question | 8, increase for noisy L3 rewards |
| `max_tokens` | Max agent response length | 1024, L3 may need more |

***

## Results

Trained Qwen3-4B-Instruct and Llama-3.2-3B-Instruct across all three levels. The rollout viewer lets you inspect individual trajectories turn by turn: each tool call, its response, and the final answer. It's the most direct way to see what the model is actually learning to do at each level and understand in depth why the reward curve is behaving as it is.

### Level 1

| Checkpoint | Reward |
| - | - |
| Start | \~0.05 |
| Step 15 | \~1.0 |

![Level 1 reward curve](https://mintcdn.com/primeintellect/WdmRRTmBxbiqzYNi/guides/patent_level_1_updated.png?fit=max\&auto=format\&n=WdmRRTmBxbiqzYNi\&q=85\&s=0bac94acae279a06122ea414911c5da7)

Saturates within 15 steps and stays there. Confirms the pipeline works end-to-end. Once the agent learns the pattern of calling `get_metadata()` and reading the response, it gets nearly everything right.

### Level 2

| Checkpoint | Reward |
| - | - |
| Start | \~0.25 |
| Step 50 | \~0.70 |

![Level 2 reward curve](https://mintcdn.com/primeintellect/WdmRRTmBxbiqzYNi/guides/patent_level_2.png?fit=max\&auto=format\&n=WdmRRTmBxbiqzYNi\&q=85\&s=ddbcdf73d30999372d86b522efb4c14d)

Clear upward trend with variance throughout. Some batches land on straightforward date subtraction, others require chaining a search into a metadata call and then doing a comparison. The curve hasn't plateaued at step 50, so longer runs will get more performance.

### Level 3

| Checkpoint | Reward |
| - | - |
| Start | \~0.30 |
| Step 100 | \~0.50 |

![Level 3 reward curve](https://mintcdn.com/primeintellect/WdmRRTmBxbiqzYNi/guides/patent_level_3.png?fit=max\&auto=format\&n=WdmRRTmBxbiqzYNi\&q=85\&s=c5f043b2b6308ab3f6f2e5359e9b3c4d)

Noisy, as expected from LLM-judged reward. Individual batches swing between 0.15 and 0.9. The upward trend is real but needs larger batches and more steps to converge. The noise has two sources: judge subjectivity, and question difficulty variation (ex: technical summary vs. cross-patent comparison are not the same task and vary in difficulty).

***

## Extending This to Other Domains

The core architecture (schema-derived tools, progressive difficulty levels, universal rubric with content-specific ground truth) is not patent-specific. To adapt it to the following categories, you can:

* **Legal case search:** Replace `read_section("Claims")` with `read_section("Holding")`. The judgment or ruling is your Level 3 answer target.
* **SEC filings:** 10-K documents have standardized sections (Risk Factors, MD\&A, Financial Statements).
* **Medical literature:** PubMed abstracts for Level 1, full-text PMC articles for Level 3. Use MeSH terms for structured metadata.
* **Enterprise knowledge bases:** Internal docs with known schemas. Level 3 judge needs domain-appropriate ground truth generation.

The main work in each new domain is: (1) acquiring the data with full text, (2) defining the question types that capture the actual analytic tasks, and (3) writing the Level 3 ground truth generation prompt with domain-specific few-shot examples.

***

All environments are published on the [Environments Hub](https://hub.primeintellect.ai/). The patent dataset is on [HuggingFace](https://huggingface.co/datasets/jessicafeiyali/wirelesspatents).


# Billing
Source: https://docs.primeintellect.ai/hosted-training/billing

How on-demand Hosted Training runs are metered and billed

On-demand Hosted Training is billed **per GPU-hour**, for the GPUs your run occupies.

## Where to find rates

Rates are set per GPU type and change over time, so they live in the CLI rather than in these docs:

```bash theme={null}
prime train models
```

The **On-Demand Capacity** table lists each purchasable GPU type with its current `$/GPU-hour`. A GPU type with no active rate isn't sold on-demand and won't appear.

When you dispatch, the CLI echoes the GPU type it resolved and the rate it will bill at, so the cost is visible before the run starts rather than first appearing on an invoice:

```bash theme={null}
prime train configs/rl.toml
```

> [!NOTE]
>
> The rate is snapshotted onto your run at dispatch. If rates change while your run is in flight, your run keeps the rate it started with.

## What gets billed

| | |
| - | - |
| **Meter starts** | When your pods are **Ready** |
| **Meter stops** | When the run reaches a terminal state or is [paused](/hosted-training/pause-resume) |
| **Granularity** | Prorated to the second, no minimum |

Queueing, scheduling, and image pull are **not** billed. A cold node pulling a multi-gigabyte prime-rl image can take many minutes — that time is on us.

## Dispatch requirements

A run is admitted against your wallet, so before dispatch you need:

* **On-demand enabled** on your account or team. On-demand training is in beta, so [request access](mailto:contact@primeintellect.ai) if it is not enabled yet
* **Spendable balance** — your balance plus any overdraft allowance and training credit grants
* **A run within your per-run GPU cap**

Credits and discounts are applied as grants against your wallet, and grants are consumed before your balance.

## Volume storage

[Volumes](/hosted-training/volumes) are billed separately from GPU-hours: **\$0.09 per GiB per month** (\$0.000125 per GiB-hour) on the volume's size, from creation until you delete it, whether or not a run is using it. See [Volume pricing](/hosted-training/volumes#pricing).

## Capacity and queueing

If the on-demand pool has no headroom, your run **queues** rather than being rejected, and is promoted automatically as capacity frees. Your wallet is re-checked at promotion.

`prime train models` shows a **Capacity** column per GPU type, either available or busy. That is a coarse hint about pool headroom, not an admission decision: whether your run starts immediately or queues also depends on how many GPUs you ask for and their topology, which the discovery endpoint does not know.

> [!NOTE]
>
> If you already hold reserved capacity for a matching GPU type, your run is placed there instead of on-demand — that hardware is already paid for.

> **[Models](/hosted-training/models)**
>
> Which models and GPU types you can dispatch on.

> **[Hosted RL](/hosted-training/hosted-rl)**
>
> Config schema, dispatch knobs, and run lifecycle.


# End-to-End Training Run
Source: https://docs.primeintellect.ai/hosted-training/end-to-end-run

Walk through a complete on-demand Hosted Training run from environment setup to results

This guide walks you through a complete Hosted Training run — choosing an environment, establishing a baseline, writing a config, dispatching the run, monitoring it, and reviewing results.

For the config schema and dispatch options in reference form, see [Hosted RL](/hosted-training/hosted-rl).

## Prerequisites

```bash theme={null}
# Install and authenticate the CLI
uv tool install -U prime
prime login
```

You also need on-demand training enabled and a funded wallet — see [Billing](/hosted-training/billing). Start from [Getting Started](/hosted-training/getting-started) if you haven't run anything yet.

Create a volume to hold the trained weights. Checkpointing is off by default and pods are torn down when a run ends, so without a volume and a `[ckpt]` block the run leaves you nothing to evaluate:

```bash theme={null}
prime volumes create research --size 100Gi
```

## Step 1: Choose an Environment

You can use an existing environment from the [Environments Hub](https://app.primeintellect.ai/dashboard/environments) or create your own. For this walkthrough we'll use `alphabet-sort` — a multi-turn game where the model must sort letters into alphabetical order. If you're new to how environments work, see [The Environment Model](/hosted-training/environment-model).

## Step 2: Run a Baseline Evaluation

Before training, evaluate the base model to establish a baseline. This confirms the environment works and shows you where the model starts:

```bash theme={null}
prime eval run primeintellect/alphabet-sort \
  -m Qwen/Qwen3.5-2B \
  -n 20 -r 1 --hosted
```

> [!TIP]
>
> A good training environment should have a baseline reward between roughly 10–80%. If the model scores 0% after many attempts, the task is too hard. If it's already at 80%+, consider harder examples or a different environment.

The evaluation runs on the platform, so there's nothing to install locally. The command prints an evaluation ID and a dashboard link where the results appear. To stream its logs:

```bash theme={null}
prime eval logs <evaluation-id> -f
```

See [Evaluating environments](/tutorials-environments/evaluating) for the full set of hosted evaluation options.

## Step 3: Choose a Model and GPU Type

```bash theme={null}
prime train models
```

This lists the models you can dispatch, plus an **On-Demand Capacity** table of purchasable GPU types with their current `$/GPU-hour` and whether a run starts now or queues. Runs boot from a cluster-side weight cache, so pick a model from that list — see [Models](/hosted-training/models).

For a first run, pick the smallest model that exercises your environment end to end. Validate cheaply, then scale.

## Step 4: Create a Training Config

Training runs are configured with a [prime-rl](/prime-rl/configuration) `.toml` file. The `[deployment]` block sizes the run and is what routes it onto Hosted Training.

```toml theme={null}
# configs/alphabet-sort.toml
name = "alphabet-sort-rl"
max_steps = 50
seq_len = 2048

[ckpt]                                 # save step 50 to the volume
interval = 50
keep_last = 1

[model]
name = "Qwen/Qwen3-4B-Instruct-2507"   # cached models in `prime train models` start warm

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[trainer.optim]
lr = 1e-6

[orchestrator]
batch_size = 128

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.source]]
name = "alphabet-sort"

[orchestrator.train.source.env.taskset]
id = "alphabet-sort"

[orchestrator.train.source.env.agent.harness]
id = "null"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

[inference]
```

The key fields:

* `[model].name` — a Hugging Face model ID or an absolute `/volume/...` path to [weights on your volume](/hosted-training/volume-models). Cached IDs start warm; uncached IDs download at startup on on-demand capacity
* `[ckpt]` — save checkpoints. Off by default; with no `[ckpt]` block nothing is written
* `[deployment]` — how many trainer and inference GPUs to reserve; required for dispatch
* `max_steps` — total training steps
* `[orchestrator].batch_size` — rollouts consumed per training batch
* `[orchestrator.train].group_size` — rollouts generated per dataset example
* `[orchestrator.train.sampling].max_completion_tokens` — max tokens the model generates per turn
* `[[orchestrator.train.source]]` — one block per training environment, carrying a nested `env` block

This config uses the native `alphabet-sort` taskset bundled with the hosted runtime and the `null` harness. The Hub package used for the baseline evaluation is a separate package; its legacy API is incompatible with the current hosted runtime. For other environments, use a runtime-compatible taskset and harness.

## Trace storage and costs

Traces uploaded by platform runs incur [Prime Traces ingestion and storage charges](/traces/pricing-and-settings), separately from training charges. Review workspace upload controls before launching. See [Viewing and deleting traces](/traces/managing-traces) to manage the resulting data.

## Step 5: Launch the Training Run

```bash theme={null}
prime train configs/alphabet-sort.toml --volume research
```

The CLI confirms the configuration, echoes the GPU type and hourly rate it resolved, and prints a dashboard link:

```
Dispatched hosted run wn2cjdrzdo6bmfqajoeuu30p

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/wn2cjdrzdo6bmfqajoeuu30p
```

Pass credentials your environment needs with `-e / --env-var` or `--env-file`:

```bash theme={null}
prime train configs/alphabet-sort.toml -e WANDB_API_KEY -e OPENAI_API_KEY
```

Config errors — bad schema, an uncached model, an oversized deployment — fail the dispatch immediately with a reason, before any GPU is reserved and before anything bills.

## Step 6: Monitor the Run

**In the terminal** — stream logs in real time:

```bash theme={null}
prime train logs <run-id> -f
```

A run has several components; pick one with `-c / --component`:

```bash theme={null}
prime train logs <run-id>                  # orchestrator (default)
prime train logs <run-id> -c trainer       # trainer (FSDP / torchrun)
prime train logs <run-id> -c inference     # vLLM inference server
prime train logs <run-id> --env <env-name> # env-server for a specific env
```

Filter with `--search`, `--regex`, `--level`, and `--since`.

**On the dashboard** — open the URL printed when the run started, for reward curves, rubric scores, reward distributions, and individual rollouts.

Key metrics to watch:

* **Reward** — the overall reward curve should trend upward over time
* **Rubric** — individual rubric component scores
* **Reward Distribution** — should shift from lower to higher values as training progresses

Stop a run early to release its GPUs and stop the meter:

```bash theme={null}
prime train stop <run-id>
```

To release the GPUs but continue from the current step later, [pause](/hosted-training/pause-resume) the run instead.

## Step 7: Review Results

### Retrieve the checkpoint

The RL run's checkpoints, logs, and other outputs are saved to the [volume](/hosted-training/volumes) when the run ends. Wait for the volume sync to finish, then download the run directory:

```bash theme={null}
prime volumes get research runs/<run-id>/ ./run-dir
```

### Export to HF format

Trainer checkpoints are DCP-sharded, so they need converting before anything else can load them. The converter is `tools/convert_dcp_to_bf16.py` in the [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) repo. It needs:

* **At least one GPU.**
* **The run's resolved config.** The converter reads model and tokenizer settings from `<run-dir>/configs/latest/resolved/trainer.json` for RL or `sft.json` for SFT. Hosted runs currently do not save this config with the checkpoints. Obtain it from Prime support and place it at the required path before converting. The launch TOML is not a substitute for the resolved JSON config.
* **Access to the base model and tokenizer.** If the resolved config uses `/volume/...` paths, make those directories available on the machine doing the conversion.

Run the converter from a prime-rl checkout on a machine with a GPU:

```bash theme={null}
uv run python tools/convert_dcp_to_bf16.py ./run-dir/checkpoints/step_50

# multi-rank, for faster gathers and models too big for one GPU
uv run torchrun --nproc-per-node 8 tools/convert_dcp_to_bf16.py ./run-dir/checkpoints/step_50
```

This writes sharded safetensors plus config and tokenizer assets to `./run-dir/checkpoints/step_50/weights`. That directory loads into an HF consumer. See [Exporting Checkpoints](/prime-rl/training#exporting-checkpoints) for FP8 variants.

> [!NOTE]
>
> Hosted SFT runs save checkpoints rather than the whole run directory. Arrange the downloaded checkpoint under `run-dir/checkpoints/step_<step>` and obtain the resolved config before running the converter.

### Evaluate against your baseline

Serve the exported weights, then point the Step 2 evaluation at them. This evaluation runs locally, so install the environment first:

```bash theme={null}
# install the environment locally
prime env install primeintellect/alphabet-sort

# serve the exported directory
uv run inference --vllm.model <run-dir>/checkpoints/step_<n>/weights

# evaluate the same environment, same sample size as Step 2
uv run vf-eval primeintellect/alphabet-sort \
  -m <run-dir>/checkpoints/step_<n>/weights \
  -b http://localhost:8000/v1 -k OPENAI_API_KEY \
  -n 20 -r 1 -s
```

`-b` points `vf-eval` at your endpoint and `-k` names the env var holding its key. Compare the reward against the baseline you recorded in Step 2.

> [!NOTE]
>
> Metrics, samples, and progress are persisted to the dashboard either way, but trainer, inference, and env-server pod logs go away with the pods when the run is torn down — pull anything you need while the run is live.

## Run Size Guidelines

Each of these omits the blocks already covered above. Add `[ckpt]` and dispatch with `--volume` to keep weights. See [Checkpoint settings](/hosted-training/volumes#checkpoint-settings) for retention settings.

### Small (validation)

Verify your environment and config work before committing to a longer run.

```toml theme={null}
max_steps = 50
seq_len = 2048

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[orchestrator]
batch_size = 128

[orchestrator.train]
group_size = 8
```

### Medium (experimentation)

Good for iterating on environment design and hyperparameters.

```toml theme={null}
max_steps = 200
seq_len = 4096

[deployment]
num_train_gpus = 2
num_infer_gpus = 2

[orchestrator]
batch_size = 256

[orchestrator.train]
group_size = 16

[wandb]
project = "my-experiment"
```

### Large (production)

```toml theme={null}
max_steps = 500
seq_len = 32768

[deployment]
type = "multi_node"
num_train_nodes = 2
num_infer_nodes = 2

[trainer.model]
attn = "flash_attention_3"

[orchestrator]
batch_size = 512

[orchestrator.train]
group_size = 16

[wandb]
project = "production"
```

> [!NOTE]
>
> `WANDB_API_KEY` is required for the `[wandb]` block to take effect — without it the block is stripped and the run trains without W\&B logging.

> **[Hosted RL](/hosted-training/hosted-rl)**
>
> Config reference, dispatch knobs, limits, and run lifecycle.

> **[Billing](/hosted-training/billing)**
>
> How GPU-hours are metered and what gets billed.

> **[Troubleshooting](/hosted-training/troubleshooting)**
>
> Solutions for common issues with Hosted Training runs.


# The Environment Model
Source: https://docs.primeintellect.ai/hosted-training/environment-model

Understand environment types, rubrics, and the RL training loop

## The Training Loop

When you launch an RL training run, three decoupled components coordinate:

* **Inference** generates model completions turn-by-turn via a vLLM server.
* **Orchestrator** samples prompts from the environment's dataset, dispatches rollouts to inference, drives the multi-turn interaction loop (calling the environment's response logic and tool execution between each model turn), scores completed rollouts via the rubric, packs training batches, and relays weight updates between trainer and inference.
* **Trainer** receives scored rollouts, computes advantages, and produces weight updates via GRPO.

One step of training:

```
Orchestrator samples prompts from the environment
  -> Inference generates a rollout (multi-turn, driven by env logic)
  -> Environment scores the rollout via its rubric
  -> Orchestrator packs the batch and computes advantages
  -> Trainer updates policy weights
  -> Orchestrator broadcasts new weights to inference
  -> Repeat
```

The full loop is asynchronous. Rollout generation at step N overlaps with training at step N-1, and a single trajectory may span multiple policy versions as weights update mid-rollout.

**The environment is the only part you write.** Everything else (batching, weight sync, GPU scheduling, async rollout management) is handled by the infrastructure. An environment plugs directly into this loop, whether you run it via Hosted Training or self-hosted with `prime-rl`.

***

## What is an Environment?

An environment is a self-contained Python module that packages three things:

1. **A dataset** of prompts (with optional ground-truth answers or metadata).
2. **A harness** that controls how the model interacts with the task: single-turn Q\&A, multi-turn tool calling, stateful sandbox sessions, etc.
3. **A rubric** of reward functions that score the model's output and produce scalar signals.

The same environment definition drives RL training, standalone evaluation (`vf-eval` locally, or `prime eval run --hosted`), and synthetic data generation. RL environments and agent evals are the same abstraction: dataset + harness + scoring rules.

Environments are distributed as versioned Python wheels declared in `pyproject.toml`, installable and shareable via the Environments Hub.

### Legacy and Verifiers v1 environment configs

Verifiers ships two packaging/configuration styles, and Hosted Training supports both:

* **Legacy (Verifiers v0) environments** expose a `load_environment(**args)` entrypoint.
* **Verifiers v1 (taskset/harness) environments** split task data and scoring from model-facing execution. In training TOML, these use `env.taskset = { id = "..." }` and optional `env.agent.harness = { id = "..." }` inside a training source.

In the Verifiers v1 model, the **taskset** owns train/eval tasks, task controls, rewards, metrics, and task-owned tools. The **harness** is the program that drives the model — the built-in `default` harness, or an agent harness such as `bash` or `codex`. Its `runtime` field selects *where* that program runs: a local `subprocess`, or a `docker`/`prime`/`modal` sandbox. The Verifiers v1 `Environment` composes a taskset and harness and plugs into the same evaluation and training loop as legacy environments (the `vf.Environment` subclasses such as `SingleTurnEnv` and `ToolEnv` described below).

Hosted Training uses the native `prime-rl` config schema: training environments are declared as an array of `[[orchestrator.train.source]]` tables, each carrying a nested `env` block. See [End-to-End Run](/hosted-training/end-to-end-run#step-4-create-a-training-config) for a worked example.

***

## The Type Hierarchy

All user-facing environment types descend from `Environment`, the abstract base class that defines the dataset, rubric, and rollout interface. `MultiTurnEnv` extends `Environment` and implements the core rollout loop — the `rollout()` method is finalized there. The concrete types (`SingleTurnEnv`, `ToolEnv`, `StatefulToolEnv`) each extend `MultiTurnEnv`, layering interaction complexity progressively.

### `SingleTurnEnv`

Prompt in, completion out, reward computed. This is `MultiTurnEnv` with `max_turns=1`. Use it for standard Q\&A benchmarks.

```python theme={null}
import verifiers as vf

dataset = vf.load_example_dataset("gsm8k")

async def correct_answer(completion, answer) -> float:
    return 1.0 if completion[-1]["content"] == answer else 0.0

env = vf.SingleTurnEnv(
    dataset=dataset,
    rubric=vf.Rubric(funcs=[correct_answer]),
)
```

### `MultiTurnEnv`

The base class for anything conversational or interactive. Override two hooks:

* **`env_response(messages, state)`**: produce the environment's next message given the conversation so far. Game logic, simulation steps, or external calls go here.
* **`@vf.stop` methods**: async methods decorated with `@vf.stop` act as termination conditions. The rollout ends when any stop condition returns `True` or `max_turns` is reached.

The rollout loop alternates between model inference and `env_response` until a stop condition fires or `max_turns` is reached.

### `ToolEnv`

Adds native tool/function-calling. Pass a list of Python functions; `verifiers` extracts JSON schemas from type hints and docstrings, wires them into the OpenAI-compatible tool-calling protocol, and dispatches calls during rollouts.

```python theme={null}
async def calculate(expression: str) -> str:
    """Evaluate a mathematical expression.

    Args:
        expression: A math expression to evaluate (e.g. "2 + 2 * 3")
    """
    return str(eval(expression))

env = vf.ToolEnv(
    dataset=dataset,
    tools=[calculate, search_tool],
    rubric=rubric,
    max_turns=10,
)
```

Tools in `ToolEnv` must be **stateless and idempotent**: each call is fully determined by its arguments. The environment terminates when the model responds without issuing any tool calls (or hits `max_turns`).

### `StatefulToolEnv`

When tools need per-rollout state (a sandbox handle, a database connection, a session token), `StatefulToolEnv` adds two hooks:

* **`setup_state(state)`**: called at rollout start to initialize a fresh state dict.
* **`update_tool_args(tool_name, args, messages, state)`**: intercepts each tool call to inject state into arguments before dispatch.

All mutable state lives in the state dict. Never store globals.

***

## Environment Structure

```
my_env/
├── my_env.py         # Implementation
├── pyproject.toml    # Dependencies + metadata
└── README.md         # Documentation
```

The entry point is `load_environment()`, which returns an `Environment` instance. This is what the training and evaluation infrastructure calls.

```python theme={null}
def load_environment(split: str = "train") -> vf.Environment:
    dataset = load_dataset("my-org/my-dataset", split=split)

    rubric = vf.Rubric(
        funcs=[correctness, format_score],
        weights=[0.8, 0.2],
    )

    return vf.ToolEnv(
        dataset=dataset,
        tools=[search_tool, execute_tool],
        rubric=rubric,
        max_turns=5,
    )
```

Expensive setup (loading datasets, building indices) goes in `load_environment()` or `__init__()`. Per-rollout setup goes in `setup_state()`.

***

## Rubrics

A `Rubric` is a set of scoring functions evaluated after each rollout. Functions can be sync or async, and receive keyword arguments like `prompt`, `completion`, `answer`, `state`, and `parser` depending on what they need.

Functions are weighted and combined into a scalar reward. You can also define non-reward metrics (token count, tool call frequency) that get logged without affecting gradients. For subjective criteria, `JudgeRubric` delegates scoring to an external LLM judge.


# Getting Started
Source: https://docs.primeintellect.ai/hosted-training/getting-started

Launch your first on-demand Hosted Training run

Train a model with reinforcement learning on Prime Intellect's infrastructure — no GPUs to manage, no cluster to reserve.

> [!NOTE]
>
> On-demand training is in **beta**. [Request access](mailto:contact@primeintellect.ai) before your first run, or dispatch will be refused.

## Pick your path

| | What it is | Access |
| - | - | - |
| **[Hosted Training](#1-install-the-cli-and-log-in)** | You submit a config, we run it on GPUs we operate, billed [per GPU-hour](/hosted-training/billing). | Beta, [request access](mailto:contact@primeintellect.ai) |
| **[Model Factory Clusters](/model-factory/clusters)** | A dedicated cluster running training, inference, Slurm and Kubernetes side by side. Use this when you need control over the stack. | [Contact us](mailto:contact@primeintellect.ai) |
| **[prime-rl](/prime-rl/overview)** | The open-source training framework, running on your own hardware. | Self-hosted |

Hosted Training is the fastest way to start, and the rest of this page covers it.

## Prerequisites

* Python 3.10+
* A [Prime Intellect account](https://app.primeintellect.ai) with on-demand training enabled and a funded wallet, see [Billing](/hosted-training/billing)

## 1. Install the CLI and log in

```bash theme={null}
# install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# install the prime CLI
uv tool install -U prime
# log in
prime login
```

## 2. See what you can run

```bash theme={null}
prime train models
```

This lists the models available to dispatch and an **On-Demand Capacity** table of purchasable GPU types with their current `$/GPU-hour`. Models on that list are already cached cluster-side and start warm. An on-demand run can still train a model that is not listed, it just downloads the weights first.

## 3. Create a volume to keep your weights

Checkpointing is **off by default**, and when a run finishes its pods are torn down. Without a volume and a `[ckpt]` block, a completed run leaves you with dashboard metrics and no model.

```bash theme={null}
prime volumes create research --size 100Gi
```

Add `[ckpt]` to save weights. The example below also limits recent checkpoints; see [Volumes](/hosted-training/volumes#checkpoint-settings) for retention, sizing, and cluster-specific requirements.

## Trace storage

Before launching, review [Prime Traces pricing and upload settings](/traces/pricing-and-settings). Traces uploaded by platform runs incur ingestion and storage charges. Stored traces can be explored in the [Traces dashboard](https://app.primeintellect.ai/dashboard/traces).

The November 6 [legacy sample storage retirement](/traces/legacy-storage) is separate from the [retirement of shared LoRA training](/hosted-training/shared-lora-retired).

## 4. Launch a run

Training runs are configured with a [prime-rl](/prime-rl/configuration) `.toml` file. The `[deployment]` block sizes the run and is what routes it onto Hosted Training.

Save this as `configs/rl.toml` — one trainer GPU and one inference GPU:

```toml theme={null}
name = "reverse-text-hosted"
max_steps = 100
seq_len = 2048

[ckpt]                                              # save step 100 to the volume
interval = 100
keep_last = 1

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[trainer.optim]
lr = 3e-6

[orchestrator]
batch_size = 64

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.source]]
name = "reverse-text"

[orchestrator.train.source.env.taskset]
id = "reverse-text"

[orchestrator.train.source.env.agent.harness]
id = "null"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

[orchestrator.renderer]
name = "prime-qwen3"

[inference]
```

Dispatch it against the volume:

```bash theme={null}
prime train configs/rl.toml --volume research
```

The CLI echoes the GPU type and hourly rate it resolved, then prints a run ID and dashboard link.

## 5. Monitor progress

**Terminal:**

```bash theme={null}
prime train logs <run-id> -f
```

**Dashboard:** open the link from the previous step for reward curves, rubric scores, and individual rollouts in real time.

Stop a run early to release its GPUs and stop the meter:

```bash theme={null}
prime train stop <run-id>
```

## 6. Get your weights

Wait for the run and its final volume sync to finish, then download its checkpoints:

```bash theme={null}
prime volumes get research runs/<run-id>/checkpoints/ ./checkpoints
```

Checkpoints are DCP-sharded and need converting before anything else can load them — see [Export to HF format](/hosted-training/end-to-end-run#export-to-hf-format).

## Next steps

> **[End-to-End Run](/hosted-training/end-to-end-run)**
>
> Full walkthrough with baseline evals, model selection, and result review.

> **[Hosted RL](/hosted-training/hosted-rl)**
>
> Config reference, dispatch knobs, and run lifecycle.

> **[Billing](/hosted-training/billing)**
>
> How runs are metered and what gets billed.

> **[Platform Overview](/platform-overview)**
>
> How environments, training, evaluation, and inference fit together.


# Hosted RL (Beta)
Source: https://docs.primeintellect.ai/hosted-training/hosted-rl

On-demand full-parameter RL training on dedicated clusters

Hosted Training updates every parameter of the model. Your run gets its own trainer, inference, and orchestrator pods, sized by the `[deployment]` block in your config.

Runs dispatch **on-demand** — no reserved cluster required — and are [billed per GPU-hour](/hosted-training/billing) for as long as the pods are running.

## Before you start

Models listed by `prime train models` are cached and start warm. On-demand runs can also download uncached Hugging Face models at startup, which adds billable time. Runs on a reserved cluster need the base model cached there, unless they load it from a volume. Both RL and Hosted SFT support `/volume/...` model paths with `--volume`; see [Volume-hosted models](/hosted-training/volume-models).

```bash theme={null}
prime train models
```

This lists the models you can dispatch, plus an **On-Demand Capacity** table of purchasable GPU types with their current `$/GPU-hour`. If the model you want is missing, [reach out](mailto:contact@primeintellect.ai) to get it added.

To see every GPU type you can dispatch on:

```bash theme={null}
prime train gpus
```

Dispatch is admitted against your wallet, so you also need on-demand enabled, a spendable balance, and a run that fits your per-run GPU cap — see [Billing](/hosted-training/billing).

Two platform limits apply by default:

| Limit | Default |
| - | - |
| GPUs per run (trainer + inference, all pods) | 64, or your per-run GPU cap if lower |
| Queued runs per user | 10 |

Reach out if your campaign needs more than that.

## Config

Hosted Training runs use the native [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) config schema unchanged — no platform-specific fields. Size the run with a `[deployment]` block (see [Launching a run](#launching-a-run)).

Minimal single-node example (1 trainer GPU + 1 inference GPU):

```toml theme={null}
name = "reverse-text-hosted"
max_steps = 100
seq_len = 2048

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[trainer.optim]
lr = 3e-6

[orchestrator]
batch_size = 64

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.source]]
name = "reverse-text"

[orchestrator.train.source.env.taskset]
id = "reverse-text"

[orchestrator.train.source.env.agent.harness]
id = "null"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

# This fine-tune's name misses the renderer map's exact match, so select explicitly.
[orchestrator.renderer]
name = "prime-qwen3"

[inference]
```

Multi-node example (2 train nodes + 2 inference nodes, each a full 8-GPU node):

```toml theme={null}
name = "qwen30b-math"
seq_len = 32768

[model]
name = "Qwen/Qwen3-30B-A3B-Thinking-2507"

[deployment]
type = "multi_node"                # required — `single_node` is the default variant
num_train_nodes = 2
num_infer_nodes = 2                # per inference replica; see `num_infer_replicas`

[trainer.model]
attn = "flash_attention_3"
ep = 8                             # expert parallel (MoE)

[trainer.optim]
type = "adamw"
lr = 1e-6

[orchestrator]
batch_size = 512
max_off_policy_steps = 8

[orchestrator.concurrency]
max_inflight = 1024                # ceiling on in-flight episodes

[orchestrator.train.sampling]
max_completion_tokens = 32768

[[orchestrator.train.source]]
name = "math"

[orchestrator.train.source.env.taskset]
id = "i3_math"

[orchestrator.train.source.env.agent.harness]
id = "null"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

[inference.vllm]
tensor_parallel_size = 8           # tensor parallel inside each inference replica
```

> [!NOTE]
>
> Training and eval environments are arrays of `source` tables, each carrying a nested `env` block (`env.taskset`, `env.agent.harness`, `env.agent.runtime`). The older flat `[[orchestrator.train.env]]` shape with an `id` field was removed from prime-rl and is rejected at dispatch — see [Environments](/prime-rl/configuration#environments) for the current shape.

> [!TIP]
>
> Multi-node runs broadcast weights over NCCL by default and auto-discover the cluster's RDMA devices — no extra config needed.

See the [prime-rl docs](/prime-rl/configuration) and [config examples](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/configs) for the full schema.

## Launching a run

Every RL config dispatches as a Hosted Training run — no flag needed. A `[deployment]` block sizes the run; without one, prime-rl's own defaults apply, same as a local run:

```bash theme={null}
prime train configs/rl.toml
```

On dispatch you get a run ID and a dashboard link:

```
Dispatched hosted run wn2cjdrzdo6bmfqajoeuu30p

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/wn2cjdrzdo6bmfqajoeuu30p
```

Pass credentials with `-e / --env-var` or `--env-file`:

```bash theme={null}
prime train configs/rl.toml -e WANDB_API_KEY -e HF_TOKEN -e OPENAI_API_KEY
```

`-e KEY` reads the value from your shell, `-e KEY=VALUE` sets it inline, and `--env-file path/to/.env` loads a whole file.

Every key you pass is written to a per-run secret and projected onto the trainer, inference, orchestrator, and env-server pods, so an environment that needs a third-party API key can read it straight from the process environment. `WANDB_API_KEY` and `HF_TOKEN` are recognized and routed to their own fields.

A few constraints on the secret map:

| Constraint | Value |
| - | - |
| Entries per run | 32 |
| Name | POSIX env-var name, ≤ 128 chars |
| Value | ≤ 32 KB |
| Total | ≤ 256 KB |

> [!NOTE]
>
> `WANDB_API_KEY` is required for the `[wandb]` block to take effect — without it the block is stripped and the run trains without W\&B logging.

Two dispatch-level knobs control the dispatch. Both accept a CLI flag or a top-level TOML key, and the flag wins:

| Knob | Flag | TOML | Default |
| - | - | - | - |
| prime-rl image build | `--image-tag v0.5.1` | `image_tag = "v0.5.1"` | `main` |
| GPU type to dispatch on | `--gpu-type H200_141GB` | `gpu_type = "H200_141GB"` | auto-pick |

Checkpointing is **off by default**. To keep model weights, add a `[ckpt]` block, create a [volume](/hosted-training/volumes), and pass its name with `--volume`. Checkpoints live at `runs/<runId>/checkpoints/`. The run directory is saved to cloud storage when the run ends; `keep_interval` and `keep_last` are optional retention settings. See [Checkpoint settings](/hosted-training/volumes#checkpoint-settings).

## Run lifecycle

A dispatched run moves through:

```
QUEUED → PENDING → CREATING → PULLING_IMAGE → RUNNING → COMPLETED / FAILED / STOPPED
```

* **QUEUED** — no cluster has GPU headroom yet. The run is admitted and promoted automatically as capacity frees; you don't need to resubmit.
* **PULLING\_IMAGE** — pods are scheduled and pulling the prime-rl image. This is multiple GB and can take many minutes on a cold node. It's expected, not a stall.

Config errors (bad schema, uncached model, oversized deployment) are caught synchronously and fail the dispatch immediately with a reason, before any GPU is reserved.

Stop a run early to release the cluster:

```bash theme={null}
prime train stop <run-id>
```

To release the cluster but continue from the current step later, [pause](/hosted-training/pause-resume) the run instead.

> [!WARNING]
>
> When a run reaches a terminal state, its cluster resources are torn down automatically. Orchestrator logs are captured to the run record first, but trainer, inference, and env-server pod logs go away with the pods — pull anything you need while the run is live. Metrics, samples, distributions, and progress are persisted and stay available in the dashboard. Trace uploads have separate [pricing and workspace controls](/traces/pricing-and-settings); see [Managing traces](/traces/managing-traces) for viewing, downloading, and deleting them. Older legacy samples are subject to the [November 6 storage retirement](/traces/legacy-storage).

## Monitoring

A Hosted Training run has several distinct components. Pick which one to read with `-c / --component`:

```bash theme={null}
prime train logs <run-id>                  # orchestrator (default)
prime train logs <run-id> -c trainer       # trainer (FSDP / torchrun)
prime train logs <run-id> -c inference     # vLLM inference server
prime train logs <run-id> --env <env-name> # env-server for a specific env
```

List the orchestrator and env-server components for a run:

```bash theme={null}
prime train components <run-id>
```

Follow and filter with `-f`, `--search`, `--regex`, `--level`, and `--since`. See [Monitoring](/hosted-training/end-to-end-run#step-6-monitor-the-run) for details.

The dashboard shows reward curves, rubric scores, and individual rollouts at `https://app.primeintellect.ai/dashboard/training/<run-id>`.

> **[End-to-End Run](/hosted-training/end-to-end-run)**
>
> Step-by-step walkthrough of a complete run, from baseline eval to results.

> **[prime-rl Configuration](/prime-rl/configuration)**
>
> Full reference for the underlying training framework config schema.


# Hosted SFT (Beta)
Source: https://docs.primeintellect.ai/hosted-training/hosted-sft

Supervised fine-tuning on Hosted Training: volumes, dataset uploads over SSH, and prime train dispatch

> [!NOTE]
>
> Hosted SFT is in **beta**. Contact Prime support if dispatch is refused.

Run `prime-rl` supervised fine-tuning on Prime's hosted GPU clusters: create a volume, copy your dataset onto it over SSH, dispatch. No `kubectl`, no cluster credentials — the trainer reads the dataset from the volume.

This walkthrough fine-tunes a small Qwen3 model to reverse text ([`reverse-text`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/reverse-text)) on one GPU. For SFT on your own infrastructure, see [Training](/prime-rl/training).

## Prerequisites

```bash theme={null}
uv tool install prime
prime login
```

**Requires `prime` CLI v0.9.0 or later.**

* A Prime account with hosted training access — Prime-side enablement; the platform deployment needs the hosted SFT path switched on.
* A base model the run can boot from: **in the model cache** (check with `prime train models`), or **weights already on the volume itself** via a `/volume/...` `[model].name` path ([Volume-hosted models](/hosted-training/volume-models)).

```bash theme={null}
prime train models
```

* A Hugging Face dataset you can download with `hf download` — with `prompt`/`completion` or `messages` columns ([Training § Dataset Format](/prime-rl/training#dataset-format)).

## 1. Create a volume

```bash theme={null}
prime volumes create research --size 100Gi
```

> [!NOTE]
>
> Size the volume for your dataset plus the checkpoints the run keeps on its disk. A checkpoint is about 11–12 bytes per parameter: the 0.6B smoke run writes \~7.2 GB. See [Choose a size](/hosted-training/volumes#choose-a-size). Grow it with `prime volumes expand <name> --size <size>`.

## 2. Put the dataset on the volume

Open a shell on the volume and download the dataset onto it. The download uses the cluster's network, not your laptop's. Shells need [a cluster allocation or a cluster your team owns](/hosted-training/volumes#what-you-can-do-with-and-without-a-cluster-allocation) and your [primary SSH key](/hosted-training/volumes#set-up-your-ssh-key):

```bash theme={null}
prime volumes ssh research --read-write
```

Inside the session (run `hf auth login` first for private datasets):

```bash theme={null}
hf download willcb/R1-reverse-wikipedia-paragraphs-v1-1000 \
  --repo-type dataset \
  --local-dir /volume/datasets/reverse-text
exit
```

Back in your local terminal, stop the session using the ID printed when it opened:

```bash theme={null}
prime volumes stop research <session-id>
```

Wait for its final sync before dispatching. Exiting SSH alone leaves the read-write session open and blocks the training run.

The trainer sees the volume at `/volume`, the same path you see in the shell. Without one, download the dataset to your machine and upload it instead: `prime volumes put <name> ./my-data datasets/` creates `datasets/my-data` (paths start at the volume root, and a trailing `/` means "into this folder"). In the config, `data.name` accepts either form: a path relative to the volume (`datasets/my-data`) or the absolute mount path (`/volume/datasets/my-data`) — the platform resolves both to the same location.

## 3. Write the config

Save this as `sft.toml` — `data.name` points at the dataset on the volume (relative to the volume root; absolute `/volume/...` paths work too):

```toml theme={null}
max_steps = 10

[ckpt]               # save step 10 to the volume
interval = 10
keep_last = 1

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"  # must be in the cluster model cache

[data]
name = "datasets/reverse-text"  # volume-relative; the platform resolves it to /volume/datasets/reverse-text
type = "sft"
seq_len = 2048        # add splits = ["train"] for multi-split datasets
batch_size = 8

[optim]
lr = 2e-5

[deployment]
type = "single_node"
num_train_gpus = 1
gpus_per_node = 1

[renderer]
name = "prime-qwen3"  # must match the model family
```

> [!NOTE]
>
> `output_dir` is injected by the platform — omit it. The walkthrough model is a cached debug sibling already fine-tuned on this dataset — treat the loss curve as a smoke test of the hosted path.
>
> This walkthrough is trainer-only. To score checkpoints against an environment while the run trains, see [Online evals](#online-evals) below.

Instead of a cached model ID, `[model].name` can point at HF-format weights already on the volume, uploaded with `prime volumes put`. RL configs support volume model paths too. Run checkpoints are in prime-rl's distributed format, so export them to HF format before using them as `[model].name` — or resume the full training state by pointing `[resume].dir` at the checkpoint's step directory on the volume (`dir = "/volume/runs/<runId>/checkpoints/step_<N>"`). See [Volume-hosted models](/hosted-training/volume-models#continue-from-a-previous-runs-checkpoint).

## 4. Dispatch

```bash theme={null}
prime train sft.toml --volume research
```

```text theme={null}
Creating Hosted Training run...
Dispatched hosted run kqeggj5dl7k85bc0mk1i4y5i

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/kqeggj5dl7k85bc0mk1i4y5i
```

Use **the run ID your dispatch printed** (`<runId>`) below. The run moves `PENDING` → `CREATING` → `RUNNING` → `COMPLETED` — expect roughly three minutes on one H200 for this walkthrough.

## 5. Watch it run

```bash theme={null}
prime train get <runId>
prime train metrics <runId>   # the loss series the dashboard renders
prime train logs <runId>      # the live trainer log
```

```text theme={null}
Run kqeggj5dl7k85bc0mk1i4y5i

  Status: COMPLETED
  Model: PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT
  Max Steps: 10
  Created: 2026-09-28 20:33
  Started: 2026-09-28 20:34
  Completed: 2026-09-28 20:37
```

```text theme={null}
step: [1, 2, 3, 4, 5, 6, 7, 8, 9]
loss/mean:       [1.108, 1.1788, 0.9996, 0.9468, 1.1179, 1.044, 1.0353, 1.0038, 0.9944]
loss/perplexity: [3.028, 3.251, 2.717, 2.577, 3.058, 2.841, 2.816, 2.729, 2.703]
```

```text theme={null}
20:35:38 [INFO] Initializing scheduler with 10 steps (type='constant')
Generating train split: 100%|████████████████████| 1000/1000 [00:00<00:00, 23213.48
examples/s]
20:35:47 [SUCCESS] Step 1 |    8.5s | Loss 1.1080 | Grad. Norm 12.9495 | LR
2.00e-05 | Throughput 0 tokens/s | MFU 0.0% | Peak Mem. 10.7/139.8 GiB (7.7%)
20:35:52 [SUCCESS] Step 3 |    2.4s | Loss 0.9996 | Grad. Norm 7.4464 | LR
2.00e-05 | Throughput 6704 tokens/s | MFU 2.9% | Peak Mem. 10.7/139.8 GiB (7.7%)
```

Ten steps on a 0.6B model with a constant LR will not descend cleanly — the point is a visible, NaN-free curve. Occasionally the last metrics row lands during teardown; terminal status and the trainer's final log lines are the success signal. Teardown is automatic. The step-10 checkpoint and the `datasets/` folder stay on the volume.

Wait for the run to end and the final volume sync to complete before downloading its checkpoint:

```bash theme={null}
prime volumes get research runs/<runId>/checkpoints/step_10/ ./step_10
```

(Usage accounting shows `0 tokens / $0.00` for SFT — a reporting quirk, not free usage.)

## Online evals

Add an `[eval]` block to score checkpoints against an environment while the run trains. The platform then runs an inference server alongside the trainer, which broadcasts its weights to it at each eval interval, plus one env server per eval source:

```toml theme={null}
[eval]
interval = 5          # evaluate every 5 trainer steps

[[eval.source]]
name = "reverse-text"
env = { taskset = { id = "primeintellect/reverse-text" } }

[inference]
[inference.vllm]
tensor_parallel_size = 1
```

Sizing: inference GPUs come out of the same allocation as the trainer, so `num_train_gpus + num_infer_gpus` must fit `gpus_per_node` on a single node. On `multi_node`, set `num_infer_nodes` to at least 1 and add an `[inference.router]` block.

```toml theme={null}
[deployment]
type = "single_node"
num_train_gpus = 4
num_infer_gpus = 4    # defaults to 1
gpus_per_node = 8
```

Eval metrics land on the run's dashboard next to the loss curve, and each component gets its own log tab (`prime train logs <runId> --component inference`). The run stays `RUNNING` until the evals triggered by the final checkpoint have drained, so expect it to outlast the last trainer step.

> [!NOTE]
>
> Online evals need a cluster on prime-rl-fft 0.24.0 or newer; dispatch is rejected with a clear error otherwise. Weight transport defaults to NCCL when `[inference]` is set, and to filesystem for LoRA runs. Traces are not yet wired for SFT eval rollouts.

## Troubleshooting

> **Online evals rejected at dispatch**
>
> The cluster's chart predates online-eval support. Remove the `[eval]` block to run trainer-only SFT, or ask Prime support when the cluster will be upgraded.
>
> ```text theme={null}
> Error: online evals ([eval]) are not supported by the prime-rl-fft chart this
> deployment installs (needs 0.24.0 or newer). Remove the [eval] block to run
> trainer-only SFT.
> ```

> **\[inference] without \[eval]**
>
> The inference pool exists only to serve online-eval rollouts, so it needs an `[eval]` block to go with it.
>
> ```text theme={null}
> Error: [inference] is only used for online evals — add an [eval] block or
> remove it.
> ```

> **Model not available on this cluster**
>
> Hosted SFT boots from the cluster model cache. Check `prime train models`, or ask Prime support to cache the model. If the weights are already on the volume, point `[model].name` at their `/volume/...` path instead — no cache entry is needed.
>
> ```text theme={null}
> Error: HTTP 400: Model 'PrimeIntellect/Qwen3-0.6B' is not available on this
> cluster. Reach out to Prime support to get it added.
> ```

> **data.name requires --volume**
>
> Real data needs a volume. `data.type = "fake"` is the one no-volume path (synthetic data, connectivity smoke test).
>
> ```text theme={null}
> Error: HTTP 400: data.name requires `--volume <name>`. Create a volume with
> `prime volumes create <name>`, then rerun `prime train sft.toml --volume <name>`.
> ```

> **Dataset missing on the volume**
>
> The trainer reads `data.name` from the volume mount — a path that is empty or missing fails at startup. Re-run step 2 and check the name matches: the directory you create under `/volume/datasets/<name>` in the SSH session is exactly the `datasets/<name>` you point `data.name` at (relative paths resolve under `/volume`, so `datasets/<name>` and `/volume/datasets/<name>` are the same location).

> **Volume ran out of space during download**
>
> Inside the shell, `hf download` fails when the session's disk is full. Exit, end the session, grow the volume, and re-run the download in a new session:
>
> ```bash theme={null}
> prime volumes stop research <session-id>
> prime volumes expand research --size <larger-size>
> prime volumes ssh research
> ```

## Limitations

* **Requires CLI v0.9.0+** — older versions misroute SFT configs against the RL schema.
* **SFT-shaped datasets** — `prompt`/`completion` or `messages` columns.
* **Cached or volume-hosted models** — a cached HF id (`prime train models`; support can add others), or weights on the volume via a `/volume/...` `[model].name` path.
* **Online evals need chart 0.24.0+** — on older clusters SFT is trainer-only and `[eval]` is rejected. No traces for eval rollouts yet.
* **Checkpoints live on the volume** — use `prime train checkpoints <runId>` to list saved checkpoints, then `prime volumes get research runs/<runId>/checkpoints/` to download them after sync.

## Scaling up

The same flow carries to multi-node — switch `[deployment]` to `multi_node` and add the parallelism knobs. The GLM-5.3 example below is a **4-step smoke config**: it checks that the multi-node path boots and the model fits in memory — it is not a throughput-tuned recipe. Save as `glm53-sft.toml`:

```toml theme={null}
max_steps = 4

[env_vars]
TRITON_CACHE_DIR = "/tmp/.cache/triton_cache"

[model]
name = "zai-org/GLM-5.3-BF16"
impl = "custom"
attn = "flash_attention_2"
cp = 8
ep = 8
optim_cpu_offload = true

[model.ac]
freq = 1

[model.ac_offloading]
max_inflight_activations = 5

[optim]
type = "sign_sgd"
lr = 5e-5
weight_decay = 0.01

[scheduler]
type = "linear"
warmup_steps = 50
decay_steps = 0

[renderer]
name = "glm-5.3"

[deployment]
type = "multi_node"
num_train_nodes = 8
gpus_per_node = 8

[model.debug]
force_balanced_routing = true  # debug only

[ckpt]               # save the final checkpoint
interval = 4
keep_last = 1

[data]
type = "sft"
name = "datasets/intellect-3-sft-10k"  # put it on the volume first (step 2); volume-relative
splits = ["math"]
batch_size = 8
micro_batch_size = 1
seq_len = 4096
```

> [!NOTE]
>
> This config targets 8×8 H200 (64 GPUs). If you adapt it for real training: remove `force_balanced_routing` — it forces round-robin expert routing and is debug-only; with `warmup_steps = 50` a 4-step run never leaves warmup, so losses are warmup-dominated rather than convergence; the `[ckpt]` block saves one checkpoint at step 4, so size the volume for it (about 11–12 bytes per parameter, see [Choose a size](/hosted-training/volumes#choose-a-size)); with this linear schedule the LR ramps to its peak at step 50 — expect a transient loss spike near the peak before it settles (9.42 at step 55 in a 60-step run). See [Scaling](/prime-rl/scaling) and [Advanced](/prime-rl/advanced) for the knobs.

> **[Hosted RL](/hosted-training/hosted-rl)**
>
> RL-style full-parameter training on a dedicated cluster — same CLI, same dashboard.

> **[prime-rl Training](/prime-rl/training)**
>
> The underlying SFT trainer: config schema, dataset format, and local runs.


# Models
Source: https://docs.primeintellect.ai/hosted-training/models

Which models and GPU types you can dispatch on Hosted Training

The model list is cluster-dependent and changes as models are added, so the CLI is the source of truth:

```bash theme={null}
prime train models
```

That gives you two things:

* **Models** — the models available to dispatch
* **On-Demand Capacity** — purchasable GPU types with their current `$/GPU-hour`, and whether a run **starts now** or **queues**

Add `--output json` for machine-readable output. For pricing mechanics, see [Billing](/hosted-training/billing).

## The weight cache

Runs boot from a cluster-side weight cache and never download weights at start-up. **A model that isn't cached can't be dispatched** — a config naming one fails at dispatch, before any GPU is reserved.

To see only the cached models for a reserved cluster you hold:

```bash theme={null}
prime train models
```

If the model you want is missing, [reach out](mailto:contact@primeintellect.ai) to get it added.

## GPU types

To list every GPU type you can dispatch on:

```bash theme={null}
prime train gpus
```

You can pin one explicitly at dispatch with `--gpu-type`, or let the platform pick. Placement favours clusters that already have your model cached, since a cache miss means a full weight download.

## Choosing a model

**Validate first.** Start with the smallest model that exercises your environment end to end — a short run on a small model catches config and reward-function bugs for a fraction of the cost. Keep `max_steps` low and watch that the loss or reward moves at all.

**Then scale.** Once the environment is proven, move to the model you actually care about and raise `batch_size` and `group_size` for a more stable reward signal. MoE models with small active parameter counts give strong performance per GPU-hour.

Size the run with the `[deployment]` block — see [Hosted RL](/hosted-training/hosted-rl#config) for single-node and multi-node examples.

## Thinking mode

Some model families support a thinking mode that produces extended chain-of-thought before the final answer. It tends to help on tasks that benefit from multi-step reasoning — math, code, logic — at the cost of longer completions, which means more tokens generated per rollout and more wall-clock per step.

Thinking is controlled through the renderer and sampling config; see [prime-rl Configuration](/prime-rl/configuration).

> **[Billing](/hosted-training/billing)**
>
> How GPU-hours are metered and where rates come from.

> **[Hosted RL](/hosted-training/hosted-rl)**
>
> Config schema and dispatch reference.


# Pause and resume runs
Source: https://docs.primeintellect.ai/hosted-training/pause-resume

Release a run's GPUs at a checkpoint and continue it later, on any cluster

Pausing a run stops it at its next training step, saves a checkpoint of that step to the run's [volume](/hosted-training/volumes), and releases its GPUs. Resuming starts a new run from that checkpoint on whichever cluster has capacity, and training continues from the step after it.

## Quickstart

Install and log in to [Prime CLI](/cli-reference/introduction) 0.16.0 or later. Pausing needs a run launched with `--volume`.

**1. Launch a run on a volume**

```bash theme={null}
prime train config.toml --volume research
```

**2. Pause it**

```bash theme={null}
prime train pause <run-id>
```

The run turns **PAUSING** while it finishes its current step, saves it, and syncs the volume.

**3. Wait for PAUSED**

```bash theme={null}
prime train get <run-id>
```

Once the run is **PAUSED**, its GPUs are released and the meter stops. This usually takes a few minutes.

**4. Resume it**

```bash theme={null}
prime train resume <run-id>
```

This starts a new run with its own ID, which reuses the paused run's config, volume, image, and GPU type and continues at the next step.

## Things to know

* Only a RUNNING run can be paused, and only if it uses prime-rl image `v0.10.1.dev11` or later. The default image qualifies. A run without a volume can only be [stopped](/hosted-training/hosted-rl).
* Pausing doesn't need a `[ckpt]` block. The pause saves its own checkpoint at `runs/<runId>/checkpoints/step_<n>/`.
* A paused run can't be unpaused or restarted in place. Resume it as a new run instead. The new run writes its outputs to its own `runs/<newRunId>/` directory on the same volume.
* Secrets aren't kept after a run starts, so pass them again when you resume: `prime train resume <run-id> -e WANDB_API_KEY`.
* To resume on a different GPU type or image, pass `--gpu-type` or `--image-tag` to `prime train resume`.
* To change the config, for example to raise `max_steps`, resume from your edited TOML with `prime train config.toml --resume-from <run-id>`. Keep the same model, since the checkpoint is loaded as is. If you also pass `--volume`, it must name the paused run's volume.
* The PAUSING phase is [billed](/hosted-training/billing) like a running run, because the run still holds its GPUs while it saves. A PAUSED run isn't billed.


# Shared LoRA Training (Retired)
Source: https://docs.primeintellect.ai/hosted-training/shared-lora-retired

The shared multi-tenant LoRA training service has been retired and replaced by on-demand Hosted Training

Shared Hosted Training for LoRA runs has been **retired**. The service trained a LoRA adapter on top of a shared, multi-tenant deployment and billed per token. It no longer accepts new runs, and its `.toml` config format is no longer dispatched.

## What to use instead

[Hosted Training](/hosted-training/getting-started) now runs **on-demand**: your run gets its own GPUs, every parameter is trained, and you're [billed per GPU-hour](/hosted-training/billing). No reserved cluster required.

Configs use the native [prime-rl](/prime-rl/configuration) schema rather than the old flat format — see [Hosted RL](/hosted-training/hosted-rl) for the config shape and [End-to-End Run](/hosted-training/end-to-end-run) for a walkthrough.

## Your existing adapters

LoRA adapters from completed runs remain downloadable from the Prime platform. Adapter **deployment for inference has been removed**: adapters can no longer be served through the Prime Inference API, and the `prime deployments` CLI commands have been retired.

To serve a fine-tuned model, self-host it with [prime-rl](/prime-rl/advanced#lora-training) on your own GPUs or on a [Model Factory Cluster](/model-factory/clusters).

## LoRA itself isn't going away

Only the hosted shared service was retired. LoRA remains a supported, actively maintained feature of [prime-rl](/prime-rl/advanced#lora-training) for both RL and SFT, which you can run on your own GPUs or on a [Model Factory Cluster](/model-factory/clusters).

> **[Getting Started](/hosted-training/getting-started)**
>
> Launch an on-demand Hosted Training run.

> **[LoRA in prime-rl](/prime-rl/advanced#lora-training)**
>
> Train LoRA adapters on your own infrastructure.


# Troubleshooting
Source: https://docs.primeintellect.ai/hosted-training/troubleshooting

Common issues and solutions for Hosted Training

This page covers common issues you may encounter when using Hosted Training, along with their solutions.

## Environment Issues

> **ModuleNotFoundError: No module named 'verifiers'**
>
> The verifiers library is not installed in your project.
>
> ```bash theme={null}
> uv add verifiers
> ```
>
> Or if you're installing a specific environment:
>
> ```bash theme={null}
> prime env install my-env
> ```

> **load\_environment not found**
>
> This usually means a module name collision — your environment name conflicts with an existing Python package, or the environment wasn't installed correctly.
>
> **Solutions:**
>
> * Rename your environment to avoid conflicts
> * Reinstall: `prime env install my-env`
> * Check that your environment module exposes a `load_environment` function at the top level

> **Environment not found on the Hub**
>
> The environment ID doesn't match any published environment.
>
> ```bash theme={null}
> # Check the exact ID
> prime env info owner/environment-name
>
> # Install a specific version
> prime env install owner/environment-name@latest
> ```
>
> Make sure you're using the correct `owner/name` format and that you have access if it's a private environment.

> **MissingKeyError when loading environment**
>
> The environment requires API keys that aren't set. The error message will list which keys are missing.
>
> **For local evaluation:**
>
> ```bash theme={null}
> export OPENAI_API_KEY=sk-...
> ```
>
> **For Hosted Training**, the recommended approach is to link secrets to your environment via the [Environments Hub](/tutorials-environments/secrets). Alternatively, pass them at dispatch:
>
> ```bash theme={null}
> prime train config.toml -e OPENAI_API_KEY        # reads from your shell
> prime train config.toml --env-file secrets.env   # loads a whole file
> ```
>
> Or use the CLI to manage secrets:
>
> ```bash theme={null}
> prime secret list              # global secrets
> prime env secret list my-env   # per-environment secrets
> ```

## Training Issues

> **Reward is always 0.0**
>
> The task is too hard for the model at its current capability level. The model can't solve any examples, so there's no reward signal to learn from.
>
> **Solutions:**
>
> * Try a larger or more capable model
> * Use easier examples (filter your dataset or adjust environment args)
> * Increase `[orchestrator.train.sampling].max_completion_tokens` — the model may need more space to reason
> * Check your rubric implementation for bugs that might always return 0
> * Run a baseline evaluation first: `uv run vf-eval my-env -m <model> -n 20 -r 1`

> **Reward is always 1.0**
>
> The task is too easy — the model already solves everything.
>
> **Solutions:**
>
> * Use harder examples or a more challenging dataset split
> * Add more demanding rubric criteria
> * Use a smaller model that has more room to improve

> **Reward is not changing / training seems stuck**
>
> There could be several causes:
>
> * **Low reward diversity:** If all rollouts for an example get the same reward, there's no contrast for the model to learn from. Increase `[orchestrator.train].group_size` (16–32) to get more variation.
> * **Learning rate too low:** Try raising `[trainer.optim].lr` — Hosted Training runs typically sit in the `1e-6` to `1e-5` range, so move in small steps.
> * **Batch size too small:** Larger batches provide more stable gradient estimates. Try `[orchestrator].batch_size = 512`.
> * **Task mismatch:** The task may not be suitable for RL training. Ensure the reward function produces a meaningful gradient of scores, not just binary 0/1.

> **Pydantic validation error in config**
>
> A field in your TOML config has the wrong type or an invalid value.
>
> **Common causes:**
>
> * String values not quoted: `name = Qwen/Qwen3-4B` → `name = "Qwen/Qwen3-4B"`
> * Integer where float expected or vice versa
> * Missing required sections like `[model]`, `[deployment]`, or `[[orchestrator.train.source]]`
>
> Double-check your config against the [prime-rl config reference](/prime-rl/configuration).

> **Disk quota exceeded (Errno 122) on a volume**
>
> The run's [volume](/hosted-training/volumes) is full. Usually a checkpoint did not fit: a full fine-tuning checkpoint takes about 11–12 bytes per parameter, and the trainer writes a new one before it deletes the oldest. The trainer crashes and retries into the same limit until the run is marked failed.
>
> **Solutions:**
>
> * Set `keep_last = 1` or `2` in `[ckpt]`, or save less often with a larger `[ckpt].interval`
> * Delete old runs' outputs: `prime volumes ssh <name> --read-write`, then remove `/volume/runs/<oldRunId>`
> * Grow the volume or use a larger one
>
> See [Volume capacity](/hosted-training/volume-capacity) to size the volume.

> **Model not available**
>
> The model you specified isn't currently supported for Hosted Training.
>
> ```bash theme={null}
> # Check available models
> prime train models
> ```
>
> Runs boot from a cluster-side weight cache, so a model that isn't cached can't be dispatched. See [Models](/hosted-training/models).

## CLI Issues

> **prime: command not found**
>
> The CLI isn't installed or isn't on your PATH.
>
> ```bash theme={null}
> # Install or reinstall
> uv tool install prime
>
> # If already installed, upgrade
> uv tool install -U prime
>
> # Verify
> prime --version   # Should be >= 0.5.15
> ```
>
> If you installed with `uv tool install` and it's still not found, make sure `~/.local/bin` is in your PATH.

> **Authentication failed / not logged in**
>
> Your CLI session may have expired.
>
> ```bash theme={null}
> prime login
> ```
>
> This opens a browser window to re-authenticate.

> **prime train fails immediately**
>
> Check the following:
>
> * Your config file is valid TOML (no syntax errors)
> * The model is available: `prime train models`
> * The environment ID is correct and accessible
> * You're authenticated: `prime login`
> * Your CLI is up to date: `uv tool install -U prime`

## Evaluation Issues

> **Rate limit exceeded (429 errors)**
>
> Reduce concurrency when running evaluations:
>
> ```bash theme={null}
> uv run vf-eval my-env -m openai/gpt-4.1-mini -n 100 -c 8
> ```
>
> The `-c` flag controls maximum concurrent requests. Lower it if you're hitting rate limits.

> **ThinkParser failures with Qwen3 models**
>
> Qwen3 and DeepSeek-R1 models have chat templates that automatically remove `<think>` tags from message history. This conflicts with `ThinkParser`.
>
> **Solution:** Use `MaybeThinkParser` or `Parser` instead of `ThinkParser` in your environment:
>
> ```python theme={null}
> # Instead of:
> parser = vf.ThinkParser(extract_fn=my_fn)
>
> # Use:
> parser = vf.MaybeThinkParser(extract_fn=my_fn)
> ```

> **Evaluation results look wrong or inconsistent**
>
> * **Check `rollouts_per_example`:** Low values (1–2) produce noisy results. Use at least 3–5 for reliable metrics.
> * **Check `num_examples`:** Very small sample sizes can be misleading.
> * **Check sampling temperature:** High temperatures produce more variation between runs.
> * **Check your rubric:** Make sure reward functions handle edge cases (empty responses, malformed outputs, etc.).

## Environment Development Issues

> **Environment works locally but fails in Hosted Training**
>
> Common causes:
>
> * **Missing dependencies:** Make sure all required packages are listed in your environment's `pyproject.toml`
> * **Missing secrets:** API keys available locally may not be set for hosted runs. Link them to your environment via the [Environments Hub](/tutorials-environments/secrets), or pass them at dispatch with `-e` / `--env-file`.
> * **Hardcoded paths:** Avoid absolute file paths in your environment code
> * **Network access:** Some external APIs may not be reachable from the hosted environment

> **How do I debug my environment?**
>
> Start with a local evaluation using verbose output:
>
> ```bash theme={null}
> uv run vf-eval my-env -m openai/gpt-4.1-mini -n 5 -r 1 -v
> ```
>
> The `-v` flag enables verbose logging. You can also test your environment directly in Python:
>
> ```python theme={null}
> from verifiers import load_environment
>
> env = load_environment("my-env")
> # Inspect the dataset, rubric, etc.
> ```

## Getting Help

If your issue isn't covered here:

* **Discord:** Join the [Prime Intellect Discord](https://discord.gg/ZTFydGWPKj) for community support and Q\&A
* **Research Support:** Fill out the [research support form](https://form.typeform.com/to/iYn9UliG) for hands-on assistance
* **Feedback:** Use the thumbs-up/down on any docs page to let us know what's helpful or missing

## Missing traces or upload failures

Check the workspace, upload setting, API permissions, and storage limit. Training completion does not by itself confirm that every trace uploaded. See [Traces troubleshooting](/traces/managing-traces#troubleshooting). For older samples, see [legacy storage retirement and exports](/traces/legacy-storage).


# Volume capacity and checkpoint sizing
Source: https://docs.primeintellect.ai/hosted-training/volume-capacity

Size a volume for your checkpoints, and fix runs that fail with Disk quota exceeded

A [volume](/hosted-training/volumes) has a fixed size. Runs that write past it fail. Full fine-tuning checkpoints are large, so size the volume before you launch a run.

## What fills a volume

| Content | Where | When it is removed |
| - | - | - |
| Trainer checkpoints | `runs/<runId>/checkpoints/step_N/` | When the run's retention settings delete them (see below) |
| Data you uploaded | for example `datasets/`, `models/` | Only when you delete it |
| Outputs of earlier runs | `runs/<oldRunId>/` | Only when you delete them; they stay after a run ends |
| RL rollout traces | under the run's directory | Never during the run; they grow until the run ends |

Every run that uses the volume shares the same space.

## How many checkpoints are on disk

These `[ckpt]` settings control how many checkpoints a run keeps. RL configs use the same fields in the shared `[ckpt]` block.

| Setting | Effect |
| - | - |
| `interval` | Save a checkpoint every N steps. When unset, the run saves one checkpoint at the end of training. |
| `keep_last` | Keep only the N most recent checkpoints. When unset, **every checkpoint is kept**. |
| `keep_interval` | Also keep every checkpoint whose step is a multiple of N, permanently. |

The trainer writes a new checkpoint **before** it deletes the oldest one. So at peak the run holds `keep_last + 1` checkpoints, plus any pinned by `keep_interval`.

## How big one checkpoint is

A trainer checkpoint holds the full training state: model weights in fp32 and the optimizer state. Plan for about **11–12 bytes per parameter**. Measured saves: \~341 GB for Qwen3-30B-A3B (30.5B parameters) and \~7.2 GB for a 0.6B model.

| Model size | One checkpoint (about 11–12 bytes/param) |
| - | - |
| 0.6B (measured) | \~7.2 GB |
| 8B | \~90–100 GB |
| 30B (measured: Qwen3-30B-A3B, 30.5B params) | \~341 GB |
| 100B | \~1.1–1.2 TB |

The bf16 weights alone are 2 bytes per parameter (\~57 GiB for Qwen3-30B-A3B). That is a lower bound and the size of exported weights. It is not the size of a trainer checkpoint.

> [!NOTE]
>
> `ckpt.skip_optimizer = true` does not make checkpoints smaller. It only skips **loading** the optimizer state when a run resumes. The checkpoint is still saved with the full state.

## Size the volume

```text theme={null}
required ≈ (keep_last + 1) × 11–12 bytes × parameters
         + uploaded datasets and models
         + outputs of other runs on the volume
         + about 20% headroom
```

If `keep_last` is unset, multiply by the number of checkpoints the run will save instead.

**Example.** Qwen3-30B-A3B (30.5B parameters) with `keep_last = 2`:

* One checkpoint is about 341 GB. Peak is 3 checkpoints: about 1 TB.
* With the formula (11–12 bytes per parameter, 20% headroom): about 1.2 TB, plus your datasets. Use a `1536Gi` (1.5 TiB) volume. Sizes are whole numbers of `Gi` or `Ti`, so write 1.5 TiB as `1536Gi`.
* On a `200Gi` volume (about 215 GB), even the **first** checkpoint does not fit. The run fails at its first save.

For RL runs, use the same checkpoint math, then add room for rollout traces. Traces are written in chunks of about 5 GiB. That is the file size per chunk, not a limit: traces keep growing for as long as the run trains. Long RL runs need extra headroom, and old runs' traces should be cleaned up.

## The size is a hard quota

The size you set with `--size` or `--volume-size` is a hard limit. A write past it fails with one of these errors:

```text theme={null}
OSError: [Errno 122] Disk quota exceeded
OSError: [Errno 28] No space left on device
```

Check what uses the space (Prime CLI 0.13.0 or later):

```bash theme={null}
prime volumes du training-outputs -h          # whole volume, and usage against its size
prime volumes du training-outputs /runs -h    # one directory
```

The volume's stored contents update when a run or read-write session ends, so `du` does not show what a run that is still going has written.

A volume that already holds more than its size cannot start a run: the run fails before training with `volume contents (...) exceed the volume size (...); run prime volumes expand`. Expand the volume or delete data, then launch again.

## What happens when a volume fills

1. The checkpoint save fails and the trainer crashes.
2. The trainer restarts from the last good checkpoint (or from the start, if there is none), trains to the next save, and hits the same limit again. Each retry spends GPU time.
3. After repeated crashes, the run is marked failed with a disk-quota error.

The run does not recover on its own. Free space or use a larger volume, then launch again.

## Fix or prevent a full volume

* **Keep fewer checkpoints.** Set `keep_last = 1` or `2` in `[ckpt]`.
* **Save less often.** Raise `[ckpt].interval`, or leave it unset to save only at the end.
* **Delete old outputs.** Open a read-write session and remove runs you no longer need:

  ```bash theme={null}
  prime volumes ssh training-outputs --read-write
  rm -rf /volume/runs/<oldRunId>
  ```

  Do not delete a checkpoint that another run resumes from with `[resume].dir`.
* **Grow the volume.** Expand it with `prime volumes expand <name> --size <size>` (see [Make a volume bigger](/hosted-training/volumes#make-a-volume-bigger)).


# Volume-hosted models
Source: https://docs.primeintellect.ai/hosted-training/volume-models

Train from model weights stored on your volume instead of the model cache

> [!NOTE]
>
> Volume-hosted models are part of the [Volumes](/hosted-training/volumes) beta.

Hosted SFT and RL runs can start from Hugging Face-format weights on a [volume](/hosted-training/volumes), including private models and weights exported from your own fine-tunes. Point `[model].name` at their absolute `/volume/...` path and attach the volume. No model-cache entry is needed.

Use this for:

* **Weights that aren't on Hugging Face**, such as private or proprietary checkpoints.
* **Continuing your own fine-tunes** from weights you exported to Hugging Face format.

## Quick start

Create a volume and upload a complete Hugging Face model directory, including `config.json`, safetensors weights, and tokenizer assets:

```bash theme={null}
prime volumes create research
prime volumes put research ./qwen3-hf models/
```

Point the config at them:

```toml theme={null}
max_steps = 100

[ckpt]
interval = 50
keep_last = 2

[model]
name = "/volume/models/qwen3-hf"      # Qwen3 weights at this path

[data]
name = "datasets/my-dataset"          # the dataset can live on the same volume
type = "sft"
seq_len = 4096

[deployment]
num_train_gpus = 1

[renderer]
name = "prime-qwen3"                  # match the staged model's family
```

Launch with the volume:

```bash theme={null}
prime train sft.toml --volume research
```

Size `[deployment]` for your model as described in [Hosted SFT](/hosted-training/hosted-sft#3-write-the-config). This example uses Qwen3 weights; choose the renderer for your model family.

## Use the model in an RL run

Start with a complete [RL training config](/hosted-training/end-to-end-run#step-4-create-a-training-config) and replace its shared model name:

```toml theme={null}
[model]
name = "/volume/models/qwen3-hf"

[orchestrator.renderer]
name = "prime-qwen3"
```

The shared name supplies the trainer, orchestrator, and inference model. If your config overrides model names for individual roles, change those names to the same directory too. Then launch with `prime train config.toml --volume research`.

Set the renderer explicitly for the model family. A filesystem path cannot identify the family from a Hugging Face model ID; the Qwen3 example above uses `prime-qwen3`.

RL configs with `[trainer]` and `[orchestrator]` blocks support volume model paths; a cached Hugging Face ID is not required for these paths.

## Rules

| Requirement | Rule |
| - | - |
| Volume required | The run must use the volume: `--volume <name>`, or `volume = "<name>"` in the config. |
| Path | Starts with `/volume/` and has at least one folder after it. Relative paths don't work here, unlike `data.name`. |
| Path characters | Each part starts with a letter or digit, followed by letters, digits, `.`, `-`, or `_`. `.` and `..` aren't allowed. |
| Weight format | A complete Hugging Face model directory, including config, weights, and tokenizer assets. A sharded training checkpoint is not a model directory: see [below](#continue-from-a-previous-runs-checkpoint). |
| Model cache | Not used, and not checked. |

The whole volume is mounted read-write, including earlier `runs/` directories. The trainer writes weight conversions to scratch space rather than overwriting the source model.

## Continue from a previous run's checkpoint

Run checkpoints are in prime-rl's sharded format, so they can't be used directly as `[model].name`. [Export a checkpoint to Hugging Face format](/hosted-training/end-to-end-run#export-to-hf-format), upload the exported directory under `models/`, and use its `/volume/models/...` path for a new SFT or RL run.

Starting from exported weights begins a new run; it does not restore the previous optimizer or training step. To resume the full training state, set `[resume].dir` to the checkpoint's step directory in the run TOML:

```toml theme={null}
[resume]
dir = "/volume/runs/<runId>/checkpoints/step_<N>"
```

## See also

* [Volumes](/hosted-training/volumes): create, upload, and download.
* [Hosted SFT](/hosted-training/hosted-sft): the walkthrough this builds on.
* [Hosted RL](/hosted-training/hosted-rl): RL configs and dispatch options.


# Volumes (Beta)
Source: https://docs.primeintellect.ai/hosted-training/volumes

Keep datasets, models, and checkpoints across Hosted Training runs

> [!NOTE]
>
> Volumes are in **beta** for Hosted RL and Hosted SFT.

A volume is named cloud storage for your training data and results. Upload datasets and models to it, use it in training runs, and download the checkpoints each run saves there. A volume belongs to your team (or to you, with no team selected) and isn't tied to any cluster.

> [!NOTE]
>
> Legacy staging is being retired; volumes work the same on every cluster.

## What you can do with and without a cluster allocation

| Task | Without a cluster allocation or team-owned cluster | With one |
| - | - | - |
| Create, list, grow, and delete volumes | Yes | Yes |
| Upload and download files (`put` / `get`) | Yes | Yes |
| Open a shell on a volume (`ssh`) | No | Yes |
| Use a volume in a training run (`--volume`) | Yes, on [on-demand](/hosted-training/billing) capacity (beta) for GPU types that support it | Yes |

On-demand training is in beta and needs beta access for your account or team. On demand, a volume works only on GPU types whose clusters support shared storage. On any other GPU type, the run is refused with a message that names the type. Pick another GPU type with `--gpu-type`, or run without `--volume`.

Shells need a cluster allocation for your account or team, or a cluster your team owns. On-demand capacity alone doesn't include SSH. Personal accounts can't own clusters, so they need an allocation. Ask Prime support if you need one.

## Quickstart

Install and log in to [Prime CLI](/cli-reference/introduction) 0.11.0 or later. Volumes use the team set in your CLI (`prime config set-team-id <team-id>`). With no team set, they belong to your personal account.

**1. Create a volume**

```bash theme={null}
prime volumes create research --size 500Gi
```

**2. Upload a dataset**

```bash theme={null}
prime volumes put research ./my-data datasets/
```

This creates `datasets/my-data` on the volume. Training runs see it at `/volume/datasets/my-data`.

**3. Optional: open a shell on the volume**

Needs [a cluster allocation or a cluster your team owns](#what-you-can-do-with-and-without-a-cluster-allocation) and an [SSH key](#set-up-your-ssh-key).

```bash theme={null}
prime volumes ssh research
```

After editing, exit the shell, run `prime volumes stop research <session-id>`, and wait for the final sync. Exiting alone leaves the session open and blocks training on the volume.

**4. Train with the volume**

Add a [checkpoint configuration](#checkpoint-settings) to save weights, then:

```bash theme={null}
prime train config.toml --volume research
```

**5. Download your checkpoints**

Wait for the run to end and its volume sync to finish, then:

```bash theme={null}
prime volumes get research runs/<runId>/checkpoints/ ./checkpoints
```

**6. Delete the volume when you're done**

```bash theme={null}
prime volumes delete research
```

## Things to know

* Each run writes checkpoints under `runs/<runId>/checkpoints/`. Previous runs are visible in the next run or shell at `/volume/runs/`. Direct `put` uploads to `runs/` are refused.
* A volume has one writer at a time: a training run, a read-write shell, or an upload. Wait for that writer to finish syncing before starting another. Read-only shells can coexist with a writer, but see the last saved copy.
* Changes made by a run or shell reach cloud storage when it ends. There is no periodic sync. Downloads during a run can return the previous saved contents.
* Each run gets a disk the size of your volume. It must fit the whole volume, including earlier runs, plus new checkpoints. See [Choose a size](#choose-a-size).
* Volumes are billed for their full size, whether or not anything uses them. See [Pricing](#pricing).

## Create a volume

Names use 1 to 40 lowercase letters, digits, and hyphens, and start and end with a letter or digit. The default size is `5Ti`, and the maximum is `20Ti`.

> [!WARNING]
>
> You pay for the size you set, not the data you store. The default `5Ti` costs about \$460.80 per month even when empty. Pass `--size` to pick a smaller volume; you can [grow it later](#make-a-volume-bigger). See [Pricing](#pricing).

```bash theme={null}
prime volumes create research
prime volumes create experiments --size 500Gi
```

The volume is ready as soon as the command returns. If you have a cluster allocation (or your team owns a cluster) and an SSH key, `create` also starts a read-only shell session in the background. That session stops after 30 minutes of no use. Pass `--no-warm` to skip it.

List your volumes:

```bash theme={null}
prime volumes list
```

```text theme={null}
┏━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Name     ┃ Size  ┃ Status  ┃ Created                          ┃ Created by ┃
┡━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ research │ 500Gi │ CREATED │ 2026-10-12T16:05:11.482000+00:00 │ Jane Doe   │
└──────────┴───────┴─────────┴──────────────────────────────────┴────────────┘
```

Add `-o json` for JSON output.

## Upload data

`put` uploads a file or folder. Paths on the volume start at its root:

```bash theme={null}
prime volumes put research ./my-data datasets/        # creates datasets/my-data
prime volumes put research ./my-data/ datasets/my-data # same, copying the folder's contents
prime volumes put research ./model.safetensors models/my-model/
```

* A trailing `/` on the volume path means "into this folder".
* A trailing `/` on a local folder copies what's inside it, not the folder itself.
* Volume paths can use letters, digits, and `._-@%+=,:`. For other names, use a [shell](#open-a-shell-on-your-volume).
* Symlinks inside a folder are skipped.
* You can't upload to `runs/`.

Uploads go straight to cloud storage, so they don't need a cluster allocation or an SSH key. While an upload runs, nobody can start another writer or delete the volume. A training run or someone else's read-write shell blocks uploads until it finishes syncing. If you have your own read-write shell open, `put` can upload through it instead; those changes reach cloud storage when you end the session.

## Download data

```bash theme={null}
prime volumes get research datasets/my-data/ ./my-data
prime volumes get research / ./everything
```

The local destination defaults to the current folder.

Direct downloads read the saved copy in cloud storage. Wait for a run or read-write shell to finish syncing to retrieve its latest changes. `get` can read ordinary files through your own read-write shell, but downloads under `runs/` always use cloud storage.

## Open a shell on your volume

A shell is handy for downloading data on the cluster's fast network (for example, with `hf download`), or for looking around and editing files. It needs [a cluster allocation or a cluster your team owns](#what-you-can-do-with-and-without-a-cluster-allocation) and an [SSH key](#set-up-your-ssh-key).

```bash theme={null}
prime volumes ssh research              # read-write (the default)
prime volumes ssh research --read-only
```

When the session starts, the whole volume is copied to the machine, including `runs/`, and you find it at `/volume`. A large volume takes a while to copy. The session becomes ready after the download completes.

* **Read-write:** changes are saved back when the session ends. Files you delete in the shell are deleted from cloud storage too. A training run or another read-write shell holds the writer lock until its final sync completes.
* **Read-only:** nothing is saved back. These sessions can coexist with a writer, subject to the session limit. They read the saved contents from when the session started.

Exiting the shell doesn't end the session. It ends after 30 minutes with no connection, or after 24 hours. To end it now:

```bash theme={null}
prime volumes sessions research
prime volumes stop research <session-id>
```

Only you can connect to or stop your sessions, even on a team volume.

Wait for the final sync after stopping a read-write session before downloading its changes or launching a training run on the same volume.

### Set up your SSH key

Shells accept one key: your account's **primary** SSH key. The CLI must point at the matching private key.

**1. Get a key pair**

Use an existing pair, or create one:

```bash theme={null}
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
```

**2. Add the public key to your account**

Open [Dashboard → Tokens → SSH Keys](https://app.primeintellect.ai/dashboard/tokens?tab=ssh-keys) and paste the contents of `~/.ssh/id_ed25519.pub`. A new key becomes your primary key. To switch later, click **Set Primary** on another key.

**3. Point the CLI at the private key**

```bash theme={null}
prime config set-ssh-key-path ~/.ssh/id_ed25519
```

The default is `~/.ssh/id_rsa`. `prime volumes ssh` prints `Using SSH key: <path>` before it connects.

If you never added a key, you can use the key pair under **System generated keys** on the same page: download the private key, run `chmod 600` on it, and point `prime config set-ssh-key-path` at it. Once you add your own key, the system-generated key stops working for shells.

## Use a volume in a training run

Pass `--volume` to a [Hosted RL](/hosted-training/hosted-rl) or [Hosted SFT](/hosted-training/hosted-sft) run:

```bash theme={null}
prime train config.toml --volume research
```

Or set `volume = "research"` at the top of the TOML, before any tables. If the volume doesn't exist, the CLI creates it. Use `--volume-size` to choose its size; without it, the new volume gets the default `5Ti` size and is [billed](#pricing) for it.

When the run starts, the whole volume is downloaded and mounted read-write at `/volume`. For example, a Hosted SFT `data.name = "datasets/my-data"` reads `/volume/datasets/my-data`. Earlier outputs are available under `/volume/runs/`. The run's changes are saved to cloud storage when it ends, after the training pods stop. The final sync runs when the run completes, stops, fails, or [pauses](/hosted-training/pause-resume), so checkpoints survive `prime train stop` and crashes.

End any read-write shell and wait for its final sync before launching the run. A training run holds the volume's writer lock until its own sync completes.

### Train from weights already on the volume

Both Hosted SFT and RL runs can load Hugging Face-format model weights from an absolute `/volume/...` path. Set `[model].name` in a Hosted SFT config, or set the shared `[model].name` in an RL config with `[trainer]` and `[orchestrator]` blocks:

```toml theme={null}
[model]
name = "/volume/models/my-model"
```

Attach the volume when launching:

```bash theme={null}
prime train config.toml --volume research
```

The directory must contain the model configuration, weights, and tokenizer assets. It does not need an entry in the cluster model cache. A sharded training checkpoint is a different format; [export it to Hugging Face format](/hosted-training/end-to-end-run#export-to-hf-format) before using it as a model directory. See [Volume-hosted models](/hosted-training/volume-models) for config examples and path rules.

### Checkpoint settings

Checkpointing is off unless you add `[ckpt]`. An empty `[ckpt]` block saves the final checkpoint; set `interval` to save during training too:

```toml theme={null}
max_steps = 100

[ckpt]
interval = 50       # save a checkpoint every 50 steps
keep_last = 2       # keep the 2 newest checkpoints
```

| Setting | What it does |
| - | - |
| `interval` | How often the run saves a checkpoint. The final checkpoint is also saved. |
| `keep_last` | How many recent checkpoints to keep. Pruned checkpoints are not recovered by the final volume sync. |
| `keep_interval` | Preserve checkpoints at multiples of this step in addition to the recent ones. It does not control upload timing. |

If you set only `keep_interval`, the final checkpoint can be pruned when its step is not a multiple of that interval. Set `keep_last` to at least `1` to keep the final checkpoint too.

> [!NOTE]
>
> Neither `keep_interval` nor `keep_last` is required. All files on `/volume` are synced when the run ends. RL runs write checkpoints, logs, and other run outputs there. Hosted runs currently do not save the resolved config needed for [checkpoint conversion](/hosted-training/end-to-end-run#export-to-hf-format). Hosted SFT writes checkpoints there; its other outputs remain on the run's scratch disk.

## Get your checkpoints back

Each run saves its checkpoints to `runs/<runId>/checkpoints/step_<N>/`:

```bash theme={null}
prime volumes get research runs/<runId>/checkpoints/ ./checkpoints
prime volumes get research runs/<runId>/checkpoints/step_100/ ./step_100
```

Wait for the final volume sync before downloading a new run's outputs. Complete trainer checkpoints contain `trainer/.metadata`. To load checkpoints outside Prime, convert them first: see [Export to HF format](/hosted-training/end-to-end-run#export-to-hf-format).

## Choose a size

Each run gets a disk the size of your volume. It must fit:

* all existing files, including datasets, models, and earlier `runs/` outputs, plus
* `keep_last + 1` checkpoints (the run writes a new one before it deletes the oldest), plus
* about 20% extra.

The `keep_last + 1` estimate assumes `keep_last` is set and `keep_interval` is unset. If you also preserve checkpoints with `keep_interval`, include those extra checkpoints. Without retention limits, size for every checkpoint the run saves.

A full fine-tuning checkpoint is about **11–12 bytes per parameter** (measured: \~341 GB for a 30B save, \~7.2 GB for a 0.6B save):

| Model size | One checkpoint |
| - | - |
| 0.6B | \~7.2 GB |
| 8B | \~100 GB |
| 30B (measured: Qwen3-30B-A3B) | \~341 GB |
| 100B | \~1.2 TB |

For example, Qwen3-30B-A3B with `keep_last = 2` holds up to 3 checkpoints, about 1 TB. With 20% extra and your datasets, use `1536Gi` (1.5 TiB). On `200Gi`, even the first checkpoint doesn't fit.

`ckpt.skip_optimizer = true` doesn't make checkpoints smaller.

For the full sizing formula, the peak checkpoint count, and how to fix a run that fails with `Disk quota exceeded`, see [Volume capacity](/hosted-training/volume-capacity).

> [!WARNING]
>
> Deleting saved files or checkpoints inside a read-write shell removes them from cloud storage when the session ends. Check what you need to keep before removing earlier runs to free space.

### Check disk usage

```bash theme={null}
prime volumes du research
```

`prime volumes du` (CLI 0.13.0 or later) shows how much of the volume's saved contents is in use. It reads cloud storage, so a live run's or session's changes appear only after it ends and syncs.

## Make a volume bigger

```bash theme={null}
prime volumes expand research --size 2Ti
```

Volumes can grow up to `20Ti` but can't shrink. Runs and shells already going keep their old size. New ones get the new size.

Billing switches to the new size from the moment the expand completes. Expanding runs the same [wallet checks](#pricing) as creating a volume.

## Pricing

Volume storage costs **\$0.09 per GiB per month** (\$0.000125 per GiB per hour, with a month counted as 30 days).

* **You pay for the volume's size, not its contents.** A `500Gi` volume costs the same whether it holds 1 GiB or 500 GiB. `prime volumes du` shows how much you use.
* **Billing runs from creation to deletion**, whether or not a run or shell is using the volume. It is prorated, with no minimum. Deleting the volume stops the charge.
* **Team volumes bill to the team's wallet**; personal volumes bill to yours.
* Volume storage is separate from the [GPU-hours](/hosted-training/billing) a training run uses.

| Size | Per month |
| - | - |
| `100Gi` | \$9.00 |
| `500Gi` | \$45.00 |
| `1Ti` | \$92.16 |
| `1536Gi` (1.5 TiB) | \$138.24 |
| `5Ti` (default) | \$460.80 |
| `20Ti` (maximum) | \$1,843.20 |

Creating or expanding a volume checks your wallet the same way a run does: you need a spendable balance, and the new hourly cost must fit within your spend limit and any team-member spending limits. If it doesn't, the command fails with `You've reached your Spend Limit` or `Insufficient funds in the wallet`. Add funds or raise the limit in the [Billing dashboard](https://app.primeintellect.ai/dashboard/billing), or choose a smaller size.

Track volume charges in the [Billing dashboard](https://app.primeintellect.ai/dashboard/billing).

## Delete a volume

> [!WARNING]
>
> Deleting a volume deletes everything on it, including every run's checkpoints.

```bash theme={null}
prime volumes delete research
```

Add `--yes` to skip the confirmation. Only team admins can delete team volumes.

You can't delete a volume while a run uses it, a shell is open on it, or an upload is running. Stop the run (`prime train stop <runId>`) or the shell (`prime volumes stop`), then try again. If the status shows `DELETE FAILED`, run `prime volumes delete` again.

## Troubleshooting

| Error | What it means | What to do |
| - | - | - |
| `SSH sessions need a cluster allocation or a team-owned cluster with shared storage, and this team has neither (on-demand capacity doesn't include SSH).` (for a personal account: `...a cluster allocation with shared storage, and this account has none...`) | Shells need a cluster allocation or a cluster your team owns. On-demand capacity alone doesn't include SSH. | Use `put` and `get` instead, or ask Prime support for a cluster allocation. Until the newer wording ships, you may see the older message `no cluster assigned to you has ReadWriteMany storage, which an SSH session needs to stage the volume; ask Prime support for one`, which means the same thing. |
| `On-demand training is in beta. Contact Prime support to request beta access for this account.` | Your account or team has no cluster allocation, so the run would use on-demand capacity, which is in beta. | Ask Prime support for beta access, or for a cluster allocation. |
| `No on-demand gpu_type=<type> cluster has ReadWriteMany storage, which a --volume run needs. Try another GPU type, or run without --volume.` | On demand, that GPU type doesn't support volumes. | Choose another GPU type with `--gpu-type`, or run without `--volume`. |
| `volume '<name>' has an active read-write session (...); use --read-only, or end that session first` | Someone else has a read-write shell open. | Use `--read-only`, or ask them to end their session. |
| `volume '<name>' has a live read-write SSH session (...); put through that session or end it first` | You tried to upload while someone else has a read-write shell open. | Upload from inside that shell, or wait for it to end. |
| `an upload to volume '<name>' is in progress until <time>Z; retry after it finishes` | An upload is running. | Wait for it to finish. A stopped upload clears within 15 minutes. |
| `volume '<name>' is mounted by N live run(s)` or `has N active volume SSH session(s)` | You tried to delete a volume that's in use. | Stop the runs or shells, then delete again. |
| `runs/ holds run outputs and is read-only; put elsewhere.` | You tried to upload to `runs/`. | Upload to another folder, such as `datasets/`. |
| `volume '<name>' is in use by a run (...) that writes to it` or `is attached read-write by a run (...)` | A training run holds the writer lock. | Wait for the run and its final volume sync to finish. |
| `volume contents (...) exceed the volume size (...); run prime volumes expand` | The saved contents do not fit in the requested disk. | Expand the volume before starting another run or shell. |
| `No space left on device` (`Errno 28`) | The volume is too small for your files plus checkpoints. | Stop the run, `prime volumes expand`, and start again. Or lower `keep_last`. |
| `volumes can only grow` | The new size isn't bigger than the current one. | Choose a larger size, up to `20Ti`. |
| `Volume SSH session limit (20) reached` | Your team has 20 shells open. | End some with `prime volumes stop`. |
| `Permission denied (publickey)` | Your CLI key doesn't match your primary key. | Repeat [Set up your SSH key](#set-up-your-ssh-key). |
| `no SSH public key is configured for your account` | Your account has no SSH key. | Add one in the dashboard. |


# Prime Inference
Source: https://docs.primeintellect.ai/inference/overview

Run models through an OpenAI-compatible API

Prime Inference provides one OpenAI-compatible API for both hosted and gateway models. Check the [model catalog](https://app.primeintellect.ai/dashboard/inference?tab=models) for current availability, serving type, and pricing; they vary by model.

## Hosted and gateway models

| Serving type | How the request is served |
| - | - |
| **Hosted** | Prime Intellect runs the model on its own inference infrastructure. `z-ai/glm-5.3` is a hosted model. |
| **Gateway** | Prime Intellect routes the request to an external inference provider. The model is not served on Prime Intellect infrastructure. |

Both use the same Prime API key and endpoint, and you select a model by its catalog ID. Pricing, limits, and supported features are set per model and shown in the catalog, so check them before choosing a model, especially if where a model runs matters for your workload.

## Get started

1. Create an API key with the **Inference** permission in the [API keys guide](/api-reference/api-keys).
2. Set `PRIME_API_KEY` in your environment. Keep it on the server, not in browser code or a public repository.
3. Send a chat completion with a model ID from the catalog. The example below uses [GLM-5.3](/inference/models/glm-5-3).

```bash theme={null}
export PRIME_API_KEY="your-api-key"

curl https://api.pinference.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $PRIME_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "z-ai/glm-5.3",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

### OpenAI Python SDK

```python theme={null}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["PRIME_API_KEY"],
    base_url="https://api.pinference.ai/api/v1",
)
response = client.chat.completions.create(
    model="z-ai/glm-5.3",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)
```

To list model IDs programmatically, call `GET https://api.pinference.ai/api/v1/models` or run `prime inference models`. Use the returned model ID exactly as shown. See the [Models API](/api-reference/inference-models) and [Chat Completions API](/api-reference/inference-chat-completions) for endpoint details. Evaluation models such as [Clef](/inference/models/clef) use the [Evaluations API](/api-reference/inference-evaluations) instead of chat completions.

## Team billing

Requests bill the API key owner's personal account unless you include `X-Prime-Team-ID`. To use team credits, obtain your team ID from the [team profile](https://app.primeintellect.ai/dashboard/team-profile) or `prime teams list`, and add the header to **each** direct API request:

```bash theme={null}
-H "X-Prime-Team-ID: your-team-id"
```

With the OpenAI SDK, set `default_headers={"X-Prime-Team-ID": "your-team-id"}` on the client. `prime config set-team-id` applies to CLI operations, not to direct HTTP or SDK requests. Check usage and balances in the [billing dashboard](https://app.primeintellect.ai/dashboard/billing). If a request returns `insufficient_funds`, check the balance of the account selected by the key and optional team header.

## Next steps

* [GLM-5.3](/inference/models/glm-5-3): use a model hosted on Prime Inference.
* [Clef](/inference/models/clef): classify and score content with a hosted evaluation model.
* [Troubleshooting](/inference/troubleshooting): resolve billing errors.
* [Evaluate models](/tutorials-environments/evaluating) against environments.


# Troubleshooting
Source: https://docs.primeintellect.ai/inference/troubleshooting

Resolve common Prime Inference errors

## Insufficient funds

> **The `insufficient_funds` error**
>
> Prime Inference returns this error when the account billed for the request does not have enough balance. The API key identifies the key owner's personal account, and the `X-Prime-Team-ID` header switches billing to that team. The Prime CLI team setting does not apply to direct API requests.
>
> ```json theme={null}
> {
>   "error": {
>     "message": "Insufficient balance (including overdraft). Please add funds to continue.",
>     "type": "insufficient_quota",
>     "code": "insufficient_funds"
>   }
> }
> ```

> **Insufficient funds on a personal account**
>
> Requests without `X-Prime-Team-ID` bill the API key owner's personal account.
>
> * Add funds to your personal balance in the [billing dashboard](https://app.primeintellect.ai/dashboard/billing)
> * Check that the API key belongs to the account you funded
> * Retry the request

> **Insufficient funds on a team**
>
> Requests bill a team only when they include `X-Prime-Team-ID`, and the key owner must have access to that team.
>
> * Add funds to the team's billing balance
> * Include `X-Prime-Team-ID` on every request, including SDK requests (see [team billing](/inference/overview#team-billing))
> * Find the team ID on the [team profile page](https://app.primeintellect.ai/dashboard/team-profile) or with `prime teams list`

## Getting help

If your issue isn't covered here:

* **Discord:** post in the #inference channel on our [Discord](https://discord.gg/ZTFydGWPKj)
* **Support:** contact [contact@primeintellect.ai](mailto:contact@primeintellect.ai)


# Introduction
Source: https://docs.primeintellect.ai/introduction

Welcome to the Prime Intellect Documentation

## Platform

Train, evaluate, and deploy AI models with our frontier research infrastructure.

> **[Hosted Training](/hosted-training/getting-started)**
>
> Run large-scale RL training on-demand, without managing infrastructure

> **[Environments Hub](/tutorials-environments/environments)**
>
> Create, share, and discover RL environments for training and evaluation

> **[Evaluations](/tutorials-environments/evaluating)**
>
> Run evaluations on your environments with hosted inference

> **[Sandboxes](/sandboxes/overview)**
>
> Secure code execution environments for AI agents

> **[Guides](/guides)**
>
> Step-by-step workflows for training and environments

## Libraries

Open source tooling for environments, evaluations, and large-scale RL training.

> **[verifiers](/verifiers/overview)**
>
> Our library for environments and evals

> **[prime-rl](/prime-rl)**
>
> Our large-scale async RL framework

## Inference

Use hosted models through an OpenAI-compatible API.

> **[Get started](/inference/overview)**
>
> Make your first inference request

> **[GLM-5.3](/inference/models/glm-5-3)**
>
> Use GLM-5.3 hosted on Prime Inference

## Compute

Access GPUs and infrastructure for AI workloads.

> **[On-Demand Cloud](/cli-reference/provision-gpu)**
>
> Deploy single GPU instances in under a minute

> **[Storage](/tutorials-storage/create-persistent-storage)**
>
> Persistent storage for datasets and checkpoints

> **[Reserved Clusters](/model-factory/monitoring)**
>
> Dedicated GPU clusters with monitoring

## Resources

> **[API Reference](/api-reference/introduction)**
>
> REST API documentation

> **[CLI Reference](/cli-reference/introduction)**
>
> Prime CLI commands

> **[Compute FAQ](/faq)**
>
> Common questions


# Model Factory Clusters
Source: https://docs.primeintellect.ai/model-factory/clusters

Dedicated clusters running training, inference, Slurm and Kubernetes side by side

Model Factory Clusters are dedicated clusters operated for you. Where [Hosted Training](/hosted-training/getting-started) gives you a managed run, a cluster gives you the machine underneath it.

> [!NOTE]
>
> Clusters are set up with our team rather than self-serve, so this page covers what they are and how to get one. Configuration and operational guides are in progress.

## What you get

* **Everything at once**: training, inference, Slurm and Kubernetes all running side by side on the same cluster, not one or the other
* **One-click inference**: deploy a model as an inference endpoint on your own cluster from the dashboard, without assembling a serving stack
* **Control over the stack**: a cluster is what you need when you want to pin the exact commit of the training framework rather than take the managed build

## Getting access

[Contact us](mailto:contact@primeintellect.ai) to discuss a cluster.

> **[Hosted Training](/hosted-training/getting-started)**
>
> Dispatch a managed, on-demand run instead.

> **[Slurm Orchestration](/model-factory/slurm-orchestration)**
>
> How Slurm is set up on Prime Intellect clusters.


# Cluster Monitoring
Source: https://docs.primeintellect.ai/model-factory/monitoring

Detect issues early and minimize downtime with production-ready monitoring

![Cluster Monitoring Dashboard](https://mintcdn.com/primeintellect/IpQNxUo8HXFlKuu0/images/reserved-clusters/dashboard.png?fit=max\&auto=format\&n=IpQNxUo8HXFlKuu0\&q=85\&s=c2a2e85e66a81c50e35974eab0fbf3cf)

## Overview

Reserved clusters come with production-ready monitoring out of the box—no setup required. Our monitoring is built to meet the [ClusterMAX™](https://www.clustermax.ai/monitoring) standard, the industry benchmark for GPU cloud infrastructure developed by SemiAnalysis.

Detect hardware issues early, minimize downtime, and keep your training runs on track with:

* **Preconfigured Grafana dashboards** covering critical GPU, node, and network metrics
* **Proactive alerting** delivered to your preferred channels the moment issues arise
* **GPU health monitoring** including XID error detection and hardware anomaly tracking

## Accessing the Dashboard

Your cluster includes a Grafana dashboard for visualizing metrics. Access is provided during the cluster handover process—once your reserved cluster is ready, you will receive the dashboard URL and credentials from the Prime Intellect team.

## Available Metrics

The monitoring dashboard includes preconfigured graphs for critical metrics across your cluster:

| Category | Metrics | Source |
| - | - | - |
| GPU | Utilization, memory, temperature, power usage, interconnect throughput (PCIe, NvLink) | [DCGM Exporter](https://github.com/NVIDIA/dcgm-exporter) |
| Node | CPU, memory, disk, and network usage | [Node Exporter](https://github.com/prometheus/node_exporter) |
| InfiniBand | Throughput, port errors, and discards | [Node Exporter](https://github.com/prometheus/node_exporter) |
| Slurm | Node states, cluster utilization, and running jobs | [Slurm Exporter](https://github.com/PrimeIntellect-ai/slurm-exporter) |

![GPU Metrics Dashboard](https://mintcdn.com/primeintellect/IpQNxUo8HXFlKuu0/images/reserved-clusters/gpu-metrics.png?fit=max\&auto=format\&n=IpQNxUo8HXFlKuu0\&q=85\&s=8253ba3e373d4f73457379531bc57da6)

## Troubleshooting with the Dashboard

When a training job fails, use the dashboard to quickly identify the root cause. In large clusters, alerts eliminate the need to manually hunt for the bad node—they tell you exactly which node and device has the issue.

### 1. Check XID Alerts First

Start with the **Alert list** in the Cluster Overview. XID alerts identify the exact node and GPU device experiencing issues, so you can pinpoint problems immediately without searching through hundreds of nodes.

XID errors are NVIDIA's GPU error codes reported by the kernel driver. Each code indicates a specific hardware or software issue—from memory errors to thermal shutdowns. Use the [NVIDIA XID Error Catalog](https://docs.nvidia.com/deploy/xid-errors/index.html) to look up the error code and understand the recommended action.

### 2. Check InfiniBand Performance

Next, expand the **InfiniBand Metrics** section and look for:

* **Port Errors** - Non-zero values indicate connectivity problems
* **Throughput drops** - May explain slow collective operations during distributed training

### 3. Review Secondary Metrics

For additional context, check:

* **GPU Temperature** - Spikes above 80°C may indicate cooling issues
* **GPU Utilization** - Sudden drops to 0% during training suggest failures
* **Slurm Nodes State** - See if any nodes are marked as "down"

### Adjust Time Range

Use the time picker (top right) to zoom into the incident window or compare with historical baseline.

## Alerting

Get notified immediately when something needs attention. Alerts are triggered when metrics exceed critical thresholds, so you can respond before issues impact your workloads.

### Notification Channels

Alerts can be delivered to your preferred channels:

* **Slack** - Receive alerts directly in your Slack workspace
* **Microsoft Teams** - Get notifications in your Teams channels
* **Webhooks** - Integrate with any provider that accepts webhooks

To configure your notification channel, contact the Prime Intellect team with your preferred destination.

Your cluster comes with battle-tested alert rules based on our experience operating large-scale GPU clusters.

### Creating Custom Alerts

You can create custom alert rules directly in Grafana for your specific workloads. Here's how:

**1. Open Alert Rules**

In your Grafana dashboard, navigate to **Alerting** → **Alert rules** → **+ New alert rule**.

**2. Define Query and Condition**

Select your data source and write a query for the metric you want to monitor. Then set the condition that triggers the alert (e.g., `GPU temperature > 80°C`).

**3. Set Evaluation Behavior**

Configure how often the rule is evaluated and how long the condition must be true before firing. This helps prevent false alarms from brief spikes.

**4. Configure Notifications**

Choose where to send alerts—use an existing contact point (Slack, Teams, webhook) or create a new one.

**5. Save and Activate**

Click **Save rule** to activate your custom alert.

For detailed documentation, see the [Grafana Alerting guide](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/).


# Slurm Orchestration
Source: https://docs.primeintellect.ai/model-factory/slurm-orchestration

Submit and manage distributed jobs on a dedicated cluster with Slurm

Slurm is an open-source workload manager and job scheduler for high-performance computing clusters. Dedicated clusters arrive with Slurm configured and shared storage mounted, so you can submit distributed jobs without setting any of it up.

# Essential Slurm Commands

Run these commands on the cluster's controller node to manage your jobs and inspect the fleet:

## View Cluster Information

```bash theme={null}
# Display information about nodes and partitions
sinfo

# Show detailed node information
sinfo -Nel

# Display partition summary
sinfo -s

# Example output:
# PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
# gpu*         up   infinite      2   idle node[001-002]
```

## GPU Resource Allocation

Prime Intellect clusters use the Generic Resource (GRES) system for GPU allocation. Understanding the correct syntax is crucial for successful job submission.

### Interactive GPU Sessions

```bash theme={null}
# Request an interactive session with GPUs (use --gres for batch scripts)
srun --gpus=1 --pty bash

# Request multiple GPUs across nodes
srun --nodes=2 --gpus-per-node=8 nvidia-smi

# Request specific node with GPUs
srun --gpus=4 --nodelist=node001 --pty bash

# Alternative GRES syntax (required for batch scripts)
srun --nodes=2 --gres=gpu:8 --ntasks-per-node=1 nvidia-smi

# Once in the session, verify GPU access
nvidia-smi

# Check available GPUs in your session
nvidia-smi -L
```

> [!NOTE]
>
> When writing batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N` to avoid InvalidAccount errors.

## Batch Job Submission

Batch jobs allow you to queue work that runs without manual intervention. Create a script file (e.g., `job.sh`) with SBATCH directives:

### Basic GPU Job Script

```bash theme={null}
#!/bin/bash
#SBATCH --job-name=gpu-test
#SBATCH --nodes=2
#SBATCH --gres=gpu:8              # IMPORTANT: Use gres for GPUs in batch scripts
#SBATCH --ntasks-per-node=1
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out        # %x=job-name, %j=job-id
#SBATCH --error=%x-%j.err

echo "Job started on $(date)"
echo "Running on nodes: $SLURM_JOB_NODELIST"

# Run nvidia-smi on all allocated nodes
srun -l nvidia-smi

# Your training or compute commands here
# srun python train.py
```

### Submit and Manage Jobs

```bash theme={null}
# Submit a batch job
sbatch job.sh

# View queued and running jobs
squeue

# View your jobs only
squeue -u $USER

# Cancel a job
scancel <job_id>

# Cancel all your jobs
scancel -u $USER

# View detailed job information
scontrol show job <job_id>
```

### Job Output Location

> [!WARNING]
>
> Job output files are written to the directory where `sbatch` was executed, which may be on the compute node's local filesystem if you're not in shared storage. Always submit jobs from the shared storage directory to ensure output accessibility.

## Example: Quick Cluster Test

Verify your Slurm cluster with these simple tests:

```bash theme={null}
# Test 1: Check all GPUs across nodes (interactive)
srun --nodes=2 --gpus-per-node=8 nvidia-smi

# Test 2: Verify task distribution across nodes
srun --nodes=2 --ntasks-per-node=2 hostname

# Test 3: Check task distribution
srun --nodes=2 --ntasks-per-node=2 bash -c 'echo "Task $SLURM_PROCID running on node $(hostname)"'
```

# Troubleshooting Common Issues

## InvalidAccount Error

If you encounter `InvalidAccount` errors when submitting batch jobs:

1. **Use correct GPU syntax**: In batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N`
2. **No accounting plugin**: Prime Intellect clusters intentionally don't use Slurm's accounting plugin since clusters are single-tenant with dedicated resources. Remove any `#SBATCH --account=` directives from your scripts
3. **Check partition availability**: Ensure the partition you're requesting exists with `sinfo`

> [!NOTE]
>
> Prime Intellect clusters run without the Slurm accounting plugin because each cluster is single-tenant with dedicated resources. This simplifies configuration and eliminates account-based resource restrictions, giving you full access to all allocated resources without quota management overhead.

### Incorrect (causes InvalidAccount)

```bash theme={null}
#!/bin/bash
#SBATCH --gpus-per-node=8  # Wrong for batch scripts
```

### Correct

```bash theme={null}
#!/bin/bash
#SBATCH --gres=gpu:8       # Correct for batch scripts
```

## Job Output Not Found

If you can't find your job output files:

* **Check working directory**: Output files are created where `sbatch` was run
* **Use absolute paths**: Specify full paths in `--output` and `--error` directives
* **Check other nodes**: If submitted from a compute node, outputs may be on that node's local storage
* **Always submit from shared storage**: Change to the shared storage directory before running `sbatch`

## Node Communication Issues

If nodes can't communicate or jobs hang:

* Verify all nodes are in `idle` state with `sinfo`
* Check node connectivity: `srun --nodelist=<node> hostname`
* Ensure shared storage is mounted on all nodes
* Consider restarting your cluster if issues persist

# Advanced Slurm Commands

## Direct Node Access

Access specific compute nodes directly for debugging or monitoring:

```bash theme={null}
# SSH into a specific node via srun
srun --nodelist=computeinstance-abc123 --pty bash

# Run commands on specific nodes
srun --nodelist=node001,node002 hostname

# Allocate resources without running a command
salloc --nodes=2 --gres=gpu:8 --time=01:00:00
# Then use srun within the allocation
srun nvidia-smi
```

## Resource Monitoring

```bash theme={null}
# View detailed node status
scontrol show node

# Check GPU allocation
scontrol show node | grep -E "NodeName|Gres"

# Monitor job efficiency
seff <job_id>

# View job accounting information
sacct -j <job_id> --format=JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode

# Check cluster utilization
sreport cluster utilization
```

## Job Arrays for Parameter Sweeps

Run multiple similar jobs with different parameters:

```bash theme={null}
#!/bin/bash
#SBATCH --job-name=param-sweep
#SBATCH --array=1-10
#SBATCH --gres=gpu:1
#SBATCH --output=sweep_%A_%a.out  # %A=array job ID, %a=array task ID

# Use SLURM_ARRAY_TASK_ID for different parameters
python train.py --seed=$SLURM_ARRAY_TASK_ID --lr=$(echo "0.001 * $SLURM_ARRAY_TASK_ID" | bc)
```

## Environment Variables

Useful Slurm environment variables available in jobs:

```bash theme={null}
echo "Job ID: $SLURM_JOB_ID"
echo "Job Name: $SLURM_JOB_NAME"
echo "Node List: $SLURM_JOB_NODELIST"
echo "Number of Nodes: $SLURM_JOB_NUM_NODES"
echo "Tasks per Node: $SLURM_NTASKS_PER_NODE"
echo "CPUs per Task: $SLURM_CPUS_PER_TASK"
echo "Submit Directory: $SLURM_SUBMIT_DIR"
echo "Task ID: $SLURM_PROCID"
echo "Node ID: $SLURM_NODEID"
```

# Shared Storage Integration

Your Slurm cluster comes with shared storage automatically mounted on all nodes. The UI displays the mount path for your shared storage directory. This ensures:

* Consistent file access across all compute nodes
* No need to manually copy data between nodes
* Simplified job submission and management
* Persistent storage for checkpoints and results

## Best Practices

> **Use the Controller Node**
>
> Always submit jobs and run Slurm commands from the controller node, not compute nodes

> **Leverage Shared Storage**
>
> Store your code, data, and outputs in the shared storage directory shown in the UI for seamless access across nodes

> **Monitor Resources**
>
> Regularly check cluster utilization with `sinfo` and `squeue` to optimize job scheduling

> **Use Job Arrays**
>
> For parameter sweeps or similar tasks, use Slurm job arrays for efficient scheduling


# Platform Overview
Source: https://docs.primeintellect.ai/platform-overview

How environments, training, evaluation, and inference fit together on Prime Intellect

Prime Intellect is an open research platform for post-training. It unifies the [Environments Hub](/tutorials-environments/environments), [Hosted Training](/hosted-training/getting-started), and [Hosted Evaluations](/tutorials-environments/hosted-evaluations) into a full-stack platform for reinforcement learning research and optimization.

The platform enables the entire lifecycle of post-training research — from large-scale agentic RL, to inference and evaluation — without needing to worry about the costs of massive GPU clusters or the details of low-level algorithm implementation.

## Core Concepts

### Environments

The platform is built around **environments**, which contain everything needed to run a model on a task:

* A **dataset** of tasks (input prompts with optional ground-truth answers)
* A **harness** for the model (tools, sandboxes, context management, multi-turn interactions)
* A **rubric** to score the model's performance (reward functions, metrics)

Environments are self-contained Python modules built with the [verifiers](https://github.com/PrimeIntellect-ai/verifiers) library. They can be used to train models with reinforcement learning, evaluate capabilities, generate synthetic data, optimize prompts, experiment with agent harnesses, and more. See [The Environment Model](/hosted-training/environment-model) for a detailed breakdown of environment types and the rollout loop.

### Environments Hub

The [Environments Hub](https://app.primeintellect.ai/dashboard/environments) is a community registry where you can browse and deploy open-source environments, or create and share your own. Environments support running evaluations with any OpenAI-compatible model endpoint, or training any supported open-weights model via Hosted Training.

The hub includes a range of pre-built environments spanning math, coding, games, search and tool use, multimodal tasks, and more — as well as validated implementations of popular benchmarks like AIME, MATH-500, and Humanity's Last Exam.

### Model Factory

**Model Factory** is the training side of the platform, built on the open-source [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) training library. It covers two things:

* **[Hosted Training](/hosted-training/getting-started)** — submit a config and we run it on GPUs we operate, on-demand and [billed per GPU-hour](/hosted-training/billing). Full-parameter RL and SFT, no cluster to reserve.
* **[Model Factory Clusters](/model-factory/clusters)** — a dedicated cluster running training, inference, Slurm and Kubernetes side by side, for when you need control over the stack.

The earlier shared multi-tenant LoRA service has been [retired](/hosted-training/shared-lora-retired).

### Hosted Evaluations

Run evaluations against any environment directly from the web UI or CLI, using Prime Inference or your own OpenAI-compatible endpoints. Results are saved and viewable in the terminal UI or on the dashboard.

## Architecture

Training is powered by the [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) framework. The three key components are:

* **Trainer** — Consumes batches and updates model weights
* **Inference** — Serves the model via an OpenAI-compatible API with live weight updating
* **Orchestrator** — Manages environment logic, schedules rollout requests, and coordinates the training loop

Each Hosted Training run gets its own trainer, inference, and orchestrator pods, sized by the `[deployment]` block in your config.

## What You Can Do

| Capability | Description |
| - | - |
| **RL Training** | Train open-weights models with reinforcement learning on your custom environments |
| **Evaluation** | Benchmark any OpenAI-compatible model against environments on the Hub |
| **Environment Development** | Build, test, and share environments using the verifiers library |
| **Synthetic Data Generation** | Generate rollout data for downstream use |

## Key Libraries

The platform integrates with two core open-source libraries:

**[verifiers](https://github.com/PrimeIntellect-ai/verifiers)** — The environment framework. Provides building blocks for datasets, rubrics, tools, sandboxes, and multi-turn interactions. Environments built with verifiers are portable across local evaluation, hosted evaluation, and Hosted Training.

**[prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)** — The training framework. Supports fully asynchronous distributed RL at scale with FSDP2 training and vLLM inference. Used both for self-hosted training on your own GPUs and as the backend for Hosted Training.

## Getting Started

1. [Create a Prime Intellect account](https://app.primeintellect.ai)
2. Install the Prime CLI: `uv tool install prime`
3. Authenticate: `prime login`
4. See what you can run: `prime train models`

> **[Getting Started](/hosted-training/getting-started)**
>
> Launch your first on-demand Hosted Training run.

> **[End-to-End Training Run](/hosted-training/end-to-end-run)**
>
> Walk through a complete Hosted Training run.


# Advanced
Source: https://docs.primeintellect.ai/prime-rl/advanced



This page covers the specialized features layered on top of the core training stack: our custom model implementations (with EP for MoE families and CP for long-context training), multimodal training, LoRA training, and disaggregated prefill/decode inference. For developer-side workflows (adding new model architectures, debugging modeling code at small scale), see [Development](/prime-rl/development).

## Table of Contents

* [Custom Modeling](#custom-modeling)
  * [Expert Parallelism Backends](#expert-parallelism-backends)
  * [Runtime Fusions](#runtime-fusions)
* [Multimodal Training](#multimodal-training)
  * [Supported Families](#supported-families)
  * [Enabling VLM Mode](#enabling-vlm-mode)
  * [Limitations](#limitations)
* [LoRA Training](#lora-training)
* [Disaggregated Prefill/Decode Inference](#disaggregated-prefilldecode-inference)

## Custom Modeling

The trainer only runs `prime-rl`'s own model implementations, selected from the HF config type. Besides dense Llama, Qwen3 and Qwen3.5, these cover the families below. Other architectures fail at trainer setup.

| Family | HF config types | EP | CP |
| - | - | - | - |
| GLM-5 / GLM-5.2 (`glm_moe_dsa`) | `zai-org/GLM-5`, `zai-org/GLM-5-FP8`, `zai-org/GLM-5.2`, `zai-org/GLM-5.2-FP8` | ✅ | ✅ |
| Qwen3 MoE | `Qwen/Qwen3-30B-A3B`, … | ✅ | ✅ |
| Qwen3.5 MoE | `Qwen/Qwen3.5-35B-A3B`, … | ✅ | ✅ |
| Qwen3.5 VLMs | see [Multimodal training](#multimodal-training) | MoE only | ✅ |
| Laguna | `poolside/Laguna-XS.2` | ✅ | ✅ |
| MiniMax M2 | `MiniMax/MiniMax-M2` | ✅ | ✅ |
| Nemotron H | `nvidia/Nemotron-3-Nano-30B-A3B`, … | ✅ | ❌ |
| Trinity (AFMoE) | `arcee-ai/Trinity-Mini`, … | ✅ | ✅ |
| GLM-4 / GLM-4.5 / INTELLECT-3 | `THUDM/GLM-4-9B-0414`, `zai-org/GLM-4.5`, `PrimeIntellect/INTELLECT-3`, … | ✅ | ✅ |
| GPT-OSS | `unsloth/gpt-oss-20b-BF16`, … | ✅ | ✅ |
| DeepSeek V4 | `deepseek-ai/DeepSeek-V4-Flash-0731` | ✅ | ✅ |

GPT-OSS uses FlashAttention 4 with learned attention sinks. Training requires SM90 or SM100/SM110 GPUs
and a BF16 checkpoint such as `unsloth/gpt-oss-20b-BF16`; the original MXFP4 checkpoints are not supported.

### Low-precision training

Dense linear precision and routed-expert precision are configured independently. `[trainer.model.quantization]` applies only to dense `Linear` modules:

* `type = "fp8"` — DeepGEMM FP8 blockwise linears (requires SM90+).
* `type = "mxfp8"` — torchao MXFP8 linears (requires SM100). `recipe` is `mxfp8_rceil` or `mxfp8_rceil_wgrad_with_hp`.

`[trainer.model.moe.compute]` selects routed-expert compute independently:

* `type = "bf16"` (default), with `backend = "torch"` (default) or `"sonicmoe"`.
* `type = "deepgemm_fp8"` (requires DeepGEMM and SM90+)
* `type = "mxfp8"` (requires `prime-kernels`, torchao, and SM100)

SonicMoE uses the upstream `sonic-moe` package (`uv sync --extra sonic-moe`) for fused BF16 expert computation. The supported model is Qwen3 MoE with the `gate_up` model fusion enabled. Backend selection requires fused gate/up weights, standard SwiGLU, and bias-free experts; incompatible expert structures raise an error during setup. It uses the same router and local, torch EP, or DeepEP dispatch as other compute backends:

```toml theme={null}
[trainer.model]
name = "Qwen/Qwen3-30B-A3B"
ep = 2

[trainer.model.fusions]
enabled = ["gate_up"]

[trainer.model.moe.compute]
type = "bf16"
backend = "sonicmoe"

[trainer.model.moe.dispatch]
type = "torch"
```

```toml theme={null}
[trainer.model.quantization]
type = "mxfp8"
recipe = "mxfp8_rceil"

[trainer.model.moe.compute]
type = "mxfp8"
recipe = "mxfp8_rceil"

[trainer.model.moe.dispatch]
type = "torch"
transport = "mxfp8"
```

All MoE compute backends accept `apply_to`:

* `"all"` (default) applies the backend to all expert groups.
* `"85%"` applies it to the first 85% of model layers, rounded down. For a 48-layer model, this selects layers 0–39.
* `[0, 1, 2, 3]` selects explicit zero-based model layer indices; `[]` selects none.

Percentages must be between 0% and 100%; explicit indices must be within the model's layer count. Non-MoE blocks in hybrid models count toward layer indices and percentages. Each selected layer uses the backend for all its routed experts. Other expert groups use BF16 compute and BF16 token transport while retaining the configured dispatch backend and expert parallelism. Dense linear quantization is configured separately.

For example, this selects routed experts in Qwen3's first four model layers:

```toml theme={null}
[trainer.model.moe.compute]
type = "mxfp8"
apply_to = [0, 1, 2, 3]
```

Backend shape checks and token alignment apply only to the selected compute path.

In RL runs, configure the same precision selection for rollouts. Inference module names can differ from the trainer's names, and inference precision is configured explicitly, not inferred from `apply_to`. Check the selected modules on both sides before comparing trainer and rollout logprobs.

GLM-5.2 adds IndexShare: the DSA sparse-attention indexer runs only on a subset of layers and the remaining layers reuse the cached top-k indices. The trainer reads this schedule from the model's `indexer_types` config field and enables the index cache automatically, so no extra config is needed.

### Expert Parallelism Backends

`[trainer.model.moe.dispatch]` selects how routed tokens are dispatched and combined:

* **`torch`** (default): torch all-to-all with `transport = "bf16"` or, when MXFP8 expert compute is selected, `transport = "mxfp8"` on SM100.
* **`deepep`**: DeepEP custom dispatch/combine kernels. Set `num_sms` and optional `token_chunk_size` in the same table. Pre-built H100/H200 binaries use CUDA 13.0 and are installed by `uv sync --all-extras`.

```toml theme={null}
[trainer.model.moe.dispatch]
type = "deepep"
num_sms = 20
token_chunk_size = 4096
```

With DeepEP, gradient clipping is currently not supported. (`optim.max_norm` is set to `None` automatically.)

### Runtime Fusions

`model.fusions` packs parameters that are always computed together into one tensor, turning several GEMMs into one. Both fusions are on by default:

* `gate_up` — each gated MoE expert's `gate_proj` and `up_proj` become one `[num_experts, 2 * intermediate_size, hidden_size]` weight, halving the routed-expert grouped GEMMs.
* `qkv` — an attention layer's `q_proj`, `k_proj` and `v_proj` (and their biases) become one linear layer.

```toml theme={null}
[trainer.model.fusions]
enabled = ["gate_up", "qkv"]   # [] disables
```

Fusions are runtime-only. Checkpoints keep the canonical parameter names and shapes, so a run can turn a fusion on or off at any point and still load its own checkpoints, and exported weights are unaffected. Only modules that support a fusion are packed; a requested fusion that no module supports logs a warning and is skipped. Fusions are skipped when LoRA is enabled.

Muon receives the packed layout as matrix partitions and orthogonalizes each logical matrix on its own, so a packed parameter trains exactly as the parameters it replaces would — including per-projection learning-rate scaling for grouped-query attention — while keeping a single momentum tensor.

The experimental `shard_fused_on_dim1 = true` shards fused 2-D weights along dim 1 under FSDP, which makes weight loading and checkpointing zero-copy: fused weights and their optimizer state are read and written in place rather than assembled into a full copy on each rank first. It requires `hidden_size` to be divisible by the FSDP shard mesh size.

## Multimodal Training

### Supported Families

The built-in VLM registry covers:

| Family | `model_type` | Vision attr | LM attr |
| - | - | - | - |
| Qwen3.5 | `qwen3_5` | `model.visual` | `model.language_model` |
| Qwen3.5-MoE | `qwen3_5_moe` | `model.visual` | `model.language_model` |

### Enabling VLM Mode

Add `[model.vlm]` and bfloat16 dtypes:

```toml theme={null}
[model]
name = "Qwen/Qwen3.5-4B"
optimization_dtype = "bfloat16"
reduce_dtype = "bfloat16"

[model.vlm]
vision_encoder_attr = "model.visual"
language_model_attr = "model.language_model"
# freeze_vision_encoder = true  # default; set false to fine-tune the encoder
```

The weight-broadcast key prefix is derived as `{language_model_attr}.layers.` automatically.

VLM training requires a registered custom PrimeRL implementation.

### Limitations

* **Vision encoder frozen by default.** The default LoRA targets do not match Qwen3.5 vision modules. Set `freeze_vision_encoder = false` to fine-tune the encoder; this is incompatible with LoRA because LoRA freezes all non-adapter parameters.
* **bfloat16 mandatory.** The trainer config validator refuses any other `optimization_dtype` / `reduce_dtype` for VLMs — vLLM serves VLMs in bfloat16 and a mismatch breaks the importance ratio.
* **Higher KL mismatch with multi-image inputs.** Expect noisier `mismatch_kl` than text-only; this is from minor numerical differences between the trainer's and vLLM's image processing.
* **Images aren't logged to monitors.** Sample logging captures the prompt text but not the actual images.

## LoRA Training

LoRA is enabled by adding `[model.lora]`:

```toml theme={null}
[model.lora]
rank = 16
alpha = 32
dropout = 0.0
```

`target_modules` defaults to a reasonable cross-family set (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `experts`, plus a few latent-projection names for Nemotron). Unknown names are silently ignored, so the defaults work across architectures. Add architecture-specific names to extend coverage (e.g. `in_proj` / `out_proj` for Mamba).

LoRA is supported across SFT and RL. NCCL weight broadcast is **not** supported with LoRA — the default NCCL transport automatically falls back to filesystem when LoRA is enabled. Broadcast dirs of LoRA runs contain the raw adapter (`adapter_model.safetensors` + `adapter_config.json`). With LoRA on, the inference server registers the adapter under the model name and serves the base model as `<model>-base`.

## Disaggregated Prefill/Decode Inference

For large MoE serving, splitting prefill and decode onto separate vLLM groups can substantially improve throughput. Pick the prefill:decode ratio based on workload shape:

| Workload | P:D ratio | Why |
| - | - | - |
| Agentic (SWE, Lean) | 3:1 | Long growing contexts → prefill-heavy |
| Non-agentic (math, chat) | 1:2 | Short prompts, long generations → decode-heavy |

Example config: [`examples/advanced/glm-5.3/swe.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/advanced/glm-5.3/swe.toml) — full RL run on `GLM-5` with P/D disaggregation behind a `vllm-router`, FP8 inference, and NCCL weight broadcast, paired with an inference config from [`examples/advanced/glm-5.3/infer/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.3/infer).

Monitor live queue depths to detect imbalance:

```bash theme={null}
curl -s http://<prefill_node>:8100/metrics | grep num_requests_waiting
curl -s http://<decode_node>:8200/metrics | grep num_requests_waiting
```

If prefill queues and decode is idle, add prefill nodes (and vice versa).

**Required setup for disaggregated P/D (NIXL/UCX).** The pip-wheel NIXL's bundled UCX segfaults on the prefill→decode KV transfer (`signal 11: invalid permissions for mapped object` in `libucs.so`) — reproduced on vLLM 0.22 and 0.23, with/without mooncake, with/without llm-d. Building NIXL against UCX 1.19.x from source is therefore **required** (not optional) for disaggregated P/D.

```bash theme={null}
salloc -N 1 --gres=gpu:1 bash -c 'bash scripts/install_nixl_from_source.sh'
uv pip install --reinstall --no-deps deps/nixl_cu13-*.whl
```

The script writes UCX 1.19 to `third_party/ucx/`; the bundled sbatch templates prepend it to `LD_LIBRARY_PATH` so it overrides the system version. Re-run both commands after every `uv sync`, since the lock pins the wheel.


# Algorithms
Source: https://docs.primeintellect.ai/prime-rl/algorithms



This page covers the math and the configurable algorithmic components: the algorithm abstraction and its algorithms, how off-policy training works, the loss components and advantage functions, curricula, and how multi-turn rollouts get merged into training samples.

## Table of Contents

* [The Algorithm Abstraction](#the-algorithm-abstraction)
  * [Model References](#model-references)
  * [The Algorithms](#the-algorithms)
  * [Customizing Components](#customizing-components)
  * [Per-Env Algorithms](#per-env-algorithms)
  * [The Algorithm Classes](#the-algorithm-classes)
* [Async / Off-Policy Training](#async--off-policy-training)
* [Loss](#loss)
  * [Loss Components](#loss-components)
  * [IPO Loss](#ipo-loss)
  * [Custom Loss](#custom-loss)
* [Advantage](#advantage)
  * [Default Advantage](#default-advantage)
  * [Hierarchical GRPO](#hierarchical-grpo)
  * [Self-Play Advantage (RAE)](#self-play-advantage-rae)
  * [Authoring an Algorithm](#authoring-an-algorithm)
  * [Reference Scoring](#reference-scoring)
* [Curricula](#curricula)
* [Multi-Turn Trajectories](#multi-turn-trajectories)
  * [Extension Property](#extension-property)
  * [Best-Effort Interleaving](#best-effort-interleaving)
  * [Renderers](#renderers)
  * [Discontinuous Trajectories](#discontinuous-trajectories)

## The Algorithm Abstraction

A training algorithm in `prime-rl` is configured under `[orchestrator.train.algo]`, where **`type` names the algorithm** (`grpo`, `opd`, `sft`, …) and the class defaults are its vetted setting. It has two parts:

1. **Sampling** (`algo.sampling`) — how train rollouts are produced: which model generates them. `source` is a [model reference](#model-references): `"policy"` (the live policy, the default) or an inline frozen hosted model. Group sizing stays on the env config (`group_size`).
2. **The per-token training signal** — credit assignment and loss routing, fused; the algorithm's own parameters sit directly on `algo`. One mapping from a finalized rollout to per-token *(loss component, weight)* pairs — the credit a token gets and the loss that consumes it are two coordinates of the same output. Group-relative algorithms compute credit on the orchestrator and ship per-token advantage streams; reference-KL algorithms query a reference model at batch-ship time (bounded concurrency) and ship its prefill logprobs for the trainer to evaluate against the live policy. The `type` determines which loss component consumes the action tokens (`rl` / `ce` / `ref_kl`) and what happens to env-provided observation tokens in multi-turn rollouts (masked out by default; `echo` trains on them with weighted CE).

The trainer is algorithm-blind: the loss is a sum of three components (rl, ce, ref\_kl), each normalized by its own global token count; per-token streams ship on the wire (the `rl_weights` / `ce_weights` / `ref_kl_weights` component weights plus the `advantages` stream on each training sample) and the trainer just executes them. Adding an algorithm never touches the dispatcher, batch packing, or trainer hot path.

### Model References

`prime-rl` hosts exactly one model: the trainable policy (`[orchestrator.model]`). Every other model an algorithm uses is an external OpenAI-compatible endpoint, declared *inline on the component that uses it*. A model reference is either the string `"policy"` (the live policy) or a frozen hosted model (`name` + `base_url`):

```toml theme={null}
[orchestrator.train.algo]
type = "opd"

[orchestrator.train.algo.teacher]   # opd's teacher: the frozen model it scores against
name = "Qwen/Qwen3-32B"
base_url = "http://localhost:8001/v1"
```

Model *roles* are algorithm-local vocabulary — each algorithm names its reference on the field where the model is actually used, and there is no shared `teacher` slot. `opd` declares a `teacher` field (the frozen model whose reverse KL the policy distills toward); `sft`'s teacher *is* its `sampling.source` (the frozen model it imitates); `opsd` self-distills against the live policy and names no model at all. No role exists outside the algorithm that declares it: the dispatcher, sink, and trainer branch on liveness alone, never on what an algorithm calls a model.

So for `opd` set `[orchestrator.train.algo.teacher]`; for `sft` set `[orchestrator.train.algo.sampling.source]`; `opsd` needs neither. `opd`'s teacher must be a frozen endpoint — it is typed `FrozenModelConfig`, so `"policy"` isn't representable (the KL would be identically zero); `opsd`'s teacher *is* the live policy by definition (self-distillation conditioned on a demonstration), so it exposes no reference to configure.

Liveness is a property of the reference, not of any role: rollouts sampled from `"policy"` get version-salted prefix caches, carry sampling logprobs for importance ratios, and age off-policy as weights update; rollouts and scores from frozen models get a stable prefix cache and never go stale. Frozen models are externally hosted (`base_url` is required) — `prime-rl` never launches or updates them, and each env's algorithm builds its own client pool to the endpoints it declares.

### The Algorithms

The `algo.type` names the algorithm, and each type's class defaults are its vetted setting — picking a type with no other keys IS the algorithm:

```toml theme={null}
[orchestrator.train.algo]
type = "grpo"  # the default
```

| `type` | Sampling | Loss | What it is |
| - | - | - | - |
| `grpo` | policy | `rl` on actions | Standard group-relative RL. |
| `max_rl` | policy | `rl` on actions | MaxRL ([arXiv:2602.02710](https://arxiv.org/abs/2602.02710)): GRPO's centered reward normalized by the group **mean** instead of the standard deviation — the gradient is unbiased for the order-`group_size` truncation of the maximum-likelihood objective, upweighting hard examples like `1/p`. |
| `rae` | policy | `rl` on actions | RAE (SPIRAL, [arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): reward minus a per-agent EMA baseline of that agent's own rewards — the estimator for multi-agent self-play envs, where the group mean would mix the agents' opposite reward scales. See [Self-Play Advantage](#self-play-advantage-rae). |
| `hierarchical_grpo` | policy | `rl` on actions | GRPO for proposer-solver envs. Solvers are compared only with attempts on the same proposed problem; proposers are compared with the other proposals in the group. See [Hierarchical GRPO](#hierarchical-grpo). |
| `opd` | policy | `ref_kl` on actions | On-policy distillation ([Thinking Machines](https://thinkingmachines.ai/blog/on-policy-distillation/)): the policy samples, per-token reverse KL against a reference model as the gradient signal. Needs a `teacher`. |
| `sft` | *(the teacher)* | `ce` on actions | Hard distillation: a frozen model generates rollouts, the policy trains with CE on its tokens. Needs a frozen `sampling.source` (the teacher it samples from). |
| `opsd` | policy | `ref_kl` on actions | SDFT ([arXiv:2601.19897](https://arxiv.org/abs/2601.19897)): the model is its own reference, conditioned on an expert demonstration. The teacher *is* the live policy (the paper's setting, no extra deployment) — no model to configure. |
| `echo` | policy | `rl` on actions + weighted `ce` on observations | ECHO: standard GRPO plus a cross-entropy loss on env-provided tokens already present in the rollout, selected by message role (needs the renderer's role attribution). Defaults to tool-response bodies at `alpha = 0.1` (ECHO's λ); set `roles` to train other roles, each at its own weight. |

### Customizing Components

Every key beyond `type` is visibly your own assembly — there is no preset layer to diverge from. The vetted setting is the class defaults; what you set is what runs:

```toml theme={null}
# echo on tool AND user feedback tokens, each at its own weight.
# Setting any role replaces the whole table.
[orchestrator.train.algo]
type = "echo"

[orchestrator.train.algo.roles.tool]
alpha = 0.25

[orchestrator.train.algo.roles.user]
alpha = 0.05
```

A new algorithm is a named class in code, not a config that points at an import path — see [Authoring an Algorithm](#authoring-an-algorithm).

Echo also takes an optional user-supplied token filter that narrows the role selection per rollout — e.g. dropping warning lines from tool output, or tokens the sampler found unlikely:

```toml theme={null}
[orchestrator.train.algo.filter]
import_path = "my_module.drop_warnings"

[orchestrator.train.algo.filter.kwargs]
patterns = ["WARNING"]
```

```python theme={null}
# my_module.py — sees the raw rollout (message text, sampling logprobs);
# returns one keep-mask per trainable branch, spanning that branch's
# token_ids. False = never echo-trained.
def drop_warnings(rollout, *, patterns: list[str]) -> list[list[bool]]: ...
```

Component compatibility is validated at config time: frozen-model sampling can only feed the `ce` loss component — the `rl` and `ref_kl` components need the live policy's own sampling logprobs for importance ratios — `opd` pointed at `"policy"` is rejected as degenerate (zero KL), `sft` without a frozen source is rejected (CE on the policy's own tokens is not a distillation target). A group-relative algorithm with `group_size = 1` produces all-zero advantages; the resulting empty batch is caught at runtime (the orchestrator warns and skips it), not at config time.

### Per-Env Algorithms

Both components resolve per environment. Each env inherits `[orchestrator.train.algo]` like any other group default: an env that sets only some params keeps the group's algorithm with those params changed, and an env that sets a different `type` runs its own algorithm. So a single run can mix algorithms across envs — e.g. GRPO on math, ECHO on a terminal env:

```toml theme={null}
[orchestrator.train.algo]
type = "grpo"

[[orchestrator.train.source]]
name = "math"  # inherits the group's grpo
env.taskset.id = "math"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"

[[orchestrator.train.source]]
name = "terminal"
env.taskset.id = "terminal"
env.agent.harness.id = "bash"
env.agent.runtime.type = "subprocess"
# this env runs its own algorithm
algo.type = "echo"
```

### The Algorithm Classes

At runtime, each env's `TrainEnv` (`prime_rl.orchestrator.envs`) resolves the `sampling.source` model into the inference pool used for train episodes, and its resolved config builds one of the named algorithm classes in `prime_rl.orchestrator.train.algo` (one module per algorithm: `algo/grpo.py`, `algo/opd.py`, …) from the algorithm config. Algorithm dispatch is keyed on `algo.type` — it names the algorithm, and each config class's defaults are its vetted parameterization:

| `algo.type` | Class | hook(s) — stage |
| - | - | - |
| `grpo` | `GRPOAlgorithm` | `score_group`: group-norm credit (optional length penalty) |
| `echo` | `EchoAlgorithm` | `score_episode`: weighted ce on observation tokens; `score_group`: group-norm credit (inherited) |
| `max_rl` | `MaxRLAlgorithm` | `score_group`: mean-normalized group credit |
| `rae` | `RAEAlgorithm` | `score_group`: per-agent EMA-baseline credit |
| `hierarchical_grpo` | `HierarchicalGRPOAlgorithm` | `score_group`: GRPO baseline per episode for solvers, per group for the proposer |
| `opd` | `OPDAlgorithm` | `score_episode`: own-context prefill under the teacher |
| `opsd` | `OPSDAlgorithm` | `score_episode`: demo-conditioned prefill under the live policy |
| `sft` | `SFTDistillAlgorithm` | no credit assignment; CE on sampled tokens |

Algorithms operate on native verifier artifacts and annotate their message graphs directly:

* `async score_episode(episode)` — rollout-local scoring as one episode arrives.
* `async score_group(episodes)` — group-relative scoring after the cohort completes.

The pipeline calls `score_group` directly and `score_episode` through `finalize_episode`, which skips episodes with no trainable sampled tokens. Advantages, reference logprobs, and named loss weights stay on verifier nodes through admission. Only admitted traces are flattened into `TrainingSample`s.

The algorithm config states which loss component its action tokens feed (`action_loss_type`, a class variable on each config class). Every class is constructed with its algorithm config plus the one host-owned resource it can't rebuild — the live policy clients (`self.clients`). Everything else an algorithm needs it builds from its own config in `setup()`: `opd` connects its frozen `teacher`; `opsd` builds the renderer for its demonstration hint (tokenizer is always the live policy's — self-distillation has no separate model). Writing your own algorithm is subclassing `Algorithm` and overriding the hooks its signal needs (see [Authoring an Algorithm](#authoring-an-algorithm)). Shared math (efficiency shaping, prefill alignment) lives as plain functions in `prime_rl.orchestrator.train.algo.advantage`.

## Async / Off-Policy Training

`prime-rl` is asynchronous by default. The trainer and inference always run one step overlapped: while the trainer is producing $\pi_n$ from rollouts at step $n$, inference is already generating the rollouts for step $n+1$ using $\pi_{n-1}$. With matched trainer and inference step times this produces fully-overlapped pipeline parallelism — neither side ever idles.

![Async pipeline: trainer step n produces \theta\_n, inference at step n samples with \theta\_\{n-1}](https://mintcdn.com/primeintellect/mV0d6IvWxHP4DqoX/prime-rl/assets/async-pipeline.png?fit=max\&auto=format\&n=mV0d6IvWxHP4DqoX\&q=85\&s=3f8bdd85e2a68455836c7e13b6c8c21a)

At step $n = 1, 2, 3, \dots$:

* **Trainer** produces policy $\pi_n$ with weights $\theta_n$ from rollouts $(x_n, y_n)$.
* **Inference** produces rollouts $(x_n, y_n)$ from policy $\pi_{\max(0,\,n-1)}$.

Step indices are 1-indexed; policy versions are 0-indexed, with $\pi_0$ the base model. At step 1 inference samples from $\pi_0$.

## Loss

### Loss Components

The training loss is a **sum of three components**, each with its own per-token weight stream and its own normalization:

$$
\mathcal{L} = \frac{\sum \mathcal{L}_{rl}}{N_{rl}} + \frac{\sum \mathcal{L}_{ce}}{N_{ce}} + \frac{\sum \mathcal{L}_{ref\_kl}}{N_{ref\_kl}}
$$

* `rl` — the configured RL loss (`[trainer.loss]`): IPO by default, or optionally [IcePop](#icepop-loss) or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, `hierarchical_grpo`, and `echo`'s action tokens).
* `ce` — masked NLL. Used for frozen-model tokens (`sft`) and env-observation tokens (`echo`).
* `ref_kl` — the per-token reverse KL to a reference model ($\log \pi_{\text{ref}} - \log \pi$) as the policy-gradient signal, importance-ratio corrected with a one-sided trust region (`opd`, `opsd`). Requires `ref_logprobs` from a [reference scoring](#reference-scoring); the scoring model must be a vLLM server (it's the only one that exposes `prompt_logprobs`).

The orchestrator stamps each sample's component membership as per-token weight streams (`rl_weights` / `ce_weights` / `ref_kl_weights` on the wire): a weight scales that component's per-token loss, `0.0` leaves the token out of the component entirely (mask *and* denominator), and components may overlap on the same token — their gradients sum. Each $N$ is the global (all-reduced) count of that component's member tokens — for `rl`, the sum of its weights, which is the same count while weights are 0/1 — so the components don't dilute each other: adding echo observation tokens never changes the rl term's effective per-token learning rate, and an sft env packed next to a GRPO env doesn't soften its gradient. Tokens of different components pack freely into the same micro batch, and a plain GRPO run ships no weight streams at all (absent streams mean rl weight 1.0 on every trainable token — the unchanged hot path). Advantages always ship per token (`advantages` on the wire), assigned as per-token streams from the start — uniform group credit is broadcast over completion tokens at assignment; algorithms with no rl credit (opd, opsd) ship none.

### IPO Loss

The default RL loss is INTELLECT Policy Optimization (IPO). It is an importance-weighted policy-gradient term. A symmetric trust region removes tokens whose absolute probability change exceeds $\epsilon$:

$$
\mathcal{L}(\theta) = -\frac{1}{N}\sum_t
\mathbb{1}\!\left(\left|\pi(y_t)-\mu(y_t)\right| \le \epsilon\right)
\tau_A \hat{A}_t \frac{\pi(y_t)}{\mu(y_t)}.
$$

$\mu$ is the policy that generated the rollout. $\pi$ is the current trainer policy. $\hat{A}_t$ is the token-level advantage. The trust region uses the sampled token probabilities, not their ratio.

The knobs under `[trainer.loss]` are:

| Knob | Default | What it does |
| - | - | - |
| `eps` | 0.3 | Maximum absolute probability change before a token is masked. |
| `adv_tau` | 1.0 | Temperature on the advantage term. |

Omit `[trainer.loss]` to use these defaults. Set `type = "ipo"` when you specify the section. The `ce` and `ref_kl` components are fixed and unaffected by `[trainer.loss]`.

See [IPO](/prime-rl/ipo) for why IPO masks by probability instead of ratio, and the GLM-4.5-Air ablation against IcePop behind `eps = 0.3`.

### IcePop Loss

IcePop is an opt-in RL loss that drops tokens whose trainer-to-inference
importance ratio falls outside a fixed acceptance band, introduced to stabilize
MoE RL in [Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale
Mixture-of-Experts Reasoning Models](https://arxiv.org/abs/2510.18855). Accepted
tokens retain the importance-weighted policy-gradient term:

$$
\mathcal{L}(\theta) = -\frac{1}{N}\sum_t
\mathbb{1}\!\left(\alpha \le \frac{\pi(y_t)}{\mu(y_t)} \le \beta\right)
\tau_A \hat{A}_t \frac{\pi(y_t)}{\mu(y_t)}.
$$

Enable it explicitly:

```toml theme={null}
[trainer.loss]
type = "icepop"
ratio_low = 0.2
ratio_high = 5.0
```

| Knob | Default | What it does |
| - | - | - |
| `ratio_low` | 0.2 | Lower accepted trainer-to-inference probability ratio. |
| `ratio_high` | 5.0 | Upper accepted trainer-to-inference probability ratio. |
| `adv_tau` | 1.0 | Temperature on the advantage term. |

### Custom Loss

`[trainer.loss] type = "custom"` replaces the `rl` component. The loss is computed **per sequence**: you write a function that takes one sequence's tensors and returns a scalar loss. The trainer iterates and aggregates. `inputs.loss_mask` selects exactly the rl member tokens (for a plain GRPO run, all trainable tokens).

```python theme={null}
# my_module.py
import torch
from prime_rl.trainer.rl.loss import LossInputs, LossOutputs

def ppo_clip_loss(inputs: LossInputs, clip_eps: float = 0.2) -> LossOutputs:
    ratio = torch.exp(inputs.trainer_logprobs - inputs.inference_logprobs)
    clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps)
    surr1 = ratio * inputs.advantages
    surr2 = clipped * inputs.advantages
    loss = -torch.min(surr1, surr2)[inputs.loss_mask].sum()
    return LossOutputs(
        loss=loss,
        metrics={
            "clip_frac": (ratio != clipped)[inputs.loss_mask].float().mean(),
        },
    )
```

Wire it up:

```toml theme={null}
[trainer.loss]
type = "custom"
import_path = "my_module.ppo_clip_loss"

[trainer.loss.kwargs]
clip_eps = 0.2
```

The dataclasses:

```python theme={null}
@dataclass
class LossInputs:
    trainer_logprobs: Float[Tensor, "seq"]      # current policy
    inference_logprobs: Float[Tensor, "seq"]    # rollout-time policy
    ref_logprobs: Float[Tensor, "seq"] | None   # set by reference-scoring algorithms
    advantages: Float[Tensor, "seq"]
    loss_mask: Bool[Tensor, "seq"]              # this component's member tokens
    loss_weights: Float[Tensor, "seq"] | None   # the component's weight stream (None = 1.0)

@dataclass
class LossOutputs:
    loss: Float[Tensor, ""]
    metrics: dict[str, Tensor]
```

Anything you put in `metrics` is averaged across sequences and logged with the other trainer metrics.

## Advantage

The per-token training signal is set by `algo.type` and the [algorithm](#the-algorithm-abstraction)'s parameters — every signal is a per-token advantage stream, varying in evaluation site (orchestrator vs. trainer). The `algo.type` values:

| Type | Component | Effect |
| - | - | - |
| `grpo` | `rl` | Group-norm: reward minus per-group baseline, optional length penalty. |
| `max_rl` | `rl` | Mean-normalized group credit (maximum-likelihood RL). |
| `rae` | `rl` | Reward minus a per-agent EMA baseline (SPIRAL's role-conditioned advantage estimation) — for multi-agent self-play envs. |
| `hierarchical_grpo` | `rl` | GRPO for proposer-solver envs: solvers are compared within one proposed problem, while proposers are compared across proposals. |
| `echo` | `rl` + `ce` | Group-norm on action tokens, plus weighted CE on env-provided tokens selected by message role (each role's `alpha` is its ECHO λ), optionally narrowed by a user filter. |
| `opd` | `ref_kl` | On-policy distillation: per-token reverse KL to a reference model (`teacher`, an inline frozen hosted model), evaluated in the trainer from shipped reference logprobs. No credit — rollouts keep `advantages = None` and ship no advantage stream; `group_size` only fans out sampling. |
| `opsd` | `ref_kl` | SDFT: per-token reverse KL to a demo-conditioned reference. No credit — rollouts keep `advantages = None` and ship no advantage stream. |
| `sft` | `ce` | Cross-entropy on the sampled tokens. Assigns no advantage — trains on every sampled token. |

### Default Advantage

The default advantage is per-group reward minus per-group baseline (DR-GRPO without std normalization). For each prompt's group of `group_size` rollouts, every token in rollout $i$ receives advantage $s_i - \bar{s}$ where $\bar{s}$ is the group mean.

This is intentionally simple — it does the right thing for most envs. Write a named algorithm class when you need group-aware shaping that depends on trajectory metadata (sub-agent rollouts, relative-rank shaping, …) — see [Authoring an Algorithm](#authoring-an-algorithm).

A **length penalty** (`length_penalty` on the `grpo`-family algorithms) can be layered on top to discourage rambling. The `linear` penalty subtracts a single `pass_rate`-scaled penalty from each reward before the GRPO baseline, combining output tokens (`num_output_tokens_weight`), input / context tokens (`num_input_tokens_weight`), and turns (`num_turns_weight`) — each normalized by the group's own max for that quantity, with `num_input_tokens_weight` and `num_turns_weight` defaulting to `0.1`.

```toml theme={null}
[orchestrator.train.algo]
type = "grpo"

[orchestrator.train.algo.length_penalty]
type = "linear"
```

A **length-weighted baseline** (`length_weighted_baseline = true` on the `grpo`-family algorithms) replaces the plain group mean with $b = \sum_i L_i s_i / \sum_i L_i$, where $L_i$ is the number of trainable (policy-sampled, loss-masked) tokens of rollout $i$, summed across all its turns; it applies after the length penalty. With token-level loss normalization, long rollouts carry more gradient weight, so this baseline makes the per-token advantage zero-mean across the group's tokens rather than across rollouts.

**Prompt-mean loss aggregation** (`loss_aggregation = "prompt"` on `grpo` and `echo`; default `"token"`). By default every `rl` token in the batch weighs the same, so prompts whose groups produce long trajectories dominate the gradient. With `"prompt"`, the loss is MiMo-V2.6's prompt-mean (Eq. 1): the mean over prompt groups $q$ of $\frac{1}{T_q}\sum_{t \in q} \mathcal{L}_{rl,t}$, where $T_q$ is the group's total trainable tokens. MiMo uses it to keep response length from growing too fast. `score_group` gives each trainable token of the group the `rl` weight $1/T_q$, so each group's weights sum to 1 and $N_{rl}$ (the summed weights) is the number of groups in the step. Zero-advantage tokens are still dropped from the `rl` component, so a group with a zero-advantage rollout keeps less than its full weight of 1. All envs with an `rl` loss in a run must use the same `loss_aggregation`.

```toml theme={null}
[orchestrator.train.algo]
type = "grpo"
loss_aggregation = "prompt"
```

### Hierarchical GRPO

GRPO gives each rollout its reward minus the average reward of comparable rollouts. In an ordinary single-agent group, every rollout answers the same task, so one group average is enough.

A proposer-solver env is different. Starting from one source task, it produces several proposed problems, then runs several solver attempts on each problem:

```text theme={null}
one source task
├── proposed problem A
│   ├── proposer trace
│   ├── solver attempt 1
│   └── solver attempt 2
└── proposed problem B
    ├── proposer trace
    ├── solver attempt 1
    └── solver attempt 2
```

The solver attempts for A should not be compared with the solver attempts for B: the two problems may have very different difficulty. Proposer and solver rewards should not be compared either: they measure different jobs.

`hierarchical_grpo` therefore chooses the average separately for each role:

| Trace | Compared with | Why |
| - | - | - |
| Solver | Other solver attempts on the same proposed problem | They attempted the same problem. |
| Proposer | Other proposer traces in the group | They started from the same source task and proposed alternatives. |

For example, if three solvers receive rewards `[1, 1, 0]` on one proposed problem, their average is `2/3` and their advantages are `[1/3, 1/3, -2/3]`. Solver rewards from other proposed problems do not affect those values. The proposers are scored separately according to how useful their problems were for the solvers, then compared with the other proposers in the group.

Configure which roles are compared within a single proposed problem with `episode_agents`. For `proposer-solver`, that role is `solver`:

```toml theme={null}
[orchestrator.train.algo]
type = "hierarchical_grpo"
episode_agents = ["solver"]

[[orchestrator.train.source]]
name = "proposer-solver"
group_size = 4  # proposed problems per source task
env.n = 4  # solver attempts per proposed problem
env.taskset.id = "proposer-solver"
env.proposer.harness.id = "null"
env.proposer.runtime.type = "subprocess"
env.solver.harness.id = "null"
env.solver.runtime.type = "subprocess"
```

`group_size` controls how many problems are proposed from each source task. `env.n` controls how many solvers attempt each proposed problem. If a comparison contains only one trace—for example, a solver when `env.n = 1`—its advantage is zero.

This algorithm is accepted only for proposer-solver envs. Use the env's `train_proposer` and `train_solver` settings if you want to train only one role.

### Self-Play Advantage (RAE)

Group-relative baselines assume the group is exchangeable attempts by one agent. A multi-agent self-play env breaks that: one episode yields one trace per agent, all trainable, and in a zero-sum game the rewards sum to \~0 whatever the policy does — the group mean carries no information, and centering against it converts any structural asymmetry (a first-mover edge) into permanent credit for one agent.

`rae` implements SPIRAL's role-conditioned advantage estimation ([arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): each agent keeps an exponential-moving-average baseline of its own rewards, and every trace's advantage is its reward minus its agent's baseline — measured against the *pre-update* baseline (the unbiased order), then folded in at `decay` (SPIRAL's α, default 0.95). The algorithm instance is per-env, so baselines are keyed per (env, agent) — the paper's per (game, role). Advantages are not normalized, and `group_size` is free (RAE needs no sibling rollouts; `group_size = 1` is fine). Baselines live in orchestrator memory and re-warm from 0 over \~`1/(1 − decay)` traces per agent after a restart.

```toml theme={null}
[orchestrator.train.algo]
type = "rae"
decay = 0.95

[[orchestrator.train.source]]
name = "kuhn-poker"
env.taskset.id = "kuhn-poker"
env.player0.harness.id = "null"
env.player0.runtime.type = "subprocess"
env.player1.harness.id = "null"
env.player1.runtime.type = "subprocess"
```

Both of `kuhn-poker`'s agents late-bind to the run's own model — shared-policy self-play against a continuously improving opponent. Pin one agent to a frozen endpoint (`env.player1.model = ...`) for asymmetric play; its traces are marked untrainable by the env and never reach the advantage computation. A single-agent env under `rae` degrades to REINFORCE with an EMA baseline.

### Authoring an Algorithm

There is no config hook that points at user code — a new credit-assignment scheme is a new named algorithm in the repo. Subclass `Algorithm`, assign credit in the scoring hook whose timing fits your signal, and register the class:

```python theme={null}
# src/prime_rl/orchestrator/algo/my_algo.py
import torch

from prime_rl.orchestrator.train.algo.base import Algorithm
from prime_rl.orchestrator.train.algo.routing import assign_advantages


class MyAlgorithm(Algorithm):
    async def score_group(self, episodes):
        traces = [trace for episode in episodes for trace in episode.traces]
        rewards = torch.tensor([trace.reward for trace in traces], dtype=torch.float32)
        advantages = ...  # one value per trace
        for trace, advantage in zip(traces, advantages.tolist(), strict=True):
            assign_advantages(trace, advantage)
```

Add a typed `MyAlgoConfig` to `prime_rl.configs.algorithm` and its discriminated union, then register `"my_algo": MyAlgorithm` in `ALGORITHM_CLASSES`. Pick `score_episode` for rollout-local work or model calls, and `score_group` for group-relative credit. `assign_advantages` takes a scalar or a list aligned to the trace graph's sampled tokens.

Curriculum admission and metrics can inspect the resulting graph-native streams after group scoring.

### Reference Scoring

`OPDAlgorithm` / `OPSDAlgorithm` do their model I/O in `score_episode`: as each episode arrives they query a reference and attach sampled-token reference logprobs to its graph nodes:

* `opd` — score each sample's own context under the `teacher` (a frozen [model reference](#model-references)) via prefill; fills `ref_logprobs` for the `ref_kl` loss component (on-policy distillation). The `teacher` is typed `FrozenModelConfig`, so `"policy"` isn't representable (the KL would be identically zero).
* `opsd` — SDFT: prepend an expert demonstration as a leading system message (`template`, with a `{demonstration}` placeholder) and score the sample under that demo-conditioned context. The sample is scored verbatim (`hint_block + token_ids`, slicing the hint's logprobs back off), so the join is BPE-clean and it's robust to tool/multimodal prompts and any number of turns. The scoring reference *is* the live policy — self-distillation names no teacher. opsd builds its own renderer to tokenize the hint block: the tokenizer is always the live policy's (not configurable — there is no separate model), and only the `renderer` family is settable (defaults to `"auto"`, resolved from the policy tokenizer; set it to match a non-auto policy renderer). The demonstration is read from the example's `info[demo_key]`, falling back to a top-level rollout field of the same name (e.g. `answer`).

```toml theme={null}
[orchestrator.train.algo]
type = "opsd"
demo_key = "demonstration"
```

Scoring runs before curriculum admission, so a rollout that is later rejected still costs its reference compute.

The orchestrator filters samples with no training signal. This includes samples with zero advantage on all RL tokens. Samples that still carry CE or reference-KL components are retained. Filtering an RL token also removes its trainer/inference mismatch-KL contribution.

`orchestrator.constant_trainer_batch_size` defaults to `true`. The orchestrator filters samples before they count toward the batch target. It collects replacements, so rollout-based batches contain `orchestrator.batch_size` training traces. Set the option to `false` to filter after collection without replacement. This setting can improve orchestrator throughput, but it produces smaller trainer batches.

## Curricula

Each training source has a `Curriculum` composed from one `TaskSampler` and any number of named `AdmissionGate`s. The sampler chooses tasks and observes every finalized `list[vf.Episode]`; each gate inspects the same native episodes. The group trains only if every gate admits it. Rejected groups remain observable while the orchestrator samples again to fill the batch.

Samplers and gates can be stateful. Their `state_dict`, `load_state_dict`, and `metrics` methods are included in orchestrator checkpoints and logged under `curriculum/<env>/`.

Three small implementations are included:

* `StandardSampler` is the default: it advances the task iterator and cycles finite tasksets in source order.
* `DifficultyPoolSampler` samples finite tasksets with replacement and tracks each task's latest valid mean group reward. Each named pool has an inclusive reward threshold and a relative per-task sampling weight; weight `0` disables sampling from that pool. Unseen tasks use neutral weight `1.0`, so pool observations affect sampling immediately without waiting for a full taskset pass.
* `AdvRangeGate` rejects a group when every trainable-token advantage falls inside `reject_min` through `reject_max`. Unlike the built-in token filter, it can reject a configurable advantage range. Groups without an advantage stream are admitted.

```toml theme={null}
[orchestrator.train.source.curriculum.sampler]
type = "difficulty_pool"

[orchestrator.train.source.curriculum.sampler.pools.hard]
threshold = 0.25
weight = 0.2

[orchestrator.train.source.curriculum.sampler.pools.normal]
threshold = 0.75
weight = 1.0

[orchestrator.train.source.curriculum.sampler.pools.easy]
threshold = 1.0
weight = 0.2

[orchestrator.train.source.curriculum.gates.low_signal]
type = "advantage_range"
reject_min = -0.05
reject_max = 0.05
```

## Multi-Turn Trajectories

For multi-turn rollouts (tool use, browser environments, long conversations), `prime-rl` records each LLM request/response as an independent **trajectory step** and merges them at training time using best-effort interleaving — with [renderers](#renderers) as the mechanism that keeps the merge safe by construction.

### Extension Property

A sequence of trajectory steps has the **extension property** when each successive step's prompt contains all previous prompts and completions as an exact prefix. The trainer relies on this property — when it holds:

* Multiple steps merge into one training sample.
* Compute scales as $O(T)$ in the trajectory length.

When it breaks (chat template strips past thinking, environment compacts context, an agent hands off to a sub-agent, etc.), the trainer starts a new training sample from that step:

* Graceful fallback to multiple samples — no corrupted data.
* Worst case (every step breaks extension) is $O(T^2)$.

### Best-Effort Interleaving

Concretely:

```
5-step trajectory where extension breaks at step 4:

steps 1–3: extension holds   → merged into Sample 1
step 4:    extension breaks  (e.g. thinking stripped from history)
steps 4–5: extension holds   → merged into Sample 2

result: 2 training samples instead of 5
```

The orchestrator enforces an **exact prefix invariant**: the prompt at turn $t$ must be the exact concatenation of prior messages exactly as the LLM originally generated them. If turn 2's prompt is `U1, A1', U2` while `A1' ≠ A1`, the orchestrator can't safely merge — either choice produces logprob drift between trainer and inference. Starting a fresh sample is the only correct behavior, so that's what happens.

### Renderers

Best-effort interleaving works because the renderer guarantees the exact-prefix invariant *by construction* — it never re-renders prior turns, so it can't lose tokens to chat-template normalization, BPE retokenization drift, or thinking stripping. A renderer turns a model's chat template into a Python object that can:

* `render_ids(messages)` — tokenize messages to ids the inference engine accepts.
* `parse_response(completion_ids)` — recover structured `(content, reasoning_content, tool_calls)` from sampled ids.
* `bridge_to_next_turn(prev_prompt_ids, prev_completion_ids, new_messages)` — extend the previous turn's tokens verbatim with the new environment turn, instead of re-rendering history.

When `bridge_to_next_turn` succeeds, the trainer sees the exact token stream the sampler produced; when it can't be proven safe (e.g. the renderer is `DefaultRenderer` and the template's stop sequence is unknown), it returns `None` and the orchestrator falls back to a full re-render — which triggers the new-sample fallback above.

A common source of breakage in the absence of a hand-coded renderer is models like Qwen3 whose chat templates strip past `<think>` blocks across user turns:

```python theme={null}
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
messages = [
    {"role": "user", "content": "U1"},
    {"role": "assistant", "content": "<think>R1</think>A1"},
    {"role": "user", "content": "U2"},
]
tok.apply_chat_template(messages[:1], tokenize=False)
# <|im_start|>user
# U1<|im_end|>

tok.apply_chat_template(messages, tokenize=False)
# <|im_start|>user\nU1<|im_end|>\n<|im_start|>assistant\nA1<|im_end|>\n<|im_start|>user\nU2<|im_end|>
# (the <think>R1</think> from turn 2 is gone)
```

Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `glm-5`, `glm-4.5`, `minimax-m2`, `deepseek-v3`, `kimi-k2`, `kimi-k2.5`, `nemotron-3`, `gpt-oss`; anything else falls back to `DefaultRenderer` (a generic `apply_chat_template` wrapper). Pick one via:

```toml theme={null}
[orchestrator.renderer]
name = "auto"   # detect from tokenizer; pass an explicit name for fine-tunes
```

A renderer outside the `renderers` package loads as a custom renderer: `name = "custom"` with `import_path = "my_module.Class"` (or `path/to/file.py:Class`), see [Custom renderers](/prime-rl/training#dataset-format).

For the full design rationale (failure modes ruled out, empirical token-identity comparison against `apply_chat_template`, when to write a hand-coded renderer), see [the renderers writeup on the Prime Intellect blog](https://www.primeintellect.ai/blog/renderers) — the canonical reference.

### Discontinuous Trajectories

Some envs are discontinuous by design — e.g. a main agent delegating to a sub-agent and getting back only a summarized result, not the sub-agent's whole conversation. Best-effort interleaving handles this naturally: each agent's contiguous turns merge, the handoff starts a new sample. The trainer never sees fabricated extension where there is none.


# Configuration
Source: https://docs.primeintellect.ai/prime-rl/configuration



Every `prime-rl` entrypoint uses [`pydantic-config`](https://github.com/PrimeIntellect-ai/pydantic-config): TOML files for reproducible base configs, CLI flags for one-off overrides.

> **AI agents working in this repo:** the equivalent runbook is at [`skills/configs/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/configs/SKILL.md), with extra runtime hints (where config classes live, validator conventions) that aren't surfaced here.

## Table of Contents

* [Sources and Precedence](#sources-and-precedence)
* [TOML Composition](#toml-composition)
* [CLI Overrides](#cli-overrides)
* [Inspecting and Validating](#inspecting-and-validating)
* [Syntax](#syntax)
  * [Booleans](#booleans)
  * [Lists](#lists)
  * [Dicts](#dicts)
  * [Optional Sub-Configs](#optional-sub-configs)
  * [None](#none)
  * [Discriminated Unions](#discriminated-unions)
  * [Environments](#environments)
  * [Environment Variables](#environment-variables)
* [Examples](#examples)

## Sources and Precedence

Field values come from three sources — Pydantic defaults, TOML files (passed with `@`), and CLI flags. They're layered in this order, with later sources winning:

1. **Defaults** declared on the Pydantic model.
2. **TOML files** passed with `@`, left to right — later files override earlier ones.
3. **CLI flags** in dotted, kebab-case form (`--model.name`).

## TOML Composition

The `@` token introduces a TOML file. Multiple `@` arguments compose left-to-right, deep-merged — unset fields in an overlay keep the base value:

```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml                      # one file
uv run rl @ base.toml @ overlay.toml                           # left to right
uv run rl --trainer @ trainer.toml --orchestrator @ orch.toml  # per-section
uv run rl @ base.toml --trainer @ trainer.toml                 # mixed
```

> Mind the space: `@ path/to/x.toml`, not `@path/to/x.toml`.

## CLI Overrides

CLI flags mirror the TOML tree using dots:

```bash theme={null}
--max-steps 50                              # top-level
--model.name Qwen/Qwen3-4B                  # nested
--trainer.optim.lr 1e-5                     # double-nested
--inference.vllm.tensor-parallel-size 4
```

> Field names are snake\_case in TOML (`max_model_len`) and kebab-case on the CLI (`--max-model-len`).

> A renamed field keeps no alias for its old name: the old spelling fails as an unknown key rather than being silently translated.

## Inspecting and Validating

```bash theme={null}
uv run rl --help                                       # full schema
uv run rl @ rl.toml --dry-run --output-dir /tmp --run.name check  # write resolved configs (JSON) to /tmp/check/configs
```

## Syntax

### Booleans

CLI uses paired flags: bare `--flag` sets `True`, `--no-flag` sets `False`. TOML must be explicit:

```bash theme={null}
uv run rl @ rl.toml --clean       # True
uv run rl @ rl.toml --no-clean    # False
```

```toml theme={null}
clean = true
```

### Lists

CLI accepts space-separated values or a JSON literal. TOML uses an array literal. Both forms target the same field:

```bash theme={null}
uv run rl @ rl.toml --trainer.model.lora.target-modules q_proj k_proj v_proj
uv run rl @ rl.toml --trainer.model.lora.target-modules '["q_proj", "k_proj", "v_proj"]'
```

```toml theme={null}
[trainer.model.lora]
target_modules = ["q_proj", "k_proj", "v_proj"]
```

Overlay TOMLs **replace** lists wholesale — an overlay that wants to add one item must still spell out the full list. For arrays of tables, see [Environments](#environments).

### Dicts

CLI takes a JSON literal. TOML uses a table. CLI dicts deep-merge with TOML dicts — CLI keys win on conflict but don't wipe the file's keys:

```bash theme={null}
uv run rl @ rl.toml --orchestrator.train.source.0.args \
  '{"dataset_name": "openai/gsm8k", "dataset_subset": "main"}'
```

```toml theme={null}
[[orchestrator.train.source]]
args.dataset_name = "openai/gsm8k"
args.dataset_subset = "main"
```

### Optional Sub-Configs

Many sub-configs are typed `SomeConfig | None`. Two patterns enable them:

* **Bare flag with defaults**: `--model.compile` or, in TOML, an empty section `[model.compile]`. The sub-config materializes with all-default values.
* **Enable and set fields together**: `--model.compile.fullgraph` (CLI) or any populated `[model.compile]` table (TOML).

To **disable** a sub-config that's on by default, use `--no-<name>` on the CLI or assign the string `"None"` in TOML (see [None](#none)). This is how `[ckpt]`, `[model.lora]`, `[model.compile]`, `[trainer.wandb]`, etc. are turned on and off.

### None

TOML has no `null`. Use the string `"None"`, which the loader coerces:

```toml theme={null}
[inference.vllm]
max_model_len = "None"
```

On the CLI: `--inference.vllm.max-model-len None`.

### Discriminated Unions

Loss, advantage, optimizer, scheduler, weight broadcast transport, and several others are discriminated unions. Set the `type` field to pick a variant:

```toml theme={null}
[trainer.optim]
type = "muon"
lr = 1e-5
mu = 0.95
```

Omit `type` to keep the default variant.

### Environments

Training and evaluation sources are arrays of tables. Set one source per environment; training sources can optionally carry sampling weights:

```toml theme={null}
[[orchestrator.train.source]]
name = "gsm8k"
ratio = 3  # 75% of batches
env.taskset.id = "gsm8k"
env.taskset.split = "train"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"

[[orchestrator.train.source]]
name = "reverse-text"
ratio = 1  # default — 25% of batches
env.taskset.id = "reverse-text"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"

[[orchestrator.eval.source]]
name = "gsm8k-eval"
env.taskset.id = "gsm8k"
env.taskset.split = "test"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"
```

Each source group — `[orchestrator.train]`, `[orchestrator.eval]`, the `[eval]` block of `sft` and the top level of `eval` — holds defaults for its sources. Every field that a group and its sources both have is a default: `env`, `sampling`, `select` and `group_size`, plus `algo` on the train group and `interval` on the eval groups of a training run. Each source inherits the fields that its group sets. One rule applies to all of them: nested blocks merge key by key, a source's own values win, and a block whose `type` (e.g. of `algo`) differs from the group's is the source's alone.

Sampling parameters use Verifiers' shared normalization: `max_tokens` and `max_completion_tokens` name the same limit, and `extra_body` parameters merge like top-level parameters. Agent overrides apply after source defaults; an explicit null clears an inherited parameter. Resolved sampling is recorded on each trace.

For live-policy training, set distribution parameters (`temperature`, `top_p`, `top_k`) on the source's `sampling` block. Every agent in that source, including non-trainable roles, may override only `max_tokens`, `reasoning_effort`, and `chat_template_kwargs`. These rules are checked when loading the config; evaluation and frozen-model sources allow general agent sampling overrides.

```toml theme={null}
[orchestrator.train]
group_size = 16

[orchestrator.train.algo]
type = "grpo"

[[orchestrator.train.source]]
env.taskset.id = "math"
group_size = 32  # this source's own value wins over the group's 16

[orchestrator.eval.env]
retries.rules = [
  {type = "ProviderError", max_retries = 3},
  {type = "SandboxError", max_retries = 3},
]
timeout.episode = 7200

[orchestrator.eval.select]  # evaluate 128 tasks of each taskset
limit = 128
```

The group `env` block holds only the knobs that every env and taskset has: `retries`, `timeout`, `max_concurrent_agents`, `interception`, and `taskset.task` and `taskset.system_prompt`. Ids and agents stay on each source. `select` picks which tasks of a source's taskset run: `include`/`exclude` by task `idx`/`ids`/`keys`/`names`, then `shuffle`, `skip` and `limit`, in that order.

`ratio` is a training-source field: it defaults to `1` (equal weight per env), and values are relative weights normalized to probabilities across envs. Eval sources of a training run carry `interval` instead, the step interval at which they fire; a standalone eval has neither.

Everything environment lives under the `env` block (verifiers' `[env]` shape): `env.taskset` configures the v1 taskset, and each agent is a field on the env — `env.agent.harness` selects how the single-agent env's tasks are run, and per-run caps are per-agent (`env.agent.max_turns`, `env.agent.timeout`, `env.agent.max_output_tokens`). A multi-agent env declares its own seats (`env.<role>.*`).

The same taskset can appear multiple times across train and eval (or with different settings) — useful for evaluating on a held-out split or comparing two configurations side by side. When it is reused, set a distinct `name` on each entry; `name` defaults to the taskset id and must be unique across all envs in the same group.

### Environment Variables

OS environment variables exported into launched component process(es). In `rl` configs, top-level `[env_vars]` applies to trainer, inference, and orchestrator:

```toml theme={null}
[env_vars]
HF_HUB_OFFLINE = "1"
TOKENIZERS_PARALLELISM = "false"
```

Component-specific tables layer on top:

```toml theme={null}
[trainer.env_vars]
NCCL_DEBUG = "INFO"
PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:False"

[inference.env_vars]
VLLM_USE_DEEP_GEMM = "1"

[orchestrator.env_vars]
PRIME_LOG_LEVEL = "debug"
```

The `rl` launcher applies these the same way in both single-node and multi-node (SLURM) runs. Precedence, low to high:

1. The launcher's own defaults — **your `env_vars` override these**.
2. Your top-level `[env_vars]`.
3. Your `[component.env_vars]`.
4. Orchestration-critical vars the launcher always sets last — `CUDA_VISIBLE_DEVICES` (GPU partitioning) and `WANDB_SHARED_*` (the single shared W\&B run) — **these cannot be overridden** from `env_vars`.

For standalone `sft` and `inference` configs, `[env_vars]` applies to that entrypoint's process(es). For disaggregated P/D inference, the role-specific [`deployment.{prefill,decode}_env_vars`](/prime-rl/inference) layer on top of any shared inference env vars.

## Examples

The shipped end-to-end examples in [`examples/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples) are the canonical, kept-up-to-date references — the rest of the repo's TOMLs (under `configs/`) are CI- and debug-internal and may drift. Each basic example directory has its own README with the full launch story; the advanced examples are config-only.

**Basic** (1–8 GPUs):

* [**Reverse Text**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/reverse-text) — `Qwen3-0.6B` reversing a chunk of text. Tiny single-turn SFT + RL; runs on a single consumer GPU in minutes.
* [**Wordle**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wordle) — `Qwen3-1.7B` playing Wordle. Multi-turn SFT + RL; 2–4 H100s.
* [**Alphabet Sort**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/alphabet-sort) — `Qwen3-4B-Instruct-2507` sorting names alphabetically. Multi-turn LoRA RL without SFT warmup; one H100.
* [**Wiki Search**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wiki-search) — `Qwen3-4B-Instruct-2507` answering trivia by searching a Wikipedia corpus. Multi-turn with tool use.
* [**Hendrycks Sanity**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/hendrycks-sanity) — `Qwen3-1.7B` on a filtered MATH subset. Useful for algorithm ablations.

**Advanced** (32–2048 GPUs, SLURM):

* [**Qwen3-30B-A3B**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/qwen3-30b-a3b) — `Qwen3-30B-A3B` on math, SWE, and tool use.
* [**GLM-4.5-Air**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-4.5-air) — `GLM-4.5-Air` on search, SWE, and terminal; includes a variant that trains the 100B+ MoE on only two nodes.
* [**Nemotron-3-Super**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/configs/advanced/nemotron-3-super) — `Nemotron-3-Super-120B` hybrid-Mamba MoE on SWE at 131k context.
* [**MiniMax-M2.5 SWE**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/configs/advanced/minimax-m2.5) — `MiniMax-M2.5` on agentic SWE.
* [**INTELLECT-3.1**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/intellect-3.1) — reproduces our INTELLECT-3.1 training run.
* [**High-throughput GLM-5**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.3) — large-scale `GLM-5`/`GLM-5.3` inference with P/D disaggregation and FP8.

### Worked Example: Compose, Override, Dry-Run

Start from a shipped base config, override two fields on the CLI, and dry-run:

```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml \
  --monitors.wandb.name my-experiment \
  --trainer.optim.lr 5e-6 \
  --output-dir /tmp/reverse-dry \
  --run.name check \
  --dry-run
```

Then inspect the resolved config:

```bash theme={null}
ls /tmp/reverse-dry/check/configs/latest/resolved/
# rl.json  trainer.json  orchestrator.json  inference.json
```

Each per-process JSON contains the final, validated configuration that the run
uses. This is also what each standalone process reads. For example, the trainer
reads `configs/latest/resolved/trainer.json`.

`configs/latest/command.txt` records the shell-safe launch command and its CLI
overrides. Each launch also remains under `configs/attempt_<n>/`. To compare
configs, dry-run a known-good base and your overlay. Then diff the two attempts.


# Development
Source: https://docs.primeintellect.ai/prime-rl/development



This page covers workflows for developing on `prime-rl` itself — running the test suite, contributing changes, and adding new model architectures with the small-scale tooling we use to iterate on MoE families without booting up a 100B+ run.

## Table of Contents

* [Test Suite](#test-suite)
  * [Layout](#layout)
  * [Running Tests Locally](#running-tests-locally)
  * [CI Workflows](#ci-workflows)
  * [Markers](#markers)
* [Pre-Commit Hooks](#pre-commit-hooks)
* [Adding a New Model](#adding-a-new-model)
  * [Implement the Modeling Code](#implement-the-modeling-code)
  * [Register a Mini Preset](#register-a-mini-preset)
  * [Run the Smoke Test](#run-the-smoke-test)
* [Adding a Custom VLM Implementation](#adding-a-custom-vlm-implementation)

## Test Suite

The test suite is split into three tiers, each with its own CI workflow.

### Layout

* **`tests/unit/`** — fast-running, hermetic tests for isolated logic: config parsing and validation, advantage / loss / scheduler / packer math, individual dataset paths, model-conversion roundtrips, etc. Tests that need a GPU are tagged with the `gpu` marker.
* **`tests/integration/`** — full-stack RL/SFT runs on a tiny model end-to-end through inference + orchestrator + trainer.
* **`tests/nightly/`** — runs the configs in [`examples/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples) every night to catch regressions in the shipped examples.

### Running Tests Locally

```bash theme={null}
uv run pytest -v                                           # everything
uv run pytest tests/unit -v                                # unit only
uv run pytest tests/integration -v                         # integration only
uv run pytest -v -m "not gpu"                              # CPU-only subset (mirrors CPU CI)
uv run pytest -v -m gpu                                    # GPU-only subset
uv run pytest tests/integration/test_reverse_text.py -vvs  # one specific scenario
```

### CI Workflows

| Workflow | Trigger | What runs | Where |
| - | - | - | - |
| [`cpu_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/cpu_tests.yaml) | every PR + push to `main` | `pytest tests/unit -m "not gpu"`, plus a slim-wheel install check that `prime-rl-configs` imports cleanly without heavy deps (no torch / vllm / transformers / wandb / verifiers / datasets / liger / loguru in `sys.modules`) | `ubuntu-latest` |
| [`gpu_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/gpu_tests.yaml) | relevant changes in non-draft PRs; full suite on push to `main` or manual dispatch | `pytest tests/unit -m gpu`, plus RL/SFT integration scenarios (`reverse_text`, `reverse_text_sft`, `reverse_text_sft_lora`, `reverse_text_lora`, `reverse_text_moe`, `reverse_text_rl_opd`, `reverse_text_rl_sft`, `alphabet_sort`) with `max-parallel: 5`; separate hosted `gsm8k_eval` job | self-hosted `prime-rl-2xh200` k8s scale set (2xH200); `ubuntu-latest` for test selection and hosted eval |
| [`nightly_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/nightly_tests.yaml) | 03:00 PST daily + manual `workflow_dispatch` (single-file filter optional) | every file in `tests/nightly/`, one matrix job per file | `research-cluster` |

The GPU + Nightly workflows skip drafts — open the PR as **Draft** until you're ready to consume CI compute, then mark it ready for review to trigger the GPU matrix.

### Markers

Two pytest markers are declared in `pyproject.toml` (`addopts = "--strict-markers"`):

* `gpu` — gate a test that needs CUDA. CPU CI uses `-m "not gpu"`; the GPU unit job uses `-m gpu`.
* `slow` — gate a test that's expensive enough you'd usually skip it locally. Deselect with `-m "not slow"`.

## Pre-Commit Hooks

Install the [pre-commit](https://pre-commit.com) hooks before your first commit so ruff check + format run on staged Python files automatically:

```bash theme={null}
uv run pre-commit install
```

## Adding a New Model

Bringing up a new model family is three steps: implement the modeling code, register a mini preset, and run the smoke test. The preset and smoke test let you iterate on the modeling code at \~0.5B scale on 1–2 GPUs instead of paying the cost of the full-size model — useful for catching bugs in modeling code, state-dict conversions, and pipeline integration before scaling.

### Implement the Modeling Code

Drop the modeling code under `src/prime_rl/trainer/models/<arch>/` (HF-compatible config, modeling, and weight conversion). Mirror the layout of an existing family — `glm4_moe/` or `qwen3_moe/` are good starting points.

### Register a Mini Preset

Add an entry to [`scripts/mini_moe.py`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/scripts/mini_moe.py) so the smoke-test workflow can build a \~0.5B test model in your architecture. The preset names the config class, picks small dimensions, and wires up the HF + prime-rl model classes plus a tokenizer source:

```python theme={null}
ARCH_PRESETS = {
    "glm4_moe": {
        "config_class": Glm4MoeConfig,
        "config_kwargs": dict(hidden_size=1024, num_hidden_layers=24, n_routed_experts=8, ...),
        "hf_model_class": HFGlm4MoeForCausalLM,
        "prime_model_class": PrimeRLGlm4MoeForCausalLM,
        "tokenizer_source": "THUDM/GLM-4-9B-0414",
    },
    # add your arch here
}
```

### Run the Smoke Test

Build the mini model. This creates a \~543M-parameter GLM-4 MoE (1024 hidden, 24 layers, 8 experts) with random weights, copies the tokenizer from the original GLM-4 model, and verifies the HF↔prime-rl roundtrip is lossless:

```bash theme={null}
uv run python scripts/mini_moe.py --arch glm4_moe --output-dir ./mini-glm-moe
```

To re-verify the roundtrip after a modeling-code change without re-creating the model:

```bash theme={null}
uv run python scripts/mini_moe.py --arch glm4_moe --output-dir ./mini-glm-moe --verify-only
```

Warm up the random-weight mini model with SFT on reverse-text so KL divergence becomes meaningful in the RL phase. Loss drops from \~12 to \~2.5 — the output won't be coherent, but the distribution is non-trivial. A pre-built SFT'd checkpoint lives at [samsja/mini-glm-moe](https://huggingface.co/samsja/mini-glm-moe) if you want to skip this step:

```bash theme={null}
uv run sft \
  --model.name ./mini-glm-moe \
  --data.name PrimeIntellect/Reverse-Text-SFT \
  --max_steps 200 \
  --optim.lr 1e-4 \
  --ckpt
```

Then run the full RL stack on reverse-text:

```bash theme={null}
uv run rl @ configs/ci/integration/reverse-text-moe/start.toml \
  --model.name samsja/mini-glm-moe \
  --inference.vllm.gpu-memory-utilization 0.7 \
  --inference.vllm.max-model-len 2048
```

What to look for:

* **No crashes.** Validates the full inference + orchestrator + trainer pipeline end-to-end.
* **Finite, non-zero KL.** Confirms the reference distribution is meaningful.
* **Loss reasonable.** Not NaN, not stuck.

Don't expect reward to climb meaningfully in 20 steps on a random model.

### Requirements for merging a new model

Before merging a new model, you need to ensure the following:

* The model is correctly registered and defines and all the required methods - such as `convert_hf_layer_to_tt` and `convert_tt_layer_to_hf`.
* The small smoke test passes.

In the PR that adds the new model, you also need to provide a table covering the KL mismatch across 20 steps on `math` environment with `batch_size=64`. All the entries in the table must lower than 0.015. If this is not met, the PR will not be merged (unless reasonable justification is provided). This is to ensure all our models are consistent and their implementations match the implementations in the inference framework.

## Adding a Custom VLM Implementation

VLM training (any run with `[model.vlm]` set, SFT or RL) is custom-implementation-only: `get_model` rejects models without a custom PrimeRL VLM class at load time. To make a new VLM family trainable, extend a custom text model with a composite VLM body. The unified dense/MoE Qwen3.5 implementation in `models/qwen3_5/` is the reference. The pieces, in dependency order:

1. **Custom text model first.** The VLM body wraps a custom `*ForCausalLM` (see [Adding a New Model](#adding-a-new-model)), so the text side — including its state-dict conversion and KL-mismatch table — comes first.
2. **Composite VLM body.** A `*VLMModel` that holds the vision encoder and custom text model, embeds tokens, runs vision, scatters image embeddings over placeholder tokens, and builds MRoPE 3D positions from `mm_token_type_ids` (the renderer owns the token→modality mapping). The unified `*ForCausalLM` dispatches on the config: composite config → VLM path, text config → text path.
3. **Always run the vision encoder.** Text-only micro-batches must feed the encoder dummy pixels and graft the result into the graph with zero contribution (`inputs_embeds + image_embeds.sum() * 0.0`) so FSDP/EP collectives stay symmetric across ranks when the encoder is trainable.
4. **Packed-boundary consumption.** Samples pack into shared rows with per-document boundaries in `seq_lens`; every custom model's `forward()` declares the typed `seq_lens`/`seq_lens_are_pre_shard` parameters (the trainer passes them unconditionally) and must honor the boundaries — varlen flash `cu_seqlens`, linear-attention state resets per document, and a loud rejection on attention paths that can't (see the packed-batch guard in any modeling file). Set `supports_packed_multimodal_training` on the VLM model once packed rows are handled — RL fails loudly at startup for VLM models without it.
5. **Registration.** Register the composite `model_type` in `_CUSTOM_VLM_MAPPING` (`models/__init__.py`) so `get_model` dispatches to the custom class, and describe the family in `VLM_REGISTRY` (`utils/vlm.py`).
6. **Context parallelism (optional).** CP-capable VLMs hold a `CPContext` on all modules which need to be CP-aware and shard embeds/positions inside the model after the vision merge; the trainers defer sharding to the model for MRoPE batches under ulysses.
7. **Validation.** Same bar as text models: the KL-mismatch table for the text path, plus an SFT run and an RL run on a real multimodal dataset (the `color-codeword` environment is the reference task).


# Eval
Source: https://docs.primeintellect.ai/prime-rl/eval



This page covers `uv run eval` — evaluating a model in one or more environments. For the online evals of a training run, see [Training](/prime-rl/training#online-evals).

> **AI agents working in this repo:** the equivalent runbook is at [`skills/eval/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/eval/SKILL.md).

## Table of Contents

* [Launch](#launch)
* [Configuration](#configuration)
* [Resume](#resume)
* [Monitors](#monitors)
  * [File](#file)
  * [Prime](#prime)
* [Metrics](#metrics)

`uv run eval` runs one epoch of every configured source and exits. It reuses the orchestrator's eval pipeline: env servers are spawned per source, episodes are admitted under the concurrency controller, and every episode streams into the run's trace stream and metrics.

## Launch

By default the model is served by Prime Inference; authenticate with `PRIME_API_KEY` or `prime login`:

```bash theme={null}
uv run eval gsm8k -n 32 -r 4 -c 8                                   # Prime Inference
uv run eval @ configs/debug/eval/single-turn.toml                   # the same shape as a TOML
```

The default endpoint comes from `PRIME_INFERENCE_URL`, then the Prime CLI's `inference_url`, then the standard Prime Inference URL. An explicit `client.base_url` wins. API keys and environment-backed headers are resolved when clients are built; Prime CLI credential and team fallbacks apply only to Prime Inference hosts, including for a separate admin URL.

To evaluate a model you serve yourself, start a `uv run inference` vLLM server (or any OpenAI-compatible API) and point the client at it:

```bash theme={null}
uv run inference --vllm.model Qwen/Qwen3-4B
uv run eval gsm8k -n 32 -r 4 -m Qwen/Qwen3-4B --client.base_url http://localhost:8000/v1
```

Single-source shorthands: `<taskset-id>` names the run's only source, `--env.<field> <value>` sets a field of that source's env block (`--env.agent.harness.id bash`, `--env.taskset.tasks '["fix-git"]'`), `-n`/`-s`/`-r` set `select.limit`/`select.shuffle`/`group_size`, `-m` the model, and `-c N` pins the concurrency band (`concurrency.min_inflight = max_inflight = N`). `<taskset-id>` cannot be combined with a TOML that defines `[[source]]` blocks; next to one, `--env.<field>` sets the shared [env block](#configuration) instead. `uv run eval -h` lists them.

Against a local vLLM deployment, set `min_inflight < max_inflight` in `[concurrency]` to dynamically adjust the number of concurrent episodes for maximum throughput. An external API exposes no vLLM `/metrics` to adapt to, so pin the concurrency there (`-c N`, i.e. `min_inflight = max_inflight`).

## Configuration

Multi-source runs use a TOML (`EvalConfig` in `packages/prime-rl-configs/src/prime_rl/configs/eval.py`). The eval block is flattened to the top level — `[[source]]`, `[client]`, `[concurrency]`, `[sampling]`, `[select]`, `group_size` — and each source takes the same `env` block as `[[orchestrator.eval.source]]`:

```toml theme={null}
model = "Qwen/Qwen3-4B"
group_size = 4

[select]             # every source inherits these
limit = 32

[client]
base_url = "http://localhost:8000/v1"

[concurrency]        # adaptive against a local vLLM deployment
min_inflight = 8
max_inflight = 256

[sampling]
max_completion_tokens = 2048

[env]                # every source inherits these
timeout.episode = 7200
retries.max_retries = 3

[[source]]
env.taskset.id = "gsm8k"
env.agent.harness.id = "bash"

[[source]]
env.taskset.id = "aime25"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"
select.include.idx = ["0:30"]
```

Per-source `group_size` and `sampling` override the top-level defaults. The top-level `[env]` block holds the env knobs that every source inherits (see [Environments](/prime-rl/configuration#environments)); a source's own `env` values win.

Each field a source sets in its `select` overrides the same field of the top-level `[select]`. `select` picks which tasks of the taskset run: `include`/`exclude` by task `idx`/`ids`/`keys`/`names`, then `shuffle`, `skip` and `limit` (see verifiers' [Selecting tasks](/deps/verifiers/docs/v1/tasksets#selecting-tasks)). Train sources take the same `select`.

Every source's env server is spawned by the eval process unless the source sets `serve.address`, in which case the server is externally managed. A spawned server binds an OS-assigned loopback port and publishes it to `configs/attempt_N/resolved/envs/eval/<name>.address`, which the eval process reads, so concurrent runs on one host never collide on a port.

## Resume

An interrupted run resumes from its trace stream. Relaunch with the same `--run.name` and `--resume`: saved episodes with `episode.ok` rejoin the epoch as if they had just arrived (stream, metrics and platform upload cover the whole epoch), and only the rollouts still owed run. Failed episodes and the ones the interruption cut off run again. Saved records that fail schema validation stop the resume before the current stream is archived.

```bash theme={null}
uv run eval @ eval.toml --run.name my-eval
# <interrupted>
uv run eval @ eval.toml --run.name my-eval --resume
```

The previous attempt's `monitors/file` is kept as `monitors/file.attempt_N`; the resumed attempt writes a fresh one. Nothing is deleted, and a resume reads every attempt's stream.

A landed episode counts toward a selected task in the same source when their content hashes (`task.hash`) match. A task whose content changes runs again even if its public `task.key` stays the same. `select` and `group_size` may change between launches: each selected occurrence keeps up to `group_size` matching episodes, and only its missing rollouts run. The resumed config is not checked against the interrupted one: any of it may be overridden, so keep the model, the sampling and each source's env the same when the landed episodes must stay comparable. Each selected occurrence owns a separate group containing its restored and new rollouts, even when the same task is selected more than once, so pass\@k and the dashboard see one group per selected occurrence. Use `--clean` to start over instead.

## Monitors

Monitors receive the run's metrics and episodes through one abstraction. The file monitor is on by default; `--monitors.wandb` logs the run to Weights & Biases; `--monitors.prime` streams it to the Prime platform.

### File

The file monitor writes the run's metrics and trace stream under `output_dir / run.name / monitors/file/` and feeds the local [dashboard](/prime-rl/training#dashboard). Env servers stream every rollout to the eval process as it happens: the trace's header when it is minted, then one delta per committed turn (the new messages and the model call behind them), per phase change (boot, setup, agent, finalize, scoring), and a preview of each model request's uncommitted messages, so a tool result shows the moment the model is asked about it rather than with the reply. The file monitor appends those deltas to `monitors/file/traces/live/<trace_id>.jsonl` and deletes the file when the episode lands in the finished trace stream, so that directory is exactly the live set and finished traces stay where they always were. The dashboard's traces tab shows live rollouts in the episode table itself: a pulsing dot in the `#` cell marks the row, a phase badge says where the rollout is (boot, setup, running, finalize, scoring), the `dispatched → arrived` column shows the dispatch time with the elapsed time in brackets, and the turn, token and branch counts grow as deltas land. The status filter toggles live and completed rows, and a live row opens in the trace viewer, which follows the rollout turn by turn (the transcript stays pinned to the newest turn until you scroll up; folded entries stay folded; the pending messages of the request in flight render dimmed with an `awaiting model` chip). The table and an open live trace poll once a second and transfer nothing while the rollout is unchanged. The progress line in `eval.log` counts the live rollouts by phase (`2 inflight episodes ... - boot 1 · running 1`). Read a live trace from a shell with `uv run python -m prime_rl.monitors.file.traces <run_dir> [<trace_id>]`. The same stream exists for RL runs, covering train and online-eval rollouts alike.

The dashboard's overview tab reads the trace stream directly. It shows one env at a time; the filter menu picks the env and whether errored episodes join the distributions. A block bar has one cell per expected episode: landed episodes (errored ones in red; click one to open it), live rollouts as pulsing outlines (click one to follow it), and the rest empty, with the percentage beside it. The expected count is what the eval runner wrote to `monitors/file/plan.json` when it counted the epoch's tasks. Below the bar, a summary row gives the headline numbers: avg\@k and pass\@k, the error and truncation rates (yellow past 10%, red past 50%), mean turns and branches in one tile, mean episode time and tokens, and the total cost when the run has costs; hover a tile for its distribution. The env's own rewards and metrics follow, a beeswarm pane each, resizable like the training charts. The usage section is one pane splitting each episode's tokens into input and output, and the timing section is the same pane over the phases of a rollout (boot, setup, model, harness, finalize, scoring, and `other` for wall time no phase accounts for): an icicle bar shows the mean split, one strip per episode below shows the split per rollout, and a legend sits above. Hovering a phase highlights it across the pane; the strips show 40 episodes by default and all of them on `show all`. A metric every episode shares appears as a chip instead of a pane, and a metric with only a few distinct values, such as a binary reward, as a counted dot plot with the count and share of each value. Every dot is an episode (or a task): hover it for its value and click it to open the trace; hover the pane's background for the summary (min, p10, median, p90, max), with a faint boxplot behind the dots. Errored episodes stay out of the distributions until the filter includes them, where they read red. The metrics tab charts every key of the run's `metrics.jsonl`: the epoch's `eval/<env>/…` summaries and the time-keyed `inference/` and `concurrency/` rows the runner samples while the epoch is in flight.

### Prime

`--monitors.prime` streams each source's epoch as an evaluation on the Prime platform:

```bash theme={null}
uv run eval gsm8k -n 32 -r 4 -c 8 --monitors.prime
```

The monitor opens one platform evaluation per env when its epoch starts, streams every episode to it as it lands, and closes it with the epoch's aggregates. The run leaves its platform identity in `monitors/prime/run.json` (the training run's id and URL, or one entry per eval epoch), and the dashboard's top bar shows a `view on platform` link from the moment the evaluation opens: a direct link for a training run or a single evaluation, a menu when a run has several.

## Metrics

Eval metrics mirror the training rollout hierarchy under the `eval/<env>` scope:

| Metric | Reading |
| - | - |
| `eval/<env>/all/<agent>/reward/mean` | mean reward over the epoch |
| `eval/<env>/all/<agent>/is_truncated/mean` | share of rollouts cut by the length limit |
| `eval/<env>/all/<agent>/is_timeout/mean` | share of rollouts stopped by a stage deadline (`<stage>_timeout`) |
| `eval/<env>/all/<agent>/has_error/mean` | share of rollouts that raised |
| `eval/<env>/all/seq_len/mean` | mean episode length in tokens |

Task-specific env metrics (e.g. `correct_answer`, `format`) appear under the same prefix. The console summary line per source (`Evaluated <env> ... Reward 0.8125`) reports the epoch mean.


# Inference
Source: https://docs.primeintellect.ai/prime-rl/inference



This page covers the inference configuration and the supported features/deployment shapes. It covers how to scale the inference server from a single GPU to 1000s of GPUs that run agentic workloads at the speed of light with all the bells and whistles configured.

## Table of Contents

* [Overview](#overview)
* [Single-Node](#single-node)
* [Multi-Node](#multi-node)
  * [Multi-replica](#multi-replica)
  * [Wide-EP](#wide-ep)
* [P/D Disaggregation](#pd-disaggregation)
* [Router](#router)
  * [Routing policies](#routing-policies)
* [Adaptive Concurrency](#adaptive-concurrency)
* [Advanced Configuration](#advanced-configuration)
  * [KV Cache Offload](#kv-cache-offload)
  * [Optimized P/D disaggregation deployment](#optimized-pd-disaggregation-deployment)
  * [Other vLLM features](#other-vllm-features)
  * [Router Replay](#router-replay)

## Overview

`prime-rl` chooses to use `vLLM` as the inference engine. We aim to stay up-to-date with the latest vLLM features, being at-most 1 version behind the latest stable release. This allows us to use the latest features from vLLM as soon as they are released - such as router replay, CPU KV cache offload, and more.

We support 3 distinct deployment shapes:

* [Single-Node](#single-node) - Runs the inference server on a single node. Useful for debugging, small scale experiments or smaller models. The default deployment shape.
* [Multi-Node](#multi-node) - Runs the inference server on multiple nodes. Useful for large scale experiments or larger models, where latency is not a concern - i.e. single turn inference, long context inference, etc.
* [Disaggregated](#pd-disaggregation) - Runs the inference server on multiple nodes, but disaggregates the prefill and decode stages. Useful for large scale experiments or larger models, where latency is a concern and multi-node deployment creates very high E2E rollout latency, such as agentic workflows.

Most of the features are supported for all deployment shapes, with few exceptions. These exceptions are rejected on validation.

Every deployment shape has the same client-facing layout: a single global router listens on `inference.server.port` and fronts all vLLM engines, which listen on `inference.backend_port` (+ rank offset). Clients always talk to one URL, regardless of how many engines run behind it.

You can select the deployment shape with `InferenceDeploymentConfig` in your config file. This is a config-field that allows you to set the deployment shape and topology knobs such as `num_nodes` and `num_replicas`.

```toml theme={null}
[inference.deployment]
type = "single_node" # or "multi_node" or "disaggregated"
```

To configure the inference server, you can use the `InferenceConfig` field. This is a config-field that allows you to set the inference server-specific knobs. Most of these are supported for all deployment shapes, with few exceptions. These exceptions are rejected on validation.

```toml theme={null}
[inference.vllm]
model = "PrimeIntellect/INTELLECT-3"
...
```

We will now walk through the supported features and deployment shapes in detail, starting with the single-node deployment.

## Single-Node

The single-node deployment is the default deployment shape. It runs the inference server on a single node. It is useful for debugging, small scale experiments or smaller models. You can configure the single-node deployment with the `SingleNodeInferenceDeploymentConfig` config-field.

```toml theme={null}
[inference.deployment]
type = "single_node"
```

The launcher starts a `vllm-router` on `inference.server.port` (default `8000`) fronting the vLLM engine on `inference.backend_port` (default `8100`). Clients connect to the router URL; admin operations (weight updates, health checks) bypass the router and hit the engine port directly — the RL entrypoint wires `orchestrator.model.client.admin_base_url` accordingly.

This deployment shape runs the inference server on a single node, if configured with NVLink enabled, it allows you more freedom in terms of parallelism configurations.

```toml theme={null}
[inference.vllm]
enable_expert_parallel = true # defaults to False
tensor_parallel_size = 2
data_parallel_size = 4

[inference.deployment]
type = "single_node"
```

We reccomend choosing your parallelism based on the expected throughput and latency requirements. High `dp` might create high latency, however it will also give you the highest throughput. This is a tradeoff you need to make based on your use case and required concurrency (see [Adaptive Concurrency](#adaptive-concurrency)). Setting `tp` to a higher value will usually give you lower latency, but the inference server also will become saturated faster with lower number of requests.

Another thing to consider, is the memory usage. You need to make sure that the model will fit into the available GPU memory. We will not go into the details on how to do this in this document. Related thing to consider, is the space for the KV cache. This will heavily affect the amount of requests your inference server can handle. You want to shard your model, either using `inference.vllm.enable_expert_parallel` or `inference.vllm.tensor_parallel_size` to maximize the available GPU memory.

You can also increase the available KV cache memory by enabling `inference.kv_cache_offload`. More details in the [Advanced Configuration](#advanced-configuration) section.

## Multi-Node

This deployment shape branches into 2 sub-shapes:

* [Multi-replica](#multi-replica) - Runs the inference server on multiple nodes, but each node runs an independent vLLM replica. You can think of this as a for-loop over single-node deployments.
* [Wide-EP](#wide-ep) - This option is gated behind `inference.vllm.enable_expert_parallel = true`. It allows you to run the inference server on multiple nodes, allowing you to use multi-node expert parallelism. This is a more advanced feature that is suitable for high-throughput, high-concurrency workloads.

### Multi-replica

This deployment shape runs the inference server on multiple nodes, but each node runs an independent vLLM replica.
Parallelism configuration is the same as the single-node deployment. The shape is defined by setting `inference.deployment.type = "multi_node"` and `inference.deployment.num_nodes` to the number of nodes you want to run the inference server on.

```toml theme={null}
[inference.deployment]
type = "multi_node"
num_nodes = 2

[inference.vllm]
model = "PrimeIntellect/INTELLECT-3"
tensor_parallel_size = 2
data_parallel_size = 4
```

This configuration will run 2 independent vLLM replicas, each with `tensor_parallel_size=2` and `data_parallel_size=4`. Routing is handled by a single global router running on the first inference node, fronting the per-rank endpoints of all replicas — either `vllm-router` (default) or the upstream `llm-d` EPP+Envoy, selected via the `[inference.router]` block. You can read more about the supported routing options in the [router](#router) section.

### Wide-EP

For huge, 200B+ scale models, you might want to use multi-node expert parallelism to maximize the KV-cache space. This deployment shape is defined by setting `inference.deployment.type = "multi_node"` and `inference.vllm.enable_expert_parallel = true`.

```toml theme={null}
[inference.deployment]
type = "multi_node"
num_nodes = 2

[inference.vllm]
model = "PrimeIntellect/INTELLECT-3"
enable_expert_parallel = true
tensor_parallel_size = 2
data_parallel_size = 8
```

This configuration will run 2 vLLM processes, each with `data_parallel_size_local = 4` and `tensor_parallel_size = 2` and expert parallelism spanning 2 nodes. The requests are again routed to these processes via the `vllm-router`.

## P/D Disaggregation

This is the most advanced deployment shape. It allows you to disaggregate the prefill and decode stages, with KV cache flowing between them. This is useful for large scale deployments, where there are high requirements on latency, such as agentic workflows spanning 100s of turns.

This deployment shape is defined by setting `inference.deployment.type = "disaggregated"` and choosing how many nodes each prefill and decode replica spans.

```toml theme={null}
[inference.deployment]
type = "disaggregated"
prefill_nodes_per_replica = 2
decode_nodes_per_replica = 2
```

Sometimes, you may want to run multiple independent vLLM instances within the prefill and decode stages. You can do this by setting `inference.deployment.num_prefill_replicas` and `inference.deployment.num_decode_replicas` to the number of role replicas you want to run.

```toml theme={null}
[inference.deployment]
type = "disaggregated"
prefill_nodes_per_replica = 2
num_prefill_replicas = 2
decode_nodes_per_replica = 2
num_decode_replicas = 1
```

Now each prefill replica spans 2 nodes and each decode replica spans 2 nodes. With 2 prefill replicas and 1 decode replica, one inference island spans 6 nodes.

For RL runs, the top-level deployment can multiply that whole inference island by setting `deployment.num_infer_replicas`. `deployment.num_infer_nodes` is inferred from the nested inference deployment when you omit it.

```toml theme={null}
[deployment] # this is a top-level RL deployment, not inference.deployment!!
type = "multi_node"
num_train_nodes = 4

num_infer_replicas = 3
```

This will run 3 inference islands, each running on 6 nodes. The total inference deployment will span 18 nodes, fronted by the single global router.

## Router

Every deployment fronts its vLLM engines with a single global router — it listens on `inference.server.port` and is the one URL clients connect to. The backend is configured via a discriminated `[inference.router]` block (`type = "vllm-router" | "llm-d"`):

```toml theme={null}
[inference.router]              # or [router] for the standalone inference entrypoint
type = "llm-d"                  # "vllm-router" (default) or "llm-d"
non_cached_tokens = 16          # llm-d only: below this many non-cached prompt tokens, skip remote prefill (P/D)

# llm-d only: base scorer weights, applied to every profile
[inference.router.scorers]
"prefix-cache-scorer" = 3.0
"active-request-scorer" = 2.0

# llm-d only: merged onto the P/D prefill profile (decode_scorer_overrides for decode)
[inference.router.prefill_scorer_overrides]
"queue-scorer" = 2.0
"kv-cache-utilization-scorer" = 2.0
```

* **`vllm-router`** (default) — our fork of [vllm-router](https://github.com/PrimeIntellect-ai/router). Knob: `policy`. The only backend supported for single-node (local) deployments.
* **`llm-d`** — the upstream [llm-d](https://llm-d.ai) Endpoint Picker (EPP) + Envoy proxy (multi-node / disaggregated SLURM deployments only). Routing combines **prefix-cache affinity** (grouped rollouts reuse a cached prefix and skip prefill) with the **`active-request-scorer`** — an in-flight load balancer that spreads requests across ranks immediately, unlike the metrics-scraped `queue-scorer` / `kv-cache-utilization-scorer` / `load-aware-scorer` (which lag and concentrate bursts of same-prefix requests). The scorer weights follow the upstream llm-d P/D guide; tune via `scorers` (base) + `prefill_scorer_overrides` / `decode_scorer_overrides` (per-profile, P/D). Does not support `enable_return_routed_experts` (router replay).

Both backends support the 2 most important things:

* Request routing - KV cache re-use and balanced routing
* P/D disaggregation - handling the prefill and decode stages separately

### Routing policies

The policies you might want to configure are:

* `sticky_least_loaded` - the default policy. A new `X-Session-ID` is assigned to the least-loaded replica, then every later turn in that rollout remains on the same replica for KV cache reuse. The orchestrator releases sessions when rollouts complete, fail, or are cancelled, including traces discarded during retries. Failed release requests are logged without disabling future cleanup.

* `consistent_hash` - hashes the `X-Session-ID` request header (sent per rollout by the verifiers clients) to pick a replica, preserving KV cache reuse without considering current load.

* `round_robin` - this policy will round-robin the requests between the available replicas. This is useful if you want to balance the load between the replicas. This might give you better results if you don't have enough rollouts to make `consistent_hash` hashing saturated.

## Adaptive Concurrency

The orchestrator continuously sizes how many episodes run against the inference engines at once. A static cap cannot be right: too low starves the engines, too high crosses into KV thrash — the engines evict prefix cache, re-prefill the evicted work, and throughput collapses. The optimal value depends on the model, the deployment, and the workload's episode lengths, and it moves as training changes the policy.

The controller's idea is to treat the engines as ground truth and probe, instead of modeling episode cost: while the engines show headroom, admit a little more; when they show pressure, back off. Three behaviors, in increasing severity:

* **Grow**: while the engines look healthy and the current cap is fully used, raise it multiplicatively. Growth is clocked by the pipeline itself — the cap rises by a fixed factor each time the in-flight pool has turned over once — so a fast single-turn workload ramps in seconds while a slow agentic workload ramps at its own pace, with no tuning per workload.
* **Trim**: episodes grow their context *while running*, so a pool that fit comfortably an hour ago can outgrow memory without a single new admission. When KV usage nears the ceiling, the controller first lowers the cap and lets completions drain the pool naturally (soft — no work lost); only if usage keeps climbing despite the closed gate does it also cancel the youngest episodes (hard — least work lost).
* **Cut**: when the engines are demonstrably overloaded — requests being preempted, or piling up in the waiting queue — cut the pool hard, cancel the excess, and hold further cuts until the system settles.

The cap starts from a safe bound derived from the engines (KV capacity divided by the maximum context length), or from `initial_inflight` when a good value is known, and always stays within `[min_inflight, max_inflight]`.

## Advanced Configuration

### KV Cache Offload

Maximizing KV-Cache space is crucial to support high-concurrency workloads. You can offload the KV cache to CPU memory (and, behind it, disk) by setting `inference.kv_cache_offload`. It is a discriminated config with two composable tiers, `cpu` and `disk`: a `cpu` tier is always required, and an optional `disk` tier is layered behind it (GPU → DRAM → disk). Disk-only is not supported.

The `type` field selects the backend:

* `native` — vLLM's built-in offloading. CPU-only uses `OffloadingConnector`; CPU+disk uses `TieringOffloadingSpec` (a CPU primary tier with a filesystem secondary tier). Fully self-contained — no extra processes.
* `mooncake` — a [Mooncake](https://github.com/kvcache-ai/Mooncake) **shared distributed store** (SLURM only). One `mooncake_master` + metadata server runs on the head inference node; every inference node runs a `mooncake_client` that contributes its DRAM (and, with `disk`, SSD) segment to that *single* pool. Because blocks are keyed by model + parallel rank + content hash (no instance id), a prefix cached by one node/replica is reusable by all of them over RDMA — pooling every node's CPU RAM into one KV cache. Use `native` for local/single-process runs.

```toml theme={null}
# Native CPU offload (reserves 128GB of CPU KV cache for this instance)
[inference.kv_cache_offload]
type = "native"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000 # 128GB

# Native CPU + disk tiering (self-contained)
[inference.kv_cache_offload]
type = "native"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000
[inference.kv_cache_offload.disk]
path = "/scratch/kv"        # disk capacity is bounded by the filesystem

# Mooncake CPU + disk (per-node distributed store, RDMA)
[inference.kv_cache_offload]
type = "mooncake"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000
[inference.kv_cache_offload.disk]
path = "/scratch/kv"
```

For `native`, `cpu.num_bytes` is the aggregate CPU KV pool for the instance (vLLM shards it across workers). For `mooncake`, `cpu.num_bytes` is the DRAM each node contributes to the shared pool (so the total pool ≈ `num_bytes × #inference-nodes`); the store uses RDMA, so it requires an RDMA-capable fabric. Enabling offload automatically enables prefix caching.

### Optimized P/D disaggregation deployment

For optimal P/D disaggregation deployment, we automatically set the decode `all2all_backend` to `deepep_low_latency` and the prefill `all2all_backend` to `deepep_high_throughput`. We currently don't support customizing all2all backends for P/D disaggragation out of the box. You can do this by overriding the slurm template only.

For KV cache transfer, we utilize the NIXL connector. This is the default and only currently supported connector. We aim to support more advanced options, such as D->P transfer, or Mooncake Connector in the future.

> **Required:** The pip-wheel NIXL's bundled UCX segfaults on the prefill→decode KV transfer. You must build NIXL against UCX 1.19.x from source — see [Disaggregated Prefill/Decode Inference](/prime-rl/advanced#disaggregated-prefilldecode-inference) in the Advanced docs for the full setup.

For configuring various knobs with environment variables, we enable you to configure prefill and decode environment variables separately. This is useful if you want to configure different environment variables for the prefill and decode stages.

```toml theme={null}
[inference.deployment]
type = "disaggregated"

[inference.deployment.prefill_env_vars]
"VLLM_ENABLE_MOE_DP_CHUNK" = "0"
"VLLM_DEEP_GEMM_WARMUP" = "skip"

[inference.deployment.decode_env_vars]
"VLLM_DEEP_GEMM_WARMUP" = "skip"
```

These are role-specific and layer on top of [`env_vars`](/prime-rl/configuration#environment-variables) shared by all inference processes regardless of role.

### Other vLLM features

The `[inference.vllm]` section is a pass-through: every key is forwarded to the vLLM server under vLLM's own argument name, whether prime-rl types it or not. Anything `vllm serve` accepts can be set here.

```toml theme={null}
[inference.vllm]
headless = true
max_num_seqs = 256
```

On the CLI the same keys are available as `--inference.vllm.max-num-seqs 256` (or `--vllm.max-num-seqs 256` for the standalone inference entrypoint); dict-valued arguments take a JSON string, e.g. `--vllm.compilation-config '{"cudagraph_mode": "NONE"}'`.

### Router Replay

Router replay works by capturing the expert routing decisions into a buffer. This buffer then gets sent to the trainer, which can use it instead of re-computing the routing. This lowers the trainer↔inference mismatch by an order of magnitude, resulting in more stable training.

To enable router replay, you can set `inference.vllm.enable_return_routed_experts = true`. Capture runs on vLLM's V2 model runner (forced automatically). Router replay is not supported with disaggregated P/D: vLLM does not capture routed experts across NIXL KV transfer.

```toml theme={null}
[trainer]
enable_router_replay = true # this will also auto-set inference.vllm.enable_return_routed_experts = true

[inference.vllm]
enable_return_routed_experts = true
```

This however is not free, it adds a significant overhead to the HTTP requests as this payload can grow quite large. We reccomend sizing up the env server pool (`orchestrator.*.source.serve.pool`) to allow for more parallelization on the verifiers side.

Currently this feature is also not supported with CPU KV cache offload, which can have negative impact on the inference throughput.

### Sampling Replay

Truncated sampling (`top_p < 1`, `top_k`) renormalizes the sampling distribution over a sampling mask of surviving token ids. The rollout logprobs reflect that (`logprobs_mode = "processed_logprobs"`), so the trainer must renormalize over the same mask — otherwise every importance ratio is biased and training collapses (DeepSeek V3.2's "Keep Sampling Mask", [arXiv:2512.02556](https://arxiv.org/abs/2512.02556) §3.1; Cognition's [SWE-1.7 post](https://cognition.com/blog/swe-1-7)). prime-rl handles this automatically: vLLM records the sampling mask at sampling time (`--return-sampling-mask`, native since vLLM 0.28) and the trainer renormalizes its logprobs over it.

```toml theme={null}
[orchestrator.train.sampling]
top_p = 0.95
top_k = 512   # optional, defaults to 512 under truncation (bounds each sampling mask)
```

That's all — there are no replay flags. Truncated train sampling makes the inference server return sampling masks (`inference.enable_return_sampling_mask`, auto-enabled), and the trainer replays them. Train-sampling `top_k` above 512 is rejected: the trainer pads each micro batch to its largest sampling mask, so the bound caps trainer memory. Configs that would break under renormalized logprobs are rejected: `opd`/`opsd` and truncation knobs smuggled via `extra_body`. Frozen-source envs are exempt.

Capture runs on vLLM's V2 model runner (forced automatically) and is engine-wide. It can run together with router replay. While sampling capture is on, vLLM rejects any request with `temperature <= 0` or without an effective `top_k > 0` — eval sampling against the training server must set `top_k` (the model's generation config often supplies one) and a non-zero temperature.

When launching the inference server standalone, set `inference.enable_return_sampling_mask = true` yourself; clients must sample with `0 < top_k <= 512`.


# Ipo
Source: https://docs.primeintellect.ai/prime-rl/ipo



IPO (INTELLECT Policy Optimization) is the default RL loss in prime-rl. This page explains why it masks tokens by probability instead of ratio, and shows the ablation behind the default `eps = 0.3`. The loss definition and its knobs are in [Algorithms § IPO Loss](/prime-rl/algorithms#ipo-loss).

## Table of Contents

* [Masking by Probability, Not Ratio](#masking-by-probability-not-ratio)
* [Ablation: IPO vs IcePop](#ablation-ipo-vs-icepop)
* [Choosing `eps`](#choosing-eps)
* [Reproducing](#reproducing)

## Masking by Probability, Not Ratio

In async RL, the trainer and the inference engine never agree exactly on a token's probability. The usual fix keeps importance sampling but drops the tokens where the two disagree too much. [IcePop](/prime-rl/algorithms#icepop-loss) measures disagreement as a ratio and masks a token when $\pi_{\text{train}}/\pi_{\text{infer}}$ falls outside a fixed band $[\alpha, \beta]$. IPO masks a token when its absolute probability moved too much:

$$
\left|\pi_{\text{train}}(y_t) - \pi_{\text{infer}}(y_t)\right| > \epsilon
$$

```python theme={null}
probs_diff = torch.exp(trainer_logprobs) - torch.exp(inference_logprobs)
is_masked = probs_diff.abs() > eps  # double-sided, independent of the advantage sign
```

A ratio treats every token the same regardless of its probability, which behaves badly at both ends:

* A rare token going from 0.01 to 0.06 has a ratio of 6×, so IcePop masks it, although it moved only 0.05 of probability mass.
* A likely token going from 0.90 to 0.20 has a ratio of 0.22, so IcePop keeps it, although it moved 0.70 of probability mass.

Treat the vocabulary as two outcomes, *this token* and *everything else*. Then $|\pi_{\text{train}} - \pi_{\text{infer}}|$ is the total-variation distance between the two engines on that split. The mask is double-sided and has one knob. [KPop](https://ringtech.notion.site/kpop) reached the same diagnosis independently and uses a binary KL instead.

## Ablation: IPO vs IcePop

GLM-4.5-Air on agentic SWE RL, with every arm on the same recipe and only `[trainer.loss]` changing:

| | |
| - | - |
| Model | `zai-org/GLM-4.5-Air` (100B MoE), BF16 training and inference |
| Task | `scaleswe` with the `bash` harness, taskset reward; eval on SWE-Bench Verified |
| Topology | 2 trainer nodes (Muon, CP 4 ulysses), 4 single-node TP=8 inference replicas with expert parallelism |
| Batch | 256 rollouts (16 groups of 16) at 131k context |
| Off-policy | up to 64 steps (`max_off_policy_steps = 64`), no router replay |
| Algorithm | GRPO with a linear length penalty (output tokens, input tokens, and turns at 0.1) |
| Arms | IcePop with $[\alpha, \beta] = [0.2, 5]$; IPO with `eps = 0.2` and `eps = 0.3` |

![IPO vs IcePop on GLM-4.5-Air SWE RL: solve rate (top) and trainer–inference KL (bottom)](https://mintcdn.com/primeintellect/IcgjaU00pP2Y_17X/prime-rl/assets/ipo/ipo-vs-icepop.png?fit=max\&auto=format\&n=IcgjaU00pP2Y_17X\&q=85\&s=8245fb3754049b117877512946efca11)

For the first 1k steps the arms are indistinguishable. IcePop's trainer–inference KL then creeps up and explodes around step 1.2k, more than 20× its baseline, and the solve rate falls from about 0.8 to about 0.3. Both IPO arms keep the mismatch low through the same window and keep training.

IcePop masks very little before it breaks: at steps 678–679 it masked 72 of 2.16M tokens (0.003%). Under a ratio band, the masked fraction stays tiny while the gap widens, so nothing pushes back once drift starts.

## Choosing `eps`

The first IPO default was `eps = 0.1`. On the run above it masked about 2% of tokens, and by step 1k its entropy had collapsed. To pick a new value without a sweep of full runs, we replayed the trainer and inference probabilities of 1.8M sampled tokens and measured what each `eps` would have masked:

![Left: sampled tokens by inference and trainer probability, with the IPO eps = 0.3 band and the IcePop ratio bounds. Right: fraction of tokens masked as a function of eps.](https://mintcdn.com/primeintellect/IcgjaU00pP2Y_17X/prime-rl/assets/ipo/eps-choice.png?fit=max\&auto=format\&n=IcgjaU00pP2Y_17X\&q=85\&s=ae3227778403a7c5700ee838fd390e45)

| `eps` | 0.1 | 0.2 | **0.3** | 0.5 | IcePop \[0.2, 5] |
| - | - | - | - | - | - |
| tokens masked | 1.98% | 0.64% | **0.25%** | 0.047% | 0.06% |

The IPO band has the same width in probability everywhere. The IcePop cone is very narrow near zero, where it masks rare tokens that barely moved, and very wide near one, where it keeps likely tokens that lost most of their mass. `eps = 0.3` masks only the tail and targets the tokens IcePop lets through. Both 0.2 and 0.3 were stable in full runs; 0.3 kept the highest entropy with the lowest mismatch KL, so it is the default.

## Reproducing

The configs are in [`examples/advanced/glm-4.5-air/ipo-ablation/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-4.5-air/ipo-ablation): a shared `base.toml` and one overlay per loss. See the [GLM-4.5-Air README](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-4.5-air) for cluster setup.

```bash theme={null}
uv run rl @ examples/advanced/glm-4.5-air/ipo-ablation/base.toml @ examples/advanced/glm-4.5-air/ipo-ablation/ipo.toml
uv run rl @ examples/advanced/glm-4.5-air/ipo-ablation/base.toml @ examples/advanced/glm-4.5-air/ipo-ablation/ipo.toml --trainer.loss.eps 0.2
uv run rl @ examples/advanced/glm-4.5-air/ipo-ablation/base.toml @ examples/advanced/glm-4.5-air/ipo-ablation/icepop.toml
```

Compare `mismatch_kl/all/mean` and the training solve rate across arms.


# Overview
Source: https://docs.primeintellect.ai/prime-rl/overview



`prime-rl` is a framework for large-scale, asynchronous reinforcement learning of large language models. It is designed to be easy to use and hackable, yet capable of training 1T+-parameter MoE models on 1000+ GPU clusters.

## Architecture

A `prime-rl` RL run is three cooperating processes:

![Architecture](https://mintcdn.com/primeintellect/mV0d6IvWxHP4DqoX/prime-rl/assets/architecture.png?fit=max\&auto=format\&n=mV0d6IvWxHP4DqoX\&q=85\&s=6950d6f6b3fc5be043397667162b2b20)

* **Inference** — vLLM-backed server (or fleet) holding the current policy. The orchestrator drives rollouts through the token-in `/inference/v1/generate` route via the [`renderers`](https://github.com/PrimeIntellect-ai/renderers) package (OpenAI-compatible chat/completions routes are also exposed for external clients). We are trying to stay up-to-date with the latest vLLM features, you can read more about the supported features and deployment options in the dedicated [inference documentation](/prime-rl/inference).
* **Orchestrator** — Lightweight CPU process that owns the data plane across many [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) training and eval environments. Each env runs in an isolated subprocess with a variable-size pool of env workers for scalability. The orchestrator drives multi-turn rollouts against the inference fleet (tool use, browsers, sandboxes, long horizons) without re-tokenizing across turns, computes advantages, packs the rollouts into training batches, and relays new weights from trainer to inference.
* **Trainer** — FSDP2 process group that consumes packed rollouts and steps the optimizer. We ship optimized custom modeling code for many MoE / dense / VLM families that unlocks advanced trainer parallelism — expert parallelism (EP, with DeepEP kernels) and context parallelism (CP) for long-sequence training — plus selective activation checkpointing, FP8 training on Hopper+, and LoRA. You can read more in the dedicated [training documentation](/prime-rl/training).

The three processes communicate through configurable transports — by default the trainer↔orchestrator rollout link uses ZMQ, and weight broadcast uses NCCL for synchronous in-memory transfer (falling back to filesystem when LoRA is enabled or no inference server is configured). Swap the rollout link to the local filesystem (`rollout_transport.type = "filesystem"`) if you want rollouts persisted to disk. See [Scaling](/prime-rl/scaling) for the deployment options.

## Installation

```bash theme={null}
curl -sSL https://raw.githubusercontent.com/PrimeIntellect-ai/prime-rl/main/scripts/install.sh | bash
```

The script clones the repo, initializes the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) / [`renderers`](https://github.com/PrimeIntellect-ai/renderers) / [`prime-envs`](https://github.com/PrimeIntellect-ai/prime-envs) submodules, installs `uv`, and runs `uv sync --all-extras`. For manual setup, or troubleshooting, see the [README](https://github.com/PrimeIntellect-ai/prime-rl#setup).

You need at least one NVIDIA GPU (RTX 3090/4090/5090, A100, H100, H200, or B200). Single-GPU runs are supported for debugging; production RL is typically 1× inference node + 1+ trainer nodes.

## Quick Run

Train an SFT-warmed `Qwen3-0.6B` on the `reverse-text` task — the env is bundled with the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) submodule so no separate install is needed. This config ships in the repo and runs on two GPUs (one for inference, one for the trainer):

```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml
```

The `rl` entrypoint reads `examples/basic/reverse-text/rl.toml`, splits it into per-process sub-configs, picks GPU 0 for inference and GPU 1 for the trainer, launches all three processes, and tees their stdout into the run directory at `outputs/<run_name>/logs/{trainer,orchestrator,inference}.log` (`run.name` auto-generates as `<envs>--<model>--<short-id>`; pass `--run.name <name>` for a predictable path). Within a minute the trainer should log `step 1` and a reward sample; after 20 steps the run completes with a final DCP checkpoint at `outputs/<run_name>/checkpoints/step_20`.

## Documentation

* **[Configuration](/prime-rl/configuration)** — TOML composition, CLI overrides, dry-run.
* **[Training](/prime-rl/training)** — Launch and observe RL and SFT runs.
* **[Inference](/prime-rl/inference)** — vLLM-backed server (or fleet) holding the current policy.
* **[Scaling](/prime-rl/scaling)** — Single-GPU through multi-node clusters via FSDP / EP / CP and SLURM.
* **[Algorithms](/prime-rl/algorithms)** — Async semantics, loss / advantage / filter plugins, trajectory merging.
* **[Advanced](/prime-rl/advanced)** — Custom modeling, multimodal, LoRA, P/D inference.
* **[Development](/prime-rl/development)** — Test suite, pre-commit hooks, adding a new model.


# Scaling
Source: https://docs.primeintellect.ai/prime-rl/scaling



This page covers how to scale `prime-rl` from a single GPU to a 1000-GPU cluster: single-node and multi-node deployments, FSDP / expert parallelism / context parallelism, and throughput benchmarking. See [Training](/prime-rl/training) for detailed documentation of the trainer configuration and [Inference](/prime-rl/inference) for the inference configuration.

## Table of Contents

* [Single-Node vs. Multi-Node Deployment](#single-node-vs-multi-node-deployment)
  * [Single-Node](#single-node)
    * [RL Placement](#rl-placement)
    * [SFT and Torchrun](#sft-and-torchrun)
  * [Multi-Node](#multi-node)
* [Parallelism Knobs](#parallelism-knobs)
  * [FSDP](#fsdp)
  * [Expert Parallelism](#expert-parallelism)
  * [Context Parallelism](#context-parallelism)
  * [Activation Checkpointing and Offloading](#activation-checkpointing-and-offloading)
  * [Optimizer Offloading](#optimizer-offloading)
  * [LM Head Chunking](#lm-head-chunking)
* [Memory-Tight Recipe](#memory-tight-recipe)
* [Max Trainer Throughput Recipe](#max-trainer-throughput-recipe)
* [SLURM](#slurm)
  * [Activation](#activation)
  * [`[deployment]` Block](#deployment-block)
  * [Examples](#examples)
  * [Custom Templates](#custom-templates)
* [Benchmarking](#benchmarking)

## Single-Node vs. Multi-Node Deployment

The `rl`, `sft`, and `inference` entrypoints all accept a `[deployment]` block (`type = "single_node"` or `"multi_node"`) that picks how the trainer / orchestrator / inference processes are placed across hardware. **Single-node** runs locally; **multi-node** currently goes through [SLURM](#slurm) — the launcher writes an sbatch script that places inference replicas, the orchestrator, and the trainer with the right rendezvous endpoints, IPs, ports, and shared-filesystem paths wired in.

### Single-Node

#### RL Placement

`rl` defaults to 1 trainer GPU and 1 inference GPU. To give inference 6 GPUs with data parallelism and the trainer the remaining 2 on an 8-GPU node:

```bash theme={null}
uv run rl @ rl.toml \
  --deployment.num-infer-gpus 6 \
  --deployment.num-train-gpus 2 \
  --inference.vllm.data-parallel-size 6
```

The launcher allocates GPUs in order from `CUDA_VISIBLE_DEVICES` (or all visible GPUs): inference first, trainer next. To target a specific physical subset, pin `CUDA_VISIBLE_DEVICES` before launching.

For quick A/B ablations on the same node, run two RL instances side-by-side in separate shells, each pinned to half the GPUs and a separate inference port, and track both in one dashboard:

```bash theme={null}
# shell 1, GPUs 0–1, default port 8000
CUDA_VISIBLE_DEVICES=0,1 uv run rl @ rl.toml --run.name exp1

# shell 2, GPUs 2–3, port 8001
CUDA_VISIBLE_DEVICES=2,3 uv run rl @ rl.toml \
  --inference.server.port 8001 \
  --orchestrator.model.client.base-url http://localhost:8001/v1 \
  --run.name exp2

# watch both side by side
uv run dashboard outputs
```

#### SFT and Torchrun

`uv run sft` handles distributed launch internally. To scale from 1 to N GPUs, set the deployment GPU count (or just let it pick up `WORLD_SIZE`). For non-default layouts, the manual equivalent is:

```bash theme={null}
uv run torchrun \
  --nproc-per-node 8 \
  --local-ranks-filter 0 \
  src/prime_rl/trainer/sft/train.py @ sft.toml
```

`--local-ranks-filter 0` keeps console output to rank 0 only; per-rank stdout/stderr is still captured in `<run_dir>/logs/latest/trainer/torchrun/`.

### Multi-Node

Multi-node deployments (RL or SFT) are launched via [SLURM](#slurm) — set `[deployment] type = "multi_node"` plus the matching `[slurm]` block, and the launcher writes the sbatch script that places inference, orchestrator, and trainer across the requested nodes with the inter-process wiring set up correctly. See [SLURM § Examples](#examples) for full configs.

## Parallelism Knobs

### FSDP

FSDP2 is the default model sharding strategy. By default the trainer fully shards parameters, gradients, and optimizer state across the data-parallel mesh. Tweakable knobs:

| Knob | Effect |
| - | - |
| `trainer.model.dp_replicate` | Number of dimensions to **replicate** instead of shard. Set to 2 to run 2-way DP replication × FSDP sharding within each replica — useful for very large clusters where pure FSDP communication dominates. |
| `trainer.model.reshard_after_forward` | If `true` (default), parameters are resharded after the forward pass to free memory; the backward pass re-gathers. Set `false` to keep params resident — faster but more memory. |
| `trainer.model.offload` | CPU offloading: `optimizer` (default) offloads optimizer state between steps; `full` offloads gradients, FP32 masters, and optimizer state and runs the optimizer (AdamW or SignSGD) on CPU during backward; `none` disables offloading. |

### Expert Parallelism

EP shards MoE expert weights across the EP mesh, dramatically reducing the FSDP communication volume per layer and improving the training throughput.

`ep` defaults to `"auto"`. At startup it loads the model config to check whether the model is MoE. For MoE models it resolves to `min(world_size // dp_replicate, 8)`: the FSDP island size, capped at 8. For non-MoE models it resolves to 1 (no-op). `"auto"` does not look at `num_experts` or `cp`; the resolved degree must still be a multiple of `cp` and divide `num_experts`, so set it explicitly when it does not. Set `ep` to an explicit integer to override:

```toml theme={null}
[trainer.model]
ep = 8  # explicit EP degree; must divide num_experts

[trainer.model.moe.dispatch]
type = "torch"
transport = "bf16"
```

For DeepEP, set `type = "deepep"` and tune `num_sms` plus optional `token_chunk_size` in the same dispatch table. Routed-expert precision is selected separately with `[trainer.model.moe.compute]` (`bf16`, `deepgemm_fp8`, or `mxfp8`).

### Context Parallelism

CP shards a single sequence across multiple GPUs along the token dimension — for long-context sequences. Prefer `ulysses`: it gets the most throughput and is the only style that works for hybrid linear-attention/Mamba models (Qwen3.5, NemotronH) and for VLMs. Each model class declares its supported styles in `cp_support`; an unsupported `cp_style`, or a model with no CP support at all, is rejected at setup.

`ulysses` head-shards Q/K/V, so the CP degree must divide `num_attention_heads`. GQA models with fewer KV heads than the CP degree (e.g. NemotronH: 32 query heads, 2 KV heads) are supported via KV-head replication; the CP degree must then be a multiple of `num_key_value_heads`. Nemotron-H redistributes its Mamba activations between sequence- and head-parallel layouts in the model-owned Mamba path; the CP degree must divide `mamba_num_heads` and `n_groups`.

```toml theme={null}
[trainer.model]
attn = "auto"                # auto = FA3 on Hopper, FA4 on datacenter Blackwell, FA2 otherwise; or flash_attention_2/3/4
cp = 2                       # CP degree
cp_style = "ulysses"         # "ring"
```

### Activation Checkpointing and Offloading

| Knob | Memory ↓ | Throughput ↓ |
| - | - | - |
| `trainer.model.ac` | large | \~25% |
| `trainer.model.ac.mode = "selective"` | medium | small |
| `trainer.model.ac_offloading` | extra | a bit more |

AC and AC offloading are enabled by default (full mode). Both AC modes retain stateful MoE routing updates and DeepEP communication that cannot safely replay. Selective AC uses the same transformer-block boundaries and additionally retains distributed communication, expensive matrix multiplications, grouped GEMMs, and supported attention kernels:

```toml theme={null}
[trainer.model.ac]
mode = "selective"
```

Set `targets` to operator names (for example, `"aten::mm"`) or namespaces (for example, `"prime_rl_collectives"`) to replace the default selective targets. Correctness-required operations remain retained.

Activation offloading still applies to tensors saved by autograd, but tensors retained by the checkpoint policy remain on the accelerator.

`ac_offloading` is also on by default with `max_inflight_activations = 5`. We've observed this feature to be very effective, lowering the peak memory usage by 30-40% in some cases, while only losing \~3-5% of throughput. To disable either, set `model.ac = "None"` or `model.ac_offloading = "None"`.

### Optimizer Offloading

State-only optimizer offload is enabled by default (`model.offload = "optimizer"`). For full offload, set `model.offload = "full"`; this keeps BF16 compute weights on GPU and runs CPU optimizer chunks as gradients become ready during backward. Full offload only supports AdamW and SignSGD (`optim.type = "sign_sgd"`) and disables gradient clipping. SignSGD is stateless, so it halves the host RAM footprint versus AdamW (8 instead of 16 bytes per parameter: FP32 master + FP32 accumulated gradient, no moments).

### LM Head Chunking

The vanilla LM head materializes a `[batch * seq, vocab]` logits tensor on every step — a major memory tax when the vocabulary is large (often >100K). `fused_lm_head_token_chunk_size` swaps in a custom fused linear + logprob/entropy kernel that streams through `chunk_size` tokens at a time, avoiding the materialization. It defaults to `8192` for RL training:

```toml theme={null}
[trainer.model]
fused_lm_head_token_chunk_size = 8192       # default
# fused_lm_head_token_chunk_size = "disabled"  # vanilla LM head
```

Drop the chunk size further when peak memory is still tight (e.g. with very long sequences); raise it to amortize kernel-launch overhead. SFT training silently disables this (not supported yet).

## Memory-Tight Recipe

The kitchen-sink config for fitting large MoE on limited GPUs at acceptable throughput. AC, AC offloading, compile, fused LM head chunking, optimizer offload, and EP auto-resolution are on by default — only CP needs to be set explicitly (and EP overridden if auto-resolution is not desired):

```toml theme={null}
[trainer.model]
ep = 8
cp = 2

[trainer.model.ac]
freq = 1

[trainer.model.ac_offloading]
max_inflight_activations = 1
```

The defaults already cover: fused LM head chunking (`8192`), `torch.compile` (fullgraph=False), AC (full mode), AC offloading (`max_inflight_activations=5`), and optimizer CPU offload. Walks through every memory lever in order: FSDP+EP shard the weights, CP shards the activations along the token dim, AC + AC offloading shrink the activation footprint, fused LM head chunks the loss, `torch.compile` reduces fragmentation, optim offload moves Adam state off GPU. Apply selectively — each knob has a throughput cost.

## Max Trainer Throughput Recipe

The trainer defaults are tuned for memory, not speed: optimizer CPU offload, AC offloading, and `torch.compile` without `fullgraph` let large models fit on few GPUs, at a throughput cost. When you have enough GPUs to keep the model state and activations on GPU, turn them off:

```toml theme={null}
[model]
offload = "none"            # optimizer states stay on GPU
ac_offloading = "None"      # activations stay on GPU
moe_router_dtype = "bfloat16"  # fullgraph needs the router inside the block's FSDP unit; an fp32 router is not

[model.compile]
fullgraph = true            # compile each block as a single graph
```

Then add nodes instead of CP (`cp = 1`) and run one full-length sequence per GPU (`micro_batch_size = 1`). Keep full AC unless selective AC fits. The SFT default router dtype is already `bfloat16`. RL defaults to an fp32 router, so check that bf16 routing is acceptable for your model before using `fullgraph` in RL.

Examples:

* [`examples/advanced/qwen3-235b-a22b/sft-max-perf.toml`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/qwen3-235b-a22b): Qwen3-235B-A22B SFT on 4 x 8 B300 at 16k, around 46% MFU.
* [`examples/advanced/glm-5.3/sft/h200/16k-12-node.toml`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.3/sft/h200): the same recipe as an overlay for GLM-5.3 SFT on 12 H200 nodes.

## SLURM

The `rl`, `sft`, and `inference` entrypoints all submit to SLURM when a `[slurm]` table is present — there's no separate entrypoint.

> **The prime-rl checkout and its `uv` venv must live on a shared filesystem** visible to every node. The generated sbatch script runs a single `uv sync --all-extras --all-packages` on the batch node (not once per node), so all ranks share that one environment — a node-local venv would leave the other nodes stale.

Compile and JIT caches are node-local by default: the launcher sets `TRITON_CACHE_DIR=/tmp/triton-<uid>` for every component, and `VLLM_CACHE_ROOT=/tmp/.vllm-cache-<uid>` and `FLASHINFER_WORKSPACE_BASE=/tmp/.flashinfer-cache-<uid>` for inference. Ranks writing one cache on a shared filesystem can crash or hang. Override them in `[env_vars]` or a component's `env_vars`.

### Activation

A SLURM config is usually a thin overlay that adds `[slurm]` (and `[deployment]` for multi-node) on top of a base config. Configs are composed left-to-right via the `@` CLI syntax — see [Configuration § TOML Composition](/prime-rl/configuration#toml-composition):

```toml theme={null}
# my_slurm.toml
output_dir = "/shared/outputs/my-rl"

[slurm]
job_name = "my-rl-run"
```

Launch:

```bash theme={null}
uv run rl @ base_rl.toml @ my_slurm.toml             # submits via sbatch
uv run rl @ base_rl.toml @ my_slurm.toml --dry-run   # writes launcher/rl.sbatch + resolved config, exits
```

Every SLURM entrypoint stores its generated scripts and coordination files under `<run_dir>/launcher/`. SLURM stdout and stderr go to `launcher/logs/`. Local launches do not create `launcher/`.

### `[deployment]` Block

`[deployment]` is a discriminated union picked by `type` — `single_node` or `multi_node` for RL/SFT, with an extra disaggregated variant for inference. RL multi-node:

```toml theme={null}
[deployment]
type = "multi_node"
num_train_nodes = 2
num_infer_nodes = 1              # optional when inference.deployment defines the node topology
gpus_per_node = 8                # default
nodes_per_fsdp_group = 1         # optional — controls FSDP island size
```

SFT multi-node:

```toml theme={null}
[deployment]
type = "multi_node"
num_train_nodes = 2
num_infer_nodes = 1  # required only for online evals
gpus_per_node = 8
```

### Examples

Full multi-node configs ship under [`examples/advanced/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced):

* [`nemotron-3-super/swe.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/configs/advanced/nemotron-3-super/swe.toml) — 4 trainer + 1 inference node RL on a 120B hybrid-Mamba MoE with NCCL weight broadcast.
* [`glm-5.3/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.3) — large-scale and P/D-disaggregated inference across the GLM-5 family.
* [`glm-4.5-air/swe-2-node.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/advanced/glm-4.5-air/swe-2-node.toml) — train a 100B+ MoE on only two nodes: a single-node sign-SGD trainer (full optimizer offload) paired with one inference replica at batch size 64. An overlay on the shared `swe-budget.toml` base — launch with `uv run rl @ swe-budget.toml @ swe-2-node.toml`.

For inference-only multi-node, set `[deployment] type = "multi_node"` on an inference TOML — each node runs an independent vLLM replica (TP and DP must fit within one node), fronted by a single global router on node 0. Point clients at the router URL the launcher prints.

### NIXL weight broadcast

Set `[weight_broadcast] type = "nixl"` to use receiver-driven NIXL weight transfer. Before the first SLURM run, install the NIXL/UCX build and the ModelExpress service binaries on the shared filesystem:

```bash theme={null}
bash scripts/install_nixl_from_source.sh
uv pip install --reinstall --no-deps deps/nixl_cu13-*.whl
bash scripts/install_modelexpress.sh
```

The generated job starts a job-scoped ModelExpress server and Redis backend on the trainer head node and passes that address to every component. To use an existing service, set `slurm.launch_modelexpress = false` and configure `weight_broadcast.host` and `weight_broadcast.port`.

The launcher requires the CUDA and InfiniBand transports from `third_party/ucx`. Each NIXL process selects the active InfiniBand port nearest its GPU; an explicitly configured `UCX_NET_DEVICES` takes precedence. Inference ranks start their pulls at different trainer ranks so concurrent workers distribute traffic across all available source rails.

ModelExpress exchanges peer metadata during startup. Weight updates reuse prepared NIXL requests, post every trainer-rank read in a transfer group concurrently, and use versioned NIXL notifications for source readiness and buffer credits.

By default, the trainer and inference worker each allocate one transfer arena. Set `weight_broadcast.overlap_transfer_and_replay = true` to allocate two arenas on both sides and replay one weight group while receiving the next. The additional arena is the size of the largest transfer group per GPU; allocation errors are reported instead of silently disabling overlap.

### Custom Templates

For unusual partitions, module loads, or environment setup, supply your own Jinja2 template:

```bash theme={null}
uv run rl @ my_config.toml --slurm.template-path path/to/my_template.sbatch.j2
```

The default templates live under [`src/prime_rl/templates/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/src/prime_rl/templates) — copy one as a starting point.

## Benchmarking

To benchmark a parallelism config before committing a multi-day run, run a short training with fake data and a step cap, and read throughput / MFU / step time / peak memory from the logs:

```bash theme={null}
# SFT trainer alone
uv run sft @ sft.toml --data.type fake --max-steps 4

# RL trainer alone (no inference involved) -- launch it under torchrun, like the `rl` launcher does
uv run torchrun --nproc-per-node=8 src/prime_rl/trainer/rl/train.py @ train.toml --data.fake --max-steps 4
```

Every step logs `Throughput`, `MFU`, and `Peak Mem.` to the console. For machine-readable numbers, the file monitor writes `monitors/file/metrics.jsonl` under the run's output directory by default (`monitors.file`); aggregate `perf/throughput`, `perf/mfu`, `time/step`, and `perf/peak_memory` from the run's `metrics.jsonl` — skip the first step, it is warmup. [`benchmarks/scripts/run_single_benchmark.py`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/benchmarks/scripts/run_single_benchmark.py) does exactly this and is what the CI benchmark matrix runs.


# Training
Source: https://docs.primeintellect.ai/prime-rl/training



This page covers everything you need to launch, observe, checkpoint, and recover a `prime-rl` training run — the RL trainer (and the distillation algorithms that run through it) and the SFT trainer. For multi-node and cluster layouts, see [Scaling](/prime-rl/scaling). For the loss math and algorithm knobs, see [Algorithms](/prime-rl/algorithms). For standalone evals, see [Eval](/prime-rl/eval).

> **AI agents working in this repo:** the equivalent runbooks are at [`skills/training/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/skills/training) — top-level routing in [`skills/training/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/SKILL.md), launch details in [`skills/training/start-run/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/start-run/SKILL.md), and check-in / restart procedures in [`skills/training/monitor-run/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/monitor-run/SKILL.md).

## Table of Contents

* [Entrypoints](#entrypoints)
* [RL Trainer](#rl-trainer)
  * [Launch](#launch)
  * [Useful Knobs](#useful-knobs)
  * [Algorithms](#algorithms)
  * [Important Metrics](#important-metrics)
* [SFT Trainer](#sft-trainer)
  * [Dataset Format](#dataset-format)
  * [Launch](#launch-1)
  * [SFT-Specific Knobs](#sft-specific-knobs)
  * [Important Metrics](#important-metrics-1)
* [Checkpointing](#checkpointing)
  * [Enabling Checkpoints](#enabling-checkpoints)
  * [Resuming a Run](#resuming-a-run)
  * [Serving Checkpoints](#serving-checkpoints)
* [Observability](#observability)
  * [Log Files](#log-files)
  * [Console Output](#console-output)
  * [Weights & Biases](#weights--biases)
  * [Platform Monitoring](#platform-monitoring)
* [Rules of Thumb](#rules-of-thumb)

## Entrypoints

| Command | Purpose | Notes |
| - | - | - |
| `uv run rl` | Wraps the trainer, orchestrator, and inference server in one launch from a merged TOML. | The default for any RL run. Runs locally for single-node experiments; submits to SLURM for single- or multi-node when `[slurm]` is set (see [Scaling § SLURM](/prime-rl/scaling#slurm)). |
| `uv run sft` | Supervised fine-tuning on a HF dataset. | Launches torchrun internally; never call torchrun directly. |
| `uv run inference` | vLLM server. | Always use this entrypoint over `vllm serve` — it adds `/update_weights`, `/load_lora_adapter`, and `/init_broadcaster`. |
| `uv run orchestrator` | Standalone orchestrator process. | Pair with a separately-launched inference server and one `env-server` per source. |
| `uv run eval` | Multi-env evals against a live inference server. | One epoch per source, pinned (or adaptive) concurrency, cursor checkpoints + `--resume`, dashboard + optional platform upload; see [Eval](/prime-rl/eval). |
| `uv run env-server` | Standalone env server for one environment. | The `rl` launcher starts these automatically (one per train/eval source; each binds an OS-assigned loopback port and publishes it to `configs/attempt_N/resolved/envs/<split>/<name>.address` for the orchestrator); only needed when running the orchestrator standalone, or for sources with an explicit `serve.address` — those are externally managed (e.g. on their own host) and the launcher expects the server to already run there. |

## RL Trainer

### Launch

The minimal RL run trains an SFT-warmed `Qwen3-0.6B` on the `reverse-text` task — the env is bundled with the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) submodule, so nothing else needs to be installed:

```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml
```

### Useful Knobs

A condensed view of the knobs you'll most often tune. For trainer-side parallelism, sampling, optimizer, and loss knobs see [Scaling](/prime-rl/scaling) and [Algorithms](/prime-rl/algorithms).

**Data and algorithm:**

| Knob | What it does |
| - | - |
| `orchestrator.batch_size` | Tasks per trainer step. |
| `orchestrator.constant_trainer_batch_size` | Keep trainer batches constant when samples have no training signal, such as zero advantage on all tokens. Enabled by default. Disable it for faster collection with variable trainer batch sizes. |
| `orchestrator.train.group_size` | Rollouts generated per task. |
| `orchestrator.max_off_policy_steps` | Maximum staleness of a trained rollout (default 8): the version a batch trains on minus the oldest version that generated the rollout, queue time included. Episodes past the bound are dropped; a group shares one dispatch version, so its episodes age out together. The main off-policy dial on long agentic rollouts — bump for throughput, lower for tighter on-policyness. Watch `off_policy/*` and `mismatch_kl/all/mean` when tuning. |
| `[orchestrator.train.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `hierarchical_grpo`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). |
| `trainer.model.freeze_moe_router` | Freeze MoE router gates (default `true` for RL, `false` for SFT). Router updates in RL shift the trainer's expert choices away from the ones inference sampled, amplifying the train-inference mismatch, so RL keeps the pretrained router. Set `false` to train it. |
| `[[orchestrator.train.source]]` | Training sources. List multiple tables for multi-env training; weight them via `ratio`. See [Configuration § Training sources](/prime-rl/configuration#training-sources-orchestratortrainsource). |
| `[[orchestrator.eval.source]]` + `orchestrator.eval.interval` | Eval environments and cadence (default every 100 steps). |

**Monitoring:**

| Knob | What it does |
| - | - |
| `log.level` | Process log level for trainer + orchestrator (`info` default; falls back to `$PRIME_LOG_LEVEL`). Set per-process via `trainer.log.level` / `orchestrator.log.level`, or globally on the `rl` entrypoint to propagate to both. |
| `orchestrator.log.vf_level` | Env-worker / [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) log level (`info` default; `debug` is noisy but useful for env debugging). |
| `--monitors.wandb` (+ `--monitors.wandb.project`, `--monitors.wandb.name`) | Enable Weights & Biases logging. See [Weights & Biases](#weights--biases). |
| `--monitors.prime` | Stream metrics and episodes to the Prime Intellect platform (Prime Lab). See [Platform monitoring](#platform-monitoring). |

**Run management:**

| Knob | What it does |
| - | - |
| `--output-dir outputs` | Directory that groups related runs. Each run writes its artifacts to its own run directory `<output_dir>/<run_name>` (`<run_dir>` below). Defaults to `$PRL_OUTPUT_DIR` if set, else `outputs`. |
| `--run.name <name>` | Run name, also the run directory name under `<output_dir>` (override the directory separately via `--run.dir`). Auto-generated as `<envs>--<model>--<short-id>` when unset, so every launch gets a fresh, readable run directory. Set an explicit name for a predictable path — required to resume the run later. |
| `--clean` | Wipe the run directory before starting. Useful when re-running a named run during iteration. |
| `--max-steps N` | Stop after `N` trainer steps. Overrides the config value. |
| `--dry-run` | Resolve + validate the full config, write per-process configs to `<run_dir>/configs/latest/resolved/`, and exit without launching. The fastest way to debug a misbehaving config. |

### Algorithms

The RL entrypoint supports several training algorithms, switched via `[orchestrator.train.algo]`'s `type` (see [Algorithms](/prime-rl/algorithms#the-algorithm-abstraction) for the full reference, model references, and per-algorithm customization):

| `algo.type` | Frozen model | Use case |
| - | - | - |
| `grpo` (default) | None | Standard group-relative RL |
| `max_rl` | None | [MaxRL](https://arxiv.org/abs/2602.02710): GRPO with mean-normalized advantages (maximum-likelihood RL) |
| `rae` | None | [SPIRAL](https://arxiv.org/abs/2506.24119)'s role-conditioned advantage estimation: reward minus a per-agent EMA baseline, for multi-agent self-play envs (e.g. `kuhn-poker`) |
| `hierarchical_grpo` | None | GRPO for proposer-solver envs: compare solvers only with attempts on the same proposed problem, and compare proposers with the other proposals in the group |
| `opd` | Required, must be vLLM (needs `prompt_logprobs`) | [On-policy distillation](https://thinkingmachines.ai/blog/on-policy-distillation/): the policy generates rollouts, the trainer minimizes per-token reverse KL to a reference model |
| `sft` | Required, any OpenAI-compatible endpoint | Hard-distill: a frozen model generates rollouts, the policy trains on its tokens |
| `opsd` | None — the live policy is its own reference (no deployment) | [SDFT](https://arxiv.org/abs/2601.19897): the model is its own reference conditioned on expert demonstrations |
| `echo` | None | GRPO plus cross-entropy on env-observation tokens |

A new algorithm is a named class in code, not a config — see [Algorithms § Authoring an Algorithm](/prime-rl/algorithms#authoring-an-algorithm).

Frozen models are declared inline on the algorithm, named where the model is used — `[orchestrator.train.algo.teacher]` for `opd` (the frozen model scored against), `[orchestrator.train.algo.sampling.source]` for `sft` (the model it samples from) — each with `name` + `base_url`. `opsd` declares no frozen model: it self-distills against the live policy. The `rl` entrypoint only manages policy inference — start frozen-model servers yourself and point `base_url` at them:

```bash theme={null}
CUDA_VISIBLE_DEVICES=1 uv run inference \
  --vllm.model <frozen-model> --server.port 8001
```

The standalone `uv run sft` entrypoint is the more traditional SFT path — pure dataset-based, no orchestrator. Use the `sft` algorithm only when you want a frozen model to generate the supervision on the fly.

### Important Metrics

Pulled from the console logs and mirrored to W\&B.

**Progress** (orchestrator):

* `reward/{all,env}/mean` — main signal. Should trend upward over hundreds of steps.
* `seq_len/{all,env}/mean` and `is_truncated/{all,env}/mean` — rollout length and truncation rate.
* `is_timeout/{all,env}/mean` — share of rollouts stopped by a stage deadline (`setup`/`agent`/`finalize`/`scoring` `_timeout`). Training counts a timed-out rollout as an error, so it stays out of the batch; eval scores it.
* `num_turns/{all,env}/mean` — for multi-turn envs.
* `empty_rollouts/{all,env}`, `errored_rollouts/{all,env}` — non-zero is fine in small numbers; sustained > 5% is a smell.
* `eval/{env}/{avg@k,pass@k}` — eval scores when `[orchestrator.eval]` is set.

**Stability** (trainer):

* `mismatch_kl/{all,env}/{mean,std,max}` — KL between trainer's current policy and the (older) inference policy that generated the rollouts. A sustained, growing mean is the early-warning sign for off-policy collapse.
* `entropy/{all,env}/mean` — too low means mode-collapse; too high means the model isn't committing.
* `is_masked/mean` — fraction of tokens masked by the IPO trust region.
* `optim/grad_norm` — spikes precede divergence; check the loss config or lower the LR.

**Performance** (trainer + orchestrator step independently):

| Source | Metric | Reading |
| - | - | - |
| trainer | `time/wait_for_batch` | **high → orchestrator bottleneck** |
| orchestrator | `time/wait_for_policy` | **high → trainer bottleneck** |

The trainer warns when batch wait time exceeds active trainer time. Add inference nodes when this warning persists. The orchestrator warns when policy wait time exceeds active orchestrator time. Add trainer nodes when this warning persists. The orchestrator also warns when it discards more than half of an episode window. This warning reports stale, errored, and no-signal counts.

## SFT Trainer

`uv run sft` runs supervised fine-tuning from a HF dataset. It shares model loaders, FSDP setup, checkpointing, and the chat-template plumbing with the RL trainer, so a typical workflow is *SFT → RL → SFT → …* without any reformatting.

### Dataset Format

Two accepted layouts:

* **Prompt-completion**: a HF dataset with `prompt` and `completion` columns ([TRL format](https://huggingface.co/docs/trl/en/dataset_formats#prompt-completion)). The trainer masks out the prompt and computes loss only over the completion.
* **Messages**: a HF dataset with a single `messages` column containing a list of chat turns. The trainer interprets the whole conversation as one sample, applies role-based loss masking, and trains over all assistant turns.

If both columns are present, `messages` takes precedence. A dataset that stores these under other names maps them in `[data.columns]`; the trainer renames the columns at load time:

```toml theme={null}
[data.columns]
messages = "conversation"
tools = "schemas"
```

**Per-message loss selection.** An optional `message_loss_mask` column contains one boolean or integer 0/1 per message. For example, `[0, 0, 0, 1]` on a user/assistant/user/assistant conversation trains only the final assistant turn while retaining the full context. For prompt-completion rows, the mask covers the concatenated prompt and completion messages. Zero excludes a message; one preserves the normal role and renderer loss settings. A missing or null mask leaves those settings unchanged. The renderer applies selection without changing token IDs, including ownership of assistant closing tokens. Invalid mask lengths or entries raise an error.

To read a differently named column, set `data.columns.message_loss_mask = "selection"`.

**Tool definitions and renderer controls.** For tool-use SFT, add a `tools` column in OpenAI function-calling format. Each row's value can be either a list of dicts or a JSON-encoded string of a list.

Renderer-backed SFT reads template controls from the typed `[renderer]` config in the SFT TOML. For example:

```toml theme={null}
[renderer]
name = "qwen3"
enable_thinking = false
```

Dataset columns can set renderer template controls per row, on top of the `[renderer]` config. `[data.columns.renderer]` maps renderer fields to columns and defaults to `reasoning_effort = "reasoning_effort"`, so a `reasoning_effort` column sets that field on renderers that have it (for example `gpt-oss`, `qwen3.8`, `deepseek-v4`). Rows with a null value use the configured renderer unchanged. This lets one run mix reasoning efforts, with the `[renderer]` value as the default for rows that do not set one:

```toml theme={null}
# The renderer field is reasoning_effort; this dataset stores it in an effort column.
[data.columns.renderer]
reasoning_effort = "effort"
```

If a model needs another template control, add it to that model's renderer config in `renderers` (for example a new field on the relevant `*RendererConfig`) and consume it in the renderer implementation.

**Custom renderers.** A renderer can also live outside the `renderers` package. Set `name = "custom"` and point `import_path` at the class, as `my_module.Class` or `path/to/file.py:Class`; relative file paths resolve against the launch directory. The class sets `config_class` to its own `BaseRendererConfig` subclass, which validates the remaining `[renderer]` keys and declares its template controls:

```toml theme={null}
[renderer]
name = "custom"
import_path = "my_renderers.glm53_no_thinking.NoThinkingGLM53Renderer"
clear_thinking = true
```

The same config selects the renderer for RL under `[orchestrator.renderer]`. The [`renderers` README](https://github.com/PrimeIntellect-ai/renderers#custom-renderers) shows the class behind this example.

**Renderer-backed tokenization.** SFT tokenization is renderer-only. The [`renderers`](/prime-rl/algorithms#renderers) package owns message-to-token conversion and loss attribution end-to-end, so position-dependent chat templates (for example templates that strip past `<think>` blocks across user turns) do not corrupt the loss mask. `[renderer]` defaults to `name = "auto"`; set a typed renderer config only when you need model-specific template controls. Hand-coded renderers ship for Qwen3, Qwen3.5, GLM-5, GLM-4.5, Kimi K2/K2.5, MiniMax M2, DeepSeek V3, Nemotron 3, GPT-OSS, and VLM families such as Qwen3-VL/Qwen3.5.

**VLM training requires a custom PrimeRL implementation.** Training a model with `[model.vlm]` set (SFT or RL) only works for models with a registered PrimeRL VLM class (currently Qwen3.5 dense and MoE).

See [Algorithms § Multi-Turn Trajectories](/prime-rl/algorithms#multi-turn-trajectories) for the full picture.

### Launch

The minimal SFT run trains `Qwen3-0.6B` on the `reverse-text` SFT dataset:

```bash theme={null}
uv run sft @ examples/basic/reverse-text/sft.toml --monitors.wandb
```

Multi-GPU and multi-node use torchrun under the hood (the `sft` entrypoint manages this for you — see [Scaling § SFT and Torchrun](/prime-rl/scaling#sft-and-torchrun) for non-default layouts; multi-node SFT goes through [SLURM](/prime-rl/scaling#slurm)).

`--monitors.prime` streams the run to the Prime Intellect platform — see [Platform Monitoring](#platform-monitoring).

### Online Evals

`uv run sft` can evaluate the model on rollout-based envs as it trains, reusing the RL orchestrator's eval machinery. Configure an `[eval]` block — the same shape as `[orchestrator.eval]`: multiple `[[eval.source]]` envs with per-source `interval` / `select` / `group_size` / sampling overrides — plus an `[inference]` block for the vLLM server:

```toml theme={null}
[eval]
interval = 25

[eval.select]
limit = 32

[[eval.source]]
name = "reverse-text"
env.taskset.id = "reverse-text"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"

[inference]

[deployment]
num_train_gpus = 1  # trainer
num_infer_gpus = 1  # inference
```

The launcher starts the inference server, one env server per eval source, and an online-eval process next to the trainer (it logs to `logs/attempt_<n>/eval.log`). NCCL is the default weight transport. The trainer broadcasts weights at startup (fail-fast) and at every step an eval env is due, Every broadcast runs the same four-stage handshake in `broadcasts/step_{n}`: the trainer offers the version (`.sender_ready`) and blocks, the online-eval process acknowledges (`.receiver_ready`), then the trainer transfers (`.started`) and commits (`.finished`). It runs the due envs sequentially per broadcast, so every epoch measures exactly one policy version. Set `[weight_broadcast] type = "filesystem"` to reload weights from disk instead. LoRA and externally managed inference use filesystem broadcast automatically. The base model is evaluated before the first step (disable with `eval.skip_first_step`), and the final broadcast always fires every env. In-flight eval episodes are cancelled by default when the next checkpoint is ready, so stale evals do not delay a weight update. Set `eval.cancel_on_new_checkpoint = false` to drain every triggered epoch instead. The trainer can idle while it waits for slow evals. They are sized by the same adaptive concurrency controller as the orchestrator; bound it with `[eval.concurrency]` (`min_inflight` / `max_inflight`; set them equal for fixed concurrency).

#### Multi-Node Trainer and Inference Pool

On a `multi_node` deployment, one SLURM job reserves `deployment.num_train_nodes + deployment.num_infer_nodes` nodes. The first `num_infer_nodes` run the inference pool, router, env servers, and online-eval process. The remaining nodes run the trainer. The inference pool runs one vLLM engine per DP rank behind one router, with `gpus_per_node / inference.vllm.tensor_parallel_size` engines per node:

```toml theme={null}
[deployment]
type = "multi_node"
num_train_nodes = 2  # trainer nodes
num_infer_nodes = 1  # inference pool + online evals

[inference.vllm]
tensor_parallel_size = 8

[slurm]
job_name = "my-run"
```

The shared script passes the trainer rank-0 hostname directly to the online-eval process for NCCL weight broadcasts. Each transfer is synchronous, but eval rollout execution overlaps with later training steps. The allocation remains active while the final eval finishes. Without `max_steps`, online-eval never sees a final broadcast, so the job remains active until walltime. Trainer and online-eval log to one shared W\&B run. The trainer creates it, and online-eval finalizes it.

### SFT-Specific Knobs

| Knob | What it controls |
| - | - |
| `data.name` | HF dataset name or local path |
| `data.batch_size` | Tokens per trainer step (packed) |
| `data.seq_len` | Per-sample sequence length |
| `loss_mask.*` | Which roles contribute to loss (system / user / assistant / tool). |
| `val.interval` | Run validation every N steps; `val.data` mirrors `data` |
| `eval.interval` | Run online evals every N steps; see [Online Evals](#online-evals) |

### Important Metrics

Pulled from the console log and mirrored to W\&B.

**Progress and loss:**

* `loss/mean`, `loss/perplexity` — main signal. Should decrease through the run.
* `val/loss`, `val/perplexity` — validation metrics when `[val]` is set, logged every `val.interval` steps.
* `eval/{env}/...` — online eval metrics when `[eval]` is set, logged at each evaluated checkpoint step.
* `progress/epoch`, `progress/num_samples`, `progress/num_tokens` — dataset progress.
* `progress/<subset>/ratio_{samples,tokens}` — when training on multiple HF subsets/splits, the realized mixing ratio.

**Stability and optimization:**

* `optim/grad_norm` — spikes precede divergence.
* `optim/lr` — LR schedule.
* For MoE: `max_vio/mean`, `max_vio/max` — mean and max over the step's microsteps of the largest load-balancing violation across layers and EP groups, computed from expert token counts summed across each EP group. `routing_confidence/mean` — mean routing confidence.
* For MoE: `expert_load/{cv,max_mean,cold_frac}/{mean,max}` — expert load per layer over the whole step, from token counts summed across all ranks, as the mean and max over MoE layers. `cv` is std/mean of the per-expert token counts, `max_mean` the busiest expert's count over the mean, and `cold_frac` the fraction of experts receiving under 0.1x the mean. A trainable router that drifts shows up as a rising `cv`/`cold_frac` (MiMo-V2.6 §5.4).

**Performance:**

| Metric | Reading |
| - | - |
| `perf/throughput`, `perf/throughput_per_gpu` | tokens/s overall and per GPU |
| `perf/mfu` | MFU |
| `perf/peak_memory` | peak GPU memory (GiB) |
| `time/step`, `time/forward_backward`, `time/save_ckpt` | step breakdown |

## Checkpointing

Checkpointing is split across processes because the orchestrator and trainer can be on different machines and on different steps at any given time. Inference is stateless.

| Process | What's saved | Where |
| - | - | - |
| Trainer | FSDP-sharded model (DCP), optimizer, scheduler, progress | `<run_dir>/checkpoints/step_{n}/trainer/` |
| Orchestrator | Progress, per-env data state | `<run_dir>/checkpoints/step_{n}/orchestrator/` |
| Inference | *nothing* — re-pushed from the latest checkpoint on restart | n/a |

### Enabling Checkpoints

Checkpointing is **off by default** to save disk. Enable it with `--ckpt`:

```bash theme={null}
uv run rl @ rl.toml --ckpt                              # default: end-of-training only
uv run rl @ rl.toml --ckpt.interval 25                  # every 25 steps
uv run rl @ rl.toml --ckpt.interval 25 --ckpt.keep-last 3  # rolling window of 3
uv run rl @ rl.toml --ckpt.interval 25 --ckpt.keep-interval 100  # …plus permanent every 100
```

### Resuming a Run

Re-run the same launch command and pass `--resume` (latest checkpoint) or `--resume.step <N>`. Resuming reuses the run directory, so the run needs a name you can point back at — launch with `--run.name` (or pass the first run's auto-generated name). Make sure `--max-steps` is at least the target final step, not the remaining delta:

```bash theme={null}
# First run: steps 1–10
uv run rl @ rl.toml --max-steps 10 --ckpt --run.name my-run

# Resume from the latest checkpoint: continue to step 20
uv run rl @ rl.toml --max-steps 20 --ckpt --resume --run.name my-run

# ...or from a specific step
uv run rl @ rl.toml --max-steps 20 --ckpt --resume.step 10 --run.name my-run

# ...or fork another run's checkpoint into a fresh run
uv run rl @ rl.toml --max-steps 20 --ckpt --run.name my-fork \
  --resume.dir outputs/my-run/checkpoints/step_10
```

### Exporting Checkpoints

Trainer checkpoints are DCP-sharded; export them to HF-format safetensors with `tools/convert_dcp_to_bf16.py`. The script reads the model config from the run's resolved config and writes sharded safetensors plus config/tokenizer assets to `<ckpt_dir>/weights` (or a second positional arg). It exports full fine-tunes only — LoRA checkpoints are rejected.

```bash theme={null}
# single process (1 GPU)
uv run python tools/convert_dcp_to_bf16.py outputs/my-run/checkpoints/step_10

# multi-rank for faster gathers and models that don't fit one GPU
uv run torchrun --nproc-per-node 8 tools/convert_dcp_to_bf16.py outputs/my-run/checkpoints/step_10
```

The exported directory loads directly into `uv run inference --vllm.model <dir>` or any HF consumer. Quantize it to blockwise FP8 (DeepSeek/GLM format, loads natively in vLLM) with `tools/convert_bf16_to_fp8.py <dir>`, or go straight from the checkpoint with `tools/convert_dcp_to_fp8.py <ckpt_dir>` (each rank quantizes its gathered slice, writes only `<ckpt_dir>/weights-FP8` — no intermediate bf16 export); dequantize an fp8-only release (e.g. GLM-5-FP8) for training with `tools/convert_fp8_to_bf16.py <dir>`.

## Observability

### Config Files

Each launch writes its command, input TOML, and resolved JSON files to
`<run_dir>/configs/attempt_<n>/`. Resumed runs keep the earlier configs.
`command.txt` uses shell-safe quoting.
`configs/latest` points to the current attempt.

### Log Files

The launcher tees every process's stdout/stderr into `<run_dir>/logs/attempt_<n>/` — every launch (fresh or resumed) gets its own numbered attempt directory, and `logs/latest` symlinks to the current one. The full layout (single-node runs skip the `node_*.log` and `router.log` files — there the router logs into `inference.log`):

```
<run_dir>/logs/latest/     # symlink -> attempt_<n>, one per launch
├── trainer.log                  # rank 0 only; symlink → trainer/node_0.log on multi-node
├── orchestrator.log             # single instance, single file
├── eval.log                     # `uv run eval` process, or the SFT online-eval process
├── inference.log                # symlink → inference/node_0.log on multi-node
├── trainer/
│   ├── node_*.log               # per-node trainer stdout (multi-node only)
│   └── torchrun/<rdzv>/attempt_0/<rank>/{stdout,stderr}.log   # per-rank
├── inference/
│   ├── node_*.log               # per-node inference stdout (multi-node only)
│   └── router.log               # the single global router (multi-node only)
└── envs/{train,eval}/<env_name>.log # one env server process per source (broker + its workers)
```

Env logs are the first place to look for env-side errors (most user code lives there). Verbosity is controlled by `orchestrator.log.vf_level`. For multi-rank trainer debugging, drop into `logs/latest/trainer/torchrun/<rdzv>/attempt_0/<rank>/{stdout,stderr}.log` — verbose and per-rank.

Live tailing from a single point (works on the head node for multi-node runs over a shared filesystem):

```bash theme={null}
tail -F <run_dir>/logs/latest/{trainer,orchestrator,eval,inference}.log
tail -F <run_dir>/logs/latest/trainer/node_*.log   # multi-node only
tail -F <run_dir>/logs/latest/inference/router.log # multi-node only
```

### Dashboard

`uv run dashboard [output_dir ...]` (default `outputs/`) serves a local web dashboard at `http://localhost:7788` with six views per run: an overview (the W\&B overview sections, read from the file monitor's `metrics.jsonl`), metrics (every key in that file, one pane each, with a regex filter), the resolved configs, a rollout trace viewer with per-token overlays (advantage, entropy, trainer/sampling mismatch, IPO stable mask, loss and content masks), merged component logs, and markdown reports from `<run>/reports/`. The trace viewer separates the transcript, the wall-clock timeline of physical prefix branches, a wall-clock terminal replay of model and tool activity, and a top-to-bottom semantic graph of labeled model-call relationships. Replay uses recorded model-call and message timestamps; because providers do not persist per-token timestamps, it reveals response text evenly across the measured call span and labels that cadence as inferred. Agent and context labels in the semantic graph show the latest and peak prompt lengths first, with cumulative token processing available below on hover and click. It only reads the run dirs, so it is safe to point at a live run; pass several output directories to track parallel experiments. A taken port automatically bumps to the next free one, so several dashboards coexist on one node.

A coding agent on the same machine can drive the open dashboard: `POST /api/view` with an on-disk address (`{"run", "tab", "step", "kind", "subset", "episode", "highlight": [...]}`) navigates every connected tab there and paints quote-anchored highlights in the trace viewer. Reports cite traces with `[^id]` markers whose JSON definitions carry the same address plus a verbatim quote; the dashboard re-checks each quote against the trace files and marks the citation verified or broken, so answers stay grounded in what is actually on disk. The `dashboard` skill documents the full contract.

### Weights & Biases

W\&B is off by default (the file monitor, which writes the metrics and the trace stream under the run's `monitors/file/` directory, is on by default):

```bash theme={null}
uv run rl @ rl.toml --monitors.wandb                      # default project, random name
uv run rl @ rl.toml --monitors.wandb.project my-proj --monitors.wandb.name run-42
uv run rl @ rl.toml --no-monitors.file                    # disable the local metric/trace files
```

The trainer and orchestrator log into a **single shared W\&B run**, so all metrics from both processes land in one place. Shared mode requires the W\&B SDK ≥ 0.19.9 and is incompatible with `monitors.wandb.offline = true`.

prime-rl deliberately logs a **large number of metrics** for maximum observability: every rollout metric is emitted per subset (`all`/`effective`), per statistic (`mean`/`max`/`min`/`p10`/`p90`), and per environment alongside a cross-env aggregate, so a multi-env run can emit thousands of series. To keep that navigable, every training run (RL and SFT) gets an **auto-created `overview` saved view** curating the handful of metrics that matter into `train`, `eval`, `stability`, and `performance` sections (with per-env breakdowns). The view is created once per project and adapts to the run's environments; if a later run uses a different set of environments, a new versioned view (`overview-v2`, …) is created instead of overwriting the first.

### Platform Monitoring

Register a run on the Prime Intellect platform (Prime Lab) and stream training metrics and episodes to the platform dashboard. Bare flag uses defaults:

```bash theme={null}
uv run rl @ rl.toml --monitors.prime
uv run sft @ sft.toml --monitors.prime   # same flag on the SFT trainer
```

Or set it in TOML:

```toml theme={null}
[monitors.prime]
name = "my-experiment"
```

The monitor is a thin layer over the [`prime-runs`](https://github.com/PrimeIntellect-ai/prime/tree/main/packages/prime-runs) SDK (installed as `prime-runs[train]`): it registers the run, streams per-step metrics, uploads every 10th step's episodes (full conversations with rewards and advantages) to the run's sample viewer, and closes the run out. Finalization is best-effort SDK delivery: a process that exits without finishing is reported crashed when the SDK's atexit hook runs, but an abrupt kill (SIGKILL, OOM, node loss) skips the report and a lost finalize is not retried — the platform owns the terminal state of attached (`RUN_ID`) and managed runs. Uploaded episodes are keyed to the platform run by the SDK; the orchestrator's own run id (the launcher's `PRL_RUN_ID`) stays on W\&B and in the local records.

On `sft` the monitor registers the run with the dataset-batched training fields (`max_steps`, `batch_size`, `seq_len`) and streams the trainer's per-step metrics and validation losses; SFT has no rollouts, so no episodes are uploaded.

Requires `PRIME_API_KEY` (`prime login` or the env var) and a team (`PRIME_TEAM_ID`, or the team selected with `prime login`) enabled for external runs. A configured monitor must work: a missing key or a team outside the allowlist fails the launch. `PRIME_RUNS_MODE=disabled` keeps the monitor configured but opens no platform run; `RUN_ID=<id>` attaches to an external run a launcher already created instead of registering a new one. Currently internal-only.

## Rules of Thumb

* **Start small.** Run `examples/basic/reverse-text/rl.toml` end-to-end on 2 GPUs before scaling. If the smoke run finishes cleanly, your install is good.
* **Batch size ≥ 64.** Smaller batches give noisy gradient estimates and the trainer's overhead-per-step dominates throughput. 64 is the practical floor; 128–512 is the range for quick ablations; production RL often runs at 1024+.
* **Group size ≥ 8.** Bigger groups (`orchestrator.train.group_size`) make it more likely that a task produces a mix of high- and low-reward rollouts, which is what gives the trainer a usable signal — if all rollouts in a group succeed or all fail, the within-group advantage collapses to zero and the trainer learns nothing from that task. Bigger groups also tighten advantage normalization. 8 is the floor; 16–32 is common.
* **Runs never share a directory.** Every launch writes to its own run directory `<output_dir>/<run_name>`, auto-named `<envs>--<model>--<short-id>` by default. Name runs you want to find again or resume with `--run.name <name>`; re-using a name blocks unless you resume or pass `--clean`.
* **Use `--dry-run` before SLURM.** Validators (e.g. CP needs flash-attention) fail fast in dry-run and slow in queue.


# Checkpoints & Forking
Source: https://docs.primeintellect.ai/sandboxes/checkpoints

Save a sandbox's filesystem, restore it, or branch it

A checkpoint saves a sandbox's filesystem so you can start new sandboxes from it later, or fork a running one to branch your work. It captures the sandbox's root filesystem only — data on mounted [volumes](./volumes) is never included. Checkpoints capture **files only**: processes start fresh and nothing in memory carries over.

A checkpoint is usable once it reaches `DURABLE`. From then on it outlives its source sandbox. Checkpoint storage is [billed separately](./overview#pricing).

> [!WARNING]
>
> Keep the source sandbox alive until the checkpoint is `DURABLE`. If it's
> deleted or hits its timeout first, the checkpoint is lost.

## Fork

The quickest way to branch a running sandbox. It checkpoints, waits for `DURABLE`, and starts a new sandbox from it:

```bash theme={null}
prime sandbox fork sbx_123 --name experiment-branch --yes
```

The fork inherits the source's CPU, memory, and timeout unless you pass `--cpu-cores`, `--memory-gb`, or `--timeout-minutes`. Use `--wait-timeout` (seconds) to cap how long it waits for the checkpoint.

## Checkpoint and restore

```bash CLI theme={null}
# 1. Create a checkpoint (prints its ID)
prime sandbox checkpoint create sbx_123

# 2. Poll until its state is DURABLE
prime sandbox checkpoint list sbx_123 --checkpoint-id <checkpoint-id>

# 3. Restore into a new sandbox
prime sandbox checkpoint restore <checkpoint-id> --name restored-workspace --yes
```

```python Python theme={null}
checkpoint = client.checkpoint(sandbox_id)
client.wait_for_checkpoint(checkpoint.id, timeout_seconds=300)

restored = client.create(
    CreateSandboxRequest(name="restored-workspace", checkpoint_id=checkpoint.id)
)
client.wait_for_creation(restored.id)
```

The image and disk size come from the checkpoint, so leave out `docker_image` and `disk_size_gb` in the SDK. Everything else (resources, env vars, timeouts) works as usual; the CLI `restore` accepts `--cpu-cores`, `--memory-gb`, `--timeout-minutes`, and `--team-id`.

`wait_for_checkpoint` returns once the checkpoint is `DURABLE`. It raises `RuntimeError` if the checkpoint fails and `TimeoutError` if the wait runs out. A timeout doesn't mean the checkpoint failed, so check it with `get_checkpoint` before deleting the source sandbox. `list_checkpoints(sandbox_id)` lists a sandbox's checkpoints.


# Run Commands & SSH
Source: https://docs.primeintellect.ai/sandboxes/commands

One-off commands, background jobs, and interactive shells

## Run a command

```bash CLI theme={null}
prime sandbox run sbx_123 --working-dir /workspace "python -c 'print(42)'"
```

```python Python theme={null}
result = client.execute_command(sandbox.id, "python -c 'print(42)'")
print(result.stdout, result.stderr, result.exit_code)
```

Responses include stdout, stderr, and exit code. Options:

| Option | CLI | Python |
| - | - | - |
| Timeout (seconds) | `--timeout` | `timeout` (default 300) |
| Working directory | `-w` / `--working-dir` | — |
| Extra env vars | `-e` / `--env KEY=VALUE` | `env={"KEY": "VALUE"}` |

A command that exceeds its timeout raises `CommandTimeoutError` in the SDK. A running command is not cut off when its 30-minute gateway token expires; the token is checked only when the command starts. For work that should outlive the request, use a background job.

## Background jobs

Background jobs keep running in the sandbox, independent of the command connection, while you poll for completion. They persist until they finish or the sandbox terminates.

```python theme={null}
import time

sandbox = client.create(
    CreateSandboxRequest(
        name="training-job",
        docker_image="python:3.11-slim",
        timeout_minutes=1440,  # 24 hours
        cpu_cores=4,
        memory_gb=16,
    )
)
client.wait_for_creation(sandbox.id)

job = client.start_background_job(sandbox.id, "python train.py --epochs 100")

while True:
    status = client.get_background_job(sandbox.id, job)
    if status.completed:
        print(status.exit_code, status.stdout)
        break
    time.sleep(30)

client.download_file(sandbox.id, "/app/model.pt", "./model.pt")
client.delete(sandbox.id)
```

Make sure the sandbox [timeout](./lifecycle#timeouts) is long enough for the job, or set it negative for no deadline.

## SSH

Open an interactive shell in a running sandbox:

```bash theme={null}
prime sandbox ssh <sandbox-id>

# Use a different shell installed in the image
prime sandbox ssh <sandbox-id> --shell bash

# Anything after -- goes straight to ssh, e.g. one remote command
prime sandbox ssh <sandbox-id> -- tail -f /workspace/train.log
```

The CLI generates an ephemeral key pair, creates a session, and connects automatically. The session is revoked and the key deleted on disconnect. OpenSSH (`ssh`) must be installed locally, and the sandbox must be `RUNNING`. By default you get a `/bin/sh` login shell as the sandbox's default user in its working directory.

Only interactive shells and remote commands are supported: port forwarding (`-L`/`-R`), agent forwarding, `scp`, and `sftp` are not. Use [file transfers](./files) to move files.

> [!WARNING]
>
> SSH sessions do **not** count as activity for the [idle timeout](./lifecycle#timeouts), so a sandbox with an open SSH connection can still be shut down.


# Files
Source: https://docs.primeintellect.ai/sandboxes/files

Move files in and out of a sandbox

```bash CLI theme={null}
# Push local files into the sandbox
prime sandbox upload sbx_123 notebooks/analysis.ipynb /workspace/

# Pull results back
prime sandbox download sbx_123 /workspace/report.csv reports/latest.csv
```

```python Python theme={null}
client.upload_file(sandbox.id, "/workspace/model.bin", "./artifacts/model.bin")
client.download_file(sandbox.id, "/workspace/report.csv", "./reports/report.csv")

# Read small files directly
contents = client.read_file(sandbox.id, "/workspace/report.csv")
print(contents.content[:80])
```

* Uploads have no fixed size limit; they are bounded by free disk space in the sandbox. They overwrite existing files and create missing parent directories. The SDK reads the whole local file into memory and times out after 300 seconds by default, so pass a larger `timeout` for big files on slow connections.
* `read_file` is capped at **10 MiB** and raises `SandboxFileTooLargeError` past it; use `download_file` for large files.
* Transfers and commands use gateway tokens that expire 30 minutes after they are issued. The CLI and SDK cache them and fetch a new one before expiry; if you call the gateway directly, request a new token with `POST /api/v1/sandbox/{sandbox_id}/auth` before `expires_at`.
* A token is checked only when a request starts. Transfers and command streams already in progress keep running past the token's expiry, and fetching a new token does not interrupt them; only new requests need a valid token.
* If you rotate credentials or hit 401s, run `prime sandbox reset-cache` (CLI) or `client.clear_auth_cache()` (SDK).


# Prime Images
Source: https://docs.primeintellect.ai/sandboxes/images

Build and manage custom Docker images for sandboxes

## What are Prime Images?

Prime Images lets you push custom Docker images to Prime's registry and use them in sandboxes. Builds happen in the cloud, so you don't need Docker running locally.

## Image references

When you create a sandbox, the image reference resolves in this order:

1. **Prime Images** — references that start with `prime/`, like `prime/<username>/<image>:<tag>`, `prime/<team-slug>/<image>:<tag>`, or the id forms (`prime/<userId>/<image>:<tag>`, `prime/team-<teamId>/<image>:<tag>`), resolve to a Prime Image you can access: your own, your team's, or one that's been published.
2. **Docker Hub** — any other reference with no registry host and no `prime/` prefix resolves as a Docker Hub image, so official images like `ubuntu:22.04` or `pytorch/pytorch:latest` work as usual.
3. **Other public registries** — references hosted on registries like `ghcr.io` or `quay.io` are not accepted directly. Transfer the image into Prime first with [`prime images push --source-image`](#push-an-image), then reference the resulting `prime/` image.

## Push an Image

```bash theme={null}
# Basic push (uses ./Dockerfile in current directory)
prime images push myapp:v1.0.0

# Make the image public when the build completes
prime images push myapp:v1.0.0 --public

# Specify a different Dockerfile
prime images push myapp:v1.0.0 --dockerfile custom.Dockerfile

# Use a different build context
prime images push myapp:v1.0.0 --context ./app

# Copy existing public images into Prime's registry
prime images push --source-image ubuntu:22.04,ghcr.io/org/app:v1

# Copy and rename the destination tag
prime images push myubuntu:22.04 --source-image quay.io/org/ubuntu:22.04
```

The CLI packages your build context, uploads it, and kicks off a remote build. You'll get a build ID to track progress.

Use `--source-image` to transfer public images from Docker Hub, GHCR, Quay, public ECR, `registry.k8s.io`, or MCR without building from a Dockerfile. Pass several at once as a comma-separated list. Google-hosted registries are rejected.

When the push completes, the CLI prints the full image reference to use when creating a sandbox. Prime Image references always start with `prime/`. By default this uses your unique slug — your username for personal images, or the team slug when you're in a team context:

```text theme={null}
prime/<your-username>/myapp:v1.0.0      # personal
prime/<team-slug>/myapp:v1.0.0          # team
```

If you haven't set a username (or your team has no slug yet), the reference falls back to the id form — `prime/<userId>/myapp:v1.0.0` or `prime/team-<teamId>/myapp:v1.0.0`. Run `prime images list` to see the exact reference for each image.

## Check Build Status

```bash theme={null}
# See all your images and their build status
prime images list
```

Status meanings:

* **PENDING** – Queued for build
* **UPLOADING** – Build context is being uploaded
* **BUILDING** – Build in progress
* **COMPLETED** – Build succeeded, image is usable
* **FAILED** – Build failed (check your Dockerfile)
* **CANCELLED** – Build was cancelled

## Use Your Image

Once the status shows **COMPLETED**, create a sandbox with it:

```bash theme={null}
# Personal image
prime sandbox create prime/<your-username>/myapp:v1.0.0 --cpu-cores 2 --memory-gb 4

# Team image
prime sandbox create prime/<team-slug>/myapp:v1.0.0 --cpu-cores 2 --memory-gb 4
```

## Push Many Images at Once

`prime images push-bulk` builds many images in one command. Builds come from a JSONL manifest (`--manifest`), a Harbor tasks directory (`--harbor`), or a Hugging Face dataset (`--hf`):

```bash theme={null}
# JSONL manifest: Dockerfile builds and public-registry source builds, one per line
prime images push-bulk --manifest builds.jsonl

# Harbor task directory (each task's environment/ folder is the build context)
prime images push-bulk --harbor ./tasks --tag latest

# Hugging Face dataset: one Dockerfile per row, names from a column
prime images push-bulk --hf org/dataset --dockerfile-column dockerfile --name-column instance_id

# Hugging Face dataset: one registry image reference per row
prime images push-bulk --hf org/dataset --column docker_image
```

Manifest lines are either Dockerfile builds:

```json theme={null}
{"image": "myapp:v1.0.0", "context": "./app", "dockerfile": "Dockerfile.prod"}
```

or public-registry source builds:

```json theme={null}
{"source": "ghcr.io/org/app:v1", "image": "myapp:v1.0.0"}
```

Useful flags:

* `--public` / `--private` – set visibility for the built images
* `--concurrency` – how many builds run in flight at once
* `--build-timeout` – seconds to wait for a single build
* `--dry-run` – resolve and print the build list without pushing
* `--failures-out` – where to write a re-runnable manifest of failed builds (default `push-bulk-failures.jsonl`)

Failed builds are written to the `--failures-out` manifest so you can re-run just the failures.

## Update an Image

`prime images update` renames an image, changes its tag or visibility, or moves it between scopes. The image contents do not move — the old reference stops resolving and the new reference points at the same stored artifacts.

```bash theme={null}
# Rename
prime images update myapp:v1 --name myapp-final

# Change the tag
prime images update myapp:v1 --tag v2

# Change visibility
prime images update myapp:v1 --public

# Move a personal image into a team
prime images update myapp:v1 --to-team team_a
```

Add `--dry-run` to preview the change without applying it, and `--yes` to skip the confirmation prompt.

## VM Images

Sandboxes are microVMs, so they boot a VM image rather than a container image. You still define images the normal way — build from a Dockerfile with `prime images push`, and the VM image builds automatically.

The first time a given image is launched, Prime converts it into a VM image. The sandbox sits in `PENDING` while that runs, which can take a few minutes; later sandboxes from the same image start normally.

A `--source-image` push also builds the VM image during the push, so the first sandbox launch from it starts right away.

## Publish or Unpublish an Image

Images are private by default. Publishing an image lets other authenticated Prime users use it in sandboxes with the `prime/` reference shown by `prime images list`.

```bash theme={null}
prime images publish myapp:v1.0.0
prime images unpublish myapp:v1.0.0
```

## Delete an Image

```bash theme={null}
prime images delete myapp:v1.0.0

# Skip confirmation
prime images delete myapp:v1.0.0 --yes
```


# Create & Manage Sandboxes
Source: https://docs.primeintellect.ai/sandboxes/lifecycle

Resources, start commands, timeouts, labels, and cleanup

## Create a sandbox

```bash CLI theme={null}
prime sandbox create python:3.11-slim \
  --name analytics-lab \
  --cpu-cores 2 \
  --memory-gb 4 \
  --disk-size-gb 20 \
  --timeout-minutes 240
```

```python Python theme={null}
sandbox = client.create(
    CreateSandboxRequest(
        name="analytics-lab",
        docker_image="python:3.11-slim",
        cpu_cores=2,
        memory_gb=4,
        disk_size_gb=20,
        timeout_minutes=240,
    )
)
client.wait_for_creation(sandbox.id)
```

* Omit `--name` to auto-generate a slug.
* Pass `--team-id` to charge a different workspace.
* Add `--yes` to skip the confirmation prompt in automation.
* In the SDK, pass `idempotency_key` to make create retries safe: requests that share a key are deduplicated, so a network failure mid-create can be re-run without producing a second sandbox.

The image can be a Docker Hub image or a [Prime Image](./images#image-references). See [Limits](./overview#limits) for resource ranges and defaults.

## Start command

Run your own process when the sandbox boots. There is no shell, so the executable and each argument are separate values:

```bash CLI theme={null}
prime sandbox create python:3.11-slim -- python serve.py --port 8000
```

```python Python theme={null}
from prime_sandboxes import StartCommand

sandbox = client.create(
    CreateSandboxRequest(
        name="api-server",
        docker_image="python:3.11-slim",
        start_command=StartCommand(executable="python", args=["serve.py", "--port", "8000"]),
    )
)
```

To get shell features (pipes, variables, `&&`), invoke a shell explicitly: `-- /bin/sh -c "python serve.py --port 8000"` on the CLI, or `StartCommand(executable="/bin/sh", args=["-c", "python serve.py --port 8000"])` in Python.

The start command is one-shot: when the process exits, the sandbox stops. Omit it to keep the sandbox up and ready for [commands](./commands).

## Timeouts

The timeout caps total lifetime (default 60 minutes). Set it to any negative value to remove the lifetime deadline.

The idle timeout terminates the sandbox sooner when nothing is using it, which is useful for agent workflows that might forget to clean up:

```bash CLI theme={null}
prime sandbox create python:3.11-slim \
  --timeout-minutes -1 \
  --idle-timeout-minutes 15
```

```python Python theme={null}
sandbox = client.create(
    CreateSandboxRequest(
        name="idle-aware",
        docker_image="python:3.11-slim",
        timeout_minutes=-1,
        idle_timeout_minutes=15,
    )
)
```

Any command, upload, download, or file read resets the idle clock. Long-running commands stay pinned for their full duration.

* Disabled by default.
* Must be between 1 and 1,440 minutes. For a finite lifetime, it must not exceed the timeout.
* SSH sessions do **not** count as activity.

A sandbox that shuts down this way reports `termination_reason="idle_timeout"` in the SDK, and **Idle Timeout** in `prime sandbox get` and on the dashboard.

## Labels

Tag sandboxes by project, team, or environment, then filter or bulk-delete by label:

```bash CLI theme={null}
prime sandbox create python:3.11-slim --label experiment --label ml-pipeline

# Must have ALL labels
prime sandbox list --label experiment --label ml-pipeline
```

```python Python theme={null}
sandbox = client.create(
    CreateSandboxRequest(
        name="tagged",
        docker_image="python:3.11-slim",
        labels=["experiment", "ml-pipeline"],
    )
)
```

## Inspect

```bash theme={null}
# Overview at a glance
prime sandbox list --status RUNNING --output table

# Rich details for one sandbox
prime sandbox get sbx_123 --output json
```

## Delete

```bash CLI theme={null}
# By ID (space or comma-separated)
prime sandbox delete sbx_123 sbx_456 sbx_789

# Everything matching ALL given labels
prime sandbox delete --label experiment --label staging --yes

# Every active sandbox you own (careful!)
prime sandbox delete --all --yes

# Team admins: the whole team, or one teammate
prime sandbox delete --all --all-users --yes
prime sandbox delete --all --user <user-id> --yes
```

```python Python theme={null}
client.delete("sbx_123")

# Bulk delete by IDs or by labels (not both)
result = client.bulk_delete(sandbox_ids=["sbx_456", "sbx_789"])
result = client.bulk_delete(labels=["experiment", "staging"])
print(f"Deleted: {result.succeeded}, Failed: {result.failed}")
```

On the CLI, use exactly one of: sandbox IDs, `--label`, or `--all`. `--all` and `--label` deletes run as a single server-side scoped delete: by default they cover your own sandboxes, `--all-users` covers the whole team, and `--user <user-id>` covers one teammate (both admin-only). The CLI prints a count preview before deleting, and per-sandbox results for explicit IDs.

## Troubleshooting

* Sandbox stuck in `PENDING` the first time you use an image? Prime is converting it into a VM image, which can take a few minutes. See [VM images](./images#vm-images).
* Stuck in `PROVISIONING`? Wait a minute, then rerun `prime sandbox list --status RUNNING`. If it stays pending, delete and recreate from a known-good image.
* Auth errors on transfers? `prime sandbox reset-cache` (CLI) or `client.clear_auth_cache()` (SDK) refreshes the gateway token after you rotate API keys.


# Network Isolation
Source: https://docs.primeintellect.ai/sandboxes/network-isolation

Restrict where a sandbox can send traffic

Restrict outbound traffic with an allowlist or a denylist. The two are mutually exclusive.

```bash CLI theme={null}
# Deny all egress, including DNS
prime sandbox create python:3.11-slim --network-deny 0.0.0.0/0

# Allow only specific destinations
prime sandbox create python:3.11-slim --network-allow api.openai.com

# Deny specific destinations, allow everything else
prime sandbox create python:3.11-slim --network-deny evil.example.com
```

```python Python theme={null}
# Deny all egress, including DNS
CreateSandboxRequest(name="isolated", docker_image="python:3.11-slim", network_allowlist=[])

# Allow only specific destinations
CreateSandboxRequest(name="allow", docker_image="python:3.11-slim", network_allowlist=["api.openai.com"])

# Deny specific destinations, allow everything else
CreateSandboxRequest(name="deny", docker_image="python:3.11-slim", network_denylist=["evil.example.com"])
```

Entries are domains (`example.com`, `*.example.com`) or IPv4 addresses/CIDRs; IPv6 is not supported. The CLI flags are repeatable. Blocked outbound calls fail with a network error.

## Change rules on a running sandbox

```bash theme={null}
# Replace the allowlist with two hosts
prime sandbox network sbx_123 --allow api.openai.com,*.example.com

# Replace the denylist; '*' blocks all outbound traffic
prime sandbox network sbx_123 --deny '*'

# Show current state
prime sandbox network sbx_123
```

Each call **replaces** the whole list, and each flag takes a single comma-separated value. Rules can be saved but not yet active on the running sandbox; the CLI keeps retrying until they apply.


# Private Networks
Source: https://docs.primeintellect.ai/sandboxes/networks

Connect sandboxes to each other over a private, rule-controlled network

## What are Private Networks?

A private network lets your sandboxes talk to each other directly. Each member is reachable from the others at `<sandbox-id>.sandbox.internal`, and the network's rules decide who may connect to whom, and on which ports. Nothing outside the network can reach its members this way, unless a rule [exposes a port to the internet](#exposing-ports-to-the-internet).

**Common use cases:**

* Run a client and a server in separate sandboxes, such as an agent and the service it is being evaluated against
* Split a workload into tiers, for example `web` sandboxes that may reach a `db` sandbox on port 5432 and nothing else
* Spin up a small cluster for distributed tests without exposing anything to the internet
* Publish a web app, API or dashboard running in a sandbox at a public HTTPS URL

> [!NOTE]
>
> Private networks connect running VM sandboxes that belong to the same team,
> or to you personally if you are not using a team. A network and all of its
> members must belong to the same owner.

## Complete Example

Here's a full working example: one sandbox serves HTTP on port 8000, and a second sandbox reaches it by name.

**1. Create two sandboxes**

```bash theme={null}
prime sandbox create python:3.11-slim --name web
prime sandbox create python:3.11-slim --name db
```

**2. Create a network and add both sandboxes**

```python Python SDK theme={null}
from prime_sandboxes import APIClient, SandboxClient

client = SandboxClient(APIClient())

network = client.create_lan_network("demo")
client.add_lan_member(network.id, "<web-sandbox-id>", tags=["web"])
client.add_lan_member(network.id, "<db-sandbox-id>", tags=["db"])
```

```bash CLI theme={null}
prime sandbox network create demo
prime sandbox network add <network-id> <web-sandbox-id> --tag web
prime sandbox network add <network-id> <db-sandbox-id> --tag db
```

**3. Start a server in one sandbox and call it from the other**

```bash theme={null}
# Serve HTTP on port 8000 in the db sandbox
prime sandbox run <db-sandbox-id> -- bash -c 'nohup python -m http.server 8000 >/dev/null 2>&1 &'

# Reach it by name from the web sandbox
prime sandbox run <web-sandbox-id> -- python -c "import urllib.request; print(urllib.request.urlopen('http://<db-sandbox-id>.sandbox.internal:8000/').status)"
```

A new network allows every member to reach every other member on every port, so the request returns `200`.

## Rules

Rules narrow who may connect to whom. Each rule is **directed** and **allow-only**: it lets members matching `from` open TCP connections to members matching `to`, on the listed ports. A connection that no rule allows is refused.

A selector is either `*` (every member) or `tag:<name>` (members carrying that tag). Leave out the ports to allow every port.

```python Python SDK theme={null}
from prime_sandboxes import LanRule

# web may reach db on 5432 and 6379; nothing else is allowed
client.set_lan_rules(
    network.id,
    [LanRule(from_="tag:web", to="tag:db", ports=[5432, 6379])],
)
```

```bash CLI theme={null}
# Rules are written FROM->TO[@PORT,...]
prime sandbox network rules <network-id> -r 'tag:web->tag:db@5432,6379'
```

Each call **replaces** the whole rule list. Pass several rules to allow several paths:

```bash theme={null}
prime sandbox network rules <network-id> \
  -r 'tag:web->tag:db@5432' \
  -r 'tag:worker->tag:db@5432' \
  -r 'tag:web->tag:worker@8080'
```

A few things to keep in mind:

* **Rules are directed.** `tag:web->tag:db` lets `web` connect to `db`, not the other way around. Replies on an allowed connection flow normally.
* **The default is open.** A new network starts with the single rule `*->*`. Replace it to restrict traffic.
* **An empty rule list blocks everything.** Members stay in the network but cannot connect to each other.

## Tags

Tags are labels on a member that rules select with `tag:<name>`. Set them when you add a sandbox. Adding a sandbox that is already a member replaces its tags.

```bash theme={null}
# Add with tags, or replace the tags of an existing member
prime sandbox network add <network-id> <sandbox-id> --tag web --tag frontend
```

Tags use lowercase letters, digits, `.`, `_` and `-`, start with a letter or digit, and are at most 63 characters long.

## Reaching Other Members

Inside a member sandbox, every other member resolves at `<sandbox-id>.sandbox.internal`. Use that name as the host for any TCP client:

```bash theme={null}
curl http://<sandbox-id>.sandbox.internal:8000/
psql -h <sandbox-id>.sandbox.internal -p 5432 -U postgres
```

* Names resolve to private addresses in `198.18.0.0/15`. These addresses are per sandbox and may change, so always connect by name.
* Only TCP is supported between members.
* A sandbox can belong to several networks at once. It can connect to a member of any of them, as each network's rules allow.

## Exposing Ports to the Internet

A rule with `from` set to `internet` publishes ports of the members matching `to` at a public HTTPS URL:

```text theme={null}
https://<port>-<sandbox-id>.sandbox-v2-prod.pinfra.io
```

Requests reach your server with their path, query and headers unchanged, so any HTTP server works, such as a web app, an API or a dashboard.

```python Python SDK theme={null}
from prime_sandboxes import LanRule

# Publish port 8080 of every member tagged web
client.set_lan_rules(
    network.id,
    [LanRule(from_="internet", to="tag:web", ports=[8080])],
)
```

```bash CLI theme={null}
prime sandbox network rules <network-id> -r 'internet->tag:web@8080'
```

Then start a server on that port and open its URL:

```bash theme={null}
prime sandbox run <web-sandbox-id> -- bash -c 'nohup python -m http.server 8080 >/dev/null 2>&1 &'
curl https://8080-<web-sandbox-id>.sandbox-v2-prod.pinfra.io/
```

A few things to keep in mind:

* **Anyone with the URL can connect.** Prime does not authenticate these requests. Add your own authentication if the service should not be public.
* **List the ports.** An `internet` rule must name its ports, and it exposes only those. Ports 49983 and 2222 are reserved for the sandbox API and SSH, and cannot be exposed.
* **Listen on `0.0.0.0`.** A server bound only to `127.0.0.1` inside the sandbox cannot be reached from its public URL.
* **HTTP only.** The URL serves HTTP requests. Other TCP services, such as databases, are reachable only from [other members of the network](#reaching-other-members).
* **The sandbox must be a member.** Only members matching `to` are exposed. Removing the sandbox from the network, changing its tags, or deleting the network stops the exposure within seconds.
* **Public and private access are separate.** An `internet` rule gives members no access to each other, and a rule like `*->*` exposes nothing publicly. Each call replaces the whole rule list, so include your private rules too:

```bash theme={null}
prime sandbox network rules <network-id> \
  -r 'internet->tag:web@8080' \
  -r 'tag:web->tag:db@5432'
```

A port that no rule exposes returns `404 Port not exposed`. A sandbox that does not exist returns the same response, so the URL reveals nothing about which sandboxes exist.

## Managing Networks

```python Python SDK theme={null}
# List networks, with sync status
for n in client.list_lan_networks():
    print(n.id, n.name, n.synced)

# Show rules and members
network = client.get_lan_network(network.id)
for member in network.members:
    print(member.sandbox_id, member.tags, member.synced)

# Remove one sandbox, or delete the whole network
client.remove_lan_member(network.id, "<sandbox-id>")
client.delete_lan_network(network.id)
```

```bash CLI theme={null}
# List networks
prime sandbox network list

# Show rules and members
prime sandbox network get <network-id>

# Remove one sandbox, or delete the whole network
prime sandbox network remove <network-id> <sandbox-id>
prime sandbox network delete <network-id>
```

Changes apply within a few seconds. Until they do, the network or member shows as not yet synced, and Prime keeps retrying in the background.

Removing a sandbox from a network cuts it off right away. Its connections to the other members are refused, including connections to addresses it looked up earlier. A sandbox that stops running leaves its networks automatically.

`AsyncSandboxClient` has the same methods for async code. Pass `team_id` to `create_lan_network` and `list_lan_networks` (or `--team-id` on the CLI) to work with a specific team. Otherwise your configured team is used.

## Limits

| Resource | Limit |
| - | - |
| Networks per sandbox | 32 |
| Rules per network | 256 |
| Ports per rule | 64 |
| Tags per member | 16 |
| Network name length | 128 characters |

## Quick Troubleshooting

* **"Only running sandboxes can join a network"**: wait until the sandbox is `RUNNING`, then add it.
* **"Only VM sandboxes can join a network"**: private networks are available for VM sandboxes only.
* **"The sandbox and the network must belong to the same team or user"**: create the network under the same team as the sandbox, or switch your configured team.
* **Name does not resolve**: names resolve only inside a sandbox that belongs to a network. Check that `prime sandbox network get <network-id>` lists the sandbox as a synced member.
* **Connection refused or reset**: no rule allows that path. Check the direction and ports with `prime sandbox network get <network-id>`.
* **Public URL returns `404 Port not exposed`**: no `internet` rule exposes that port on the sandbox. Check that the sandbox is a synced member whose tags match the rule's `to`, and that the port is listed.
* **Public URL returns `502`**: the port is exposed, but nothing inside the sandbox accepts connections on it. Check that your server is running and listens on `0.0.0.0`.


# Sandboxes Overview
Source: https://docs.primeintellect.ai/sandboxes/overview

What sandboxes can do, how to launch one, and what it costs

Prime Sandboxes are disposable, hardware-isolated microVMs for AI agents, benchmarking, agentic RL, and quick experiments. Each one boots an OCI/Docker image as its root filesystem, so it runs anything a real VM can: Docker, systemd, background services, kernel-dependent workloads. Thousands can start concurrently in seconds.

![Sandbox Dashboard](https://mintcdn.com/primeintellect/27DA8i_ikZGh6lgH/images/sandboxes/sandbox_dashboard.png?fit=max\&auto=format\&n=27DA8i_ikZGh6lgH\&q=85\&s=e6959f8950f0355be2688738a5a20403)

## Features

> **[Create & manage](/sandboxes/lifecycle)**
>
> Resources, start commands, lifetime and idle timeouts, labels, bulk delete

> **[Run commands & SSH](/sandboxes/commands)**
>
> One-off commands, long-running background jobs, interactive SSH

> **[Files](/sandboxes/files)**
>
> Upload, download, and read files

> **[Env vars & secrets](/sandboxes/secrets)**
>
> Inject config and encrypted credentials that never appear in API responses

> **[Network isolation](/sandboxes/network-isolation)**
>
> Allow or deny outbound traffic by domain or CIDR, live-updatable

> **[Private networks](/sandboxes/networks)**
>
> Let sandboxes reach each other at `<sandbox-id>.sandbox.internal`, with per-port rules

> **[Public ports](/sandboxes/networks#exposing-ports-to-the-internet)**
>
> Serve an HTTP app running in a sandbox at a public HTTPS URL

> **[Checkpoints & forking](/sandboxes/checkpoints)**
>
> Save a sandbox's filesystem, restore it later, or fork it

> **[Volumes](/sandboxes/volumes)**
>
> Persistent named storage you attach to a sandbox and publish to share

> **[Custom images](/sandboxes/images)**
>
> Build images in the cloud or copy them from public registries

> **[Tunnels](/sandboxes/tunnel)**
>
> Expose a local or in-sandbox service at a public HTTPS URL

> **[Python SDK](/sandboxes/sdk)**
>
> Sync and async clients, fleets, typed errors

> **GPUs (coming soon)**
>
> GPU sandboxes need an explicit grant; `--gpu-count` above 0 errors until then

## Quickstart

Log in first with `prime login`, using an API key with sandbox permissions.

```bash CLI theme={null}
prime sandbox create python:3.11-slim --timeout-minutes 120
prime sandbox list
prime sandbox run <sandbox-id> "python --version"
prime sandbox delete <sandbox-id>
```

```python Python theme={null}
from prime_sandboxes import SandboxClient, CreateSandboxRequest, APIClient

client = SandboxClient(APIClient())
sandbox = client.create(
    CreateSandboxRequest(
        name="quickstart", docker_image="python:3.11-slim", timeout_minutes=120
    )
)
client.wait_for_creation(sandbox.id)
print(client.execute_command(sandbox.id, "python --version").stdout)
client.delete(sandbox.id)
```

Python snippets on the feature pages assume this `client`; see the [Python SDK](./sdk) for the async client. Runnable demos live in [prime/examples](https://github.com/PrimeIntellect-ai/prime/tree/main/examples).

## Pricing

Sandboxes are billed while running. For our new launch, these rates are valid through December 22, 2026:

* **CPU**: \$0.02 per vCPU per hour
* **Memory**: \$0.0125 per GiB per hour
* **Disk**: \$0.0002 per GiB per hour

Example: 1 vCPU, 2 GiB RAM, 5 GiB disk → \$0.046/hour.

VM images and filesystem checkpoints are billed for storage separately:

* **Storage**: \$0.10 per GiB per month, billed hourly while the image or checkpoint exists
* **Upload**: a one-time charge equal to one day of storage, applied when an image build completes or a checkpoint becomes `DURABLE`

Track usage and billing in the [Prime Billing dashboard](https://app.primeintellect.ai/dashboard/billing).

## Limits

### Per-sandbox

| Resource | Min | Max | Default |
| - | - | - | - |
| vCPUs | 1 | 16 | 1 |
| Memory | 128 MiB | 64 GiB | 1 GiB |
| Disk | 2 GiB | 128 GiB | 5 GiB |
| GPUs | 0 | 8 | 0 |
| Timeout | 1 min | Unlimited | 60 min |
| Idle timeout | 1 min | 1,440 min (24 hr) | Disabled |

CPU is allocated in whole vCPUs; fractional values are truncated, and anything below 1 becomes 1. Set the timeout to any negative value for no lifetime deadline. A finite timeout must be at least as long as the idle timeout.

### Per-account

| Resource | Default limit |
| - | - |
| Active sandboxes | 1,024 |
| Total vCPUs | 4,096 |
| Total memory | 4,096 GiB |
| Total storage | 32,768 GiB |
| Sandbox creations | 102,400 per hour |
| Creation burst | 192 per 10 seconds |

Account limits apply across all running sandboxes. If you need higher limits, contact support.


# Python SDK
Source: https://docs.primeintellect.ai/sandboxes/sdk

Sync and async clients, fleets, and error handling

The SDK ships two clients with identical methods: `SandboxClient` for synchronous scripts and `AsyncSandboxClient` for concurrent workloads. Both are importable from `prime_sandboxes`. Every feature page shows the sync client; the async client takes the same calls with `await`.

## Sync client

```python theme={null}
from prime_sandboxes import SandboxClient, CreateSandboxRequest, APIClient

client = SandboxClient(APIClient())

sandbox = client.create(
    CreateSandboxRequest(name="sdk-demo", docker_image="python:3.11-slim")
)
client.wait_for_creation(sandbox.id)
print(client.execute_command(sandbox.id, "python -c 'print(42)'").stdout)
client.delete(sandbox.id)
```

## Async client

```python theme={null}
import asyncio
from prime_sandboxes import AsyncSandboxClient, CreateSandboxRequest

async def main() -> None:
    async with AsyncSandboxClient() as sandboxes:
        sandbox = await sandboxes.create(
            CreateSandboxRequest(name="sdk-demo", docker_image="python:3.11-slim")
        )
        await sandboxes.wait_for_creation(sandbox.id)
        result = await sandboxes.execute_command(sandbox.id, "python -c 'print(42)'")
        print(result.stdout.strip())
        await sandboxes.delete(sandbox.id)

asyncio.run(main())
```

## Launch a fleet

```python theme={null}
async def create_many(images: list[str]) -> None:
    async with AsyncSandboxClient() as sandboxes:
        requests = [
            CreateSandboxRequest(
                name=f"batch-{i}", docker_image=image, idempotency_key=f"batch-{i}"
            )
            for i, image in enumerate(images, start=1)
        ]
        created = await asyncio.gather(*[sandboxes.create(req) for req in requests])
        await sandboxes.bulk_wait_for_creation([sbx.id for sbx in created])
        print("Ready:", ", ".join(sbx.name for sbx in created))
```

`bulk_wait_for_creation` polls via the list endpoint, backing off automatically if the API throttles you. `idempotency_key` makes create retries safe. Tear down with [`bulk_delete`](./lifecycle#delete).

For a full script, see `prime/examples/sandbox_async_demo.py`.

## Error handling

All exceptions are importable directly from `prime_sandboxes`.

### Sandbox lifecycle errors

Raised when a sandbox is no longer `RUNNING`. `SandboxOOMError`, `SandboxTimeoutError`, and `SandboxImagePullError` are subclasses of `SandboxNotRunningError`, so catch the base class for broad handling.

| Exception | Cause | Typical Fix |
| - | - | - |
| `SandboxNotRunningError` | Operation attempted on a non-running sandbox (terminated, errored, or timed out) | Check sandbox status before operating on it |
| `SandboxOOMError` | Sandbox killed due to out-of-memory | Increase `memory_gb` in `CreateSandboxRequest` or optimize memory usage |
| `SandboxTimeoutError` | Sandbox exceeded its `timeout_minutes` and was terminated | Increase the timeout or split work into smaller tasks |
| `SandboxImagePullError` | Docker image could not be pulled | Verify the image name, tag, and that you can access the image |

### Operation errors

Raised when the sandbox is still running but the operation itself fails.

| Exception | Cause | Typical Fix |
| - | - | - |
| `CommandTimeoutError` | `execute_command` exceeded its `timeout` parameter | Increase the per-command timeout, or use `start_background_job` for long tasks |
| `UploadTimeoutError` | File upload timed out | Check network conditions; pass a larger `timeout` for big files |
| `DownloadTimeoutError` | File download timed out | Check file size and network conditions |
| `SandboxFileNotFoundError` | File does not exist at the given path | Check the path before uploading, downloading, or reading |
| `SandboxFileTooLargeError` | Read exceeded the read size limit | Use `download_file` instead of `read_file` for large files |

### API errors

| Exception | Cause | Typical Fix |
| - | - | - |
| `APIError` | Base class for all API errors (non-2xx response, malformed response, network failure) | Inspect the error message for details |
| `UnauthorizedError` | Invalid or expired API key (HTTP 401) | Check `PRIME_API_KEY` or re-run `prime login` |
| `PaymentRequiredError` | Insufficient balance (HTTP 402) | Top up your account balance |
| `APITimeoutError` | API request timed out before receiving a response | Retry the request; check network connectivity |


# Environment Variables & Secrets
Source: https://docs.primeintellect.ai/sandboxes/secrets

Pass configuration and credentials into a sandbox

```bash CLI theme={null}
prime sandbox create python:3.11-slim \
  --env APP_ENV=staging \
  --env LOG_LEVEL=debug \
  --secret API_KEY=sk-abc123 \
  --secret DB_PASSWORD=hunter2
```

```python Python theme={null}
sandbox = client.create(
    CreateSandboxRequest(
        name="configured-runner",
        docker_image="python:3.11-slim",
        environment_vars={"APP_ENV": "staging", "LOG_LEVEL": "debug"},
        secrets={"API_KEY": "sk-abc123", "DB_PASSWORD": "hunter2"},
    )
)
```

Both are injected as standard environment variables. The CLI flags take `KEY=VALUE` and can repeat.

* **Environment variables** are stored in plain text and visible when you inspect the sandbox.
* **Secrets** are encrypted at rest, never returned in API responses, and obfuscated in CLI output. Use them for API keys, passwords, and anything you wouldn't want in logs.

For a single command, pass env vars on the command instead: see [Run a command](./commands#run-a-command).


# Prime Tunnels
Source: https://docs.primeintellect.ai/sandboxes/tunnel

Expose local services to the internet through secure reverse proxy

## What is Prime Tunnels?

Prime Tunnels creates secure, public URLs for services running on your local machine. You start a local server, create a tunnel, and get an HTTPS URL that anyone can access.

**Common use cases:**

* Expose a local development server to the internet
* Receive webhooks from external services during development
* Share a work-in-progress app with teammates

> [!NOTE]
>
> Hosted evaluations can use Prime Tunnels too. Launch the run with
> `--allow-tunnel-access`, then create the tunnel from inside the hosted
> sandbox using the same API or CLI flow described here. The flag only grants
> tunnel permissions to the temporary `PRIME_API_KEY`. See
> [Hosted Evaluations](/tutorials-environments/hosted-evaluations).

## Complete Example

Here's a full working example: start a local HTTP server and expose it to the internet.

**1. Start a local server**

```bash theme={null}
# Create a simple HTML file
echo '<h1>Hello from Prime Tunnels!</h1>' > index.html

# Start Python's built-in HTTP server on port 8000
python -m http.server 8000
```

**2. Create a tunnel (in another terminal)**

```python Python SDK theme={null}
import asyncio
from prime_tunnel import Tunnel

async def main():
    tunnel = Tunnel(local_port=8000)
    await tunnel.start()

    print(f"Public URL: {tunnel.url}")
    print("Press Ctrl+C to stop")

    # Keep running until interrupted
    try:
        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        pass
    finally:
        await tunnel.stop()

asyncio.run(main())

```

```bash CLI theme={null}
prime tunnel start --port 8000
```

**3. Access your server**

Open the printed URL (e.g., `https://t-0-abc123def456.tunnel.pinfra.io`) in any browser. Your local server is now accessible from anywhere.

## Context Manager

For automatic cleanup, use the async context manager:

```python theme={null}
async with Tunnel(local_port=8000) as tunnel:
    print(f"URL: {tunnel.url}")
    # Tunnel is active here
# Tunnel automatically stopped
```

## Tunnel Properties

```python theme={null}
tunnel = Tunnel(local_port=8000)
await tunnel.start()

tunnel.tunnel_id  # "t-0-abc123def456"
tunnel.url        # "https://t-0-abc123def456.tunnel.pinfra.io"
```

## Protecting a Tunnel with Basic Auth

By default, anyone with the URL can reach your tunnel. To require credentials, provide a username when creating the tunnel — a strong password is **auto-generated server-side** and shown exactly once. All public traffic must then include HTTP basic auth; requests without valid credentials get a `401 Unauthorized`.

```python Python SDK theme={null}
tunnel = Tunnel(local_port=8000, http_user="alice")
await tunnel.start()

print(tunnel.url)
print(tunnel.http_password)  # auto-generated
```

```bash CLI theme={null}
prime tunnel start --port 8000 --auth alice
```

The CLI prints the generated password when the tunnel starts:

```
Tunnel started successfully!
URL: https://t-0-abc123def456.tunnel.pinfra.io
Tunnel ID: t-0-abc123def456
Basic auth user: alice
Basic auth password: vBI6ioyAM3_X4KWaEug4TPSbzovK8eYb
Save this password - it is shown only once and cannot be retrieved later
```

Browsers prompt for the credentials automatically. For programmatic access, send the `Authorization` header:

```bash theme={null}
curl -u alice:vBI6ioyAM3_X4KWaEug4TPSbzovK8eYb https://t-0-abc123def456.tunnel.pinfra.io
```

## CLI Commands

```bash theme={null}
# Start a tunnel
prime tunnel start --port 8000

# Start a tunnel protected by HTTP basic auth (password is auto-generated)
prime tunnel start --port 8000 --auth alice

# List active tunnels
prime tunnel list

# Stop a tunnel
prime tunnel stop t-0-abc123def456
```

## Rate Limits

The tunnel API enforces per-user rate limits over a 60-second window.

| Endpoint | Method | Limit |
| - | - | - |
| `/api/v1/tunnel` | `POST` | 600 requests / 60s |
| `/api/v1/tunnel` | `GET` | 300 requests / 60s |
| `/api/v1/tunnel/{id}` | `GET` | 300 requests / 60s per tunnel |
| `/api/v1/tunnel/{id}` | `DELETE` | 100 requests / 60s |
| `/api/v1/tunnel` | `DELETE` | 100 requests / 60s (bulk) |

Requests over the limit return `429 Too Many Requests`. There is also a per-account cap on the number of concurrent tunnels (`tunnel_limit` on the wallet) - delete unused tunnels before creating new ones.


# Volumes
Source: https://docs.primeintellect.ai/sandboxes/volumes

Persistent named storage you can attach to sandboxes

A volume is named storage that outlives any sandbox. Create it once, attach it to sandboxes at a mount path, write to it, and publish to share the result with future sandboxes.

> [!NOTE]
>
> Sandbox volumes (`prime sandbox volumes`) are separate from [Hosted Training volumes](/hosted-training/volumes), which back Model Factory training runs.

## How a volume works

* A sandbox that attaches a volume gets a **private copy** of it, taken from the volume's latest checkpoint when the sandbox is created. Its writes are invisible to every other sandbox, and a running sandbox never picks up later publishes.
* Volumes attach **read-write** (the default) or **read-only** (`:ro`). A read-only attachment cannot write or publish.
* **Publishing** turns one sandbox's private writes into a new volume checkpoint. Only sandboxes created after the publish completes see the new data. A volume that was never published is empty.
* Volumes use [checkpoints](/sandboxes/checkpoints) under the hood: each publish saves the volume's contents as a checkpoint.
* Published data survives sandbox termination; unpublished writes do not. **Publish before deleting a sandbox** whose writes you want to keep.
* A volume is a separate disk: it doesn't count against the sandbox's `disk_size_gb`, and a sandbox checkpoint captures the root filesystem only, never a mounted volume. Each volume holds up to **100 GiB**.

## Quickstart

```bash CLI theme={null}
# Create an empty volume
prime sandbox volumes create my-data

# Attach it to a new sandbox at /data and write to it
prime sandbox create python:3.11-slim --volume my-data:/data
prime sandbox run <sandbox-id> "bash -c 'echo hello > /data/hello.txt'"

# Publish: the sandbox's private writes become a new volume checkpoint
prime sandbox volumes publish my-data --sandbox <sandbox-id>

# A new sandbox sees the published data; :ro mounts it read-only
prime sandbox create python:3.11-slim --volume my-data:/data:ro
prime sandbox run <sandbox-id-2> "cat /data/hello.txt"
```

```python Python theme={null}
from prime_sandboxes import (
    APIClient,
    CreateSandboxRequest,
    SandboxClient,
    VolumeMount,
)

client = SandboxClient(APIClient())

client.volumes.create("my-data")

writer = client.create(
    CreateSandboxRequest(
        name="writer",
        docker_image="python:3.11-slim",
        volumes=[VolumeMount(name="my-data", mount_path="/data")],
    )
)
client.wait_for_creation(writer.id)
client.execute_command(writer.id, "bash -c 'echo hello > /data/hello.txt'")

# Publishes and waits until the new checkpoint is durable
client.volumes.publish(writer.id, "my-data")

reader = client.create(
    CreateSandboxRequest(
        name="reader",
        docker_image="python:3.11-slim",
        volumes=[VolumeMount(name="my-data", mount_path="/data", read_only=True)],
    )
)
client.wait_for_creation(reader.id)
print(client.execute_command(reader.id, "cat /data/hello.txt").stdout)
```

Pass `--volume NAME:/mount/path[:ro]` up to 8 times per sandbox. `:ro` mounts the volume read-only; writable mounts have no suffix.

## Import a local directory

```bash theme={null}
prime sandbox volumes create my-data --from ./dataset
```

`--from` needs `rsync` on your machine. The CLI copies the directory through a short-lived helper sandbox and publishes it as the volume's first checkpoint; the helper is deleted once the publish is confirmed durable. If the volume already exists, the directory is copied into its current published contents.

## Publish rules

* `publish` waits until the new checkpoint is `DURABLE` by default. With `--no-wait` (CLI) or `wait=False` (SDK) it returns immediately — poll `prime sandbox checkpoint get <checkpoint-id>` or `client.get_checkpoint(...)` until `DURABLE`. A publish that times out can still finish.
* If a sandbox tries to publish after the volume's checkpoint has advanced past the one it attached, publishing fails with 409. A sandbox can still publish again on top of its own earlier publishes. To recover, re-attach in a fresh sandbox, or publish with `--force` to make this sandbox's entire copy the volume's new checkpoint. `--force` replaces the published contents; it does not merge files.
* Read-only attachments cannot publish.
* If a publish fails, the volume keeps its previous checkpoint, and the sandbox keeps its private writes.

## Manage volumes

```bash CLI theme={null}
prime sandbox volumes list --num 50
prime sandbox volumes get my-data
prime sandbox volumes delete my-data
```

```python Python theme={null}
page = client.volumes.list(page=1, per_page=50)
print(page.volumes, page.has_next)

volume = client.volumes.get("my-data")
print(volume.head_checkpoint_id)  # None until the first publish is durable

client.volumes.delete("my-data")
```

* `list` paginates with `--page` / `--num` (max 100 per page); `--json` prints raw fields.
* `delete` fails with 409 while any active sandbox has the volume attached, and removes the volume and all of its published data. Deleting a team volume requires a team admin.

### Scopes and permissions

* Volumes are personal by default. Pass `--team-id` (CLI) or `team_id` (SDK) to work in a team; without it, the CLI uses the team from your config, if any.
* Team members can create, attach, and publish team volumes; only team admins can delete them.
* API tokens need the `sandbox:read` scope to list or get volumes and `sandbox:write` to create, delete, or publish.

### Names and mount paths

* Volume names: 1–40 characters, lowercase letters, digits, and hyphens; must start and end with a letter or digit; unique within their scope.
* Mount paths: absolute, not `/`, unique per sandbox, and not nested inside each other.

## Common errors

| Error | Cause | Fix |
| - | - | - |
| 404 `Volume not found` | No volume with that name in your scope | Check `prime sandbox volumes list` and your team scope |
| 409 `A volume with this name already exists in this scope` | Name taken on create | Pick another name or use the existing volume |
| 409 `Volume is mounted by one or more sandboxes` | Delete while attached | Delete or stop the attached sandboxes first |
| 400 `Read-only attachments cannot publish` | Publishing a `:ro` mount | Attach writable, then publish |
| 404 `Volume is not attached to this sandbox` | Publishing a name this sandbox doesn't have | Check the sandbox's `--volume` mounts |
| 409 `Another sandbox is publishing this volume` | Concurrent publish | Wait for the other publish to finish |
| 409 on publish, volume checkpoint advanced | The volume's checkpoint advanced after this sandbox attached | Publish with `--force` to replace the volume's contents, or re-attach in a fresh sandbox |

Published volume data is billed as [checkpoint storage](/sandboxes/overview#pricing).


# Legacy storage retirement
Source: https://docs.primeintellect.ai/traces/legacy-storage

Export existing evaluation and training samples before November 6, 2026

The platform's legacy evaluation and training sample storage will be retired on **Friday, November 6, 2026**. Request and download your archive before that date to keep a copy of your existing data. Exporting and downloading the archive is free.

**Existing data is not migrated automatically.** The archive preserves a copy for your own records; it cannot be imported directly into Prime Traces. Any new uploads to Prime Traces incur the [normal ingestion and storage charges](/traces/pricing-and-settings).

## Continue using legacy storage during the transition

Legacy sample storage remains available until retirement. Runs using the [Prime Runs opt-out controls](/traces/pricing-and-settings#opt-out-for-a-run-or-client) send their samples to legacy storage instead of Prime Traces during this period.

Legacy storage is distinct from the legacy verifiers environment runtime. Selecting an environment runtime is not the same as choosing trace storage. The retirement described here concerns evaluation and training sample storage, not model checkpoints or adapters.

## Request and download an archive

1. For team data, select the appropriate team workspace first. Then open your [personal settings](https://app.primeintellect.ai/dashboard/profile?tab=profile) or [team settings](https://app.primeintellect.ai/dashboard/team-profile).
2. Select **Request export** in the export banner. You can switch tabs or refresh while the archive is being prepared.
3. Once it is ready, return to the same page and select **Download archive**.
4. Confirm that the archive has downloaded to your device. Requesting an export alone does not save a local copy.

Repeat these steps for every workspace where you have legacy data.

## Archive limitations

The archive is a copy of legacy data, not a Prime Traces import package. Do not send it directly to `prime traces upload`. The new service expects [supported trace or episode records](/traces/uploading#supported-records).

Free archive export does not include free ingestion into Prime Traces. Prime Traces itself also has no egress fees.

If you cannot see the export banner, an export fails, or you are unsure which data is affected, contact [support@primeintellect.ai](mailto:support@primeintellect.ai) before the retirement date.


# Viewing and deleting traces
Source: https://docs.primeintellect.ai/traces/managing-traces

Inspect, download, search, and delete stored traces

## View and search

Open the [Traces dashboard](https://app.primeintellect.ai/dashboard/traces) and select the workspace that owns the data. You can also query with the [SDK or CLI](/traces/overview#python-sdk).

```bash theme={null}
prime traces list --run-id your-run-id
prime traces get <trace_id> --raw
prime traces search "Traceback" --run-id your-run-id
```

Search is case-sensitive literal search within one run, with at least three characters. Cross-run search, regular expressions, and arbitrary JSON-path queries are not supported. Results may omit records that have not finished indexing; inspect search coverage rather than treating an empty result as proof that the data is absent.

## Download a trace

```python theme={null}
from prime_traces import TracesClient

with TracesClient() as client:
    client.download_raw("your-trace-id", "trace.json")
```

This downloads an individual stored trace. It is separate from the [legacy archive export](/traces/legacy-storage); the SDK does not currently provide a working bulk-export method. Prime Traces has no egress fees.

## Delete a trace

Download any copy you need to keep before deletion. Check the workspace and trace ID carefully.

```python theme={null}
from prime_traces import TracesClient

with TracesClient() as client:
    client.delete("your-trace-id")
```

[Deleting a trace](/api-reference/traces/traces/delete-a-trace) deletes every stored copy of that trace. The API returns `202` once it is no longer readable; physical storage is reclaimed in the background. Deleting an already absent trace returns `404` (`NotFoundError` in the SDK).

## Delete a run's traces

Stop any producer still uploading to the run before deleting its traces.

```python theme={null}
from prime_traces import TracesClient

with TracesClient() as client:
    client.delete_run("your-run-id")
```

[Deleting a run's traces](/api-reference/traces/traces/delete-a-runs-traces) removes the traces present when the service looks up the run. **Episodes are kept.** This is a trace deletion operation, not a request to delete the training or evaluation record.

An upload that finishes while deletion is running may or may not be included. An upload that finishes after deletion returns is not included. The API returns `202` once the affected traces are no longer readable. A run with no traces returns `404`, including a repeated deletion.

> [!NOTE]
>
> Disabling uploads does not delete existing traces. Deletion acceptance, physical storage reclamation, and billing updates are separate events; a `202` response confirms that the affected traces are no longer readable, not that a billing update has completed.

## Troubleshooting

| Symptom | What to check |
| - | - |
| Missing traces | Check the personal/team workspace and run ID. Inspect uploader errors; do not assume a completed training run guarantees every trace uploaded successfully. |
| Storage limit exceeded | Review the workspace's [Usage Limits](https://app.primeintellect.ai/dashboard/billing?tab=limits). Delete unneeded data or review the configured limit. Physical reclamation runs in the background. |
| SDK/CLI reads fail after opting out | Account-level opt-out disables those reads; existing data remains viewable in the dashboard. |
| Validation rejected | Confirm the [record format](/traces/uploading#supported-records) and the API error. Do not retry the same malformed input unchanged. |
| Permission denied | Check API scopes, team membership, and workspace upload settings. |
| Search finds no match | Check case, run ID, and indexing coverage. Search is not a full raw-file scan. |

If you need help reconciling storage usage or charges after deletion, contact [support@primeintellect.ai](mailto:support@primeintellect.ai) with the workspace, run or trace ID, and deletion time.


# Traces Overview
Source: https://docs.primeintellect.ai/traces/overview

Store and explore evaluation, training, and agent traces

Prime Traces stores the traces from evaluation, training, and agent workloads so you can view, search, download, and delete them. Open the [Traces dashboard](https://app.primeintellect.ai/dashboard/traces) in your personal or team workspace.

* **Trace**: one rollout or request, with its messages, model calls, score, and metadata.
* **Episode**: a group of traces produced together, such as a multi-agent rollout.
* **Run**: traces that share a `run_id`, with totals such as trace count and average reward.

Prime Traces currently supports the **verifiers v1 trace format**. An arbitrary JSONL file, an OpenTelemetry export, or a legacy sample archive is not a substitute for that format. See [Uploading traces](/traces/uploading) for supported inputs.

> [!NOTE]
>
> Prime Traces is enabled by default for platform runs that upload traces. Uploads incur ingestion and storage charges. See [Pricing and settings](/traces/pricing-and-settings) to review the rates or disable uploads for your workspace.

> [!WARNING]
>
> Legacy evaluation and training sample storage will be retired on **November 6, 2026**. Existing data is not migrated automatically. [Request and download your legacy archive](/traces/legacy-storage) before that date.

## Python SDK

Create an API key with the **Prime Traces** permissions you need. The SDK reads credentials from `PRIME_API_KEY` or your `prime login` configuration. Set `PRIME_TEAM_ID` for a team's traces. See [authentication and workspace selection](/traces/uploading#authentication-and-workspace).

```bash theme={null}
uv add prime-traces
```

Upload a supported trace file using the [upload guide](/traces/uploading), then query its run:

```python theme={null}
from prime_traces import TracesClient

with TracesClient() as client:
    page = client.list(run_id="your-run-id", reward_min=0.9)
    for summary in page.items:
        print(summary.trace_id, summary.score)

    if page.items:
        raw = client.get_raw(page.items[0].trace_id)

    matches = client.search("Traceback", run_id="your-run-id")
```

## CLI

```bash theme={null}
prime traces list --run-id your-run-id
prime traces get <trace_id> --raw
prime traces search "Traceback" --run-id your-run-id
```

## Next steps

* [Uploading traces](/traces/uploading): supported formats, authentication, SDK, and CLI.
* [Pricing and settings](/traces/pricing-and-settings): charges, workspace limits, and upload controls.
* [Viewing and deleting traces](/traces/managing-traces): search, downloads, deletion, and troubleshooting.
* [Legacy storage retirement](/traces/legacy-storage): keep a copy of your existing evaluation and training data.


# Pricing and settings
Source: https://docs.primeintellect.ai/traces/pricing-and-settings

Understand trace charges, workspace limits, and automatic upload controls

## Pricing

Prime Traces is priced by data volume:

| Usage | Price |
| - | - |
| Ingestion | **\$1 per GiB ingested** |
| Storage | **\$0.10 per GiB per month stored**, billed hourly |
| Requests | **No request fees** |
| Egress | **No egress fees** |

One GiB is 1,073,741,824 bytes. Ingestion is charged when data is ingested; storage depends on the volume retained over time. For illustration, ingesting 10 GiB and retaining it for approximately one month costs approximately \$11: \$10 ingestion and \$1 storage.

Trace charges are separate from training, inference, and other platform usage. Exporting legacy samples is free, but data subsequently uploaded to Prime Traces incurs the rates above.

## Control run uploads

Prime Traces is enabled by default. Traces uploaded by platform runs incur ingestion and storage charges.

To disable uploads for a workspace:

1. Select the appropriate personal or team workspace.
2. Open [Billing → Usage Limits](https://app.primeintellect.ai/dashboard/billing?tab=limits).
3. Disable **Upload runs to Prime Traces**.

Turning off uploads does **not** delete stored traces. Existing traces remain viewable in the dashboard; account-level opt-out also disables trace reads through the SDK and CLI. See [Viewing and deleting traces](/traces/managing-traces) to manage existing data.

## Opt out for a run or client

The controls below are documented in the [Prime Runs SDK at CLI release v0.9.4](https://github.com/PrimeIntellect-ai/prime/blob/v0.9.4/packages/prime-runs/README.md#opting-out-of-prime-traces). Update older installations before using them; separately installed SDKs must also include these controls.

For a single run, pass `traces_opt_out=True` to your existing `prime_runs.init(...)` call. For a shell or job:

```bash theme={null}
export PRIME_TRACES_OPT_OUT=true
```

For runs using this machine's Prime configuration:

```bash theme={null}
prime config set-traces-opt-out true
```

Client setting precedence is: explicit `init()` argument, then `PRIME_TRACES_OPT_OUT`, then the config file. These client controls do not override an account-level opt-out. They control Prime Runs routing; they are not a substitute for workspace permissions on standalone Traces API requests.

Opted-out runs use legacy sample storage while that service remains available. This is a temporary compatibility path, not permanent free trace storage. See the [November 6 retirement and export instructions](/traces/legacy-storage).

To allow uploads again, enable them for the workspace and remove or set to `false` any applicable client opt-out. Change the setting before starting the run.

## Usage limits

Check the selected workspace's [Usage Limits](https://app.primeintellect.ai/dashboard/billing?tab=limits) when an upload fails because storage is over its limit. Verify the workspace first, then review its configured limit and stored data. See [deletion and quota troubleshooting](/traces/managing-traces#troubleshooting).

A usage limit and a pricing rate are different: the limit controls available capacity; the rates above determine charges for accepted usage.


# Uploading traces
Source: https://docs.primeintellect.ai/traces/uploading

Upload supported traces with the Python SDK or Prime CLI

## Automatic and explicit uploads

Online runs using the Prime Runs SDK upload traces to Prime Traces by default when the account has access. Run metadata and training metrics still use the Platform API. Local files are separate: writing a `traces.jsonl` file to disk does not itself upload it.

For local evaluations, follow the upload instructions for the producer and format you are using. Legacy `vf-eval` results and `prime eval push` workflows are distinct from directly uploading verifiers v1 traces. See [legacy storage retirement](/traces/legacy-storage).

Uploads are subject to [ingestion and storage charges](/traces/pricing-and-settings). To control automatic run uploads, see [upload settings](/traces/pricing-and-settings#control-run-uploads).

## Authentication and workspace

1. Create an [API key](/api-reference/api-keys) with **Prime Traces** read and/or write permissions (`traces:read`, `traces:write`) for the operations you need.
2. Authenticate with `prime login` or set `PRIME_API_KEY`.
3. Select the intended workspace. Set `PRIME_TEAM_ID` for team data; the SDK also reads the team from your CLI configuration. Check `prime config view` before uploading.

```bash theme={null}
prime login
prime config view
export PRIME_TEAM_ID="your-team-id"
```

To return to personal context, remove a shell override with `unset PRIME_TEAM_ID` and clear a configured team with `prime config remove-team-id`. The destination workspace owns the uploaded data and its usage.

## Supported records

Prime Traces currently supports **verifiers v1 traces and episodes**. A JSONL extension describes the file container, not its schema. Preserve the producer's records rather than converting legacy `prompt` / `completion` / `reward` rows by renaming fields.

The [SDK upload example](https://github.com/PrimeIntellect-ai/prime/blob/v0.9.4/packages/prime-traces/examples/basic_usage.py) includes a small service-compatible record and validation handling. It is a reduced example, not the complete verifiers schema. For actual runs, use the trace or episode objects emitted by verifiers, or their serialized output.

Trace records require a non-empty `id` and numeric `timing.start`; other validation rules also apply. Passing those two checks alone does not establish that a legacy record is a complete verifiers trace. The [upload API reference](/api-reference/traces/ingestion/upload-traces) describes the request and validation errors.

## Python SDK

```bash theme={null}
uv add prime-traces
```

The SDK accepts producer objects exposing `to_record()` as well as serialized records:

```python theme={null}
from prime_traces import LineFormat, TracesClient

# `traces` and `episodes` are objects emitted by your verifiers run.
with TracesClient() as client:
    receipts = client.upload_records(traces, context={"source": "my-eval"})

    # For episode records, select the episode line format explicitly.
    episode_receipts = client.upload_records(
        episodes, line_format=LineFormat.EPISODE
    )
```

For a JSONL file you have prepared with one supported trace record per line (called `trace-records.jsonl` in this example):

```python theme={null}
from prime_traces import TracesClient

with TracesClient() as client:
    receipts = client.upload_file("trace-records.jsonl", context={"source": "my-eval"})
```

Verifiers v1 (`vf`) saves completed rollout **episodes** in `traces.jsonl`, with each episode containing its traces. For this file, select `LineFormat.EPISODE` explicitly:

```python theme={null}
from prime_traces import LineFormat, TracesClient

with TracesClient() as client:
    receipts = client.upload_file(
        "traces.jsonl", line_format=LineFormat.EPISODE
    )
```

prime-rl (`prl`) also saves rollout episodes through its file monitor; the output paths and chunking depend on the version and configuration. Use episode format for completed episode records, not for metrics, indexes, or live trace-delta files. See the [Verifiers episode writer](https://github.com/PrimeIntellect-ai/verifiers/blob/main/verifiers/v1/utils/trace_store.py) and [prime-rl file monitor](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/src/prime_rl/monitors/file/monitor.py).

Choose the line format from the records in the file, not its filename.

## CLI

For a file containing one supported trace record per line:

```bash theme={null}
prime traces upload trace-records.jsonl
```

Use `prime traces upload --help` to inspect the options in your installed version, including line format. See [CLI installation](/cli-reference/introduction) to update the CLI.

## Verify the upload

Open the [Traces dashboard](https://app.primeintellect.ai/dashboard/traces) in the same workspace, or query the run ID recorded in the uploaded data:

```bash theme={null}
prime traces list --run-id your-run-id
```

A validation error means the input needs correcting; retrying an unchanged file will not fix it. For permissions, workspace, and quota problems, see [troubleshooting](/traces/managing-traces#troubleshooting).


# Create & Upload Environment
Source: https://docs.primeintellect.ai/tutorials-environments/create

Learn how to create and upload environments to Prime Intellect's Environments Hub

> [!NOTE]
>
> This page describes the verifiers v0 workflow (`load_environment`, `vf-eval`). v0 is deprecated and the Hub lists v0 environments under **Legacy**. Build new environments on [verifiers v1](/verifiers/v1/getting_started) instead; pushing them to the Hub works the same way with `prime env push`.

## Prerequisites

Ensure you have:

1. **Prime CLI** installed and configured
   * See [CLI Overview](/cli-reference/introduction) for setup instructions.
2. **Username** set on your [profile](https://app.primeintellect.ai/dashboard/profile)
3. **Authenticate** the CLI:
   ```bash theme={null}
   prime login
   ```

## Creating a New Environment

### Initialize Environment

Create a new environment with the verifiers starter template:

```bash theme={null}
uvx --from verifiers vf-init <your-env-name>
```

Inside a project that already has verifiers installed (for example a `prime lab setup` workspace), `uv run vf-init <your-env-name>` does the same.

This creates a template for a Python module with:

* A [README.md](http://README.md) file (displayed on the Environments Hub)
* A `pyproject.toml` file for managing dependencies, versioning, tags, description, etc.
* A Python file containing stub code for a `load_environment` function which returns a `vf.Environment` object — this will be the entrypoint for downstream applications to use your Environment, and should be used encapsulate any necessary preprocessing, resource provisioning, exposing configurable args, etc.

### Develop Your Environment

After initialization, you can modify and test your environment.

If your environment needs API keys or credentials, see [Secrets](/tutorials-environments/secrets) for the recommended setup.

To install your environment locally, you can run:

```bash theme={null}
uv pip install -e .
```

To test/evaluate the environment:

```bash theme={null}
uv run vf-eval my-environment
```

> [!NOTE]
>
> Make sure to follow the [verifiers library patterns](https://github.com/PrimeIntellect-ai/verifiers) when implementing your environment. Your environment should inherit from appropriate base classes and implement required methods.

### Upload Your Environment

Once you've developed and tested your environment, push it to the Environments Hub:

```bash theme={null}
# Inside the environments/<your-env-name>/ directory
prime env push
```

This will upload your environment under your user account. You can also upload it to a team using:

```bash theme={null}
prime env push --team <team-username>
```

> [!NOTE]
>
> Once uploaded, your environment will be available for installation by others using `prime env install owner/environment-name`, unless you use the `--visibility=PRIVATE` flag.

## URL Dependencies

If your environment depends on packages from Git repositories (not published to PyPI), you need to use the PEP 440/508 format directly in your `dependencies` list.

### Correct Format

```toml theme={null}
[project]
dependencies = [
    "verifiers",
    "tau2 @ git+https://github.com/sierra-research/tau2-bench.git",
]
```

### Incorrect Format

Do **not** use `[tool.uv.sources]` for URL dependencies that need to work with the Environments Hub:

```toml theme={null}
# ❌ This won't work - uv.sources is not embedded in wheel metadata
[tool.uv.sources]
tau2 = { git = "https://github.com/sierra-research/tau2-bench.git" }
```

The `[tool.uv.sources]` section is uv-specific and not included in the wheel package metadata. When users install your environment, the URL dependency information would be lost.

> [!NOTE]
>
> The Environments Hub automatically extracts URL dependencies from your wheel and passes them as direct requirements during installation. This works around uv's security restriction on transitive URL dependencies from registry packages.

## Version Management

### Updating Your Environment

When you make changes to your environment:

1. Update the version in `pyproject.toml`
2. Push the updated environment:

```bash theme={null}
prime env push
```

The system will automatically create a new version while keeping previous versions available.

> [!TIP]
>
> You can use `prime env push --auto-bump` to automatically increment the version number for you, so you don't need to manually update `pyproject.toml` each time.

## Getting Help

* Check the [verifiers GitHub repository](https://github.com/PrimeIntellect-ai/verifiers) for examples
* Review the [verifiers documentation](/verifiers/overview) for detailed guides
* Join our [Discord community](https://discord.gg/ZTFydGWPKj) for support


# Environment Variables
Source: https://docs.primeintellect.ai/tutorials-environments/environment-variables

Configure plain-text key-value pairs injected into your environment at runtime

Environment variables are plain-text key-value pairs injected into your environment at runtime. They are suited for non-sensitive configuration — things like model names, difficulty levels, dataset paths, or feature flags.

For sensitive values such as API keys, use [Secrets](/tutorials-environments/secrets) instead.

## Managing Variables

Variables are managed per-environment under the **Secrets** tab.

1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab and select **Variables**
3. Click **Add Variable**
4. Set a **Name**, **Value**, and optional description

Variables can be edited or deleted at any time by anyone with write access to the environment.

## Managing via CLI

```bash theme={null}
prime env var list owner/my-env                                     # list all variables
prime env var create owner/my-env --name MY_VAR --value hello       # create a variable
prime env var update <var-id> owner/my-env --value new-value        # update a variable
prime env var delete <var-id> owner/my-env                          # delete a variable
```

## Naming Rules

Variable names follow env-var conventions:

* Must start with an uppercase letter
* Can contain uppercase letters, digits, and underscores only
* Example: `MODEL_NAME`, `DIFFICULTY`, `MAX_EXAMPLES`

## Where Variables Are Available

Variables are injected automatically for all hosted services:

* Hosted Evaluations
* Hosted Training

## Variables vs Secrets

| | Environment Variables | Secrets |
| - | - | - |
| **Stored as** | Plain text | Encrypted |
| **Visible after creation** | Yes | No |
| **Use for** | Non-sensitive config | API keys, credentials |
| **Scope** | Per-environment | Per-environment or global |

When both a variable and a secret share the same name, the secret takes priority. See [Secrets](/tutorials-environments/secrets) for the full precedence rules.

## Conflict Rules

A variable name cannot conflict with an existing linked secret on the same environment. If you try to create a variable with a name already used by a linked secret, the platform will reject it. Rename or unlink the secret first.


# Overview
Source: https://docs.primeintellect.ai/tutorials-environments/environments

Create, manage and share environments for reinforcement learning and evaluation

> [!NOTE]
>
> **The Hub is moving to verifiers v1.** Environments built on the verifiers v0 API carry a **Legacy** tag on the [Environments Hub](https://app.primeintellect.ai/dashboard/environments), and the runtime toggle above the list shows [verifiers v1](/verifiers/v1/overview) or Legacy packages on their own. They still work with `prime env install`, `prime env pull` and `prime env info`, but v0 is deprecated in verifiers and is being retired.
>
> New environments should target v1. `prime env push` works unchanged: a push is listed as v1 or legacy from the package's `verifiers` requirement, and `--runtime v0|v1` overrides that. A v1 package that shows as Legacy only needs a re-push from a current CLI, or its runtime changed in the package settings. Environment Actions, the automated checks that used to run on every push, have been removed. To move an existing environment to v1, follow the [v0 to v1 migration guide](https://github.com/PrimeIntellect-ai/verifiers/blob/main/skills/create-environments/SKILL.md).

## About

Environments Hub is a community-powered platform for aggregating and showcasing environments, both for RL training and downstream evaluation.
You can view all available environments on the [Environments Hub](https://app.primeintellect.ai/dashboard/environments).\


### Motivation

There are a few inter-related issues we see with the current ecosystem for both evals and RL environments which we're aiming to address with the Environments Hub:

* Despite the rapidly growing interest in training LLMs with RL, there is currently **no established community platform** for exploring and sharing train-ready environments.
* Environment implementations are often tied to a specific training RL stack and can be difficult to adapt to a new trainer.
* Popular evaluation suites (lm\_eval, lighteval, openbench, simple-evals, HELM) offer convenient entrypoints into many single-turn Q\&A evals, but these suites generally **lack support for tasks which are agentic in nature** or require complex infrastructure setups (TAU-bench, TerminalBench, SWE-bench), resulting in a proliferation of independent eval repos without shared entrypoints or specs.
* RL environments and agent evals are **basically the same thing** (dataset + harness + scoring rules), but current open-source efforts generally treat them as fundamentally separate.
* Realistic agent environments can be complex pieces of software requiring **dependencies and versioning**, and are ill-served by monorepo structures for environment collections which can quickly become unmaintainable.

With the Environments Hub, we’ve built a community platform that doubles as a proper Python package registry. Environments are modules which declare dependencies in a `pyproject.toml` and are distributed as wheels. By adopting the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) spec, development efforts can focus on task-specific components (datasets, tools or harnesses, reward functions) and automatically leverage existing infrastructure for running evaluations or [training models with RL](https://github.com/primeintellect-ai/prime-rl).

### Resources

* **Prime CLI**: Command-line tool to install, upload and manage environments
  * Github: [https://github.com/PrimeIntellect-ai/prime](https://github.com/PrimeIntellect-ai/prime)
  * Documentation: [CLI Overview](/cli-reference/introduction)
* **Verifiers**: A library of modular components for creating RL environments and training LLM agents.
  * Github: [https://github.com/PrimeIntellect-ai/verifiers](https://github.com/PrimeIntellect-ai/verifiers)
  * Documentation: [Verifiers Docs](/verifiers/overview)
* **Prime RL**: A library for large-scale RL training with FSDP.
  * Github: [https://github.com/PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)

### Support

Join the [Prime Intellect discord](https://discord.gg/primeintellect) to discuss, share feedback, and ask any questions.

> **[Create Environment](/tutorials-environments/create)**
>
> Build and upload custom environments to the registry

> **[Install Environment](/tutorials-environments/install)**
>
> Install and use environments in your projects


# Evaluating Environments
Source: https://docs.primeintellect.ai/tutorials-environments/evaluating

Guide to running evaluations with Prime CLI using Prime Inference or custom model endpoints

You can evaluate environments against various language models through Prime Inference or other OpenAI-compatible providers. Run evaluations locally with verifiers' `vf-eval`, or launch them as hosted evaluations on the platform with `prime eval run --hosted`. This guide covers both workflows, model selection, and best practices.

## Trace storage and legacy results

Local files, legacy evaluation samples, and Prime Traces are different storage paths. The `vf-eval -s` / `prime eval push` examples below describe legacy evaluation results; they do not convert those results into verifiers v1 traces. For v1 trace and episode uploads, use [Uploading traces](/traces/uploading).

Platform runs that upload to Prime Traces incur [ingestion and storage charges](/traces/pricing-and-settings). Review workspace upload settings before running evaluations. Existing legacy samples must be [exported before November 6, 2026](/traces/legacy-storage).

## Quick Start: Running Your First Evaluation

### Prerequisites

1. **Python 3.10–3.13** — Required for the Prime CLI and verifiers
2. **Install Prime CLI** — Follow the [installation guide](/cli-reference/introduction)
3. **Set up API keys** — Configure your Prime API key via `prime login`; if you plan to use another provider, also export the provider key you will reference with `--api-key-var`
4. **Install an environment** — Use `prime env install owner/environment`

### Basic Evaluation

```bash theme={null}
# List available environments
prime env list --owner primeintellect

# Install an environment
prime env install primeintellect/reverse-text@latest

# Run a basic evaluation
uv run vf-eval reverse-text -s
```

> [!TIP]
>
> Local runs are not uploaded automatically. With `-s`, results are saved under `outputs/evals/`; upload a run with `prime eval push <run-dir> --env <owner>/<name>` (see [Push Local Results](#push-local-results)).

### Hosted Evaluation Quick Start

If your environment is already published to the Environments Hub, you can run it remotely on Prime-managed infrastructure:

```bash theme={null}
prime eval run primeintellect/gsm8k --hosted
```

Use `--follow` to stream hosted logs until completion:

```bash theme={null}
prime eval run primeintellect/gsm8k --hosted --follow
```

See [Hosted Evaluations](/tutorials-environments/hosted-evaluations) for the full dashboard and CLI workflow.

## Running Evaluations from the CLI

### Basic Syntax

```bash theme={null}
# Local run
uv run vf-eval ENVIRONMENT [OPTIONS]

# Hosted run
prime eval run ENVIRONMENT --hosted [OPTIONS]
```

Local runs use verifiers' `vf-eval` from your project's environment (for example a `prime lab setup` workspace). It takes the options below, but it does not install environments: install them first with `prime env install`. `vf-eval` runs v0 environments; to run a verifiers v1 package locally, use `uv run eval` (see [Verifiers v1 evaluation](/verifiers/v1/evaluation)). `prime eval ENVIRONMENT --hosted` is a shorthand for `prime eval run ENVIRONMENT --hosted`.

### Available Models

To see all available models for evaluation:

```bash theme={null}
prime inference models
```

**Example models:**

| Model | Notes |
| - | - |
| `openai/gpt-4.1-mini` | Fast, cost-effective (default) |
| `openai/gpt-4.1` | Higher quality |
| `anthropic/claude-sonnet-4.5` | Strong reasoning |
| `meta-llama/llama-3.3-70b-instruct` | Open-weight, balanced |
| `deepseek/deepseek-r1-0528` | Advanced reasoning |
| `qwen/qwen3-235b-a22b-instruct-2507` | Large MoE model |
| `google/gemini-2.5-flash` | Fast multimodal |

> [!NOTE]
>
> Model availability and pricing may change. Always run `prime inference models` to get the current list with pricing.

### Core Parameters

**environment** `string` *required*

Environment to evaluate. Supported forms:

* **Full slug** (e.g., `primeintellect/gsm8k`) — recommended for hosted runs and Hub environments
* **Short name** (e.g., `gsm8k`) — local-first resolution for installed environments
* **TOML config path** (e.g., `configs/eval/gsm8k.toml`) — for config-driven runs

**--model, -m** `string`

Model to use for evaluation. Default: `openai/gpt-4.1-mini`

See `prime inference models` for all available models.

**--num-examples, -n** `integer`

Number of examples to evaluate. Default: 5

**--rollouts-per-example, -r** `integer`

Number of rollouts per example for statistical significance. Default: 3

### Advanced Options

**--max-concurrent, -c** `integer`

Maximum concurrent requests to the inference API. Default: 32

**--max-retries** `integer`

Maximum number of automatic retries with exponential backoff when rollouts fail due to transient infrastructure errors (e.g., sandbox timeouts, API failures).

**--max-tokens, -t** `integer`

Maximum tokens to generate per request. If unset, uses model default.

**--temperature, -T** `number`

Sampling temperature (0.0–2.0). Higher values = more randomness.

**--sampling-args, -S** `string`

JSON string with additional sampling arguments.

Example: `'{"enable_thinking": false, "max_tokens": 256}'`

**--env-args, -a** `string`

Environment-specific arguments as JSON.

Example: `'{"difficulty": "hard"}'`

### Output and Storage Options

**--verbose, -v** `boolean`

Enable verbose output for detailed logging.

**--save-results, -s** `boolean`

Save evaluation results to `outputs/evals/` (local runs).

**--save-every, -f** `integer`

Save dataset every N rollouts. Useful for checkpointing. Default: 1

**--save-to-hf-hub, -H** `boolean`

Save results to Hugging Face Hub.

**--hf-hub-dataset-name, -D** `string`

Specify Hugging Face Hub dataset name.

### Hosted Evaluation Options

**--hosted** `boolean`

Run the evaluation on the platform instead of locally. Requires a published environment.

**--follow** `boolean`

Follow hosted evaluation status and stream logs until completion. Only valid with `--hosted`.

**--poll-interval** `number`

Polling interval in seconds for hosted status and log streaming. Only valid with `--hosted`.

**--timeout-minutes** `integer`

Optional timeout in minutes for a hosted evaluation. Default: 1440 (24 hours). Min: 120. Max: 1440.

**--allow-sandbox-access** `boolean`

Allow sandbox read/write access for hosted evaluations.

**--allow-instances-access** `boolean`

Allow hosted evaluations to create and manage instances.

**--allow-tunnel-access** `boolean`

Allow hosted evaluations to create and manage tunnels from inside the
sandbox. This adds tunnel scopes to the temporary `PRIME_API_KEY`.

**--custom-secrets** `string`

JSON object of additional secrets to inject into a hosted run.

**--eval-name** `string`

Custom display name for a hosted evaluation.

`--api-base-url` and `--api-key-var` from the model configuration section also work with `--hosted`. When you use a custom endpoint for a hosted run, provide the referenced API key inside the remote sandbox via an environment secret or `--custom-secrets`.

## End-to-End Example

Here's a complete workflow from installation to viewing results:

```bash theme={null}
# 1. Install an environment
prime env install primeintellect/reverse-text@latest

# 2. Run evaluation with a fast model (20 examples, 2 rollouts each) and save results
uv run vf-eval reverse-text -m openai/gpt-4.1-mini -n 20 -r 2 -s

# 3. Upload the run to the platform
prime eval push outputs/evals/reverse-text--openai--gpt-4.1-mini/<run-id> --env primeintellect/reverse-text

# 4. List your evaluation runs
prime eval list

# 5. Get details of a specific evaluation
prime eval get <eval-id>

# 6. View samples from an evaluation
prime eval samples <eval-id>
```

**Example output:**

```
Running evaluation: reverse-text
Model: openai/gpt-4.1-mini
Examples: 5 | Rollouts: 1 | Concurrency: 32

Progress: 100%|████████████████████████████| 5/5 [00:12<00:00, 2.45s/it]

Results:
  Average Score: 0.80
  Total Samples: 5
  Successful: 4
  Failed: 1

Results saved to: outputs/evals/reverse-text--openai--gpt-4.1-mini/...
```

## Using TOML Configs for Multi-Environment Evals

For reproducible evals, you can pass a TOML config file instead of individual CLI flags:

```bash theme={null}
uv run vf-eval configs/eval/my-benchmark.toml
```

Local TOML configs can define one or more `[[eval]]` entries, which makes them useful for benchmark suites and multi-environment comparisons:

```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20

[[eval]]
env_id = "primeintellect/gsm8k"
rollouts_per_example = 2

[[eval]]
env_id = "primeintellect/alphabet-sort"
rollouts_per_example = 2
```

When you use a TOML config, per-eval settings in `[[eval]]` override global defaults at the top of the file.

For the full config schema, precedence rules, and advanced options like ablations and endpoint registries, see [Verifiers Evaluation](/verifiers/legacy/evaluation).

## Hosted Evaluations from the CLI

Hosted eval runs are useful when you want the platform to execute the environment remotely and keep logs on the platform.

```bash theme={null}
prime eval run primeintellect/gsm8k \
  --hosted \
```

Hosted runs can also target a custom OpenAI-compatible endpoint:

```bash theme={null}
prime eval run primeintellect/gsm8k \
  --hosted \
  -m openai/gpt-4.1-mini \
  --api-base-url https://api.openai.com/v1 \
  --api-key-var OPENAI_API_KEY \
  --custom-secrets '{"OPENAI_API_KEY":"..."}'
```

When `--api-base-url` is set on a hosted run, Prime still hosts the evaluation sandbox, but model billing comes from your external provider instead of Prime Inference. See [Hosted Evaluations](/tutorials-environments/hosted-evaluations) for the full hosted billing details.

You can also launch a hosted run from a TOML file:

```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20
rollouts_per_example = 2

[[eval]]
env_id = "primeintellect/gsm8k"
env_args = { split = "test" }
```

```bash theme={null}
prime eval run configs/eval/gsm8k-hosted.toml --hosted
```

Hosted run management commands:

```bash theme={null}
prime eval logs <eval-id> -f
prime eval stop <eval-id>
```

## Managing Evaluation Results

### List Evaluations

```bash theme={null}
# List all your evaluations
prime eval list

# Filter by environment
prime eval list --env gsm8k

# Output as JSON
prime eval list --output json

# Paginate results
prime eval list --num 20 --page 2
```

### Get Evaluation Details

```bash theme={null}
# Get full details of an evaluation
prime eval get <eval-id>

# Pretty-print output
prime eval get <eval-id> --output pretty
```

### View Samples

```bash theme={null}
# Get samples from an evaluation
prime eval samples <eval-id>

# Paginate samples
prime eval samples <eval-id> --page 2 --num 50
```

### Push Local Results

Local `vf-eval` runs are not uploaded automatically. Push a saved run to view it on the platform. Uploads need an environment on the Environments Hub: when the run used a short or local environment name, pass the Hub slug with `--env`.

```bash theme={null}
# Push a specific run
prime eval push outputs/evals/reverse-text--openai--gpt-4.1-mini/<run-id> --env primeintellect/reverse-text

# Auto-discover and push runs under outputs/evals/
# (finds short-name runs; for owner/name runs, pass the run directory)
prime eval push --env primeintellect/reverse-text
```

## Model Selection Guide

When choosing models for evaluation, consider:

* **Task complexity** — Harder tasks benefit from larger, reasoning-capable models
* **Cost** — Smaller models are significantly cheaper for large-scale evals
* **Throughput** — Some models handle high concurrency better than others

### Self-hosting with vLLM

> [!NOTE]
>
> For most users, we recommend Prime Inference for easier setup. Consider self-hosting only for specialized requirements or very large-scale evaluations.

Self-hosting makes sense when you:

* Need specific model variants or custom fine-tuned models
* Require maximum cost efficiency for very large evaluations (1M+ examples)
* Are testing smaller models not available via API

#### Configuring for Self-Hosted Models

```bash theme={null}
# Point to your vLLM instance
uv run vf-eval reverse-text \
  -m Qwen/Qwen3-4B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  --api-key-var CUSTOM_API_KEY \
```

#### Recommended Self-Hosted Models

**High Performance (MoE with small active parameters):**

```bash theme={null}
uv run vf-eval reverse-text \
  -m Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  -n 2000 -c 96
```

**Balanced Performance:**

```bash theme={null}
uv run vf-eval wordle \
  -m Qwen/Qwen3-4B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  -n 5000 -c 128
```

#### vLLM Configuration Example

```bash theme={null}
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --max-model-len 32768 \
  --max-num-seqs 256 \
  --tensor-parallel-size 2 \
  --quantization fp8
```

## Environment-Specific Dependencies

Some environments may require additional dependencies beyond the base installation. Check the environment's documentation or use `prime env info` to see requirements:

```bash theme={null}
prime env info primeintellect/math_python
```

If an environment needs specific packages (e.g., sympy for math verification), install them before running evaluations.

## Troubleshooting

### Common Issues

> **Rate Limiting**
>
> **Error:** "Rate limit exceeded" or 429 responses
>
> **Solution:** Reduce concurrency
>
> ```bash theme={null}
> uv run vf-eval reverse-text -n 100 -c 8
> ```

> **Insufficient Balance**
>
> **Error:** "Insufficient balance" or payment required
>
> **Solution:** Add funds to your Prime Intellect account or use cheaper models
>
> ```bash theme={null}
> uv run vf-eval reverse-text -m openai/gpt-4.1-mini -n 1000
> ```

> **Environment Not Found**
>
> **Error:** Environment not found or installation failures
>
> **Solution:** Verify environment exists and reinstall
>
> ```bash theme={null}
> prime env list --owner primeintellect
> prime env uninstall reverse-text
> prime env install primeintellect/reverse-text@latest
> ```

> **Python Version Issues**
>
> **Error:** Installation failures or import errors
>
> **Solution:** Ensure you're using Python 3.10–3.13
>
> ```bash theme={null}
> python --version  # Should be 3.10, 3.11, 3.12, or 3.13
> ```

### Performance Tips

1. **Start Small:** Begin with `-n 5` to test your setup
2. **Monitor Costs:** Check token usage before large evaluations
3. **Use Appropriate Models:** Match model capability to task complexity
4. **Optimize Concurrency:** Balance speed vs. rate limits (default 32 is usually good)
5. **Save Results:** Pass `-s` to save local results, then upload them with `prime eval push <run-dir> --env <owner>/<name>`

### Integration with Other APIs

You can use other OpenAI-compatible API providers:

```bash theme={null}
# DeepSeek API
uv run vf-eval math \
  -m deepseek-reasoner \
  --api-base-url https://api.deepseek.com/v1 \
  --api-key-var DEEPSEEK_API_KEY

# OpenRouter API
uv run vf-eval reverse-text \
  -m meta-llama/llama-3.1-405b-instruct \
  --api-base-url https://openrouter.ai/api/v1 \
  --api-key-var OPENROUTER_API_KEY
```

## Next Steps

> **[Environments Hub](/tutorials-environments/environments)**
>
> Learn more about creating and managing environments

> **[Inference API Reference](/api-reference/inference-models)**
>
> Detailed API documentation for inference endpoints

> **[Prime CLI Reference](/cli-reference/introduction)**
>
> Complete CLI command reference

> **[Creating Environments](/tutorials-environments/create)**
>
> Build your own evaluation environments


# Getting Started
Source: https://docs.primeintellect.ai/tutorials-environments/getting-started

Create, manage and share environments for reinforcement learning and evaluation

To interact with environments on the Environments Hub, you'll need to set up your account on Prime Intellect, and install the Prime CLI.

### Prerequisites

Ensure you have:

1. **Prime CLI** installed and configured
   * See [CLI Overview](/cli-reference/introduction) for setup instructions.
2. **Username** set on your [profile](https://app.primeintellect.ai/dashboard/profile)
3. **Authenticate** the CLI:
   ```bash theme={null}
   prime login
   ```

### Quick Overview

```bash theme={null}
# List available environments
prime env list

# Get information about an environment
prime env info owner/environment-name

# Install an environment
prime env install owner/environment-name

# Create a new environment (verifiers starter template)
uvx --from verifiers vf-init my-new-environment
```

> **[Create Environment](/tutorials-environments/create)**
>
> Build and upload custom environments to the registry

> **[Install Environment](/tutorials-environments/install)**
>
> Install and use environments in your projects


# Hosted Evaluations
Source: https://docs.primeintellect.ai/tutorials-environments/hosted-evaluations

Run evaluations on the platform from the dashboard or the Prime CLI

Hosted Evaluations run your environment on Prime-managed infrastructure and store the run in Prime Evals. You can launch them either from the Environments Hub UI or directly from the CLI with `prime eval run --hosted`.

## What hosted evaluations are for

Use hosted evaluations when you want Prime to handle the execution environment for you:

* Run a published environment without setting up local Python dependencies
* Evaluate large jobs against a Hub environment slug
* Monitor logs remotely and share runs through the platform
* Grant temporary sandbox, instance, or tunnel permissions for tool-using environments

> [!NOTE]
>
> Hosted evaluations require an environment that is already published to the Environments Hub. If you only have a local environment, push it first with `prime env push`.

## Prerequisites

Before running a hosted evaluation, make sure you have:

1. **A published environment** on the Environments Hub
   ```bash theme={null}
   prime env push
   ```
2. **Write access** to that environment
3. **Prime CLI installed and authenticated** if you plan to use the CLI flow
4. **Billing configured for your chosen inference path**
   * Prime account balance if you are using Prime Inference
   * Or an external provider API key if you are using `--api-base-url` with a custom OpenAI-compatible endpoint

## Trace storage

Hosted evaluation trace uploads use Prime Traces when enabled for the workspace. Review [pricing and upload settings](/traces/pricing-and-settings) before launching. Trace ingestion and storage are separate from the inference or sandbox charges described below.

View stored traces in the [Traces dashboard](https://app.primeintellect.ai/dashboard/traces). For older sample data, follow the [legacy export instructions](/traces/legacy-storage) before November 6, 2026.

## Quick start with the CLI

The new hosted eval flow is built into `prime eval run`.

```bash theme={null}
prime eval run primeintellect/gsm8k --hosted
```

This creates a hosted run on the platform instead of executing the evaluation locally.

### Follow logs until completion

```bash theme={null}
prime eval run primeintellect/gsm8k --hosted --follow
```

With `--follow`, the CLI keeps polling the run, streams hosted logs, and exits when the evaluation reaches a terminal state.

### Run from a TOML config

Hosted evals also support TOML configs.

```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20
rollouts_per_example = 2

[[eval]]
env_id = "primeintellect/gsm8k"
env_args = { split = "test" }

[[eval]]
env_id = "primeintellect/alphabet-sort"
```

Run it with:

```bash theme={null}
prime eval run configs/eval/benchmark-hosted.toml --hosted
```

### Use a custom OpenAI-compatible endpoint

Hosted evaluations can also run on Prime-managed infrastructure while sending model requests to your own OpenAI-compatible endpoint.

```bash theme={null}
prime eval run primeintellect/gsm8k \
  --hosted \
  -m openai/gpt-4.1-mini \
  --api-base-url https://api.openai.com/v1 \
  --api-key-var OPENAI_API_KEY \
  --custom-secrets '{"OPENAI_API_KEY":"..."}' \
  --follow
```

Use `--api-key-var` to name the environment variable that contains your provider key inside the hosted sandbox. Then provide that secret either through the environment's stored secrets or with `--custom-secrets` for a one-off run.

> [!NOTE]
>
> `--api-base-url` only changes the inference endpoint. The environment still runs inside a Prime-hosted sandbox.

## Hosted-only CLI options

These flags only apply when you pass `--hosted`:

| Flag | Description |
| - | - |
| `--follow` | Stream hosted logs and wait for completion |
| `--poll-interval` | Polling interval for hosted status/log streaming |
| `--timeout-minutes` | Optional timeout in minutes for the hosted run. Default: 1440 (24 hours). Min: 120. Max: 1440. |
| `--allow-sandbox-access` | Allow sandbox read/write access |
| `--allow-instances-access` | Allow instance creation and management |
| `--allow-tunnel-access` | Allow hosted evaluations to create and manage tunnels from inside the sandbox |
| `--custom-secrets` | JSON object of secrets injected for the hosted run |
| `--eval-name` | Custom display name for the hosted evaluation |

### Example: Environment Args and Custom Secrets

```bash theme={null}
prime eval run my-team/browser-agent \
  --hosted \
  -m anthropic/claude-sonnet-4.5 \
  -a '{"task":"checkout"}' \
  --custom-secrets '{"SHOP_API_KEY":"..."}' \
  --allow-sandbox-access \
  --timeout-minutes 45
```

Use `--custom-secrets` for run-specific values. Secrets already configured on the environment continue to work as usual.

> [!NOTE]
>
> Hosted eval `env_args` are passed to `load_environment()`, similar to training `[[env]].args`. Use them for custom environment settings such as `split`, `difficulty`, tool configuration, or other environment-specific overrides supported by your environment.

## Monitoring and managing hosted runs

After starting a hosted evaluation, the CLI prints the evaluation id and platform URL.

> [!NOTE]
>
> `--follow` only streams logs and waits for completion. Hosted evaluations do not yet have a training-style checkpoint or restart workflow.

### List evaluations

```bash theme={null}
prime eval list
```

The list output includes a **Type** column so you can distinguish `HOSTED` and `LOCAL` evaluations.

### Inspect one run

```bash theme={null}
prime eval get <eval-id>
prime eval samples <eval-id>
```

### Stream logs for an existing hosted run

```bash theme={null}
prime eval logs <eval-id> -f
```

### Stop a running hosted evaluation

```bash theme={null}
prime eval stop <eval-id>
```

## Running a hosted evaluation from the dashboard

You can still launch the same workflow from the Environments Hub UI.

### Step 1: Open the environment

1. Go to the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Open your environment
3. Go to the **Evaluations** tab
4. Click **Run Hosted Evaluation**

### Step 2: Choose a model

Select an inference model for the run.

![Model selection interface showing available inference models for hosted evaluations](https://mintcdn.com/primeintellect/65ctwFCi0zBOXZ8H/images/hosted-evals-models.png?fit=max\&auto=format\&n=65ctwFCi0zBOXZ8H\&q=85\&s=1d5da6eae6daf063e4059d7d474f6037)

### Step 3: Configure the run

Set the number of examples, rollouts per example, and any environment arguments.

![Hosted evaluation configuration form with examples, rollouts, and environment arguments](https://mintcdn.com/primeintellect/65ctwFCi0zBOXZ8H/images/hosted-evals-config.png?fit=max\&auto=format\&n=65ctwFCi0zBOXZ8H\&q=85\&s=e2b2e244b9b44c639b4f83c0b1fb46cd)

If the environment needs to expose a service during the run, expand **Permissions** and enable **Allow tunnel access for this evaluation**.

> [!NOTE]
>
> Environment secrets linked in the Hub are exposed automatically during hosted evaluation runs. You only need `--custom-secrets` when launching a CLI run with additional per-run secrets.

### Step 4: Monitor progress

You will be redirected to the evaluations list where you can watch the run status.

![Evaluations list showing hosted evaluation runs and statuses](https://mintcdn.com/primeintellect/65ctwFCi0zBOXZ8H/images/hosted-evals-runs.png?fit=max\&auto=format\&n=65ctwFCi0zBOXZ8H\&q=85\&s=fed5c6f0b4983229a4942da3eb439767)

### Step 5: Review results

Completed runs show aggregate metrics and per-sample outputs in Prime Evals.

![Hosted evaluation result page showing metrics and example-level outputs](https://mintcdn.com/primeintellect/65ctwFCi0zBOXZ8H/images/hosted-evals-success.png?fit=max\&auto=format\&n=65ctwFCi0zBOXZ8H\&q=85\&s=cc89676406f7c8db9a88a85d4700cd97)

## Failure modes

When a hosted evaluation fails, the platform surfaces the error message and logs.

Common causes:

1. **Environment code errors** — import failures, dependency issues, invalid verifier logic
2. **Missing permissions** — the run needs sandbox, instance, or tunnel access but those flags were not enabled
3. **Missing secrets** — environment-linked or custom secrets were not available
4. **Timeouts** — the run exceeded the configured or platform timeout
5. **Inference issues** — temporary provider or model errors

> [!TIP]
>
> Start with a small hosted run first, such as `-n 5 -r 1`, then scale up once logs and scores look correct.

## Pricing

Hosted evaluations support two billing modes:

* **Prime Inference (default)** — If you do not pass `--api-base-url`, the run uses Prime Inference pricing for the selected model. The hosted evaluation sandbox runtime is not billed separately.
* **Custom OpenAI-compatible endpoint (CLI/API)** — If you pass `--api-base-url`, the hosted evaluation is billed as sandbox compute on Prime while your external provider bills model tokens directly. In this mode you must provide the provider key through an environment secret or `--custom-secrets`.

In either mode, total cost still depends on:

* The selected model or endpoint pricing
* Prompt and completion token usage
* `num_examples × rollouts_per_example`
* Any extra tool usage triggered by the environment

Trace uploads also incur [Prime Traces ingestion and storage charges](/traces/pricing-and-settings), independent of the inference billing mode. For upload failures or storage limits, see [Traces troubleshooting](/traces/managing-traces#troubleshooting).

## When to use dashboard vs CLI

* Use the **dashboard** when you want the simplest point-and-click flow
* Use the **CLI** when you want reproducible commands, TOML configs, log following, or automation in scripts/CI


# Install & Use Environment
Source: https://docs.primeintellect.ai/tutorials-environments/install

Learn how to install and use environments from the Prime Intellect Environments Hub

### Basic Installation

Install an environment from the Environments Hub:

```bash theme={null}
prime env install <owner>/<environment-name>
```

### Examples

```bash theme={null}
# Install latest version
prime env install will/wordle

# Install specific version
prime env install will/wordle@0.1.3
```

### Alternative Installation Methods

Use `prime env info <owner>/<environment-name>` to see all available installation methods and usage examples for a specific environment.

E.g. for [will/wordle](https://app.primeintellect.ai/dashboard/environments/will/wordle):

```bash theme={null}
# Direct pip installation from wheel URL
pip install https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl

# UV pip installation
uv pip install https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl

# Add to existing project with UV
uv add wordle@https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl
```

## Using Installed Environments

### In Python

Once installed, you can use the environment in your Python code:

```python theme={null}
from verifiers import load_environment

# Load the environment
env = load_environment("will/wordle")

# Use the environment for evaluation or training
results = env.evaluate(examples=100, rollouts_per_example=1)
```

> [!TIP]
>
> More examples and detailed usage documentation can be found in the [verifiers documentation](https://github.com/PrimeIntellect-ai/verifiers).

## Version Management

### Working with Versions

Install specific versions:

```bash theme={null}
# Install specific version
prime env install will/wordle@0.1.3

# Install latest (default)
prime env install will/wordle
```

### Upgrading Environments

To upgrade an already installed environment, simply run `prime env install` again:

```bash theme={null}
# Upgrade to latest version
prime env install will/wordle

# Upgrade to specific version
prime env install will/wordle@0.2.0
```

## Download Source Code

If you want to inspect, modify, or contribute to an environment, you can download its source code:

```bash theme={null}
# Download the latest version
prime env pull owner/environment-name
```

**Example:**

```bash theme={null}
prime env pull will/wordle
```

## Getting Help

For additional support and examples:

* **Verifiers Documentation**: [Verifiers Docs](/verifiers/overview)
* **GitHub Repository**: [https://github.com/PrimeIntellect-ai/verifiers](https://github.com/PrimeIntellect-ai/verifiers)
* **Discord Community**: Join our [Discord](https://discord.gg/ZTFydGWPKj) for community support


# Manage Collaborators
Source: https://docs.primeintellect.ai/tutorials-environments/manage-collaborators

Add collaborators to private environments on the Environments Hub.

## Overview

You can now share private environments directly with collaborators. Collaborators have read-only access today and can browse metadata, artifacts, and version history while we finish support for writer permissions.

Access collaborator management from each environment's **Settings → Collaborators** section.

![Add collaborator modal](https://mintcdn.com/primeintellect/rTc6MQfagrrwbUJa/images/collaborator-add-modal.png?fit=max\&auto=format\&n=rTc6MQfagrrwbUJa\&q=85\&s=76d25920d5332bcbc011671ddeece6af)

## Add a collaborator

1. Open the environment in the Environments Hub and switch to the **Collaborators** tab.
2. Select **Add collaborator**.
3. Enter a Prime username or team name. Each owner and team can find these values in their profile page.
4. Confirm to grant read-only access.

Once a collaborator is listed, they can run `prime env info <environment-name>` in the Prime CLI to review environment details and pull instructions.

## Manage collaborator access

* Remove collaborators at any time from the same tab.
* Collaborators currently inherit read-only permissions; writer access is coming soon.

## Troubleshooting

* **Can't find a collaborator?** Ask them to copy the username or team name exactly as it appears in their profile.
* **Need to share broadly?** Create a team that includes everyone who needs access and add the team identifier once.
* **Seeing an error?** Reach out via our support chat with the environment slug and the identifier you attempted to add.


# Secrets
Source: https://docs.primeintellect.ai/tutorials-environments/secrets

Create and link secrets across your environments

Secrets are encrypted values injected into your environment at runtime. They are the recommended way to supply API keys and other credentials to environments.

For non-sensitive configuration, use [Environment Variables](/tutorials-environments/environment-variables) instead.

## Secret Types

There are two ways to add a secret to an environment:

1. **Direct environment secrets**
   * Created directly on a specific environment
   * Only available to that environment
2. **Linked global secrets**
   * Created once under [**Keys & Secrets**](https://app.primeintellect.ai/dashboard/tokens?tab=secrets)
   * Can be linked to one or more environments
   * Useful for credentials shared across several environments, such as judge-model keys

## Where Secrets Are Used

Secrets are injected automatically for all hosted services:

* Hosted Evaluations
* Hosted Training

You do **not** need to pass them via `--env-args`.

## Adding a Direct Secret

1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab
3. Click **Add Secret**
4. Set a **Name**, **Value**, and optional description

## Creating and Linking a Global Secret

### 1) Create a global secret

1. Go to [**Dashboard → Keys & Secrets**](https://app.primeintellect.ai/dashboard/tokens?tab=secrets)
2. Click **Add Secret**
3. Set a **Name**, **Value**, and optional description

### 2) Link it to an environment

1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab
3. Click **Link Global Secret**
4. Select the secret you want to link

Once linked, the secret is available to that environment at runtime.

## Managing via CLI

### Global secrets

```bash theme={null}
prime secret list                                   # list your global secrets
prime secret create --name MY_KEY --value sk-...    # create a global secret
prime secret update <secret-id> --value sk-new      # update value
prime secret delete <secret-id>                     # delete
```

### Environment secrets

```bash theme={null}
prime env secret list owner/my-env                          # list all secrets (direct + linked)
prime env secret create owner/my-env --name MY_KEY --value sk-...  # add a direct secret
prime env secret update owner/my-env --id <id> --value sk-new      # update a direct secret
prime env secret delete owner/my-env --id <id>             # delete a direct secret
prime env secret link <global-secret-id> owner/my-env      # link a global secret
prime env secret unlink <global-secret-id> owner/my-env    # unlink a global secret
```

## Precedence and Conflict Rules

Each environment has three types of values injected at runtime:

| Type | Encrypted | Description |
| - | - | - |
| [**Environment variables**](/tutorials-environments/environment-variables) | No | Plain-text key-value pairs, configured on the **Variables** tab under **Secrets** |
| **Linked global secrets** | Yes | Encrypted values linked from your global secrets |
| **Direct environment secrets** | Yes | Encrypted values set directly on the environment |

### Runtime precedence

If the same name appears in more than one place, the highest-priority value wins:

1. **Environment variables** (lowest)
2. **Linked global secrets**
3. **Direct environment secrets** (highest)

### Name validation

All three types share the same naming format:

* Must start with an uppercase letter
* Can contain uppercase letters, digits, and underscores only
* Example: `MY_API_KEY`

### Conflict checks

The platform prevents name collisions. When adding or linking a secret, it will be rejected if the name is already taken by:

* An existing direct secret
* Another linked secret
* An environment variable

## Scope and Access

* **Personal secrets** can be linked to your personal environments
* **Team secrets** can be linked to environments within the same team

## Troubleshooting

> **Secret not found when linking**
>
> Make sure the secret exists in the same scope (personal vs team) as the environment.

> **A direct secret with the name ... already exists**
>
> Rename or remove the existing direct secret first, then link the global secret.

> **An environment variable with the name ... already exists**
>
> Rename or remove that environment variable before adding or linking the secret.

> **Secret missing at runtime**
>
> Confirm the secret appears on the environment **Secrets** tab and that your environment code reads the correct variable name.


# Create persistent storage
Source: https://docs.primeintellect.ai/tutorials-storage/create-persistent-storage

Create storage that can be shared between instances

Persistent storage allows you to create data volumes that persist beyond the lifecycle of individual instances and can be attached to different instances as needed.

# Step-by-Step Guide

**1. Navigate to the Storage tab**

Go to the `Storage` tab in your Instances page, and click on `Create Disk`.

![Storage page](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/shared-disk-page.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=43ed59ce569010a61fbaef5d2dc59c84)

**2. Choose a provider to provision the disk and select the datacenter where the disk should be created.**

![Select storage provider](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/provision-disk-list.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=d0afb70c008c6516ffae5e2df2191235)

![Select storage datacenter](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/provision-disk-list2.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=23fc46a8f59a637e6b0bead138c8d29d)

**3. Configure disk size**

Enter your desired disk size and click `Create Disk`.

![Enter disk size](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/provision-disk-size-field.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=14bed9a0b533d2bcd812ca832f8ccc1c)

**4. Wait for disk provisioning**

The disk will begin provisioning. You can only attach disks that have an `Active` status.

![Shared disk page with disk](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/shared-disk-page-with-disk.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=1a0c4cfdea8a557133eb5c5576f3037f)


# Use persistent storage with instances
Source: https://docs.primeintellect.ai/tutorials-storage/use-persistent-storage-with-instances

How to attach and use persistent storage with instances

Attach persistent storage volumes to your instances to provide durable, shareable storage that persists beyond instance lifecycles.

# Step-by-Step Guide

**1. Prerequisites**

* An active persistent disk with `Active` status
* Available instances in the same provider and location as your disk

![Storage page](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/shared-disk-page-with-disk.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=1a0c4cfdea8a557133eb5c5576f3037f)

**2. Filter instances by your existing disks**

Go to `Deploy GPU Instance`. Use the `Filter by your existing disks` toggle to show only instances in locations where you have persistent disks available.

![Filter by existing disks](https://mintcdn.com/primeintellect/VH4Zhq0MPOuc1_ND/images/filter-by-existing-disks.png?fit=max\&auto=format\&n=VH4Zhq0MPOuc1_ND\&q=85\&s=f581ad2b445e07c0a0543bca2774a569)

**3. Select a compatible provider**

Look for providers with the `Add Shared Filesystem` button, indicating disk compatibility.

![Instance with disk](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/provider-with-disk.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=fd935a45dd07945233164a33076949ab)

> [!NOTE]
>
> Persistent disks are location-specific. Ensure you're selecting instances in the same provider and datacenter as your disk.

**4. Select your existing disk and provision the instance**

![Provision with disk](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/provision-instance-with-disk.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=1cc7f4906d6efb7b2146885aa61e7d7e)

**5. After the instance deploys, check the instance details to find your shared disk's mounting path. The same disk can be attached to multiple instances simultaneously.**

![Instances with disks](https://mintcdn.com/primeintellect/u4gd9w_CirhPniY2/images/shared-disk/instances-with-shared-disks.png?fit=max\&auto=format\&n=u4gd9w_CirhPniY2\&q=85\&s=7eb6173b324776876b270f21deefcd48)


# Overview
Source: https://docs.primeintellect.ai/verifiers/overview



> **[GitHub Repository](https://github.com/PrimeIntellect-ai/verifiers)**
>
> Verifiers Repo

verifiers is a framework for defining tasks, running agents and harnesses, scoring them on set tasks, and using those for evaluations and reinforcement learning.

## Documentation

* **[verifiers v1](/verifiers/v1/overview)** — The documentation for verifiers v1 (`verifiers.v1`), built around tasksets, harnesses, and traces. This is what environments should target. The legacy v0 stack (`import verifiers as vf`) has been removed.


# Architecture
Source: https://docs.primeintellect.ai/verifiers/v1/architecture



verifiers is built out of the following parts:

A server-backed evaluation or prime-rl **orchestrator** creates worker processes and distributes rollout requests among them. The client owns the taskset. It loads the tasks once and ships each task to the workers, which owns the runtime containing the agent(s) to produce a trace out of the data.

The orchestrator and workers are managed by verifiers and prime-rl themselves and thus offer few configurable knobs.

The **rollout** is the executable combination of one loaded task, the harness, and any tools. Each rollout has an independent trace and runtime state. verifiers has several runtimes which you can use for most tasksets:

* The `subprocess` runtime runs the rollouts in Python subprocesses locally. Thus, it is meant for debugging purposes, as there might be side effects during runtime, such as one subprocess altering the config files of the harness, which then affects the other subprocesses.
* The `docker` runtime runs the rollouts in docker containers on your local machine.
* The `podman` runtime is the same implementation driving the Podman CLI, for hosts without Docker.
* The `apptainer` runtime runs the rollouts in unprivileged Apptainer instances on the host network, as on HPC clusters. It has no egress policy.
* Sandbox runtimes, such as `prime`, `modal`, or `e2b`, are meant for production, especially for training or higher concurrency evaluation. These runtimes run remotely.

For offline Docker/Podman use on Linux, cache the task image and, if it lacks Python 3, `docker.io/library/python:3.11-alpine` for host callbacks. Restricted execution also needs the cached `localhost/verifiers-network:1` image, built during the first online startup.

The harness runs inside the rollout runtime to interact with the taskset. The harness does *not* call the provider endpoint directly. Instead, model traffic goes through an **interception server** over a local connection or [Prime Tunnel](https://docs.primeintellect.ai/sandboxes/tunnel).

The interception server receives all these requests and then sends them over to the actual API, e.g. the OpenAI responses endpoint. It uses the endpoint that the harness expects, so Codex will use OpenAI Responses, while Claude Code will use the Anthropic Messages API.

The interception server, however, allows several things beyond just replaying the correct API response:

* Traces are built live, thus allowing the collection of the trajectories as they happen
* Setting sampling parameters in harnesses that don't necessarily expose those settings
* Intercepting and rewriting tool responses or server-side web search results to block reward hacks


# Env
Source: https://docs.primeintellect.ai/verifiers/v1/env



An `Env` defines the control flow between `Agents`. In the simplest case, it is just a `SingleAgentEnv` where a single agent solves a task from a taskset.

Its core signature is `Env.run(task: Task, agents: Agents)` -> None — it is passed an initial task and pre-initialized agents and then programs the full multi-agent control flow; every finished agent run automatically joins the resulting `Episode`, which holds all the traces of all the agents.

```python theme={null}
class Env(ABC):
    @abstractmethod
    async def run(self, task: Task, agents: Agents) -> None:
        """Run a single multi-agent episode."""
        ...
```

verifiers comes with different pre-built `Env`s to use:

* `IsolatedVerifierEnv` runs one solver, transfers only declared artifacts into a
  fresh configured runtime, and runs deterministic task scoring there.
* The `AgenticJudgeEnv` defines the sequential interaction between a solver and judge agent. The judge can re-use the same runtime after the solver (`SharedAgenticJudgeEnv`) or use its own, new runtime `IsolatedAgenticJudgeEnv`.
* The `UserSimEnv` models users as agents, and the episode is a turn-by-turn conversation between the user and assistant agents.
* The `BestOfNEnv` runs n independent attempts at the same task, then marks which attempt achieved the highest reward (best) and whether any attempt crossed a success threshold (pass\_at\_n), which is useful for rejection sampling and pass\@k evaluation.

## Isolated deterministic verification

Select `--env.id isolated-verifier` when the task's score must not run in the
solver's sandbox. It is still a one-agent run: the environment records one solver
trace and starts no verifier agent, model, or harness.

```bash theme={null}
uv run vf-eval my-task --env.id isolated-verifier --env.agent.runtime.type docker
```

Task authors use the existing task API. Declare every solver output the verifier
needs in `TaskData.artifacts`, then implement deterministic `@vf.reward` and
`@vf.metric` methods. A runtime parameter makes the fresh verifier box available:

```python theme={null}
from pathlib import Path

import verifiers.v1 as vf


PRIVATE_TESTS = Path("tests/test_solution.sh").read_bytes()


class CodeTask(vf.Task[CodeData]):
    async def stage_verifier(self, runtime: vf.Runtime) -> None:
        await runtime.write("/tmp/test_solution.sh", PRIVATE_TESTS)

    @vf.reward
    async def tests(self, runtime: vf.Runtime) -> float:
        result = await runtime.run(["bash", "/tmp/test_solution.sh"], {})
        return float(result.exit_code == 0)


task = CodeTask(
    CodeData(
        prompt="Fix the implementation.",
        artifacts=[vf.Artifact(source="src")],
    )
)
```

The lifecycle is fixed:

1. The task and harness run normally, including task `finalize` and harness metrics,
   but task metrics and rewards are deferred.
2. The environment collects the declared paths and `/logs/artifacts`, then destroys
   the solver runtime.
3. It creates a fresh task controller and provisions either the same resolved
   container/runtime policy or the independently configured verifier runtime, runs
   task `setup`, restores the artifacts at their original paths, runs task
   `stage_verifier`, reapplies the execution network policy, and runs task metrics
   and rewards onto the solver trace.

The verifier runtime must be Docker, Prime, or another container runtime; absolute
artifact restoration is intentionally refused on the host subprocess runtime.
By default the verifier uses the solver's resolved runtime policy. Set
`--env.verifier.runtime.*` to independently choose its runtime type, image, resources,
and network policy; `--env.verifier.env` can independently set its process environment.
Relative artifacts require matching solver and verifier workdirs because artifacts are
restored without path translation; absolute artifacts permit different workdirs.
Configured model-backed task judges are rejected: use deterministic metrics/rewards
here, or an agentic/judge environment when a model must judge the result.
`--env.verifier.retries` retries fresh verifier attempts after setup, restoration,
staging, or scoring failures (default: 2).


# Evaluation
Source: https://docs.primeintellect.ai/verifiers/v1/evaluation



To evaluate any taskset, use the `eval` entrypoint:

```bash theme={null}
uv run vf-eval primeintellect/terminal-bench-2
```

You can also use `.toml` files for configuration:

```toml theme={null}
model = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B"

[sampling]
temperature = 1.0

[env.taskset]
id = "primeintellect/terminal-bench-2"

[env.agent.harness]
id = "codex"
version = "0.116.0"

[env.agent.runtime]
type = "docker"
```

Validate the config by using `uv run vf-eval @ config.toml --dry-run`. To run the evaluation, use `uv run vf-eval @ config.toml`.

Use dotted arguments to set values using the CLI, e.g. `--sampling.temperature 0.5`. CLI arguments overwrite toml arguments when both are present.

The output from evaluations are written into `outputs/<env>--<model>--<harness>/<uuid>/` by default, where `<env>` is the taskset, prefixed by the paired env id when `--env.id` sets one (use `output_dir` to overwrite the folder). The folder contains the used `config.toml`, all the episodes in `traces.jsonl`, as well as logs of the run and workers in `logs/attempt_<n>/eval.log` — one directory per launch attempt (a resume starts a new one), with `logs/latest` pointing at the current attempt.

## Common config values

* `model` — the model id to evaluate, e.g. `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B`
* `sampling` — generation params passed to the model, e.g. `sampling.temperature`
* `env.taskset.id` — pick the taskset (or the positional `eval <taskset-id>`)
* `env.agent.harness.id` — pick the agent's harness (`[env.agent.harness]` in TOML)
* `select` — which tasks to evaluate (see [Selecting tasks](/verifiers/v1/tasksets#selecting-tasks)).
  `-n` sets `select.limit`: not setting it means all tasks, and an infinite taskset (a
  procedural generator, e.g. `wordle`) requires it. `-s` sets `select.shuffle` (fixed seed)
* `num_rollouts` — rollouts per task
* `verbose` — log at debug instead of info
* `rich` — the live dashboard (default); `--no-rich` streams logs to the console and
  prints each trace as JSON at the end
* `rich.show_logs` — replace the dashboard's per-rollout rows with a live tail of the
  attempt's logs (`logs/latest/eval.log`), the env workers' lines included
* `push` — upload the run to the Prime Intellect platform as it goes (default; `--no-push`
  keeps it local). `run.attach <evaluation-id>` streams into a run the platform already
  created instead of opening a new one — this is how a hosted evaluation's sandbox runs
  the same command; it is not something a local run sets by hand

## Resuming evaluations

`--resume <output-dir>` re-runs only the rollouts a previous run left missing or errored, appending to that run's own `traces.jsonl`. It reloads the run's saved `config.toml` verbatim, so it takes no other arguments. Good rollouts are kept, while errored ones are dropped and redone.

## Disabling tools

Almost every harness comes with a `disabled_tools` list, which can be used to disable one or multiple tools:

```toml theme={null}
[env.agent.harness]
disabled_tools = ["shell_tool"]
```

The names of these tools are set by the respective harness. Consult the relevant documentation for the given harness for the relevant name(s). Some harnesses do not offer support to disable tools.

## Skills

Harnesses whose program supports SKILL.md skills natively (e.g. Claude Code, Codex) take a `skills` list. A local skill folder is uploaded to `<skills dir>/<folder name>`. A `{runtime = "..."}` entry copies the contents of a directory already inside the runtime into the program's skill discovery directory:

```toml theme={null}
[env.agent.harness]
skills = [{runtime = "/opt/skills"}, "path/to/my-skill"]
```

Tasks can supply the same sources through `TaskData.skills`, for example `skills=[{"runtime": "/opt/skills"}]`. Task sources are installed first, followed by harness sources; later files override matching earlier files. Each run gets its own installed skills. Sources are resolved after task setup, and missing source directories fail the run.

Setting `skills` on a task or harness without native harness skill support fails up front.


# Gepa
Source: https://docs.primeintellect.ai/verifiers/v1/gepa



verifiers offers built in support for [GEPA](https://github.com/gepa-ai/gepa), an algorithm that optimizes a system prompt to maximize the downstream reward for a given taskset:

```bash theme={null}
uv run vf-gepa reverse-text
```

`gepa` runs GEPA where a number of rollouts are done before a teacher LLM reflects on the results to propose a better `Task.system_prompt` without any gradient based training. It runs against native v1 tasksets.

GEPA reuses the same `env` (taskset + agent) / `client` / `sampling` config as eval, so the `.toml` config remains very similar:

```toml theme={null}
model = "deepseek/deepseek-v4-flash"

[env.taskset]
id = "reverse-text"

[env.agent.harness]
id = "bash"

[sampling]
temperature = 1.0
```

Validate the config by using `uv run vf-gepa @ config.toml --dry-run`. To run GEPA, use `uv run vf-gepa @ config.toml`. CLI arguments overwrite toml arguments when both are present.

## Common config values

* `model` / `-m` — model for the rollouts under optimization (default: `deepseek/deepseek-v4-flash`, same as eval)
* `reflection_model` / `reflection_client` — model/endpoint that proposes new prompts (default: reuse `model` / `client`)
* `num_train` / `num_val` — train tasks for reflection minibatches and held-out val tasks for the pareto frontier (defaults: 100 / 50)
* `max_total_rollouts` — total rollouts the run may spend (default: 500)
* `max_concurrent` / `-c` — caps how many episodes are in flight at once (default: 128)

## Output

Results go under `outputs/<env>--<model>--<harness>/<uuid>/`, matching `eval`.
The best system prompt is printed when the run finishes and written to `best_system_prompt.txt` in that folder.

Hand it back to eval or training via the config-layer taskset system prompt:

```bash theme={null}
uv run vf-eval reverse-text \
  --env.taskset.system-prompt outputs/<run>/best_system_prompt.txt
```

## Limitations

**Tasksets** — GEPA optimizes `Task.system_prompt`, so the taskset must provide one. Tasksets that bake instructions into the user `prompt` instead (e.g. `gsm8k`) are not supported out of the box.

**Harnesses** — any eval harness works. With `APPENDS_SYSTEM_PROMPT`, the optimized prompt is used as a system message but otherwise is folded into the user prompt.


# Getting Started
Source: https://docs.primeintellect.ai/verifiers/v1/getting_started



verifiers runs locally with `uv`. Install it, clone the repo, and sync dependencies:

```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/PrimeIntellect-ai/verifiers.git
cd verifiers
uv sync
```

You can now run tasksets directly, e.g. `uv run vf-eval <taskset-id>`, and scaffold new ones with `uv run vf-init <name>`.

## Skills

To equip your agent with the necessary knowledge, we highly recommend the skills in this repository's [`skills/`](https://github.com/PrimeIntellect-ai/verifiers/tree/main/skills) directory (alongside [`AGENTS.md`](https://github.com/PrimeIntellect-ai/verifiers/blob/main/AGENTS.md)). They are more comprehensive than these docs, which are meant for human consumption.


# Harbor
Source: https://docs.primeintellect.ai/verifiers/v1/harbor



verifiers supports single-step Harbor 0.24.0 tasks via the `HarborTaskset` class. Tasks declaring `steps` are rejected; use `tasks` to select single-step tasks from mixed datasets. Creating a Harbor-based taskset is straightforward in most cases:

```python theme={null}
import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset


# Set the dataset to the same name as registered in the Harbor registry
class TerminalBench2Config(HarborConfig):
    dataset: str = "terminal-bench/terminal-bench-2"


# The data will get loaded automatically
class TerminalBench2Taskset(
    HarborTaskset, vf.Taskset[HarborTask, TerminalBench2Config]
):
    pass
```

You can also write custom code for your tasksets. Override each task's `image` and `verifier_image` by rebuilding it around an updated copy of its data:

```python theme={null}
from pathlib import Path
from typing import Literal

import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset

IMAGE_TEMPLATE = "registry.example.com/openthoughts/{task}:latest"
VERIFIER_IMAGE_TEMPLATE = "registry.example.com/openthoughts/{task}-verifier:latest"


class OpenThoughtsTBLiteConfig(HarborConfig):
    dataset: Literal["openthoughts/openthoughts-tblite"] = (
        "openthoughts/openthoughts-tblite"
    )
    # Load Dockerfile-only tasks before replacing their images below.
    ignore_dockerfile: bool = True


class OpenThoughtsTBLiteTaskset(
    HarborTaskset, vf.Taskset[HarborTask, OpenThoughtsTBLiteConfig]
):
    def load(self) -> list[HarborTask]:
        # Task data is frozen, so rebuild each task around an updated copy.
        return [
            HarborTask(
                task.data.model_copy(
                    update={
                        "image": IMAGE_TEMPLATE.format(
                            task=Path(task.data.task_dir).name
                        ),
                        "verifier_image": VERIFIER_IMAGE_TEMPLATE.format(
                            task=Path(task.data.task_dir).name
                        ),
                    }
                ),
                task.config,
            )
            for task in super().load()
        ]
```

Only include the fields you want to replace. `verifier_image` applies to tasks that declare a separate verifier and must contain the complete `/tests` suite, including `/tests/test.sh`. A separate verifier prefers `[verifier.environment].docker_image`, then `tests/Dockerfile` or `tests/docker-compose.yaml`, then the agent environment. A verifier section that only declares resources inherits the agent image and stages the task package's tests. Dedicated verifier builds are rejected unless `ignore_dockerfile` explicitly enables fallback. When `verifier_image` is `None`, the verifier uses the task's current `image` and stages the packaged tests. These changes leave `task.toml` unchanged.

To create and reuse images for your tasks, build the Dockerfile with Docker, push it to a registry, and set the resulting image reference in the task data.

On the `prime` runtime any pullable image reference just works: the first sandbox to use an image makes the platform build and cache what it needs from it (for VM sandboxes this build can take \~10 minutes — the eval dashboard marks affected rollouts as `build` and a warning is logged); every later sandbox on the same reference starts in seconds.

## Additional features

By default, each task's declared agent and verifier timeouts are ignored (`ignore_timeouts = true`): Harbor task timeouts are authored against Harbor's own runtime, so enforcing them confounds model capability with the speed of your inference stack. Set `ignore_timeouts = false` (or pass `--no-env.taskset.ignore-timeouts`) to apply them, e.g. for a faithful comparison against the Harbor implementation.

With `ignore_timeouts = false`, every Harbor taskset can also be modified with a `timeout_multiplier`, and any Harbor taskset with a `resource_multiplier`:

```toml theme={null}
[env.taskset]
id = "MY_TASKSET"
ignore_timeouts = false
timeout_multiplier = 2.0
resource_multiplier = 2.0
```

The `timeout_multiplier` multiplies both the agent and verifier timeout, while the `resource_multiplier` multiplies the task's CPU, memory and disk space. You might want to use these multipliers when the tasks set too tight limits and/or the agent is slow.

## Rewards

`reward.json` takes precedence over `reward.txt`. An existing JSON file must contain
an object of finite numeric values; invalid, unreadable, or oversized JSON fails
grading without falling back to text. When JSON is absent, `reward.txt` supplies
the numeric `reward` value. A verifier's nonzero exit does not override a valid
reward file.

The `reward` key supplies the scalar reward by default. To select another key:

```toml theme={null}
[env.taskset.task]
reward_key = "accuracy"
```

Other named values are retained as trace metrics. A missing selected key fails
grading; named values are never implicitly summed. The same selection applies to
shared and separate verifiers.

## Docker Compose

Select `runtime.type = "docker"` to run Compose tasks locally.

With the default Harbor env, tasks containing `environment/docker-compose.yaml`
run their topology through Harbor on local Docker, Prime VMs, or Modal's VM runtime.
Local Docker requires `--env.trust-compose`: task definitions can mount host files
and request Docker privileges, so only enable it for trusted packages. Local Compose
receives Docker connection settings and infrastructure variables rather than the
evaluator's full environment. Task-local `.env` files and declared task env remain available.

Compose preserves service entrypoints, commands, dependencies, health checks,
networking, and volumes; the agent executes in a single `main` container.
Host networking is unsupported. Runtime defaults preserve the authored image and
working directory; task settings and non-default runtime overrides take precedence.

For Prime, set `runtime.type = "prime"`. One VM hosts Docker and all
services, and its network policy applies to every service after trusted setup.
For Modal, set `runtime.type = "modal"`; Compose uses the SDK's experimental VM
backend with Docker support and requires `network_access = true` and access to that
backend. Local Docker Compose also requires unrestricted networking. GPU Compose
tasks are unsupported.

The runtime's CPU and memory settings size the entire remote sandbox, so allow room
for sidecars. Prime also applies the disk request; Modal has no disk-size setting.
Prebuilt service images must be Docker-pullable inside the sandbox; services with a
`build` stanza are built there. Prime-only VM image references cannot serve as inner
container images. Prime VM ports cannot be published externally. Modal publishes
main's runtime service port through its encrypted tunnel, including when main shares
another service's network namespace.

A taskset can set a task's `compose_host_image` to a VM image that hosts the Docker
daemon instead of the stock one. Docker is installed only when the image lacks it, and
`docker save` archives shipped in `/opt/verifiers/compose-images/` load before the
services start, so services referencing their tags pull nothing from a registry.

The Harbor environment removes the entire project or remote sandbox before
separate grading, which retains the ordinary fresh verifier runtime. A verifier
without its own image inherits the resolved main image; a fresh copy also inherits
main's working directory. Images built only
inside a cloud host must be published separately and declared in the verifier environment.

Compose projects are owned by the Harbor environment; agents borrow the existing
Docker main container. Failures retry with a fresh project through
`--env.retries`, rather than retrying an agent inside the same project.

## Network policies

Harbor's effective agent network policy is applied to Docker or Prime VM harness
runtimes. An `[agent].network_mode` override takes precedence over the `[environment]`
baseline; legacy `[environment].allow_internet` is normalized by Harbor's schema.

| Harbor mode | Task network policy |
| - | - |
| `public` | Sets the task allowlist to `["*"]`, leaving the evaluator policy intact. |
| `no-network` | Sets the task allowlist to `[]` (framework routes only). |
| `allowlist` | Sets the task allowlist to `allowed_hosts`. |

Trusted task and harness setup remains online. The policy starts immediately before the
agent and stays active through finalization and scoring. Interception and MCP URLs are
added automatically in allowlist and framework-only modes. Concrete task/runtime
allowlists retain their shared entries, while blocklists combine; framework-only access on
either side takes precedence, and concrete allowlists cannot be combined with blocklists.
Docker framework routes take precedence over deny rules, while ordinary Prime deny rules
are applied unchanged and may block a matching route. Restricted Harbor tasks require
Docker or a Prime VM; Prime accepts host-level entries. Provider-resolved URLs are retained
when their initial destination matches the effective policy. OpenAI Responses web search and
Anthropic web search/fetch receive wildcard host allowlists translated to provider domains;
policies that cannot be translated without widening still disable them. Every other hosted
tool and provider-held resource remains disabled.

## Artifacts and collect hooks

`--env.taskset.artifact-max-bytes` sets the total artifact archive budget per solver across all services (default: 32 MiB), including the `/logs/artifacts/` convention directory. Increase it for tasks that transfer trained checkpoints or VM disk files. The budget also applies when scoring is deferred to a separate verifier.

Prime VM bounded reads stream binary data. Collected archives remain in host memory for grading and are excluded from persisted traces.

`artifacts = [...]` and `[[verifier.collect]]` are read from `task.toml` ([Harbor Docs](https://www.harborframework.com/docs/run-jobs/results-and-artifacts)). Each entry's `service` selects the source runtime, defaulting to `main`; additional services come from the Harbor-owned Compose project. Main's hooks and artifacts are collected during `finalize`. For separate grading, the Harbor environment then stops main and collects sidecar evidence; a shared verifier grades in main and skips sidecar entries. Declared paths and main's `/logs/artifacts/` convention directory are restored at their original paths in the grader.

Artifact roots from different services must not overlap, since they share the grader's filesystem.

Two deliberate differences from `harbor run`:

* **A failing collect hook fails the rollout.** Harbor logs it and carries on, because there the output is observability; here it is a grading input, and a silently absent file makes the verifier score a stale state.
* **`destination` has no effect.** It positions a file in Harbor's host trial directory; verifiers has no trial directory (the trace is the record), and Harbor never lets `destination` affect verifier-side placement.

## Separate verifier environments

`[verifier].environment_mode = "separate"` grades in a second box the agent never touched, instead of the one it worked in ([Harbor Docs](https://www.harborframework.com/docs/tasks/verifier)). The harbor env — this taskset's default — grades such tasks in `finalize`: the solver plays the task as usual, its declared artifacts and the `/logs/artifacts/` convention directory are collected while its box is alive, the box is torn down, and the env then provisions a fresh box, restores those artifacts, prepares the verifier tests, and grades there, recording the verifier's rewards and metrics onto the solver's trace. The grading box derives from the solver's runtime policy unless `--env.verifier.runtime.*` names its own; setup, restoration, staging, and scoring failures retry per `--env.verifier.retries` before the episode fails. The score is read from `/logs/verifier/reward.json` — a finite number, or an object of finite numbers: with a `reward` key that key is the score and the rest are recorded as metrics; without one every key is recorded as a separate reward. Missing or invalid, it falls back to `reward.txt`.

Which image the verifier boots from follows Harbor: a declared `[verifier.environment]` if there is one, otherwise a fresh copy of `[environment]`, which is the task's own image. A dedicated `[verifier.environment].docker_image` must contain the complete `/tests` suite, including `/tests/test.sh` and its dependencies; that suite runs without uploading packaged tests, matching Harbor. When grading in a fresh copy of the solver image (including the `ignore_dockerfile` fallback), `/tests` is replaced with the task package's tests so stale image files cannot affect grading. Solver artifacts rooted under `/tests` are rejected before the tests run, and old reward files are cleared in either case.

A declared `[verifier.environment]` needs a pullable `docker_image`. Without one Harbor would build the verifier image from `tests/Dockerfile`, and verifiers never builds images — so build and push it yourself and name the resulting reference, exactly as for `[environment]`. `ignore_dockerfile` grades in the agent's image instead, which means the verifier runs somewhere the task never declared; it warns when it does.

Under any other env, a separate-verifier task refuses to grade in the agent's box rather than silently losing its isolation. `ignore_separate_verifier = true` forces every task back into shared grading, trading the isolation for one sandbox per task.

## Shortcomings

verifiers does not have parity with Harbor yet, so some features are missing and currently being worked on. The most notable missing features right now are:

* Outside Compose, image `ENTRYPOINT`s are replaced with a keepalive, so `[environment.healthcheck]` cannot depend on entrypoint-based setup or services
* Switching to a different verifier-phase network policy for a *shared* verifier ([Harbor Docs](https://www.harborframework.com/docs/tasks/network-policy)); a separate verifier's own policy is applied
* Building a verifier image from `tests/Dockerfile`, which Harbor does when a declared `[verifier.environment]` names no `docker_image`. A separate verifier image itself is supported — it just has to be pre-built and pullable (see above), because verifiers never builds images
* Multi-step tasks ([Harbor Docs](https://www.harborframework.com/docs/tasks/multi-step))


# Harnesses
Source: https://docs.primeintellect.ai/verifiers/v1/harnesses



verifiers supports a range of harnesses out of the box, including Claude Code, Codex, the tool-enabled `bash` harness, the CDP-driven `browser_use` harness, and the minimal tool-less `null` harness. However, you may want to build a custom one or extend the selection of third‑party harnesses.

## A minimal harness implementation

```python theme={null}
from verifiers.v1.clients import ModelContext
from verifiers.v1.configs.harness import HarnessConfig
from verifiers.v1.harness import Harness
from verifiers.v1.runtimes import ProgramResult, Runtime
from verifiers.v1.task import TaskData
from verifiers.v1.trace import Trace


class MyHarnessConfig(HarnessConfig):
    # These are the values that the users are allowed to set and change.
    version: str = "0.0.1"


class MyHarness(Harness[MyHarnessConfig]):
    # Set the system prompt of the task as the harness system message; else add it to the first user message
    APPENDS_SYSTEM_PROMPT = True
    # When the taskset exports a toolset, they are added as MCP. To show that your harness is able to install MCPs, you have to set this flag to true.
    SUPPORTS_MCP = True
    # Allow user simulation, mostly implemented by supporting the ACP protocol.
    SUPPORTS_RESUME = True

    async def setup(self, runtime: Runtime) -> None:
        # Install the harness in its rollout runtime
        await runtime.run(["sh", "-c", "echo installing..."], {})

    async def launch(
        self,
        ctx: ModelContext,
        trace: Trace,
        runtime: Runtime,
        endpoint: str,
        secret: str,
        mcp_urls: dict[str, str],
        data: TaskData,
    ) -> ProgramResult:
        # Run the harness in its respective runtime to completion
        # The model (interception) endpoint is in endpoint
        # mcp_urls are the URLs of the tools from the toolset (if registered)

        # Resolve the task's prompt (and system prompt) for this harness
        _, prompt = self.resolve_prompt(data)

        # Example: Use the harness, but overwrite the endpoint to use the interception server and the custom model name
        env = {
            **self.config.env,
            "HARNESS_BASE_URL": endpoint,
            "HARNESS_API_KEY": secret,
            "HARNESS_BASE_MODEL": ctx.model,
        }
        # Run the harness to completion inside the selected runtime.
        return await runtime.run_program(["<HARNESS_BINARY>", str(prompt or "")], env)
```


# Overview
Source: https://docs.primeintellect.ai/verifiers/v1/overview



verifiers is a framework for defining tasks, running agents and harnesses, scoring them on set tasks, and using those for evaluations and reinforcement learning.

The following concepts are important when creating or running tasksets, be it for evals or training:

## Environment Hub

The [Environment Hub](https://app.primeintellect.ai/dashboard/environments?ex_sort=most_stars) is Prime Intellect's collection of user-created tasksets which are installable and ready to use with verifiers.

## Taskset

A taskset is the collection and loader for the work to evaluate or train on. Each task combines a serializable `TaskData` row (prompt, files, references, resource requirements) with its task class's behavior (lifecycle hooks, tools, metrics, and rewards). The taskset's `load()` method constructs those objects and declares their task/config types through `Taskset[TaskT, ConfigT]`.

## Harness

A harness is the program the model is run in, e.g. Claude Code, Codex or mini-swe-agent.

## Agent

An Agent is the combination of harness × model × runtime policy which produces a `Trace`.

## Environment

An environment has one or multiple agents and defines the control flow among them.

## Toolset

A set of tools defined by the taskset that are installed as MCP servers into the harnesses that support them.

## Trace

A trace records the message graph, rewards, metrics, errors, and one per-call record (`ModelCall`) per provider exchange (its model, sampling, finish reason, usage, timing, and any error), etc. When using verifiers for training with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl), it stores additional information such as tokens and logprobs, built incrementally using [renderers](https://github.com/PrimeIntellect-ai/renderers).

## Documentation

* [Getting started](/verifiers/v1/getting_started) - How to install verifiers and the needed skills.
* [Architecture](/verifiers/v1/architecture) — How verifiers works behind-the-scenes
* [Tasksets](/verifiers/v1/tasksets) — How to create tasksets
  * [Harbor Tasksets](/verifiers/v1/harbor) — How to create Harbor-based tasksets
* [Evaluation](/verifiers/v1/evaluation) — How to evaluate tasksets
* [Harnesses](/verifiers/v1/harnesses) — How to build custom harnesses
* [Agent](/verifiers/v1/agent) — How to run standalone agents
* [Env](/verifiers/v1/env) — How to build multi-agent environments


# Tasksets
Source: https://docs.primeintellect.ai/verifiers/v1/tasksets



A taskset defines the work to be done, which will be solved by the agent in a *harness* running in a *runtime*.

You can scaffold a new taskset with the following:

```bash theme={null}
uv run vf-init addition-v1
```

The generated package has two important files:

```text theme={null}
environments/addition_v1/addition_v1/
├── __init__.py  # exports the taskset entry point
└── taskset.py   # defines the data, tasks, and taskset
```

The command also supports:

* `-p`, `--path <dir>` — parent directory, default: `./environments`
* `-T`, `--add-tool` — also scaffold a `vf.Toolset` tool server at `servers/tool.py`
  * Use this to create custom tools which are installed into supported harnesses via MCP.
* `-H`, `--add-harness` — also scaffold a custom `vf.Harness` at `harness.py`, selectable via `--env.agent.harness.id <name>`
  * Prefer a built-in harness unless the model needs to run inside a custom program.

> For a production-scale catalog of tasksets, see the companion [`research-environments`](https://github.com/PrimeIntellect-ai/research-environments) repository.

## An example taskset

Tasksets are made of the following components:

* The **Taskset** loads the actual **Tasks** from a dataset using the `load()` function. It can be configured with the **TasksetConfig**, to e.g. load a certain split. Configs are exposed to the user and thus should only contain configurable values.
* A **Task** defines the scoring, stop conditions, setup, judging etc. of the task to solve. It also gets the tools or user config. It gets configured by a **TaskConfig**, e.g., to set a specific judge model.
* The **TaskData** is the immutable object that holds the actual data, i.e., the prompts, images, expected outputs etc., as well as other information such as timeouts (if set).

The following taskset generates addition questions and checks whether the model returned the exact answer.

```python theme={null}
import verifiers.v1 as vf


class AdditionData(vf.TaskData):
    # One immutable row in the dataset, including its reference answer.
    answer: int


class AdditionTask(vf.Task[AdditionData]):
    # @vf.reward denotes the scoring function for the task.
    # It needs the trace, which contains the whole message graph, including function calls, user messages etc.
    # It returns the reward for the single task based on this function.
    @vf.reward
    async def exact_match(self, trace: vf.Trace) -> float:
        return float(trace.last_reply == str(self.data.answer))


class AdditionConfig(vf.TasksetConfig):
    # Values users can configure for the whole taskset.
    num_tasks: int = 100


# The Taskset itself
class AdditionTaskset(vf.Taskset[AdditionTask, AdditionConfig]):
    # The loading function for the actual tasks
    def load(self) -> list[AdditionTask]:
        return [
            AdditionTask(
                AdditionData(prompt=f"What is {i} + {i}?", answer=2 * i),
                self.config.task,
            )
            for i in range(self.config.num_tasks)
        ]
```

If a config class is not explicitly created, it means that no configurable, custom values are exposed to the user. In this example, there is no `vf.TaskConfig`, so no task values (like judge models) are configurable.

The scaffold also exports the taskset from `addition_v1/__init__.py`:

```python theme={null}
from addition_v1.taskset import AdditionTaskset

__all__ = ["AdditionTaskset"]
```

The exported `AdditionTaskset` is what verifiers loads and makes discoverable for evaluation.

## Data and configuration

Keep values on the narrowest object that needs them:

* Put load-time values shared across the dataset, such as its split, name, seed, or size, on `TasksetConfig`.
* Put values used by every task during execution or scoring under `TasksetConfig.task`.

```python theme={null}
class AdditionTaskConfig(vf.TaskConfig):
    tolerance: float = 0.0


class AdditionTask(vf.Task[AdditionData, vf.State, AdditionTaskConfig]):
    @vf.reward
    async def exact_match(self, trace: vf.Trace) -> float:
        error = abs(float(trace.last_reply) - self.data.answer)
        return float(error <= self.config.tolerance)


class AdditionConfig(vf.TasksetConfig):
    num_tasks: int = 100
    task: AdditionTaskConfig = AdditionTaskConfig()
```

These values can be overridden with `--env.taskset.num-tasks` and `--env.taskset.task.tolerance`, or with the equivalent TOML fields (`[env.taskset]`).

## Task identity

A task's `idx` is its index in the `load()` stream; it is set automatically. Set `TaskData.id` to a durable ID from the source (e.g. a dataset's instance ID) and `TaskData.name` to a readable name; both are optional.

A task's `hash` identifies its exact serialized data. Its `key` provides stable identity across runs and defaults to the hash. Override `Task.key` with a durable source ID when task data contains run-local fields such as `idx`; keys must be unique within a taskset. Both values are recorded on traces.

## Selecting tasks

Views on a taskset pick which tasks an iteration yields. They are lazy and chain in any order:

```python theme={null}
taskset.include(idx=["0:100"]).exclude(names=["broken"]).shuffle(seed=0).take(5)
```

`include` keeps and `exclude` drops the tasks named by `idx` (ints and Python slices `start:stop:step`, e.g. `"100:"`, `":50"`, `"::2"`; no negative positions), `ids`, `keys` or `names`; a task matches when any list names it. An entry that matches no task logs a warning once the stream was read.

The eval, debug, validate and GEPA entrypoints take a `select` block and apply it with `taskset.select(config.select)`, which chains the views in a fixed order: `include`, `exclude`, `shuffle`, `skip`, `limit`:

```bash theme={null}
uv run vf-eval gsm8k --select.limit 50 --select.shuffle         # 50 random tasks (short: -n 50 -s)
uv run vf-eval gsm8k --select.include.idx 0:100                 # the first 100 tasks
uv run vf-eval gsm8k --select.shuffle --select.skip 100 --select.limit 100   # a random split disjoint from the first
```

```toml theme={null}
[select]
include = { idx = ["0:500"] }
exclude = { names = ["broken-task"] }
shuffle = true
limit = 100
```

Because the shuffle comes before `skip` and `limit`, a larger `limit` extends the same selection instead of drawing a new one.

## Lazy and infinite tasksets

`load()` may be a generator instead of returning a list: yield each task as it's built. Consumers iterate the taskset lazily (`eval -n 5` builds 5 tasks, not the whole set, unless `shuffle` needs all of them) — so a generator pays off whenever building a task is expensive.

A procedural taskset can keep yielding forever. Declare `INFINITE = True` so consumers know the stream never ends — infinity is inherent to the taskset, not a config knob; how many tasks a run takes is the run's choice (`-n`), not the taskset's:

```python theme={null}
import itertools
from collections.abc import Iterator


class AdditionTaskset(vf.Taskset[AdditionTask, vf.TasksetConfig]):
    INFINITE = True

    def load(self) -> Iterator[AdditionTask]:
        for i in itertools.count():
            yield AdditionTask(
                AdditionData(prompt=f"What is {i} + {i}?", answer=2 * i),
                self.config.task,
            )
```

Two rules follow from infinity: a run over an infinite taskset must be bounded with `take` (`-n` on the CLI) or an `include` of only closed `idx` ranges — omitting both is an error — and `shuffle` needs a bound before it: there is no whole set to sample from otherwise. `select` applies `limit` after the shuffle, so on the CLI only closed `include.idx` ranges can bound it (`--select.include.idx 0:1000 -s -n 50`); in Python, `take(n).shuffle()` works too. The generator runs once, client-side (the eval entrypoint or the prime-rl orchestrator pulls tasks off it and ships each task's data to the env server), so nothing needs to re-produce the same sequence across processes; keep `load()` deterministic only if you want `--resume` to regenerate the same first `n` tasks (see `alphabet_sort`, `color_codeword`, or the built-in `textarena` taskset).

## Adding Tools

Some tasksets require custom tools, which are bundled as a `vf.Toolset` (similar to how a `vf.Taskset` bundles `vf.Task`). Tools are exposed as MCP servers to the given harness and thus need a harness which exposes MCP support (via `SUPPORTS_MCP`).

You can create them like this (remember the bootstrapping with `uv run vf-init MY_ENV -T`):

```python theme={null}
DATABASE = None


class SearchToolset(vf.Toolset[vf.SharedToolsetConfig]):
    TOOL_PREFIX = "search"

    @vf.tool
    async def query(self, text: str) -> list[str]:
        """Search the task corpus."""
        return DATABASE.search(text)


# User-configurable knobs
class SearchConfig(vf.TasksetConfig):
    tools: vf.SharedToolsetConfig = vf.SharedToolsetConfig()


class SearchTaskset(vf.Taskset[vf.Task, SearchConfig]):
    @classmethod
    def toolsets(cls, config: SearchConfig) -> list[vf.Toolset]:
        return [SearchToolset(config.tools)]
```

Taskset tools are shared by a worker's rollouts. Tools can also be set per task.

## Using Judges

If your reward is semantic, use an LLM judge.

```python theme={null}
import verifiers.v1 as vf
from functools import cached_property


class Task(vf.Task):
    answer: str


class CorrectnessJudge(vf.Judge[bool]):
    # The rubric for the judge
    prompt = """Question: {question}
    Answer: {answer}
    Response: {response}
    Correct? Reply yes or no."""

    # Parse the response from the judge
    def parse(self, response: vf.JudgeResponse[bool]) -> bool:
        return "yes" in response.text


class JudgedData(vf.TaskData):
    answer: str


class JudgedTaskConfig(vf.TaskConfig):
    # The judge inherits base_url and api keys from the client config
    judge: vf.JudgeConfig = vf.JudgeConfig(model="openai/gpt-5-mini")


class JudgedTask(vf.Task[JudgedData, vf.State, JudgedTaskConfig]):
    @vf.reward()
    async def correct(self, trace: vf.Trace) -> float:
        # Keeping judge configuration on TaskConfig makes it overridable from CLI/TOML.
        judge = CorrectnessJudge(self.config.judge)
        result = await judge.evaluate(
            trace=trace,
            question=self.data.prompt_text,
            answer=self.data.answer,
            # give the last assistant message to the judge
            response=trace.last_reply,
        )
        return float(result.parsed)


class SetConfig(vf.TasksetConfig):
    task: JudgedTaskConfig = JudgedTaskConfig()


class JudgeTraceTaskset(vf.Taskset[JudgedTask, SetConfig]):
    def load(self) -> list[JudgedTask]:
        return [
            JudgedTask(
                JudgedData(prompt="What is 2+2?", answer="4"),
                self.config.task,
            )
        ]
```

To override the judge model, set `env.taskset.task.judge.model` in your config (it is a string).

## Plugging hooks via config

Stop conditions, metrics, and rewards can also be plugged into any task from config, without touching the taskset's code. Each entry names an async function by import path (`pkg.module.function`, `pkg.module:function`, or `path/to/file.py:function`); like decorated hooks, functions declare what they need by parameter name (`task`, `trace`, `runtime` — stops receive the trace only):

```toml theme={null}
[env.taskset.task.stops]
single_turn = { fn = "my_hooks.py:two_turns" }

[env.taskset.task.metrics]
reply_length = { fn = "my_hooks.py:reply_length", priority = 10 }

[env.taskset.task.rewards]
exact_match = { fn = "my_hooks.py:exact_match", weight = 0.5 }
```

Plugged hooks merge with the task's decorated `@vf.stop` / `@vf.metric` / `@vf.reward` methods; a plugged hook replaces a decorated one with the same name, so an existing signal can be swapped out (like `single_turn` above). Rewards take a `weight`, and every entry takes a `priority` (higher runs first); unset fields keep the replaced method's values.

Leaving `fn` out keeps the task's own decorated method and only overrides its metadata — for example, turning an existing zero-weight reward into a training signal:

```toml theme={null}
[env.taskset.task.rewards]
exact_match = { weight = 1.0 }
```

## Beyond one agent

One episode doesn't have to be one agent run: agents, the control flow between agents, and cross-agent rewards are the environment's job — see [The Env](/verifiers/v1/env).