# List Audit Logs
Source: https://docs.primeintellect.ai/api-reference/admin-audit-logs/list-audit-logs
https://api.primeintellect.ai/openapi.json get /api/admin/audit-logs
List recorded admin operations, newest first, filtered by the query.
# Force Terminate Cluster Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/force-terminate-cluster-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/clusters/{cluster_id}/force
Force an orphaned cluster's status to TERMINATED (keeps the DB row).
# Get Cluster
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/get-cluster
https://api.primeintellect.ai/openapi.json get /api/admin/clusters/{cluster_id}
Return full DB details for one cluster (columns + nodes + SSH keys).
# List Cluster Node Logs
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/list-cluster-node-logs
https://api.primeintellect.ai/openapi.json get /api/admin/clusters/{cluster_id}/node-logs
Return paginated ClusterNodeLog entries for one cluster, newest first.
# List Clusters
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/list-clusters
https://api.primeintellect.ai/openapi.json get /api/admin/clusters
List non-terminated clusters (auto-scoped to the caller's grants).
# Request Certificate
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/request-certificate
https://api.primeintellect.ai/openapi.json post /api/admin/clusters/{cluster_id}/authorizations/certificate
Sign the requesting admin's SSH public key via Vault.
If a ``publicKey`` is provided in the body, it is signed directly.
Otherwise the admin's DB-stored SSH public key is used.
# Terminate Cluster Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-clusters/terminate-cluster-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/clusters/{cluster_id}
Terminate a cluster via its provider.
# Force Terminate Disk Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-disks/force-terminate-disk-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/disks/{disk_id}/force
Force an orphaned disk's status to TERMINATED (keeps the DB row).
# Get Disk
Source: https://docs.primeintellect.ai/api-reference/admin-disks/get-disk
https://api.primeintellect.ai/openapi.json get /api/admin/disks/{disk_id}
Return DB details for a single disk.
# List Disks
Source: https://docs.primeintellect.ai/api-reference/admin-disks/list-disks
https://api.primeintellect.ai/openapi.json get /api/admin/disks
List non-terminated disks (auto-scoped to the caller's grants).
# Terminate Disk Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-disks/terminate-disk-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/disks/{disk_id}
Terminate a disk via its provider.
# Send Bulk Email
Source: https://docs.primeintellect.ai/api-reference/admin-emails/send-bulk-email
https://api.primeintellect.ai/openapi.json post /api/admin/emails/send
Send a templated email to the given users and to the contact address of
the given teams (subject + body, personalized with each recipient's name).
Authorizes ``emails:create`` per recipient, so a scoped grant may email only
the users/teams it names while a broad grant may email anyone.
# Import Inventory Nodes
Source: https://docs.primeintellect.ai/api-reference/admin-inventory/import-inventory-nodes
https://api.primeintellect.ai/openapi.json post /api/admin/inventory/nodes
Create or update inventory nodes in bulk. Optionally assign them to a cluster.
This is an upsert. The service matches a node by (name, siteId). For an
existing node, it overwrites the host vars. With ansible_groups, it replaces
the cluster additional.ansibleGroups. Because the operation is an upsert,
both inventory:create and inventory:update are required and scoped to
clusterId. The audit records the acting admin. The prime-admin import command
also needs inventory:read for resolution and collision preflight.
# List Clusters
Source: https://docs.primeintellect.ai/api-reference/admin-inventory/list-clusters
https://api.primeintellect.ai/openapi.json get /api/admin/inventory/clusters
List inventory clusters (auto-scoped to the caller's grants).
# List Nodes
Source: https://docs.primeintellect.ai/api-reference/admin-inventory/list-nodes
https://api.primeintellect.ai/openapi.json get /api/admin/inventory/nodes
List inventory nodes with their ACTIVE cluster assignment. The import
command uses this to preflight hostname collisions and to report drift.
# List Sites
Source: https://docs.primeintellect.ai/api-reference/admin-inventory/list-sites
https://api.primeintellect.ai/openapi.json get /api/admin/inventory/sites
List inventory sites for the caller's grants. The operator uses a site id
for node import.
# Request Inventory Certificate
Source: https://docs.primeintellect.ai/api-reference/admin-inventory/request-inventory-certificate
https://api.primeintellect.ai/openapi.json post /api/admin/inventory/{cluster_id}/authorization/certificate
Sign the requesting admin's SSH public key for an inventory cluster.
If a ``publicKey`` is provided in the body, it is signed directly. Otherwise
the admin's DB-stored SSH public key is used. Signing is gated on the
cluster's ``vaultSshCaEnabled`` flag.
# Force Terminate Pod Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-pods/force-terminate-pod-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/pods/{pod_id}/force
Force an orphaned pod's status to TERMINATED (keeps the DB row).
# Get Pod
Source: https://docs.primeintellect.ai/api-reference/admin-pods/get-pod
https://api.primeintellect.ai/openapi.json get /api/admin/pods/{pod_id}
Return DB details for a single pod.
# List Pods
Source: https://docs.primeintellect.ai/api-reference/admin-pods/list-pods
https://api.primeintellect.ai/openapi.json get /api/admin/pods
List non-terminated pods (auto-scoped to the caller's grants).
# Terminate Pod Endpoint
Source: https://docs.primeintellect.ai/api-reference/admin-pods/terminate-pod-endpoint
https://api.primeintellect.ai/openapi.json delete /api/admin/pods/{pod_id}
Terminate a pod via its provider.
# Get Team
Source: https://docs.primeintellect.ai/api-reference/admin-teams/get-team
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}
Return DB details for a single team.
# Get Team Billing
Source: https://docs.primeintellect.ai/api-reference/admin-teams/get-team-billing
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/billing
Return the team's wallet balance, settings, and configured usage limits.
# Get Team Transaction
Source: https://docs.primeintellect.ai/api-reference/admin-teams/get-team-transaction
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/transactions/{transaction_id}
Detailed view of a single wallet transaction.
# List Team Billing History
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-billing-history
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/billing-history
Paginated billing (per-resource charge) history for a team.
# List Team Clusters
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-clusters
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/clusters
Paginated list of the team's non-terminated clusters.
# List Team Disks
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-disks
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/disks
Paginated list of the team's non-terminated disks.
# List Team Members
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-members
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/members
List every member of the team with their role and join date.
# List Team Pods
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-pods
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/pods
Paginated list of the team's non-terminated pods.
# List Team Transactions
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-team-transactions
https://api.primeintellect.ai/openapi.json get /api/admin/teams/{team_id}/transactions
Paginated wallet transaction history for a team, newest first.
# List Teams
Source: https://docs.primeintellect.ai/api-reference/admin-teams/list-teams
https://api.primeintellect.ai/openapi.json get /api/admin/teams
List teams (auto-scoped to the caller's grants).
# Update Team
Source: https://docs.primeintellect.ai/api-reference/admin-teams/update-team
https://api.primeintellect.ai/openapi.json patch /api/admin/teams/{team_id}
Edit a team: name/url/bio, set slug, set email, grant beta.
# Topup Wallet
Source: https://docs.primeintellect.ai/api-reference/admin-transactions/topup-wallet
https://api.primeintellect.ai/openapi.json post /api/admin/transactions/topup
Credit (TOPUP) or debit (REFUND) a user/team wallet. ``amount`` is in
stored units (USD * 10000).
# Get Current User
Source: https://docs.primeintellect.ai/api-reference/admin-users/get-current-user
https://api.primeintellect.ai/openapi.json get /api/admin/users/whoami
Identity + envelope for the calling admin API key.
Intentionally bypasses ``require_admin_permission``: any valid, non-
expired, non-disabled admin API key whose owner is still ADMIN/MANAGER
can call this. Returning the resolved permission envelope is what makes
a freshly-minted scoped key able to discover what it can do.
# Get User
Source: https://docs.primeintellect.ai/api-reference/admin-users/get-user
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}
Return DB details for a single user.
# Get User Billing
Source: https://docs.primeintellect.ai/api-reference/admin-users/get-user-billing
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/billing
Return the user's wallet balance, settings, and configured usage limits.
# Get User Transaction
Source: https://docs.primeintellect.ai/api-reference/admin-users/get-user-transaction
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/transactions/{transaction_id}
Detailed view of a single wallet transaction.
# List User Billing History
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-billing-history
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/billing-history
Paginated billing (per-resource charge) history for a user.
# List User Clusters
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-clusters
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/clusters
Paginated list of the user's non-terminated clusters.
# List User Disks
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-disks
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/disks
Paginated list of the user's non-terminated disks.
# List User Pods
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-pods
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/pods
Paginated list of the user's non-terminated pods.
# List User Teams
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-teams
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/teams
List the teams the user is a member of, filtered to the teams the caller
is scoped to read.
# List User Transactions
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-user-transactions
https://api.primeintellect.ai/openapi.json get /api/admin/users/{user_id}/transactions
Paginated wallet transaction history for a user, newest first.
# List Users
Source: https://docs.primeintellect.ai/api-reference/admin-users/list-users
https://api.primeintellect.ai/openapi.json get /api/admin/users
List users (auto-scoped to the caller's grants).
# Update User
Source: https://docs.primeintellect.ai/api-reference/admin-users/update-user
https://api.primeintellect.ai/openapi.json patch /api/admin/users/{user_id}
Edit a user: ban/unban, blacklist, grant beta, name/url/bio, set slug.
# Update User Limits
Source: https://docs.primeintellect.ai/api-reference/admin-users/update-user-limits
https://api.primeintellect.ai/openapi.json patch /api/admin/users/{user_id}/usage-limits
Update any subset of the user's wallet usage limits.
# Ingest Events
Source: https://docs.primeintellect.ai/api-reference/agent_analytics/ingest-events
https://api.primeintellect.ai/openapi.json post /api/v1/agent-analytics/events
# Upload Session
Source: https://docs.primeintellect.ai/api-reference/agent_traces/upload-session
https://api.primeintellect.ai/openapi.json put /api/v1/agent-traces/sessions/{session_id}
Overwrite the stored JSONL for this session.
Body is raw NDJSON; the CLI sends the entire local session file each call.
Content-Length is checked before the body is read so oversized requests
are rejected without buffering bytes.
# API keys
Source: https://docs.primeintellect.ai/api-reference/api-keys
How to generate and use authentication keys with our API
## Overview
This guide explains how to generate and manage API keys for authenticating requests to our API. Proper use of API keys is critical for securing your application and controlling access.
## Generating an API Key
To generate API Key, navigate to [Settings -> API Keys](https://app.primeintellect.ai/dashboard/tokens), and click the `Generate New Key +` button.
You can assign fine-grained permissions to each key, allowing you to create multiple keys with specific scopes or roles. This enhances the security of your application by limiting each key's responsibilities. Additionally, you can set an expiration date for each key. We strongly recommend **always** setting an expiration date to avoid using indefinitely valid keys.
Ensure that all API keys are stored securely and never shared in untrusted environments or with third-party applications.
Once a key is generated, you will be able to copy it for immediate use.
The key value is displayed only once. If you lose it, you will not be able to retrieve it again.
## Revoking an API Key
If you no longer need a particular API key go to [Settings -> API Keys](https://app.primeintellect.ai/dashboard/tokens), find your key on the list and click on `Remove` button on the right side.
# Get Disks Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-disks-availability
https://api.primeintellect.ai/openapi.json get /api/v1/availability/disks
# Get Gpu Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-gpu-availability
https://api.primeintellect.ai/openapi.json get /api/v1/availability/gpus
# Get Gpu Summary
Source: https://docs.primeintellect.ai/api-reference/availability/get-gpu-summary
https://api.primeintellect.ai/openapi.json get /api/v1/availability/gpu-summary
Get GPU pricing summary data grouped by GPU type and instance count.
# Get Legacy Cluster Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-legacy-cluster-availability
https://api.primeintellect.ai/openapi.json get /api/v1/availability/clusters
# Get Legacygpu Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-legacygpu-availability
https://api.primeintellect.ai/openapi.json get /api/v1/availability/
# Get Multinode Availability
Source: https://docs.primeintellect.ai/api-reference/availability/get-multinode-availability
https://api.primeintellect.ai/openapi.json get /api/v1/availability/multi-node
# Get Multinode Summary
Source: https://docs.primeintellect.ai/api-reference/availability/get-multinode-summary
https://api.primeintellect.ai/openapi.json get /api/v1/availability/multi-node-summary
Get multi-node (cluster) pricing summary data grouped by GPU type and instance count.
# Token usage and cost for a single RFT run
Source: https://docs.primeintellect.ai/api-reference/billing/token-usage-and-cost-for-a-single-rft-run
https://api.primeintellect.ai/openapi.json get /api/v1/billing/runs/{run_id}/usage
Return the running token + cost totals for one RFT run.
Mirrors the per-row training data shown on the billing page (Training
tokens, Inference input/output tokens, price per million tokens, and
total cost) so an agent can poll this endpoint to monitor a live run.
# Wallet balance + most recent billing rows
Source: https://docs.primeintellect.ai/api-reference/billing/wallet-balance-+-most-recent-billing-rows
https://api.primeintellect.ai/openapi.json get /api/v1/billing/wallet
Return the wallet's current balance and most recent billing rows.
All resource types are included (compute, training, inference, disks,
sandboxes, images) — same source-of-truth `Billing` table the
dashboard's Billing History tab reads. Sorted by `lastBilledAt` desc.
# Get Availability Information
Source: https://docs.primeintellect.ai/api-reference/check-gpu-availability
How to check GPU, cluster, and disk availability and pricing
Before you start, ensure that you have [API Key](./api-keys) with `Availability -> Read` permission
## Available Endpoints
The availability API provides two endpoints to query different types of resources:
* **`/api/v1/availability/gpus`** - Get GPU instance availability
* **`/api/v1/availability/disks`** - Get standalone disk availability
## Retrieving GPU Availability Data
Suppose you want to check pricing options for a single **H100 GPU**, with the location restricted to the **United States** or **Canada**. To do this, send a request to our availability endpoint as shown below:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/gpus?regions=united_states®ions=canada&gpu_count=1&gpu_type=H100_80GB' \
--header 'Authorization: Bearer your_api_key'
```
You can generate request samples using our interactive [`Availability API`](./availability/get-gpu-availability)
documentation
This is a **GET** request, and it requires an **Authorization: Bearer** token with your API key. The request accepts query parameters, allowing you to filter GPUs by region, GPU count, GPU type, and other criteria. In this case, the filters include **regions**, **gpu\_count**, and **gpu\_type**.
### Query Parameters
All availability endpoints support pagination and the following filters:
| Parameter | Type | Description |
| ---------------- | ------------- | --------------------------------------------------------------------------- |
| `regions` | List\[string] | Filter by region(s) (e.g., `united_states`, `canada`, `europe_west`) |
| `gpu_count` | integer | Desired number of GPUs |
| `gpu_type` | string | GPU model (e.g., `H100_80GB`, `A100_80GB`) |
| `socket` | string | GPU socket type (e.g., `PCIe`, `SXM`) |
| `security` | string | Security type: `secure_cloud` |
| `data_center_id` | string | Filter by specific data center ID |
| `cloud_id` | string | Filter by provider's cloud ID |
| `disks` | List\[string] | Filter by disk IDs (see [Disk Filtering](#filtering-by-user-disks) section) |
| `page` | integer | Page number (default: 1, min: 1) |
| `page_size` | integer | Results per page (default: 100, max: 100) |
## Understanding the GPU Response Object
The GPU availability endpoint returns a paginated response with a list of available GPU configurations and the total count. Here is an example of the JSON response you might receive:
```json Example response theme={null}
{
"items": [
{
"cloudId": "NVIDIA H100 PCIe",
"gpuType": "H100_80GB",
"socket": "PCIe",
"provider": "runpod",
"region": "united_states",
"dataCenter": "US-KS-2",
"country": "US",
"gpuCount": 1,
"gpuMemory": 80,
"disk": {
"minCount": 80,
"defaultCount": 80,
"maxCount": 1000,
"pricePerUnit": 0.00014,
"step": 1,
"defaultIncludedInPrice": false,
"additionalInfo": null
},
"vcpu": {
"defaultCount": 16
},
"memory": {
"defaultCount": 251
},
"internetSpeed": null,
"interconnect": null,
"interconnectType": null,
"provisioningTime": null,
"stockStatus": "Low",
"security": "secure_cloud",
"prices": {
"onDemand": 2.69,
"isVariable": false,
"currency": "USD"
},
"images": [
"ubuntu_22_cuda_12",
"cuda_12_1_pytorch_2_2",
"cuda_11_8_pytorch_2_1",
"stable_diffusion",
"flux",
"axolotl",
"bittensor",
"vllm_llama_8b",
"vllm_llama_70b",
"vllm_llama_405b"
],
"isSpot": null,
"prepaidTime": null
}
],
"totalCount": 247
}
```
For a full breakdown of the response schema, see the [`Get Gpu Availability
Endpoint`](./availability/get-gpu-availability) documentation
### Response Structure
#### `items`
An array containing the GPU availability records for the current page. Each item represents a unique GPU configuration from a provider.
#### `totalCount`
The total number of GPU configurations matching your filters across all pages.
For example, if `totalCount` is 247 and `page_size` is 100, you know there are 3 pages total.
Let's walk through the most important fields in each GPU availability item:
#### `provider`
Indicates the company or platform providing the GPU. You may encounter multiple offers from the same provider.
#### `cloudId`
A unique identifier provided by the GPU provider. This value is required if you plan to provision the GPU through our provisioning API.
#### `region`
The geographic region where the GPU is located (e.g., `united_states`, `europe_west`). This helps you select resources closer to your users.
#### `dataCenter`
Optional but necessary if present in the response. If the provider operates multiple datacenters with the same `cloudId`, you must specify which data center to use when provisioning the GPU.
#### Resource Specifications: `disk`, `sharedDisk`, `vcpu`, `memory`
Each resource field contains detailed specification information with the following attributes:
* **`minCount`**: Minimum allowed value for this resource
* **`maxCount`**: Maximum allowed value for this resource
* **`defaultCount`**: Default value provided by the provider
* **`pricePerUnit`**: Cost per unit per hour (if customizable)
* **`step`**: Increment step for adjusting the value
* **`defaultIncludedInPrice`**: Whether the default value is included in the base price
* **`additionalInfo`**: Extra information about pricing or constraints
**`disk`**: Instance-local disk storage in GB. This storage is attached to your GPU instance.
**`vcpu`**: Number of virtual CPUs allocated to the instance.
**`memory`**: RAM size available in GB.
#### Cost Calculation Example
In the example above, your server will include `16 vCPUs` and `251GB of RAM`, with customizable `disk` space. If the disk space is adjustable, its cost is calculated separately from the **prices** property. Here's an example breakdown of total costs:
```
GPU cost: $2.69
vcpu cost: $0.00
memory cost: $0.00
disk cost: $0.0112 (80 units * $0.00014)
Total cost: $2.7012
```
The total cost varies based on the disk space that you send to the provisioning API. The value can range between `minCount` and `maxCount` adjustable by `step` and you'll pay `pricePerUnit/hr` for every unit you want to use.
Some providers offer predefined server configurations, with the `defaultIncludedInPrice` set to `true`. This means the `defaultCount` is included in the base price, but any changes (even reductions) will incur additional charges.
#### `interconnect` and `interconnectType`
For multi-GPU setups, these fields specify the interconnect speed (in Gbps) and technology type (e.g., `InfiniBand`, `NVLink`) used for GPU-to-GPU communication.
#### `provisioningTime`
Estimated time in minutes required to provision and start the GPU instance.
#### `prices` and `security`
GPUs are provided by providers that maintain security standards and are hosted in secured datacenters. The price is defined in `prices.onDemand`.
Some providers may also use dynamic pricing for their instances. If the field `isVariable` is set to `true`, the price may fluctuate based on demand or currency conversion.
#### `prepaidTime`
If set, you will be pre-charged for the total number of hours specified in this field at the time of ordering, regardless of actual usage. After the prepaid time expires, the standard hourly rate will apply.
## Filtering by User Disks
The `disks` query parameter allows you to filter GPU availability based on your existing disks with persisted data. This is particularly useful when you want to provision a GPU instance that can attach to disks where you've already stored your datasets, models, or other important data.
### How It Works
When you provide disk IDs via the `disks` parameter, the API:
1. Fetches the disk information for the provided disk IDs
2. Filters GPU availability to show only instances that can be provisioned in locations where your disks are available for attachment
3. Applies rest of the filters to narrow down the results.
### Example Request
Suppose you have two disks with IDs `clhxy6aw80000j8080gdf8kqv` and `clhxy9bz50001j8080hdf9lrw` containing your training datasets in different datacenters. You can find GPUs that can attach to either disk:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/gpus?disks=clhxy6aw80000j8080gdf8kqv&disks=clhxy9bz50001j8080hdf9lrw&gpu_type=H100_80GB' \
--header 'Authorization: Bearer your_api_key'
```
This will return H100 GPUs available in the same locations where your disks are stored, making it easy to reuse your persisted data.
### Use Cases
* **Data Persistence**: Reuse datasets, models, and checkpoints stored on existing disks
* **Quick Provisioning**: Attach pre-loaded data disks to new GPU instances without re-uploading
## Checking Disk Availability
The disk availability endpoint allows you to query standalone network-attached storage options across different providers and datacenters.
### Endpoint
```bash theme={null}
GET https://api.primeintellect.ai/api/v1/availability/disks
```
### Example Request
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/disks?regions=united_states&page=1&page_size=50' \
--header 'Authorization: Bearer your_api_key'
```
### Query Parameters for Disk Availability
| Parameter | Type | Description |
| ---------------- | ------------- | ----------------------------------------- |
| `regions` | List\[string] | Filter by region(s) |
| `data_center_id` | string | Filter by specific data center ID |
| `cloud_id` | string | Filter by provider's cloud ID |
| `page` | integer | Page number (default: 1, min: 1) |
| `page_size` | integer | Results per page (default: 100, max: 100) |
### Disk Availability Response
The disk endpoint returns a paginated response with disk availability objects and the total count:
```json Example disk response theme={null}
{
"items": [
{
"cloudId": null,
"provider": "hyperstack",
"dataCenter": "US-1",
"country": "US",
"region": "united_states",
"spec": {
"minCount": 0,
"defaultCount": 0,
"maxCount": 100000,
"pricePerUnit": 0.00015,
"step": 1,
"defaultIncludedInPrice": false,
"additionalInfo": null
},
"stockStatus": "Available",
"security": "secure_cloud",
"isMultinode": false
}
],
"totalCount": 15
}
```
### Key Fields in Disk Response
#### `spec`
Contains the disk size specification with the same structure as resource specifications in GPU responses:
* **`minCount`**: Minimum disk size in GB
* **`maxCount`**: Maximum disk size in GB
* **`pricePerUnit`**: Cost per GB per hour
* **`step`**: Size increment (usually 1 GB)
#### `isMultinode`
Indicates whether this disk can be attached to multiple instances simultaneously. This is useful for shared data scenarios where multiple GPU instances need access to the same dataset.
#### `security`
Disks are hosted on secure cloud infrastructure.
### Disk Cost Calculation
Disk costs are straightforward to calculate:
```
Disk cost per hour = disk_size_gb * pricePerUnit
```
For example, a 500GB disk with `pricePerUnit: 0.00015`:
```
500 GB * $0.00015/GB/hr = $0.075/hr
```
## Pagination
All availability endpoints support pagination to handle large result sets efficiently:
* Use `page` to specify which page of results to retrieve (1-indexed)
* Use `page_size` to control how many results per page (max 100)
* Results are returned in consistent order for predictable pagination
Example with pagination:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/gpus?gpu_type=A100_80GB&page=2&page_size=50' \
--header 'Authorization: Bearer your_api_key'
```
# Create Disk
Source: https://docs.primeintellect.ai/api-reference/disks/create-disk
https://api.primeintellect.ai/openapi.json post /api/v1/disks/
# Delete Disk
Source: https://docs.primeintellect.ai/api-reference/disks/delete-disk
https://api.primeintellect.ai/openapi.json delete /api/v1/disks/{disk_id}
# Get Disk
Source: https://docs.primeintellect.ai/api-reference/disks/get-disk
https://api.primeintellect.ai/openapi.json get /api/v1/disks/{disk_id}
# List Disks
Source: https://docs.primeintellect.ai/api-reference/disks/list-disks
https://api.primeintellect.ai/openapi.json get /api/v1/disks/
# Update Disk
Source: https://docs.primeintellect.ai/api-reference/disks/update-disk
https://api.primeintellect.ai/openapi.json patch /api/v1/disks/{disk_id}
# Bulk Delete Evaluations
Source: https://docs.primeintellect.ai/api-reference/evals/bulk-delete-evaluations
https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/bulk-delete
# Create Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/create-evaluation
https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/
Create a new evaluation
This endpoint supports:
- Environment evaluations: Provide environments
- Prime RL evaluations: Provide run_id
- Suite evaluations: Provide suite_id
Ownership:
- If team_id is provided in request, the evaluation will be owned by the team
- Otherwise, the evaluation will be owned by the authenticated user
# Delete Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/delete-evaluation
https://api.primeintellect.ai/openapi.json delete /api/v1/evaluations/{evaluation_id}
# Finalize Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/finalize-evaluation
https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/{evaluation_id}/finalize
Mark an evaluation as complete and compute final statistics.
# Get Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/get-evaluation
https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/{evaluation_id}
Get detailed information about a specific evaluation.
# Get Samples
Source: https://docs.primeintellect.ai/api-reference/evals/get-samples
https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/{evaluation_id}/samples
Get samples for a specific evaluation
# List Evaluations
Source: https://docs.primeintellect.ai/api-reference/evals/list-evaluations
https://api.primeintellect.ai/openapi.json get /api/v1/evaluations/
Get a list of evaluations owned by the authenticated user or their teams
By default, returns all evaluations the user has access to (personal + all teams).
# Push Samples
Source: https://docs.primeintellect.ai/api-reference/evals/push-samples
https://api.primeintellect.ai/openapi.json post /api/v1/evaluations/{evaluation_id}/samples
Push evaluation samples
This endpoint can be called multiple times to stream samples as they're generated.
# Update Evaluation
Source: https://docs.primeintellect.ai/api-reference/evals/update-evaluation
https://api.primeintellect.ai/openapi.json put /api/v1/evaluations/{evaluation_id}
Update an existing evaluation
# Submit Feedback
Source: https://docs.primeintellect.ai/api-reference/feedback/submit-feedback
https://api.primeintellect.ai/openapi.json post /api/v1/feedback
# Cancel Hosted Evaluation Route
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/cancel-hosted-evaluation-route
https://api.primeintellect.ai/openapi.json patch /api/v1/hosted-evaluations/{evaluation_id}/cancel
Cancel a running hosted evaluation.
# Create Hosted Evaluation
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/create-hosted-evaluation
https://api.primeintellect.ai/openapi.json post /api/v1/hosted-evaluations
Create and start a hosted evaluation.
# Get Hosted Evaluation Logs Route
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/get-hosted-evaluation-logs-route
https://api.primeintellect.ai/openapi.json get /api/v1/hosted-evaluations/{evaluation_id}/logs
Get real-time logs from the sandbox running a hosted evaluation.
# Get Inference Models
Source: https://docs.primeintellect.ai/api-reference/hosted-evaluations/get-inference-models
https://api.primeintellect.ai/openapi.json get /api/v1/hosted-evaluations/models
Get available models from Prime Inference API for hosted evaluations
# Build Vm Image
Source: https://docs.primeintellect.ai/api-reference/images/build-vm-image
https://api.primeintellect.ai/openapi.json post /api/v1/images/{image_name}/{image_tag}/vm-build
Build a VM image from an existing container image.
Converts the image's current registry content into a VM image without
rebuilding the container image. If the image already has a VM artifact,
this triggers a rebuild while keeping the registered artifact available
until the replacement completes. Requires VM sandboxes to be enabled for
the owning account and a linux/amd64 image; for team images, only the
image creator or team admins may trigger it. Poll /images/build/{build_id}
for progress.
## Rate Limit
300 requests per 60 seconds per IP and token.
# Delete User Image
Source: https://docs.primeintellect.ai/api-reference/images/delete-user-image
https://api.primeintellect.ai/openapi.json delete /api/v1/images/{image_name}/{image_tag}
Delete a user image from the database and Google Artifact Registry.
For team images, only the image creator or team admins can delete.
## Rate Limit
100 requests per 60 seconds per IP and token.
# Get Build Status
Source: https://docs.primeintellect.ai/api-reference/images/get-build-status
https://api.primeintellect.ai/openapi.json get /api/v1/images/build/{build_id}
Get the status of an image build.
# Initiate Image Build
Source: https://docs.primeintellect.ai/api-reference/images/initiate-image-build
https://api.primeintellect.ai/openapi.json post /api/v1/images/build
Initiate an image build process.
Returns a presigned URL to upload the build context (tar.gz file).
After uploading, call /images/build/{build_id}/start to begin the build.
## Rate Limit
150 requests per 60 seconds per IP and token.
# List Image Builds
Source: https://docs.primeintellect.ai/api-reference/images/list-image-builds
https://api.primeintellect.ai/openapi.json get /api/v1/images/builds
List all image builds for the current user.
# List User Images
Source: https://docs.primeintellect.ai/api-reference/images/list-user-images
https://api.primeintellect.ai/openapi.json get /api/v1/images
List images and builds for the current user, scoped by team context. Returns personal images by default, or team images if teamId is provided.
Pagination note: `limit`, `offset`, and `totalCount` count logical images
(unique `owner/imageName:imageTag`). A single image may produce multiple
`data[]` rows (one per artifact type plus any active builds), so
`len(data)` can exceed `limit` and `totalCount` reflects images, not rows.
The optional `search` parameter narrows results (case-insensitively) to
images whose name, tag, or `name:tag` reference contains the given
substring.
Setting `ownerScope=platform` lists only org-less platform images and
builds; this requires platform admin access.
## Rate Limit
300 requests per 60 seconds per IP and token.
# Start Image Build
Source: https://docs.primeintellect.ai/api-reference/images/start-image-build
https://api.primeintellect.ai/openapi.json post /api/v1/images/build/{build_id}/start
Start the image build process after uploading the build context.
This creates a sandbox with Docker-in-Docker to build and push the image.
# Update Images
Source: https://docs.primeintellect.ai/api-reference/images/update-images
https://api.primeintellect.ai/openapi.json patch /api/v1/images
Update one or many logical images: visibility, name/tag, and owner.
Applies up to 100 independent patches; each item names its source image
exactly and may change `visibility`, rename `name`/`tag`, move a personal
image into one of the caller's teams, or promote a personal/team image to
an org-less platform image (platform admins only; platform images are
always PUBLIC).
Renames and owner moves update the whole logical group (container + VM
artifacts and their linked builds) atomically without moving the backing
artifacts: the old reference stops resolving and the new reference
resolves to the same content. An owner move settles storage billing to
the source wallet at the cutover instant and bills the destination
afterwards. Destinations are never overwritten (`destination_exists`).
`dryRun` performs resolution, authorization, collision detection, and
quota projection without writing; a valid request with item-specific
failures still returns 200 with per-item errors.
## Rate Limit
50 requests per 60 seconds per IP and token.
# Chat Completions
Source: https://docs.primeintellect.ai/api-reference/inference-chat-completions
Generate text responses using language models
Create model responses for chat conversations using OpenAI-compatible API.
## Base URL
```
https://api.pinference.ai/api/v1
```
## Authentication
All requests require a Bearer token in the Authorization header:
```bash theme={null}
Authorization: Bearer your_api_key
```
### Team Account Usage
When using a team account, you must include the `X-Prime-Team-ID` header. Without this header, requests default to your personal account instead of your team account.
```bash theme={null}
X-Prime-Team-ID: your-team-id-here
```
Find your Team ID on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile).
## Create Chat Completion
Generate a response from a language model given a conversation history.
### Request
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
}'
# With team account (add X-Prime-Team-ID header)
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "X-Prime-Team-ID: your-team-id-here" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
}'
```
```python Python theme={null}
import openai
# Personal account
client = openai.OpenAI(
api_key="your-api-key-here",
base_url="https://api.pinference.ai/api/v1"
)
# Team account (required: set X-Prime-Team-ID header)
client = openai.OpenAI(
api_key="your-api-key-here",
base_url="https://api.pinference.ai/api/v1",
default_headers={
"X-Prime-Team-ID": "your-team-id-here"
}
)
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "user", "content": "What is the capital of France?"}
]
)
print(response.choices[0].message.content)
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': 'Bearer your-api-key-here',
'Content-Type': 'application/json'
// Add 'X-Prime-Team-ID': 'your-team-id-here' for team accounts
},
body: JSON.stringify({
model: 'meta-llama/llama-3.1-70b-instruct',
messages: [
{ role: 'user', content: 'What is the capital of France?' }
]
})
});
const data = await response.json();
console.log(data.choices[0].message.content);
```
### Parameters
| Parameter | Type | Required | Description |
| ------------- | ------------ | -------- | ------------------------------ |
| `model` | string | Yes | Model ID to use for completion |
| `messages` | array | Yes | Conversation messages |
| `max_tokens` | integer | No | Maximum tokens to generate |
| `temperature` | number | No | Sampling temperature (0-2) |
| `stream` | boolean | No | Enable streaming responses |
| `stop` | string/array | No | Stop sequences |
### Response
```json theme={null}
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1693721698,
"model": "meta-llama/llama-3.1-70b-instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 8,
"total_tokens": 20
}
}
```
## Streaming
Enable real-time response streaming by setting `stream: true`:
```python Python theme={null}
stream = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")
```
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [{"role": "user", "content": "Tell me a story"}],
"stream": true
}'
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': 'Bearer your-api-key-here',
'Content-Type': 'application/json'
},
body: JSON.stringify({
model: 'meta-llama/llama-3.1-70b-instruct',
messages: [{ role: 'user', content: 'Tell me a story' }],
stream: true
})
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
console.log(chunk);
}
```
## Advanced Parameters
### Temperature Control
```python theme={null}
# Deterministic output
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Solve: 2+2=?"}],
temperature=0.1
)
# Creative output
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Write a poem"}],
temperature=0.9
)
```
### System Messages
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "system", "content": "You are a helpful math tutor."},
{"role": "user", "content": "Explain calculus"}
]
)
```
## Error Handling
### Rate Limit (429)
```json theme={null}
{
"error": {
"message": "Rate limit exceeded",
"type": "rate_limit_error",
"code": "rate_limit_exceeded"
}
}
```
### Invalid Model (400)
```json theme={null}
{
"error": {
"message": "Invalid model specified",
"type": "invalid_request_error",
"code": "invalid_model"
}
}
```
### Context Length Exceeded (400)
```json theme={null}
{
"error": {
"message": "Context length exceeded",
"type": "invalid_request_error",
"code": "context_length_exceeded"
}
}
```
# Models
Source: https://docs.primeintellect.ai/api-reference/inference-models
List and retrieve language models for inference
The Models API allows you to list and retrieve information about available language models in the Prime Intellect Inference service.
## Base URL
```
https://api.pinference.ai/api/v1
```
## Authentication
All requests require a Bearer token in the Authorization header:
```bash theme={null}
Authorization: Bearer your_api_key
```
### Team Account Usage
When using a team account, you must include the `X-Prime-Team-ID` header. Without this header, requests default to your personal account instead of your team account.
```bash theme={null}
X-Prime-Team-ID: your-team-id-here
```
Find your Team ID on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile).
## List Models
Returns a list of all available models that you can use for inference requests.
### Request
```bash cURL theme={null}
curl -X GET https://api.pinference.ai/api/v1/models \
-H "Authorization: Bearer $API_KEY"
# With team account (add X-Prime-Team-ID header)
curl -X GET https://api.pinference.ai/api/v1/models \
-H "Authorization: Bearer $API_KEY" \
-H "X-Prime-Team-ID: your-team-id-here"
```
```python Python theme={null}
import openai
# Personal account
client = openai.OpenAI(
api_key="your-api-key-here",
base_url="https://api.pinference.ai/api/v1"
)
# Team account (required: set X-Prime-Team-ID header)
client = openai.OpenAI(
api_key="your-api-key-here",
base_url="https://api.pinference.ai/api/v1",
default_headers={
"X-Prime-Team-ID": "your-team-id-here"
}
)
models = client.models.list()
for model in models.data:
print(f"Model: {model.id}")
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/models', {
headers: {
'Authorization': 'Bearer your-api-key-here'
// Add 'X-Prime-Team-ID': 'your-team-id-here' for team accounts
}
});
const models = await response.json();
console.log(models);
```
### Response
```json theme={null}
{
"object": "list",
"data": [
{
"id": "meta-llama/llama-3.1-70b-instruct",
"object": "model",
"owned_by": "meta",
"created": 1693721698
},
{
"id": "anthropic/claude-3-5-sonnet-20241022",
"object": "model",
"owned_by": "anthropic",
"created": 1693721698
}
]
}
```
## Get Model Details
Retrieve detailed information about a specific model.
### Request
```bash cURL theme={null}
curl -X GET https://api.pinference.ai/api/v1/models/meta-llama/llama-3.1-70b-instruct \
-H "Authorization: Bearer $API_KEY"
```
```python Python theme={null}
model = client.models.retrieve("meta-llama/llama-3.1-70b-instruct")
print(model)
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/models/meta-llama/llama-3.1-70b-instruct', {
headers: {
'Authorization': 'Bearer your-api-key-here'
}
});
const model = await response.json();
console.log(model);
```
### Response
```json theme={null}
{
"id": "meta-llama/llama-3.1-70b-instruct",
"object": "model",
"owned_by": "meta",
"created": 1693721698
}
```
## Error Responses
### Model Not Found (404)
```json theme={null}
{
"error": {
"message": "The model 'invalid-model' does not exist",
"type": "invalid_request_error",
"code": "model_not_found"
}
}
```
### Authentication Error (401)
```json theme={null}
{
"error": {
"message": "Invalid API key",
"type": "authentication_error",
"code": "invalid_api_key"
}
}
```
# API Overview
Source: https://docs.primeintellect.ai/api-reference/introduction
The Prime Intellect API provides programmatic access to our full platform. Get started by [setting up an API Key](./api-keys).
## Inference API
Access frontier models via an OpenAI-compatible API.
View available models and their capabilities
Generate text with frontier models
## Compute API
Provision and manage GPU instances across providers.
View GPU resources and availability across providers
Deploy GPU instances with customizable configurations
Monitor and control your active GPU instances
Create persistent network-attached disks
## Sandboxes API
Run code securely in isolated containers.
Learn about sandboxes and their use cases
Create and manage sandboxes programmatically
# Managing Disks
Source: https://docs.primeintellect.ai/api-reference/managing-disks
How to create and manage network-attached storage disks
Before you start, ensure that you have [API Key](./api-keys) with `Disks -> Read and write` permission
## Overview
Network-attached disks provide persistent storage that can be shared across multiple GPU instances. Disks persist independently from instances, making them ideal for storing datasets, model checkpoints, and other data that needs to survive instance termination.
## Finding Available Disk Options
Before creating a disk, use the disk availability endpoint to find available storage options across different providers and datacenters.
### Query Disk Availability
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/disks?regions=united_states&page=1&page_size=50' \
--header 'Authorization: Bearer your_api_key'
```
### Understanding the Response
The response returns available disk configurations:
```json theme={null}
[
{
"cloudId": null,
"provider": "hyperstack",
"dataCenter": "US-1",
"country": "US",
"region": "united_states",
"spec": {
"minCount": 0,
"defaultCount": 0,
"maxCount": 100000,
"pricePerUnit": 0.00015,
"step": 1,
"defaultIncludedInPrice": false,
"additionalInfo": null
},
"stockStatus": "Available",
"security": "secure_cloud",
"isMultinode": false
}
]
```
### Key Fields
* **`cloudId`**: Identifies the storage configuration (if applicable)
* **`provider`**: The provider offering the storage (e.g., `hyperstack`, `runpod`)
* **`dataCenter`**: Location where the disk will be created
* **`spec.minCount`**: Minimum disk size in GB
* **`spec.maxCount`**: Maximum disk size in GB
* **`spec.pricePerUnit`**: Cost per GB per hour
* **`isMultinode`**: Whether the disk can be attached to multiple instances
Filter by `regions` or `data_center_id` to find disks in specific locations where you plan to run your GPU instances.
## Creating a Disk
Once you've identified a suitable disk configuration from the availability API, create a disk using the information from the availability response.
### Request Body Structure
```json theme={null}
{
"disk": {
"size": 500,
"name": "training-dataset",
"dataCenterId": "US-1"
},
"provider": {
"type": "hyperstack"
}
}
```
### Parameters
| Parameter | Required | Description |
| ------------------- | -------- | -------------------------------------------------------------------------------- |
| `disk.size` | Yes | Disk size in GB (must be between `minCount` and `maxCount` from availability) |
| `disk.name` | No | Human-friendly name for the disk (auto-generated if not provided) |
| `disk.cloudId` | Yes\* | Cloud ID from the availability response (\*if provided in availability response) |
| `disk.dataCenterId` | Yes\* | Data center ID from availability (\*if provided in availability response) |
| `provider.type` | Yes | Provider type from availability response (e.g., `hyperstack`, `runpod`) |
### Example: Creating a 500GB Disk
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/disks/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"disk": {
"size": 500,
"name": "ml-training-data",
"dataCenterId": "US-1"
},
"provider": {
"type": "hyperstack"
}
}'
```
### Response
On successful creation, you'll receive a `201 Created` response with the disk details:
```json theme={null}
{
"id": "fdb5205fd9b14c9d804d3f70b4c96da0",
"name": "ml-training-data",
"createdAt": "2025-11-07T21:07:15.183000",
"updatedAt": "2025-11-07T21:07:15.183000",
"terminatedAt": null,
"status": "PROVISIONING",
"providerType": "hyperstack",
"size": 500,
"info": {
"cloudId": null,
"country": "US",
"isMultinode": false,
"dataCenterId": "US-1"
},
"priceHr": 0.075,
"stoppedPriceHr": 0.075,
"provisioningPriceHr": 0.0,
"userId": "clvb0itli0000oxevorvgrpfn",
"teamId": null,
"walletId": "clvb0iv0c0004oxev5mnt1xiv",
"pods": [],
"clusters": []
}
```
### Cost Calculation
The hourly cost is calculated as:
```
Hourly Cost = size (GB) × pricePerUnit (from availability)
```
For a 500GB disk at \$0.00015/GB/hr:
```
500 GB × $0.00015/GB/hr = $0.075/hr
```
Disks are billed continuously from creation until termination, regardless of whether they're attached to an instance.
## Listing Your Disks
Retrieve all disks associated with your account:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/disks/?limit=50&offset=0' \
--header 'Authorization: Bearer your_api_key'
```
### Query Parameters
| Parameter | Default | Description |
| --------- | ------- | ---------------------------------------- |
| `limit` | 50 | Number of results per page (max 200) |
| `offset` | 0 | Number of results to skip for pagination |
### Response
```json theme={null}
{
"totalCount": 1,
"data": [
{
"id": "fdb5205fd9b14c9d804d3f70b4c96da0",
"name": "ml-training-data",
"createdAt": "2025-11-07T21:07:15.183000",
"updatedAt": "2025-11-07T21:08:01.934000",
"terminatedAt": null,
"status": "ACTIVE",
"providerType": "hyperstack",
"size": 500,
"info": {
"cloudId": null,
"country": "US",
"isMultinode": false,
"dataCenterId": "US-1"
},
"priceHr": 0.075,
"stoppedPriceHr": 0.075,
"provisioningPriceHr": 0.0,
"userId": "clvb0itli0000oxevorvgrpfn",
"teamId": null,
"walletId": "clvb0iv0c0004oxev5mnt1xiv",
"pods": [],
"clusters": []
}
],
"limit": 50,
"offset": 0
}
```
## Getting a Single Disk
Retrieve detailed information about a specific disk:
```bash theme={null}
curl --request GET \
--url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
--header 'Authorization: Bearer your_api_key'
```
### Response
```json theme={null}
{
"id": "fdb5205fd9b14c9d804d3f70b4c96da0",
"name": "ml-training-data",
"createdAt": "2025-11-07T21:07:15.183000",
"updatedAt": "2025-11-07T21:08:01.934000",
"terminatedAt": null,
"status": "ACTIVE",
"providerType": "hyperstack",
"size": 500,
"info": {
"cloudId": null,
"country": "US",
"isMultinode": false,
"dataCenterId": "US-1"
},
"priceHr": 0.075,
"stoppedPriceHr": 0.075,
"provisioningPriceHr": 0.0,
"userId": "clvb0itli0000oxevorvgrpfn",
"teamId": null,
"walletId": "clvb0iv0c0004oxev5mnt1xiv",
"pods": [],
"clusters": []
}
```
The response includes:
* **`pods`**: Array of pod IDs currently using this disk
* **`clusters`**: Array of cluster IDs currently using this disk
* **`info`**: Additional metadata about disk location and capabilities
## Updating a Disk
Currently, you can update the disk name:
```bash theme={null}
curl --request PATCH \
--url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"name": "updated-training-dataset"
}'
```
### Response
```json theme={null}
{
"id": "fdb5205fd9b14c9d804d3f70b4c96da0",
"name": "updated-training-dataset"
}
```
## Deleting a Disk
Terminate a disk when you no longer need it. This will permanently delete the disk and all its data.
```bash theme={null}
curl --request DELETE \
--url https://api.primeintellect.ai/api/v1/disks/fdb5205fd9b14c9d804d3f70b4c96da0 \
--header 'Authorization: Bearer your_api_key'
```
### Response
```json theme={null}
{
"status": "TERMINATED"
}
```
**Disk deletion is permanent and irreversible**. All data stored on the disk will be lost. Make sure to backup any important data before deleting a disk.
**Detachment Timing**: The disk detachment process can take longer than instance termination. After terminating an instance, you may need to wait a few moments before the disk becomes available for deletion or reuse.
## Disk Status Values
Disks go through different statuses during their lifecycle:
| Status | Description |
| -------------- | --------------------------------------------------------- |
| `PROVISIONING` | Disk is being created |
| `PENDING` | Disk status is changing |
| `ACTIVE` | Disk is ready and can be attached to instances or deleted |
| `STOPPED` | Disk is stopped |
| `ERROR` | An error occurred during disk operations |
| `DELETING` | Disk is being deleted |
| `TERMINATED` | Disk has been deleted |
| `UNKNOWN` | Disk status is unknown (cannot access the current status) |
After terminating an instance, the disk detachment process may take a few moments to complete. You may need to wait briefly before the disk becomes available for deletion or reattachment.
## Common Use Cases
### Shared Training Data
Create large disks with your training datasets that can be reused across multiple training runs. This is ideal for storing preprocessed data, large image datasets, or any training data that needs to be accessed by multiple GPU instances. By keeping your data on a persistent disk, you avoid having to re-upload or regenerate datasets for each new training session.
### Model Checkpoints and Artifacts
Use disks to persist model checkpoints, trained weights, and training artifacts. This is especially valuable when running spot instances or interruptible workloads, as your progress remains safe even if the instance is terminated. You can resume training from the last checkpoint by simply attaching the disk to a new instance.
### Multi-Node Distributed Training
For distributed training across multiple GPUs or nodes, use multinode-compatible disks that can be accessed simultaneously by multiple instances. This enables shared access to training data and synchronized checkpointing across your distributed training cluster. Check the `isMultinode` field in the disk availability response to verify support.
## Best Practices
### Location Planning
* Create disks in the same datacenter where you plan to run GPU instances
* Use the [availability endpoint](./check-gpu-availability) with `disks` filter to find compatible GPUs
### Size Planning
* Start with the size needed - you cannot resize disks after creation
* Consider future growth when selecting disk size
* Remember that larger disks cost more per hour
### Data Management
* Regularly backup important data from disks
* Use descriptive names to identify disk contents
## Related Documentation
* [Check GPU Availability](./check-gpu-availability) - Filter GPUs by disk location
* [Provision GPU](./provision-gpu) - Attach disks when creating instances
* [Managing Instances](./managing-pods) - Manage instances with attached disks
# Managing Pods
Source: https://docs.primeintellect.ai/api-reference/managing-pods
How to get pods, statuses and delete instances
Before you start, ensure that you have [API Key](./api-keys) with `Instances -> Read` permission (or `Read and
Write` if you want to `Delete` instances)
## Retrieving Existing Pods
To view a list of active instances, use the [Get Pods](./pods/get-pods) endpoint.
```bash theme={null}
curl --request GET \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key'
```
This is a paginated request, with a default `limit` of **100** results. To adjust the number of results per page, use query parameters:
```bash theme={null}
curl --request GET \
--url https://api.primeintellect.ai/api/v1/pods/?offset=0&limit=10 \
--header 'Authorization: Bearer your_api_key'
```
This will return a JSON object containing the total number of instances (`total_count`), the specified `limit` and `offset` and the list of pods under `data` (check out [Pod schema](./pods/get-pod) for details)
```json theme={null}
{
"total_count": 15,
"offset": 0,
"limit": 10,
"data": [...]
}
```
### Retrieving a Specific Pod
If you know the `podId`, you can retrieve details for that specific pod:
```bash theme={null}
curl --request GET \
--url https://api.primeintellect.ai/api/v1/pods/my_pod_id \
--header 'Authorization: Bearer your_api_key'
```
## Checking Pod Status
After provisioning an instance, retrieve its status to obtain port mappings and connection details using the [Get Pod Status](./pods/status) endpoint. This request also supports checking multiple instance statuses simultaneously:
```bash theme={null}
curl --request GET \
--url https://api.primeintellect.ai/api/v1/pods/status/?pod_ids=my_first_pod_id&pod_ids=my_second_pod_id \
--header 'Authorization: Bearer your_api_key'
```
It returns a `data` with the list of object like:
```json theme={null}
{
"podId": "my_first_pod_id",
"providerType": "primeconpute",
"status": "ACTIVE",
"sshConnection": "root@135.23.125.123 -p 22",
"costPerHr": 3.52,
"primePortMapping": [
{
"internal": "22",
"external": "22",
"protocol": "TCP",
"usedBy": "SSH",
"description": ""
}
],
"ip": "135.23.125.123",
"installationFailure": null,
"installationProgress": 100
}
```
If the instance has additional open ports, these will be listed under `primePortMapping`.
In case of an error, `installationFailure` will be set. Our system will attempt automatic retries, but issues related to the image or configuration may require further action.
## Deleting a Pod
To delete an instance you no longer need, use the [Delete Pod](./pods/delete-pod) endpoint.
```bash theme={null}
curl --request DELETE \
--url https://api.primeintellect.ai/api/v1/pods/my_pod_id \
--header 'Authorization: Bearer your_api_key'
```
Once a pod is terminated, it is moved to history and becomes inaccessible via the standard [Get Pod](./pods/get-pod) or [Get Pods](./pods/get-pods) endpoints. To view historical data, use the [Get Pods History](./pods/get-pods-history) endpoint.
# Create Pod
Source: https://docs.primeintellect.ai/api-reference/pods/create-pod
https://api.primeintellect.ai/openapi.json post /api/v1/pods/
# Delete Pod
Source: https://docs.primeintellect.ai/api-reference/pods/delete-pod
https://api.primeintellect.ai/openapi.json delete /api/v1/pods/{pod_id}
# Get Pod
Source: https://docs.primeintellect.ai/api-reference/pods/get-pod
https://api.primeintellect.ai/openapi.json get /api/v1/pods/{pod_id}
# Get Pod Logs Api
Source: https://docs.primeintellect.ai/api-reference/pods/get-pod-logs-api
https://api.primeintellect.ai/openapi.json get /api/v1/pods/{pod_id}/log
Get logs for a pod.
Args:
tail: Number of most recent logs to return (default: 100)
# Get Pods
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods
https://api.primeintellect.ai/openapi.json get /api/v1/pods/
# Get Pods History
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods-history
https://api.primeintellect.ai/openapi.json get /api/v1/pods/history
# Get Pods Status
Source: https://docs.primeintellect.ai/api-reference/pods/get-pods-status
https://api.primeintellect.ai/openapi.json get /api/v1/pods/status
# Provision Instance
Source: https://docs.primeintellect.ai/api-reference/provision-gpu
How to provision an instance using availability data
Before you start, ensure that you have [API Key](./api-keys) with `Instances -> Read and write` permission
## Retrieving offers from the availability API
For an in-depth guide to using the availability endpoint, refer to [This Guide](./check-gpu-availability)
Our goal is to provision H100 GPU using availability data. First, we need to call the availability endpoint to get the current offers:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/gpus?gpu_type=H100_80GB®ions=united_states®ions=canada&gpu_count=1' \
--header 'Authorization: Bearer your_api_key'
```
We will use the Hyperstack provider and the following offer:
```json theme={null}
{
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"provider": "hyperstack",
"dataCenter": "CANADA-1",
"country": "CA",
"gpuCount": 1,
"gpuMemory": 80,
"disk": {
"minCount": null,
"defaultCount": 100,
"maxCount": null,
"pricePerUnit": null,
"step": null,
"defaultIncludedInPrice": null,
"additionalInfo": null
},
"vcpu": {
"minCount": null,
"defaultCount": 180,
"maxCount": null,
"pricePerUnit": null,
"step": null,
"defaultIncludedInPrice": null,
"additionalInfo": null
},
"memory": {
"minCount": null,
"defaultCount": 180,
"maxCount": null,
"pricePerUnit": null,
"step": null,
"defaultIncludedInPrice": null,
"additionalInfo": null
},
"internetSpeed": null,
"interconnect": null,
"interconnectType": null,
"provisioningTime": null,
"stockStatus": "Available",
"security": "secure_cloud",
"prices": {
"onDemand": 1.9,
"isVariable": null,
"currency": "USD"
},
"images": [
"ubuntu_22_cuda_12",
"cuda_12_1_pytorch_2_2",
"cuda_11_8_pytorch_2_1",
"stable_diffusion",
"flux",
"axolotl",
"bittensor",
"vllm_llama_8b",
"vllm_llama_70b",
"vllm_llama_405b"
],
"isSpot": null,
"prepaidTime": null
}
```
This GPU configuration has fixed resources, so we don't have to worry about those and just use the default ones.
## Creating the Instance Request Body
First, lets go through the [**Create Pod**](./pods/create-pod) endpoint and explain how it works. The request requires a `body` with `pod`, `provider` and optional `team` definitions.
### `pod`
The `pod` object defines the instance’s characteristics:
```json theme={null}
"pod": {
"name": "My first pod",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud"
}
```
We can choose any `name`, but the rest of the parameters is copied from the availability offer. Since this offer includes a `dataCenterId` and `country`, we're going to pass those values during provisioning, as it indicates that the provider has GPUs with the same `cloudId` available in different locations. We also copy the rest of the GPU definition data:
* `gpuType -> gpuType`
* `socket -> socket`
* `gpuCount -> gpuCount`
* `security -> security`
Last thing to do is to select an `image`. Available values are stored within the `images` property of the availability offer. We're going to select the default `ubuntu_22_cuda_12` image.
### `provider`
```json theme={null}
"provider": {
"type": "hyperstack"
},
```
The `provider` object is straightforward, we only need to specify the `type`, which in our case is `hyperstack`.
### `team`
```json theme={null}
"team": {
"teamId": "my_team_id"
}
```
If you want to assign the pod to a specific `team`, include the team object.
You can find the `teamId` on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile)
### `sharedWithTeam`
```json theme={null}
"sharedWithTeam": true
```
Set to `true` to share the instance with **all members** of the team. When enabled, every team member's SSH keys are added to the instance, giving them direct access. Requires `team.teamId` to be set.
### `teamMemberIds`
```json theme={null}
"teamMemberIds": ["clvb1jtmk0001pxfv1abc2def", "clvb2kumr0002qygw3bcd4efg"]
```
Alternatively, provide a list of specific user IDs to share the instance with. Only those members' SSH keys will be added. Use the [List Team Members](./teams) endpoint to get user IDs.
You can use either `sharedWithTeam` for all members or `teamMemberIds` for specific members. If both are provided, `sharedWithTeam` takes precedence.
## Sending the Create Request
With all parts configured, the final request looks like this:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "My first pod",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud"
},
"provider": {
"type": "hyperstack"
}
}'
```
On successful completion, you should receive a `200 OK` response with the pod details in the [Response Body](./pods/create-pod).
## Modifying instance resources
If the availability offer allows resource customization, you can adjust the default resources during provisioning. Below is an example of a modified resource configuration:
```json theme={null}
"disk": {
"minCount": 50,
"defaultCount": 100,
"maxCount": 1000,
"pricePerUnit": 0.0003,
"step": 10,
"defaultIncludedInPrice": false,
"additionalInfo": null
},
"vcpu": {
"minCount": 4,
"defaultCount": 16,
"maxCount": 32,
"pricePerUnit": 0.004,
"step": 2,
"defaultIncludedInPrice": true,
"additionalInfo": null
},
```
This configuration enables changes to `disk` and `vcpu` specifications.
Be aware that default `vcpu` is included in price (`defaultIncludedInPrice` set to `true`) the default disk is not, which results in an **additional cost of \$0.03** when using the default disk size.
### Increase disk size
To increase the disk size, set a new value in the `pod` property when sending the create request. For this example, since the provider supports increments of **10** with a minimum of **50**, we'll set the disk size to **200**. This adjustment will affect the total hourly cost as follows:
```
GPU cost: $2.69
vcpu cost: $0.00 (we're not paying for vcpu because `defaultIncludedInPrice == true`)
memory cost: $0.00
disk cost: $0.06 (200 units * $0.0003)
Total cost: $2.75
```
So the final request will look like:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "My first pod",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud",
"diskSize": 200
},
"provider": {
"type": "hyperstack"
}
}'
```
### Modifying vcpu
This case is a little more complicated. Because `defaultIncludedInPrice` allows us to use default of **16 vcpus for free**, there are 2 options in which we're going to pay additional amount for vcpus
#### Increasing vcpu
Raising `vcpu` to **20** will increase the cost beyond the base instance price:
```
GPU cost: $2.69
vcpu cost: $0.08 (20 units * $0.004)
memory cost: $0.00
disk cost: $0.03 (100 units[default] * $0.0003)
Total cost: $2.80
```
#### Decreasing vcpu
Reducing `vcpu` to **10** can also increase costs compared to the default configuration. This is because some servers use predefined containers, and altering configurations may incur additional fees, making it more economical to use the default setup:
```
GPU cost: $2.69
vcpu cost: $0.04 (10 units * $0.004)
memory cost: $0.00
disk cost: $0.03 (100 units[default] * $0.0003)
Total cost: $2.76
```
### Example request with adjusted disk and vcpu
With both `disk` and `vcpu` increased our request will look like:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "My first pod",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud",
"diskSize": 200,
"vcpus": 20
},
"provider": {
"type": "hyperstack"
}
}'
```
and the total cost breakdown is as follows:
```
GPU cost: $2.69
vcpu cost: $0.08 (20 units * $0.004)
memory cost: $0.00
disk cost: $0.06 (200 units[default] * $0.0003)
Total cost: $2.83
```
## Dynamic pricing
Certain offers feature dynamic pricing, meaning that rates may fluctuate throughout the instance’s lifetime due to factors such as:
* Price being pegged to a foreign currency and fluctuating with exchange rates
* Payment in alternative currencies or tokens
* Market-based pricing
If an offer supports dynamic pricing, `prices -> isVariable` will be set to `true`. In this case, it’s recommended to specify a `maxPrice` when provisioning the instance to set a cap:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "My first pod",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud",
"maxPrice": 2.70,
},
"provider": {
"type": "hyperstack"
}
}'
```
This configuration limits provisioning to instances at or below the specified price. However, **prices may still vary during the instance’s lifetime**.
## Attaching Existing Disks
You can attach existing disks with persisted data when provisioning a new GPU instance. This is particularly useful when you want to reuse datasets, models, or checkpoints stored on network-attached disks without re-uploading them.
### Finding Compatible GPUs
Before provisioning, use the [availability endpoint](./check-gpu-availability) with the `disks` filter to find GPU instances that can attach to your existing disks:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/availability/gpus?disks=clhxy6aw80000j8080gdf8kqv&gpu_type=H100_80GB' \
--header 'Authorization: Bearer your_api_key'
```
This returns only GPU offers available in the same location as your disk.
### Provisioning with Disks
To attach disks when creating an instance, add the `disks` array to your request body:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "Training instance with data",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"security": "secure_cloud"
},
"provider": {
"type": "hyperstack"
},
"disks": ["clhxy6aw80000j8080gdf8kqv"]
}'
```
### Response
On successful provisioning, you'll receive a response with the instance details and attached disk information:
```json theme={null}
{
"id": "11de9260fb52451085f3936c20b0ffdb",
"userId": "clvb0itli0000oxevorvgrpfn",
"teamId": null,
"walletId": "clvb0iv0c0004oxev5mnt1xiv",
"name": "Training instance with data",
"providerType": "hyperstack",
"status": "PROVISIONING",
"installationStatus": "PENDING",
"installationFailure": null,
"installationProgress": null,
"createdAt": "2025-11-07T23:36:36.391000",
"updatedAt": "2025-11-07T23:36:36.418000",
"terminatedAt": null,
"gpuName": "H100_80GB",
"gpuCount": 1,
"socket": "PCIe",
"priceHr": 1.9,
"stoppedPriceHr": null,
"provisioningPriceHr": 0.0,
"environmentType": "ubuntu_22_cuda_12",
"customTemplateId": null,
"clusterId": null,
"primePortMapping": [
{
"internal": "22",
"external": "22",
"protocol": "TCP",
"usedBy": "SSH",
"description": "SSH access"
},
{
"internal": "*",
"external": "*",
"protocol": "TCP",
"usedBy": null,
"description": "All other ports"
}
],
"sshConnection": null,
"ip": null,
"resources": {
"memory": 180,
"disk": 1250,
"sharedDisk": null,
"vcpus": 28
},
"attachedResources": [
{
"resourceType": "DISK",
"id": "clhxy6aw80000j8080gdf8kqv",
"status": "UNATTACHED",
"isDetachable": true,
"mountPath": "/data",
"resourcePath": "/dev/vdc",
"size": 500,
"isShared": false
},
{
"resourceType": "DISK",
"id": "ephemeral",
"status": "ATTACHED",
"isDetachable": false,
"mountPath": "/ephemeral",
"resourcePath": "/dev/vdb",
"size": 750,
"isShared": false
}
],
"isSpot": null,
"autoRestart": null
}
```
The response includes:
* **`attachedResources`**: Array showing both your persistent disk (`clhxy6aw80000j8080gdf8kqv`) and the ephemeral instance disk
* **`resources.disk`**: Total disk space including both persistent and ephemeral storage (500GB + 750GB = 1250GB)
* **Disk status**: `UNATTACHED` initially, will become `ATTACHED` once the instance is fully provisioned, and then `MOUNTED` when disk it mounted to the working directory.
The persistent disk will be automatically attached and mounted at `/data` once the instance completes provisioning. The ephemeral disk is immediately available at `/ephemeral`.
### Multiple Disks
You can attach multiple disks to a single instance:
```bash theme={null}
curl --request POST \
--url https://api.primeintellect.ai/api/v1/pods/ \
--header 'Authorization: Bearer your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"pod": {
"name": "Multi-disk training",
"cloudId": "n3-H100x1",
"gpuType": "H100_80GB",
"socket": "PCIe",
"gpuCount": 1,
"image": "ubuntu_22_cuda_12",
"dataCenterId": "CANADA-1",
"country": "CA",
"security": "secure_cloud"
},
"provider": {
"type": "hyperstack"
},
"disks": [
"clhxy6aw80000j8080gdf8kqv",
"clhxy9bz50001j8080hdf9lrw"
]
}'
```
### Use Cases
* **Training with Large Datasets**: Attach disks containing preprocessed training data
* **Model Checkpointing**: Resume training from checkpoints stored on persistent disks
* **Shared Data Access**: Use multi-node compatible disks across multiple instances
* **Spot Instances**: Keeping your data when running spot instances
- Only disks you own can be attached to your instances
- Disks must be in an `ACTIVE` state
- Some providers may have limitations on the number of disks that can be attached
# Create Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/create-secret
https://api.primeintellect.ai/openapi.json post /api/v1/secrets/
Create a new global secret.
The secret will be encrypted and stored securely. If teamId is provided,
the secret will be associated with that team (requires team membership).
# Delete Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/delete-secret
https://api.primeintellect.ai/openapi.json delete /api/v1/secrets/{secret_id}
Delete a secret.
This operation cannot be undone. Secrets that are linked to environments
will be automatically unlinked.
# Get Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/get-secret
https://api.primeintellect.ai/openapi.json get /api/v1/secrets/{secret_id}
Get a specific secret by ID.
Note: The secret value is not returned, only metadata.
# List Secrets
Source: https://docs.primeintellect.ai/api-reference/secrets/list-secrets
https://api.primeintellect.ai/openapi.json get /api/v1/secrets/
List global secrets for the authenticated user or a team.
If teamId is provided, returns secrets for that team (requires team membership).
Otherwise, returns the user's personal secrets.
# Update Secret
Source: https://docs.primeintellect.ai/api-reference/secrets/update-secret
https://api.primeintellect.ai/openapi.json patch /api/v1/secrets/{secret_id}
Update an existing secret.
You can update the name, value, and/or description.
# Delete Ssh Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/delete-ssh-key
https://api.primeintellect.ai/openapi.json delete /api/v1/ssh_keys/{key_id}
# Get Ssh Keys
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/get-ssh-keys
https://api.primeintellect.ai/openapi.json get /api/v1/ssh_keys/
# Set Primary Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/set-primary-key
https://api.primeintellect.ai/openapi.json patch /api/v1/ssh_keys/{key_id}
# Upload Ssh Key
Source: https://docs.primeintellect.ai/api-reference/ssh-keys/upload-ssh-key
https://api.primeintellect.ai/openapi.json post /api/v1/ssh_keys/
# Teams
Source: https://docs.primeintellect.ai/api-reference/teams
How to list teams and team members using the API
Before you start, ensure that you have an [API Key](./api-keys) with `Teams -> Read` permission
## Listing Your Teams
Retrieve the teams you belong to using the `GET /user/teams` endpoint.
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/user/teams' \
--header 'Authorization: Bearer your_api_key'
```
This is a paginated request with a default `limit` of **100** results. Use query parameters to adjust:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/user/teams?offset=0&limit=50' \
--header 'Authorization: Bearer your_api_key'
```
### Response
```json theme={null}
{
"total_count": 2,
"offset": 0,
"limit": 50,
"data": [
{
"teamId": "clvb0itli0000oxevorvgrpfn",
"name": "My Team",
"slug": "my-team",
"role": "ADMIN",
"createdAt": "2025-01-15T10:30:00.000000"
},
{
"teamId": "clvb1jtmk0001pxfv1abc2def",
"name": "Research Lab",
"slug": "research-lab",
"role": "MEMBER",
"createdAt": "2025-02-01T14:20:00.000000"
}
]
}
```
### Response Fields
| Field | Type | Description |
| ----------- | ------ | ---------------------------------------------- |
| `teamId` | string | Unique team identifier |
| `name` | string | Team display name |
| `slug` | string | Team URL slug |
| `role` | string | Your role in the team (`ADMIN` or `MEMBER`) |
| `createdAt` | string | ISO 8601 timestamp of when you joined the team |
## Listing Team Members
Retrieve the members of a team using the `GET /teams/{team_id}/members` endpoint. You must be a member of the team to access this endpoint.
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/teams/{team_id}/members' \
--header 'Authorization: Bearer your_api_key'
```
This is a paginated request with a default `limit` of **100** results. Use query parameters to adjust:
```bash theme={null}
curl --request GET \
--url 'https://api.primeintellect.ai/api/v1/teams/{team_id}/members?offset=0&limit=50' \
--header 'Authorization: Bearer your_api_key'
```
### Response
```json theme={null}
{
"total_count": 2,
"offset": 0,
"limit": 50,
"data": [
{
"userId": "clvb0itli0000oxevorvgrpfn",
"userName": "Alice",
"userEmail": "alice@example.com",
"role": "ADMIN",
"joinedAt": "2025-01-15T10:30:00.000000"
},
{
"userId": "clvb1jtmk0001pxfv1abc2def",
"userName": "Bob",
"userEmail": "bob@example.com",
"role": "MEMBER",
"joinedAt": "2025-02-01T14:20:00.000000"
}
]
}
```
### Response Fields
| Field | Type | Description |
| ----------- | ------ | --------------------------------------------------- |
| `userId` | string | Unique user identifier |
| `userName` | string | User's display name |
| `userEmail` | string | User's email address |
| `role` | string | Team role (`ADMIN` or `MEMBER`) |
| `joinedAt` | string | ISO 8601 timestamp of when the user joined the team |
You can find your `teamId` on your [Team's Profile page](https://app.primeintellect.ai/dashboard/team-profile) or by calling `GET /user/teams`.
# List Team Members
Source: https://docs.primeintellect.ai/api-reference/teams/list-team-members
https://api.primeintellect.ai/openapi.json get /api/v1/teams/{team_id}/members
# Check Docker Image
Source: https://docs.primeintellect.ai/api-reference/template/check-docker-image
https://api.primeintellect.ai/openapi.json post /api/v1/template/check-docker-image
Check whether a docker image is accessible (optionally with registry credentials).
# List Registry Credentials
Source: https://docs.primeintellect.ai/api-reference/template/list-registry-credentials
https://api.primeintellect.ai/openapi.json get /api/v1/template/registry-credentials
List registry credentials owned by the requesting user or their teams.
# Create Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/create-dedicated-run
https://api.primeintellect.ai/openapi.json post /api/v1/training/runs
Dispatch a dedicated full-FT prime-rl run on a registered PrimeCluster.
Access is gated by ClusterAllocation - the picker in
`training_service.create_dedicated_run` rejects callers whose
team/user has no matching allocation on a live PrimeCluster. When
`team_id` is set the caller must additionally be a member of that
team (mirrors the LoRA/shared dispatch path).
Config validation (validator schema, hub env ids, HF repo names,
per-run GPU cap) runs sync so bad configs return 400 immediately.
Cluster-side probes (GPU capacity, model cache) and the helm
install run async in a Cloud Task; callers poll
`GET /api/v1/rft/runs/{run_id}` for status transitions PENDING ->
CREATING -> RUNNING (or FAILED with errorMessage on a cluster-side
rejection).
# Delete Dedicated Run
Source: https://docs.primeintellect.ai/api-reference/training/delete-dedicated-run
https://api.primeintellect.ai/openapi.json delete /api/v1/training/runs/{run_id}
Delete a dedicated run: helm uninstall + namespace delete + DB cleanup.
Idempotent. Safe to call repeatedly. Cleans up the linked Job row too
(which uninstalls the helm release on the customer cluster). Soft-
deletes the RFTRun row (matches the existing run-delete UX - Billing
refs are preserved). Access is per-row: `verify_run_ownership`
accepts personal runs the caller owns and team runs the caller is
a member of.
# Get Available Fft Models
Source: https://docs.primeintellect.ai/api-reference/training/get-available-fft-models
https://api.primeintellect.ai/openapi.json get /api/v1/training/available-fft-models
Models pre-cached on the caller's eligible PrimeClusters for FFT dispatch.
Same principal collapse as the dispatch picker (team wins when
present, otherwise personal) and the same cluster filter (uncordoned
+ heartbeat fresh + non-zero ClusterAllocation) so what surfaces
here is what dispatch would actually land on. Only PRESENT-cache
clusters contribute — ABSENT / ERROR would 400 the dispatch and
NULL (unprobed) hasn't reported a manifest yet.
Each model carries the list of clusters it's warm on with `gpu_type`
+ `cache_synced_at` so callers can pin the follow-up dispatch to a
specific GPU family without hitting the "picker landed on a cluster
that has the GPU but not this model" reject path.
# Get Available Gpu Types
Source: https://docs.primeintellect.ai/api-reference/training/get-available-gpu-types
https://api.primeintellect.ai/openapi.json get /api/v1/training/available-gpu-types
Distinct GPU types the caller could dispatch a dedicated FFT run on.
Same principal collapse as the dispatch picker (team wins when
present, otherwise personal) so what the dropdown offers matches
what dispatch would land on. When `team_id` is set the caller must
be a team member, matching the create-run gate.
# Bulk Delete Tunnels Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/bulk-delete-tunnels-endpoint
https://api.primeintellect.ai/openapi.json delete /api/v1/tunnel
Bulk soft-delete multiple tunnels by IDs or labels.
## Rate Limit
100 requests per 60 seconds per user.
# Create Tunnel Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/create-tunnel-endpoint
https://api.primeintellect.ai/openapi.json post /api/v1/tunnel
Create a new tunnel for exposing a local service.
## Rate Limit
600 requests per 60 seconds per user.
# Delete Tunnel Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/delete-tunnel-endpoint
https://api.primeintellect.ai/openapi.json delete /api/v1/tunnel/{tunnel_id}
Delete a tunnel and disconnect any active clients.
## Rate Limit
100 requests per 60 seconds per user.
# Get Tunnel Status Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/get-tunnel-status-endpoint
https://api.primeintellect.ai/openapi.json get /api/v1/tunnel/{tunnel_id}
Get status of a specific tunnel.
## Rate Limit
300 requests per 60 seconds per tunnel.
# List Tunnels Endpoint
Source: https://docs.primeintellect.ai/api-reference/tunnel/list-tunnels-endpoint
https://api.primeintellect.ai/openapi.json get /api/v1/tunnel
List all active tunnels for the current user.
## Rate Limit
300 requests per 60 seconds per user.
# Get Whoami
Source: https://docs.primeintellect.ai/api-reference/user/get-whoami
https://api.primeintellect.ai/openapi.json get /api/v1/user/whoami
# List My Teams
Source: https://docs.primeintellect.ai/api-reference/user/list-my-teams
https://api.primeintellect.ai/openapi.json get /api/v1/user/teams
# Set Username Slug
Source: https://docs.primeintellect.ai/api-reference/user/set-username-slug
https://api.primeintellect.ai/openapi.json patch /api/v1/user/slug
# Get Availability Information
Source: https://docs.primeintellect.ai/cli-reference/check-gpu-availability
How to check GPU and disk availability and pricing
Before you start, ensure you have set your API keys and SSH key path as outlined in the [CLI introduction](/cli-reference/introduction).
## Retrieving GPU Availability Data
To see all available GPU types and their current pricing across all regions, simply run:
```bash theme={null}
prime availability list
```
This command will display a table showing all available GPU configurations, including details like GPU type, count, location, price per hour, and stock status.
## Retrieving Disk Availability Data
To see all available disk storage options across providers and datacenters:
```bash theme={null}
prime availability disks
```
This displays available persistent storage configurations with pricing, maximum capacity, and multinode support information.
## Retrieving Filtered GPU Availability Data
You can then use filters like `--gpu-type`, `--regions`, or `--gpu-count` to narrow down the results. Here are all available filters:
### GPU Availability Filters
* `--gpu-type`: Filter by specific GPU model
* Example: `--gpu-type H100_80GB`
* Optional string value
* `--gpu-count`: Filter by number of GPUs
* Example: `--gpu-count 2`
* Optional integer value
* `--regions`: Filter by geographic regions
* Example: `--regions united_states,canada` or `--regions united_states --regions canada`
* Optional, can specify multiple regions
* `--socket`: Filter by socket type
* Example: `--socket PCIe`
* Accepts values: `PCIe`, `SXM2`, `SXM3`, `SXM4`, `SXM5`
* Optional string value
* `--disks`: Filter GPUs by disk IDs (shows GPUs in same location as disks)
* Example: `--disks disk-id-1 --disks disk-id-2`
* Optional, can specify multiple disk IDs
* Useful for finding GPUs compatible with your existing disks
* `--group-similar`: Group similar configurations from same provider
* Example: `--no-group-similar`
* Default: true
* Optional boolean flag
### Example with Multiple Filters
```bash theme={null}
prime availability list --gpu-type H100_80GB --regions united_states --socket PCIe --no-group-similar
```
### Finding GPUs Compatible with Your Disks
If you have existing disks and want to find GPUs in the same datacenter/region:
```bash theme={null}
# First, list your disks to get their IDs
prime disks list
# Then filter GPUs by disk IDs
prime availability list --disks disk-id-1 --disks disk-id-2
```
This ensures you see only GPU configurations that can access your existing disks.
## Retrieving Filtered Disk Availability Data
### Disk Availability Filters
* `--regions`: Filter by geographic regions
* Example: `--regions united_states,eu_north`
* Optional, can specify multiple regions
* `--data-center-id`: Filter by specific datacenter
* Example: `--data-center-id US-1`
* Optional string value
* `--cloud-id`: Filter by cloud configuration ID
* Example: `--cloud-id "Provider Storage A"`
* Optional string value
### Example with Filters
```bash theme={null}
prime availability disks --regions united_states --data-center-id US-1
```
## Understanding the Terminal Output
### GPU Availability Output
Here's an example of what the command output looks like:
| ID | GPU Type | GPUs | Socket | Provider | Location | Stock | Price/Hr | Memory (GB) | Security | vCPUs | RAM (GB) |
| ------ | ---------- | ---- | ------ | -------- | -------- | ----- | -------- | ----------- | ------------- | ----- | -------- |
| 346663 | H100\_80GB | 2 | PCIe | runpod | CA | Low | \$5.40 | 160 | secure\_cloud | 32 | 502 |
| 551ffd | H100\_80GB | 2 | PCIe | runpod | US | Low | \$5.40 | 160 | secure\_cloud | 32 | 502 |
Each row represents a unique GPU configuration available for deployment. The output includes:
* A unique ID for the configuration
* GPU specifications (type, count, socket)
* Provider and location information
* Current stock status
* Pricing and hardware details
### Disk Availability Output
For disk availability, the output includes:
| ID | Provider | Location | Stock | Price/Hr/GB | Max Size (GB) | Is Multinode |
| ------ | ---------- | ------------- | --------- | ------------ | ------------- | ------------ |
| 7d2232 | runpod | US (US-WA-1) | Available | \$0.00011111 | 8192 | Yes |
| 472b26 | hyperstack | NO (NORWAY-1) | Available | \$0.00009700 | 100000 | No |
Each row represents available disk storage with:
* A unique ID for the configuration
* Provider offering the storage
* Location (country and datacenter)
* Stock availability status
* Price per GB per hour
* Maximum disk size supported
* Whether the disk supports multinode (can be attached to multiple instances)
# Configuration
Source: https://docs.primeintellect.ai/cli-reference/config-cli
Configure your Prime CLI settings
The Prime CLI provides several commands to manage your configuration settings. All configuration commands are accessed through the `prime config` command group.
## View Configuration
View your current configuration settings:
```bash theme={null}
prime config view
```
This displays a table with your current settings, including:
* API Key (partially masked)
* Team ID
* Base URL
* SSH Key Path
## API Key Management
Set your Prime Intellect API key:
```bash theme={null}
prime config set-api-key
```
## Team Settings
Configure your team ID for team-based access:
```bash theme={null}
prime config set-team-id
```
Switch back to your personal account:
```bash theme={null}
prime config remove-team-id
```
## Sharing Settings
Configure whether new instances are automatically shared with all team members:
```bash theme={null}
prime config set-share-resources-with-team true
```
When enabled, `prime pods create` will automatically set `sharedWithTeam` on new instances without needing to pass `--share-with-team` each time. Set to `false` to disable:
```bash theme={null}
prime config set-share-resources-with-team false
```
## Connection Settings
Set the API endpoint URL:
```bash theme={null}
prime config set-base-url
```
Configure SSH key path:
```bash theme={null}
prime config set-ssh-key-path
```
## Reset Configuration
Reset all settings to their default values:
```bash theme={null}
prime config reset
```
This will remove all your custom settings, including your API key. You'll need to reconfigure the CLI after resetting.
## Configuration Options
| Command | Description | Default |
| ------------------------------- | ---------------------------------- | ------------------------------- |
| `view` | Display current configuration | - |
| `set-api-key` | Set your API key | - |
| `set-team-id` | Set team ID for team access | - |
| `remove-team-id` | Switch to personal account | - |
| `set-base-url` | Set API base URL | `https://api.primeintellect.ai` |
| `set-ssh-key-path` | Set SSH private key path | `~/.ssh/id_rsa` |
| `set-share-resources-with-team` | Auto-share new instances with team | `false` |
| `reset` | Reset to default settings | - |
All configuration commands will prompt for values if not provided as command-line arguments.
# Environments Hub
Source: https://docs.primeintellect.ai/cli-reference/environments
Create, install and manage environments from the Environments Hub
For documentation on how to use the Prime CLI with the Environments Hub, refer to the [Environments Hub Tutorial](/tutorials-environments/getting-started).
# Overview
Source: https://docs.primeintellect.ai/cli-reference/introduction
Command line interface for managing Prime Intellect compute, RL environments and code sandboxes.
The Prime Intellect CLI provides a powerful command-line interface for managing all of our Prime Intellect offerings.
Including:
* Managing compute resources on our platform
* Creating, publishing and installing RL environments from our environment hub
* Managing code sandboxes for secure code execution
Check out our [open source repository](https://github.com/PrimeIntellect-ai/prime-cli) on GitHub.
## Quick Start
1. Install uv (if not already installed):
```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
```
2. Install the CLI with uv:
```bash theme={null}
uv tool install prime
```
3. Authenticate:
```bash theme={null}
prime login
```
Alternatively you can also manually set your API key using `prime config set-api-key`
4. Configure SSH key for pod access:
```bash theme={null}
prime config set-ssh-key-path
```
You can generate your ssh keys on app.primeintellect.ai under the /dashboard/profile section. We recommend creating a dedicated directory for your Prime Intellect keys in your home directory, like `/Users//pi-keys`.
5\. Verify your configuration:
```bash theme={null}
prime config view
```
### Alternative Installation: Using pip
If you prefer not to use uv:
```bash theme={null}
pip install prime
```
## Key Features
* **GPU Resource Management**: Query and filter available GPU resources
* **Pod Management**: Create, monitor, and terminate compute pods
* **SSH Access**: Direct SSH access to running pods
* **Team Support**: Manage resources across team environments
## Basic Commands
```bash theme={null}
# List available GPUs
prime availability list
# Filter by GPU type
prime availability list --gpu-type H100_80GB
# Create a pod
prime pods create
# List your pods
prime pods list
# SSH into a pod
prime pods ssh
```
For detailed documentation on each command and feature, explore the sections above.
View and filter available GPU resources across providers using simple commands.
Create, monitor, and manage compute pods
directly from your terminal.
Configure connection settings and team accounts.
# Managing Disks
Source: https://docs.primeintellect.ai/cli-reference/managing-disks
How to create and manage network-attached storage using the CLI
Before you start, ensure that you have [API Key](/api-reference/api-keys) with `Disks -> Read and write` permission
## Overview
Network-attached disks provide persistent storage that can be shared across multiple GPU instances. Disks persist independently from instances, making them ideal for storing datasets, model checkpoints, and other data that needs to survive instance termination.
## Checking Disk Availability
Before creating a disk, check available storage options:
```bash theme={null}
prime availability disks
```
Filter results using these options:
* `--regions`: Geographic locations (e.g., united\_states,eu\_north)
* `--data-center-id`: Specific datacenter (e.g., US-1)
* `--cloud-id`: Cloud configuration ID
Example with filters:
```bash theme={null}
prime availability disks --regions united_states
```
### Example Output
| ID | Provider | Location | Stock | Price/Hr/GB | Max Size (GB) | Is Multinode |
| ------ | ----------- | ----------------------- | --------- | ------------ | ------------- | ------------ |
| c008ad | runpod | US (US-CA-2) | Available | \$0.00011111 | 8,192 | Yes |
| 4e50ab | runpod | US (US-IL-1) | Available | \$0.00011111 | 8,192 | Yes |
| 4eb3b5 | runpod | US (US-KS-2) | Available | \$0.00011111 | 8,192 | Yes |
| 334eaa | runpod | US (US-MO-1) | Available | \$0.00011111 | 8,192 | Yes |
| d9c8e5 | runpod | US (US-NC-1) | Available | \$0.00011111 | 8,192 | Yes |
| 241b41 | runpod | US (US-TX-3) | Available | \$0.00011111 | 8,192 | Yes |
| 7d2232 | runpod | US (US-WA-1) | Available | \$0.00011111 | 8,192 | Yes |
| a869c3 | hyperstack | US (US-1) | Available | \$0.00009700 | 100,000 | No |
| 8121df | crusoecloud | US (us-southcentral1-a) | Available | \$0.00011546 | 8,192 | No |
| 81cdd2 | crusoecloud | US (us-east1-a) | Available | \$0.00011546 | 8,192 | No |
| 2f139f | dc\_roan | US | Available | \$0.00007000 | 8,192 | Yes |
**Column Descriptions:**
* **ID**: Short identifier for easy disk creation (use with `--id` parameter)
* **Provider**: Storage provider offering the disk (runpod, hyperstack, crusoecloud, dc\_roan)
* **Location**: Country code and datacenter identifier
* **Stock**: Current availability status
* **Price/Hr/GB**: Cost per gigabyte per hour (hourly rate × disk size = total hourly cost)
* **Max Size (GB)**: Maximum disk capacity supported by the provider
* **Is Multinode**: Whether the disk can be attached to multiple instances simultaneously
## Creating a Disk
Create a new persistent disk:
```bash theme={null}
prime disks create --id c008ad --size 500
```
### Available Options
Short ID from availability disks list
Disk size in GB (must be within provider's min/max limits)
Human-friendly name for the disk (auto-generated if not provided)
Team identifier for the disk
Skip confirmation prompt
### Creation
```bash theme={null}
prime disks create \
--id 7d2232
--size 500 \
--name ml-training-data
```
Disks are billed continuously from creation until termination. Make sure to terminate disks when no longer needed.
## Listing Your Disks
View all your disks:
```bash theme={null}
prime disks list
```
### Output Options
Display as JSON:
```bash theme={null}
prime disks list --output json
```
With pagination:
```bash theme={null}
prime disks list --limit 50 --offset 0
```
The output shows:
* Disk ID, name, and size
* Current status
* Provider and location
* Creation date and age
* Hourly cost
* Attached pods and clusters
## Getting Disk Details
View detailed information about a specific disk:
```bash theme={null}
prime disks get
```
Display as JSON:
```bash theme={null}
prime disks get --output json
```
The detailed view includes:
* Complete disk configuration
* Current status and pricing
* List of attached pods
* List of attached clusters
* Creation and update timestamps
* Location information
## Updating a Disk
Update the disk name:
```bash theme={null}
prime disks update --name new-disk-name
```
Currently, only the disk name can be updated. Disk size cannot be changed after creation.
## Terminating a Disk
Terminate a disk when you no longer need it:
```bash theme={null}
prime disks terminate
```
Skip confirmation prompt:
```bash theme={null}
prime disks terminate --yes
```
**Disk termination is permanent and irreversible**. All data stored on the disk will be lost. Make sure to backup any important data before terminating a disk.
**Detachment Timing**: After terminating an instance with attached disks, you may need to wait a few moments for the disk to detach before you can terminate it or attach it to another instance.
## Disk Status Values
Disks go through different statuses during their lifecycle:
| Status | Description |
| -------------- | ------------------------------------------------------------ |
| `PROVISIONING` | Disk is being created |
| `PENDING` | Disk status is changing |
| `ACTIVE` | Disk is ready and can be attached to instances or terminated |
| `STOPPED` | Disk is stopped |
| `ERROR` | An error occurred during disk operations |
| `DELETING` | Disk is being terminated |
| `TERMINATED` | Disk has been terminated |
| `UNKNOWN` | Disk status is unknown (cannot access the current status) |
## Using Disks with Instances
### Attach Disks During Instance Creation
```bash theme={null}
prime pods create \
--id 346663 \
--disks disk-id-1 \
--disks disk-id-2
```
Learn more in [Provision Instance](/cli-reference/provision-gpu).
### Best Practices
**Location Planning**
* Create disks in the same datacenter where you plan to run GPU instances
* Use `prime availability disks` to find compatible storage options
**Size Planning**
* Start with the size you need - you cannot resize disks after creation
* Consider future growth when selecting disk size
* Remember that larger disks cost more per hour
**Data Management**
* Use descriptive names to identify disk contents
* Regularly backup important data from disks
* Terminate unused disks to avoid unnecessary costs
**Multinode Disks**
* Use multinode-capable disks for distributed training
* Check the `isMultinode` field in availability output
* Ensure your provider supports simultaneous access if needed
## Common Use Cases
### Shared Training Data
Create large disks with your training datasets that can be reused across multiple training runs:
```bash theme={null}
# Create a 1TB disk for training data
prime disks create --size 1000 --name training-datasets --id 7d2232
# Attach to multiple training instances
prime pods create --id gpu-id --disks disk-id-1
```
### Model Checkpoints
Use disks to persist model checkpoints and training artifacts:
```bash theme={null}
# Create disk for checkpoints
prime disks create --size 200 --name model-checkpoints --id 7d2232
# Use with training instance
prime pods create --id gpu-id --disks checkpoint-disk-id
```
### Multi-Node Distributed Training
Use multinode-compatible disks for distributed training:
```bash theme={null}
# Find multinode-capable disk storage
prime availability disks --regions united_states
# Create multinode disk (look for "Yes" in Is Multinode column)
prime disks create --size 500 --name shared-data --id multinode-disk-id
# Attach to multiple nodes
prime pods create --id gpu-id-1 --disks shared-disk-id
prime pods create --id gpu-id-2 --disks shared-disk-id
```
## Related Documentation
* [Check Availability](/cli-reference/check-gpu-availability) - Find available disk options
* [Provision Instance](/cli-reference/provision-gpu) - Attach disks when creating instances
* [API Disk Management](/api-reference/managing-disks) - API reference for disk operations
# Provision Instance
Source: https://docs.primeintellect.ai/cli-reference/provision-gpu
How to provision an instance using availability data
Before you start, ensure that you have [API Key](/api-reference/api-keys) with `Instances -> Read and write` permission
## Checking GPU Availability
First, check available GPU configurations:
```bash theme={null}
prime availability list
```
Filter results using these options:
* `--gpu-type`: Specific GPU model (e.g., H100\_80GB)
* `--regions`: Geographic locations (e.g., united\_states,canada)
* `--gpu-count`: Number of GPUs needed
* `--socket`: Socket type (PCIe, SXM4, etc.)
* `--provider`: Name of a provider
## Creating an Instance
Create a new instance interactively:
```bash theme={null}
prime pods create
```
### Available Options
Short ID from availability list
Cloud ID from provider
GPU model (e.g., H100\_80GB, A100)
Number of GPUs required
Instance name (alphanumeric and dashes only)
Storage size in GB
Number of virtual CPUs
RAM in GB
Operating system image
Team identifier for the instance
Environment Variable in form `KEY=value` - this can be used multiple times
Attach existing disk IDs to the instance. Can be specified multiple times for multiple disks.
* Example: `--disks disk-id-1 --disks disk-id-2`
Share the instance with all team members. Requires a team to be set via `--team-id` or `prime config set-team-id`. Team members' SSH keys will be added to the instance.
Interactively select specific team members to share the instance with. Displays a numbered list of team members and prompts for selection. Enter comma-separated numbers or `all` for everyone.
### Interactive Creation Process
If options are not provided, the CLI will guide you through:
1. GPU Selection
2. Configuration Options
3. Resource Allocation
4. Image Selection
5. Team Assignment
The CLI will show default values and valid ranges for resources like disk size, vCPUs, and memory based on the selected GPU configuration.
### Attaching Persistent Disks
To attach existing persistent disks to your instance:
```bash theme={null}
prime pods create \
--id 346663 \
--disks disk-id-1 \
--disks disk-id-2
```
This is useful for:
* Accessing shared datasets across multiple instances
* Persisting model checkpoints and training artifacts
* Resuming training from previous sessions
Use `prime disks list` to see your available disks and their IDs. Learn more about creating and managing disks in [Managing Disks](/cli-reference/managing-disks).
## Sharing with Team Members
When working in a team, you can share instances so that other members' SSH keys are added to the instance, giving them direct access.
### Share with all team members
```bash theme={null}
prime pods create --share-with-team
```
### Interactively select members
```bash theme={null}
prime pods create --add-members
```
This displays a numbered list of team members (excluding yourself) and prompts you to select which members to share with. Enter comma-separated numbers (e.g. `1,3,5`) or type `all` to share with everyone.
### Auto-share by default
To always share new instances with your team without passing a flag each time:
```bash theme={null}
prime config set-share-resources-with-team true
```
Sharing flags require a team to be configured. Set one with `prime config set-team-id` or pass `--team-id` when creating the instance.
## Managing Your Instance
Monitor instance status:
```bash theme={null}
prime pods status
```
SSH into your instance:
```bash theme={null}
prime pods ssh
```
Configure your SSH key path in CLI settings and run chmod 400 on your private\_key.pm before using SSH.
Terminate instance:
```bash theme={null}
prime pods terminate
```
You'll be prompted to confirm before termination.
# Teams
Source: https://docs.primeintellect.ai/cli-reference/teams
Manage and view your teams and team members
## List Your Teams
View the teams you belong to:
```bash theme={null}
prime teams list
```
### Available Options
Maximum number of teams to return (default: 100)
Number of teams to skip for pagination (default: 0)
Output format: `table` (default) or `json`
### Example Output
| ID | Name | Slug | Role | Created |
| ------------------------- | ------------ | ------------ | ------ | ------------------- |
| clvb0itli0000oxevorvgrpfn | My Team | my-team | ADMIN | 2025-01-15T10:30:00 |
| clvb1jtmk0001pxfv1abc2def | Research Lab | research-lab | MEMBER | 2025-02-01T14:20:00 |
## List Team Members
View the members of a specific team:
```bash theme={null}
prime teams members
```
This uses the team from your config. To specify a different team:
```bash theme={null}
prime teams members --team-id clvb0itli0000oxevorvgrpfn
```
### Available Options
Team ID to list members for. Uses the configured team ID if not specified.
Output format: `table` (default) or `json`
### Example Output
| User ID | Name | Email | Role | Joined |
| ------------------------- | ----- | --------------------------------------------- | ------ | ------------------- |
| clvb0itli0000oxevorvgrpfn | Alice | [alice@example.com](mailto:alice@example.com) | ADMIN | 2025-01-15T10:30:00 |
| clvb1jtmk0001pxfv1abc2def | Bob | [bob@example.com](mailto:bob@example.com) | MEMBER | 2025-02-01T14:20:00 |
Set a default team with `prime config set-team-id` so you don't need to pass `--team-id` every time.
# FAQ
Source: https://docs.primeintellect.ai/faq
Frequently Asked Questions
## Instance Types and Pricing
An On-Demand instance is a non-interruptible virtual machine that you can deploy and terminate at any time, paying only for the compute time you use.
A Spot instance is a cost-effective virtual machine that uses unused cloud capacity. It offers discounts of up to 90% compared to On-Demand instances, but it can be interrupted if the capacity is needed by other users. Spot instances are ideal for workloads that are flexible and can tolerate interruptions, such as batch jobs, data analysis, or fault-tolerant applications.
Each Pod has an hourly cost depending on the provider and GPU type. Your Prime Intellect credits are deducted every minute while the Pod is active. If your credits run out, your Pods will be automatically deleted. We highly recommend setting up our auto-top-up feature in your [billing settings](https://app.primeintellect.ai/dashboard/billing) to ensure balances are automatically topped up as needed.
## Cloud Providers and Infrastructure
We are constantly adding new providers! You can find the current list of providers in our app.
Instances typically launch within a few minutes, but the exact time may vary depending on the provider.
More detailed information on spin-up time is shown on your instance card.
We don't provide formal SLAs at this time, as the underlying infrastructure relies on various providers. If you experience repeated downtime or unreliable performance, contact support. We'll work with the provider to resolve the issue or offer refunds/credits when disruptions stem from provider-side instability. For more reliability, choose Secure Cloud providers or verify uptime and past performance before deploying long-term workloads.
## Data Management and Storage
You can pause and resume instances on supported providers (currently Runpod), allowing you to save state and continue later at a lower cost than keeping the instance running. When resuming, note that GPU resources may not be immediately available, leading to wait times. Look for the "Pause" button on the instance details page. ⚠️ Important: If you terminate (rather than pause) your instance, you will lose all data.
For paused instances only, data can be persisted in a special directory that is shown in your instance details in the dashboard. For example, Runpod stores persistent data in the /workspace directory. Note that this only works when pausing an instance - if you terminate the instance, all data will be lost. Always back up critical data to external storage (e.g., S3) before terminating or pausing.
No, once an instance is terminated, all data stored on it is permanently deleted and cannot be recovered.
While most providers let you configure storage during instance creation, very large volumes may not always be honored. If your requested storage isn't fully available or doesn't match what's provisioned, contact support. We'll verify the provider's limitations and may offer guidance or alternative providers that can meet larger volume requirements.
## Configuration and Connectivity
Yes. Many providers allow you to configure CPU, RAM, and storage during instance creation. However, some have limits. Hover over the pricing or configuration fields on the instance creation page to see what's available. We're also adding filters to search by CPU, RAM, and storage soon.
First, ensure you have the correct SSH key from your instance's "SSH Connection" button and set permissions on your local machine correctly with chmod 400 private\_key.pem. For Jupyter notebooks, simply click the provided link on your instance page to access the notebook interface directly in your browser. If you still face issues, contact support with error details.
You can click the Port Information button on your instance to see which ports are open:
Yes. Certain providers (e.g., Latitude, Cudo, Scaleway, Hyperstack, Oblivus, Massedcompute, Datacrunch) already have open ports. We're also adding the ability to specify custom port mappings. Check provider details or the FAQ to see if port configuration is supported. If not, you may need to wait for the upcoming port-exposure feature rollout.
Some providers support docker-in-docker configurations and system-level services, while others do not. Check the provider's documentation or contact support if you encounter issues. We're working on better clarifications and tutorials to help you understand what's supported in your chosen environment.
## Billing and Payments
If your instance fails due to provider issues (such as failing to start, crashing early, or being impossible to terminate from your dashboard), contact support with the instance details. We typically provide refunds or credits when the issue stems from our system or provider instability. Please contact us in the support chat by typing REFUND as your first word, then describe your issue in detail and include screenshots.
When you sign up or top up credits, look for the promo code input field. If you've already signed up, the promo code field is typically found in the billing/top-up section. If you have trouble finding it, contact support and we'll manually credit your account.
We've introduced a "Teams" feature where you can invite multiple users and set a dedicated billing email. Create a team from your dashboard's team settings, add members, and specify the billing email (e.g., [billing@company.com](mailto:billing@company.com)) so that all invoices are sent there. This helps consolidate usage and manage costs for multiple users under one account.
Currently, most billing occurs in USD and via standard payment methods. If you need invoicing in other currencies or want to explore alternative digital payment options, please contact support. We're actively considering these features and may arrange manual invoicing or special payment methods in the future.
## Troubleshooting
If your instance fails to launch or remains "pending" for an extended period, first try terminating it and creating a new one. If the problem persists, contact support with the instance details. In cases where the issue is on the provider's side, we can often provide refunds or credits.
## Multi-Node Cloud
Our Multi-Node clusters are GPU instances in our cloud featuring NVIDIA H100 SXM5 GPUs with 80GB memory each, available in configurations of up to 256 GPUs per cluster. These instances are deployed on our Secure Cloud infrastructure with the following specifications:
**Premium Configuration:**
* 3.2Tbps INFINIBAND connectivity
* 104 CPU cores per node
* 16TB storage per node
* \~\$52.80/hr
**Value Configuration:**
* 100Gbps ETHERNET connectivity
* 104 CPU cores per node
* 12TB storage per node
* \~\$40.80/hr (CHEAPEST CLUSTER option)
All clusters are hosted in United States data centers with an estimated spin-up time of \~30 minutes. Those instances can spin up on-demand. For further custom cluster options use the new **"Custom GPU Quote"** feature.
You can follow our step-by-step guide on how to deploy and use a multi-node cluster for your use case [here](tutorials-multi-node-cluster/deploy-multi-node).
Yes, you can reserve larger instance clusters for longer durations to ensure availability and reduce costs. Use the **"Custom GPU Quote"** feature in your dashboard to request a quote for long-term reservations. You can also reach out to [contact@primeintellect.ai](mailto:contact@primeintellect.ai) for additional information.
## GPU Providers
We are actively expanding our network of GPU providers, including both individual compute providers and larger cloud platforms. For more information, please contact us at [contact@primeintellect.ai](mailto:contact@primeintellect.ai).
# Browser Environments
Source: https://docs.primeintellect.ai/guides/browser-environments
Guide to building and training browser-based environments with Prime Intellect and Browserbase.
# Multimodal Browser Environments in verifiers
**BrowserEnv now supports vision-based web browsing via Computer Use Agent (CUA) mode.**
Both modes are modes in verifiers and we provide examples for trying both. This guide covers the two interaction modes, how to get started, and walks through two real environments: a lightweight eval (`bb_demo.py`) and a full benchmark for training (`webvoyager_no_anti_bot.py`). It also covers BrowserEnv-specific pieces of RL training, including DOM and CUA training configurations and how to integrate with Lab.
***
## Two Modes of Browser Interaction
`BrowserEnv` is a unified `StatefulToolEnv` subclass that supports two operating modes, selected via the `mode` parameter. Both modes use [Browserbase](https://browserbase.com) as the browser provider.
### DOM Mode: Natural Language Control
DOM mode uses the [Stagehand](https://github.com/browserbase/stagehand) SDK to translate natural language instructions into browser actions. Stagehand runs its own LLM internally (configured via `stagehand_model`, defaults to `openai/gpt-4o-mini`) to interpret the page DOM and execute the appropriate operations.
The agent's tool surface is high-level and semantic:
* `navigate(url)`: go to a URL
* `observe(instruction)`: find possible actions matching a natural language description
* `act(instruction)`: execute an action described in natural language (click a button, fill a form)
* `extract(instruction, schema_json)`: extract structured data from the page
The agent never sees the rendered page. It works through Stagehand's abstraction of the DOM, which means it requires no coordinates, and no screenshots. This is fast and effective when pages have reliable semantic HTML.
DOM mode works best when actionable page state is exposed semantically through Stagehand. Visually ambiguous cases, especially overlays or elements that are easier to disambiguate by pixels, can still be easier in CUA.
**Stagehand routing:** When `proxy_model_to_stagehand=False` (default), Stagehand uses its own `stagehand_model` and `MODEL_API_KEY`. When `proxy_model_to_stagehand=True`, BrowserEnv injects the rollout client's model name, base URL, and API key into `observe`, `act`, and `extract`, so those Stagehand calls run through the same client/model endpoint as the rollout model.
### CUA Mode: Vision-Based Control (Multimodal)
CUA mode gives the agent a live screenshot of the rendered page after every action. The agent sees pixels and acts via screen coordinates, the same way a human would interact with a screen.
The agent's tool surface is low-level and coordinate-based:
* `click(x, y, button)`: click at screen coordinates
* `double_click(x, y)`: double-click at coordinates
* `type_text(text)`: type text into the focused element
* `keypress(keys)`: press keyboard keys
* `scroll(x, y, scroll_x, scroll_y)`: scroll at a position
* `goto(url)`: navigate to a URL
* `back()` / `forward()`: browser history navigation
* `wait(time_ms)`: wait for a specified duration
* `screenshot()`: capture the current page state
Each action returns a multimodal response: a text status block (URL, viewport dimensions, success/error) and a base64 PNG screenshot encoded as an `image_url` content block. The model processes both. After the tool call returns, `BrowserEnv` moves screenshot parts out of tool messages into a trailing user message. When `keep_recent_screenshots` is set, older screenshots are replaced with `[Screenshot removed to save context]` placeholders.
**When to use it:** Tasks that require visual understanding, such as navigating unfamiliar UIs, interacting with canvas-rendered apps, clicking elements that lack semantic markup, or any workflow where a human would need to *look* at the screen to proceed.
### Side-by-Side Comparison
| Aspect | DOM Mode | CUA Mode |
| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| **Control paradigm** | Natural language via Stagehand | Vision-based screen coordinates |
| **Observation space** | Text (DOM abstractions) | Multimodal (screenshots + text) |
| **Server requirement** | None (Stagehand SDK direct) | CUA server (auto-deployed or manual) |
| **Extra API key** | `MODEL_API_KEY` by default, or the rollout client when proxied | The CUA server template can forward `OPENAI_API_KEY` internally if it is present |
| **Best for** | Structured web interactions | Visual/complex UIs |
| **Speed** | Faster (direct DOM manipulation) | Slower (screenshot round-trips) |
***
## Getting Started
### Installation
Both example environments require the `browser` extra:
```bash theme={null}
# Install verifiers with browser support
uv pip install "verifiers[browser]"
```
### Environment Variables
```bash theme={null}
# Required for both modes
export BROWSERBASE_API_KEY="your-api-key"
export BROWSERBASE_PROJECT_ID="your-project-id"
# Required for LLM judge model - pick your favorite model/your custom judge model, here we use OpenAI as an example
export OPENAI_API_KEY="your-openai-key"
# Required for DOM mode only (Stagehand's internal LLM) - pick your favorite model provider
export MODEL_API_KEY="your-llm-api-key"
```
### Running the Examples
Both examples ship with the same task: navigate to the Prime Intellect homepage and read the headline. Note to do the below, you need to have a prime account.
```bash theme={null}
# DOM mode
prime eval run browser-dom-example -m openai/gpt-4.1-mini
# CUA mode (pre-built image, fastest and recommended)
prime eval run browser-cua-example -m openai/gpt-4.1-mini
```
***
## CUA Mode Execution Backends
CUA mode requires a **CUA server** — a lightweight TypeScript HTTP service that bridges the Python browser agent and the browser. Under the hood, Browserbase uses [Understudy](https://github.com/browserbase/stagehand/tree/main/packages/core/lib/v3/understudy), a custom and powerful Chrome DevTools Protocol (CDP) driver developed at Browserbase as part of [Stagehand](https://github.com/browserbase/stagehand), the AI browser automation framework. Understudy was built to overcome limitations in Playwright's built-in automation capabilities (e.g. more granular input control and richer session introspection). Because Understudy's CDP engine is written in TypeScript, a thin server layer is needed to expose it as a REST API that the Python environment can call. The CUA server is that layer — it receives coordinate-based action requests over HTTP, translates into CDP commands via Understudy, and returns screenshots back to the agent.
Three execution modes are available, from fastest to most flexible:
### 1. Pre-built Docker Image (Default)
Uses `deepdream19/cua-server:latest` with everything pre-installed. Startup takes \~5–10 seconds. This is the recommended approach.
```bash theme={null}
prime eval run browser-cua-example -m openai/gpt-4.1-mini -b https://api.openai.com/v1 -k OPENAI_API_KEY
```
### 2. Binary Upload
Builds and uploads a custom CUA server binary to a sandbox at runtime. Startup takes \~30–60 seconds. Use this when you need a modified server.
```bash theme={null}
prime eval run browser-cua-example -m openai/gpt-4.1-mini -a '{"use_prebuilt_image": false}'
```
### 3. Manual Server (Local Development)
Connect to a locally running CUA server. Start the server yourself, then point the environment at it:
```bash theme={null}
# Terminal 1: start the CUA server
cd assets/templates/browserbase/cua && pnpm dev
# Terminal 2: run with sandbox disabled
prime eval run browser-cua-exampl openai/gpt-4.1-mini -a '{"use_sandbox": false}'
```
### Screenshot Management
CUA mode captures a screenshot after every action. Two parameters control how screenshots flow:
* `save_screenshots` (bool): persist every screenshot to disk as timestamped PNGs in `screenshot_dir` (defaults to `./screenshots`). The CUA example defaults this to `False`; the `BrowserEnv` class itself defaults to `True`.
* `keep_recent_screenshots` (int | None): how many recent screenshots to retain in the conversation context window sent to the model. Defaults to `2`. Set to `None` to keep all (higher token cost).
***
## Example Environments
### [`bb_demo`](https://app.primeintellect.ai/dashboard/environments/prime/bb-demo): Quick Eval with Click Visualization
`bb_demo.py` is a lightweight single-task environment designed for quick CUA mode evaluation and debugging. It serves as a hello world for multimodal browser environments and pairs a simple browsing task with a custom `CUAMode` subclass that annotates saved screenshots with click markers.
**Task.** The agent is asked to navigate to the Prime Intellect website, find the blog page, and summarize the latest post. A `JudgeRubric` backed by `gpt-4.1-mini` evaluates whether the agent's response adequately describes the blog content, returning `1.0` for "yes" and `0.0` otherwise.
**Click marker overlay.** The environment subclasses `CUAMode` as `ClickMarkerCUAMode`, which overrides `click()` and `double_click()` to store the target (x, y) coordinates before each action. When the resulting screenshot is saved to disk, `_save_screenshot_with_marker()` uses Pillow to composite a visual marker (concentric circles, crosshairs, coordinate label) onto the PNG. The marker is drawn on the *saved* file only. The screenshot returned to the model in conversation context is unmodified, keeping the agent's visual input clean while giving developers an annotated record of every click.
**Subclassing pattern.** `ClickMarkerBrowserEnv` intercepts `BrowserEnv.__init__` when `mode="cua"`. It manually extracts all CUA-specific parameters from `kwargs`, bypasses the parent's mode setup by calling `StatefulToolEnv.__init__` directly, then constructs and registers the custom `ClickMarkerCUAMode`. For DOM mode, it delegates entirely to the parent. This pattern is useful any time you need to swap in a custom mode implementation without forking `BrowserEnv`.
**Usage:**
```python theme={null}
from bb_demo import load_environment
env = load_environment(
save_screenshots=True,
screenshot_dir="./my_screenshots",
mark_clicks=True,
click_marker_radius=15,
)
```
Requires `pip install Pillow`. Without it, screenshots still save without markers and the environment shows a warning at startup.
### [`webvoyager_no_anti_bot`](https://app.primeintellect.ai/dashboard/environments/browserbase/webvoyager-no-anti-bot): Training-Scale Benchmark
WebVoyager is a 600-task web navigation benchmark spanning real websites (Allrecipes, Amazon, Apple, ArXiv, GitHub, Google Flights/Maps/Search, ESPN, and more). This version filters out 43 tasks from `dictionary.cambridge.org` that are blocked by Cloudflare anti-bot protection, leaving 93.3% of the original dataset intact. It supports both DOM and CUA modes, making it a good candidate for RL training runs. Here is an [example training run](https://app.primeintellect.ai/training/shared/v8uz1wu40av5bssrqz3mi203) with Qwen3-VL-8B-Instruct:
**Dataset.** Tasks are loaded from a local JSONL file (`WebVoyager_data_clean.jsonl`). Each row has a natural language task (`ques`), a starting URL (`web`), a website name (`web_name`), and a task ID. There are no ground-truth answers because WebVoyager is task-completion-based rather than answer-matching. You can filter by website with `web_filter` (e.g., `web_filter="Amazon"`) and limit the number of examples with `num_examples`.
**Evaluation.** Since there are no explicit answers, the environment uses a task-completion judge. The agent's entire multi-turn trajectory is rendered into a structured text transcript by `WebVoyagerTrajectoryParser`, a custom `vf.Parser` subclass. This transcript includes assistant messages, tool calls with normalized arguments, and truncated tool results, while images are excluded. The transcript is capped at 12,000 characters to keep judge context manageable. The judge prompt instructs the LLM to verify that the agent navigated to the correct site, performed the required actions, and reached the requested end state. If the agent made no tool calls at all, the reward is automatically `0.0` without consulting the judge.
**Transcript rendering.** `render_webvoyager_transcript()` walks the completion messages and emits a line-by-line log: `ASSISTANT:` for text, `TOOL_CALL name({args})` for tool invocations, and `TOOL_RESULT:` for tool responses (truncated to 500 characters each). This keeps the judge grounded in what actually happened rather than what the agent claims happened. The judge prompt explicitly says to treat unsupported assertions as insufficient evidence.
**Mode flexibility.** `load_environment()` accepts `mode="dom"` or `mode="cua"` and passes all relevant configuration through to `BrowserEnv`. Both modes work against the same dataset and rubric.
**Usage:**
```bash theme={null}
# All 600 tasks, DOM mode
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini
# CUA mode, filtered to Amazon tasks
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"mode": "cua", "web_filter": "Amazon"}'
# 10 examples for a quick test
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"num_examples": 10}'
```
Note: the full 600-task suite takes a while to run. For initial testing, use `num_examples` or `web_filter` to scope it down.
***
## Building Your Own Browser Environment
Both examples follow the standard verifiers environment contract: a Python module exposing `load_environment(**kwargs) -> vf.Environment`. The pattern is:
1. **Define a dataset**: a HuggingFace `Dataset` with `question`, `answer`, and optionally `start_url` and `task_id` columns. For task-completion benchmarks where there's no ground-truth answer (like WebVoyager), set `answer` to an empty string and rely on a task-completion judge.
2. **Define a rubric**: typically a `JudgeRubric` with an LLM judge. For answer-matching tasks, the judge compares agent output to the expected answer. For task-completion tasks, subclass `vf.Parser` to render the trajectory into a judge-friendly transcript and evaluate whether the task was actually done.
3. **Construct a `BrowserEnv`**: pass `mode`, dataset, rubric, system prompt, and any mode-specific configuration.
4. **Package it**: add a `pyproject.toml` with `verifiers[browser]>=0.1.8` as a dependency.
```python theme={null}
import verifiers as vf
from verifiers.envs.integrations.browser_env import BrowserEnv
from datasets import Dataset
def load_environment(max_turns: int = 15, **kwargs) -> vf.Environment:
dataset = Dataset.from_dict({
"question": ["What is the title of the first blog post?"],
"answer": ["Expected answer here"],
"start_url": ["https://example.com/blog"],
})
rubric = vf.JudgeRubric(judge_model="gpt-4o-mini", judge_prompt="...")
rubric.add_reward_func(your_judge_func, weight=1.0)
return BrowserEnv(
mode="cua", # or "dom"
dataset=dataset,
rubric=rubric,
max_turns=max_turns,
system_prompt="...",
**kwargs,
)
```
Then install and evaluate:
```bash theme={null}
uv pip install -e ./environments/your_env
prime eval run your-env -m openai/gpt-4.1-mini
```
***
## RL Training with Browser Environments
Browserbase integrates with Prime Intellect through `BrowserEnv`. Browserbase handles the cloud browsers, `BrowserEnv` wraps those sessions as a verifiers environment, and Prime runs the evaluation or RL training loop.
In practice, that means the same BrowserEnv configuration can be used in both evaluation and training. The training-specific questions are usually:
* whether you want `mode="dom"` or `mode="cua"`
* whether Stagehand should keep its own model or be routed through the rollout model
* which browser/session settings you want in the `[[env]]` config
### Before You Train
1. Validate the environment with evals first. Reward quality matters more than running a large training job quickly.
2. Check reward distribution. If everything is `0.0` or everything is `1.0`, the run will not teach you much.
3. Start with a small training run before scaling up browser minutes and GPU time.
### Training Target in This Repo
`webvoyager_no_anti_bot` is the training-scale BrowserEnv example in this repo.
* 600 tasks in the filtered WebVoyager dataset
* supports both `mode="dom"` and `mode="cua"`
* uses task-completion reward rather than answer matching
* renders assistant text, tool calls, and truncated tool results into the judge transcript
* returns `0.0` reward immediately if a rollout makes no tool calls
That makes it the cleanest local reference for comparing DOM and CUA training without changing the benchmark itself.
Example Prime DOM training run: [WebVoyager RL run](https://app.primeintellect.ai/training/shared/aixtf0gfgstmasfsugb9x3bg).
Example Prime CUA training run: [WebVoyager RL run](https://app.primeintellect.ai/training/shared/v8uz1wu40av5bssrqz3mi203).
### DOM Training
A DOM sample run can be done with the following configuration:
* model: `Qwen/Qwen3-4B-Instruct-2507`
* `batch_size = 64`
* `rollouts_per_example = 8`
* environment id: `browserbase/webvoyager-no-anti-bot`
By default, Stagehand keeps using its own `stagehand_model` and `MODEL_API_KEY`. If you want the trained rollout model to also handle Stagehand's DOM operations, enable `proxy_model_to_stagehand=true`. In that configuration, `observe`, `act`, and `extract` are routed through the same client/model endpoint as the rollout model.
The checked-in sample does not add extra BrowserEnv args. Add them when you want shared Stagehand routing:
```toml theme={null}
model = "Qwen/Qwen3-4B-Instruct-2507"
env_files = ["secrets.env"]
max_steps = 100
batch_size = 64
rollouts_per_example = 8
[sampling]
max_tokens = 1024
[[env]]
id = "browserbase/webvoyager-no-anti-bot"
args = { mode = "dom", proxy_model_to_stagehand = true }
```
### CUA Training
A vision-language sample run can be done with the following configuration:
* model: `Qwen/Qwen3-VL-4B-Instruct`
* `batch_size = 32`
* `rollouts_per_example = 4`
* environment id: `prime/webvoyager-no-anti-bot`
* BrowserEnv args: `mode = "cua"`, `max_turns = 10`, `viewport_width = 800`, `viewport_height = 600`, `keep_recent_screenshots = 2`, `memory_gb = 6`
Sample `.toml` file:
```toml theme={null}
model = "Qwen/Qwen3-VL-4B-Instruct"
env_files = ["secrets.env"]
max_steps = 100
batch_size = 32
rollouts_per_example = 4
[sampling]
max_tokens = 512
[[env]]
id = "prime/webvoyager-no-anti-bot"
args = { mode = "cua", max_turns = 10, viewport_width = 800, viewport_height = 600, keep_recent_screenshots = 2, memory_gb = 6 }
```
### Credentials and Workflow Notes
* Always provide `BROWSERBASE_API_KEY` and `BROWSERBASE_PROJECT_ID` when BrowserEnv is creating Browserbase sessions.
* DOM mode needs `MODEL_API_KEY` unless you route Stagehand through the rollout model with `proxy_model_to_stagehand=true`.
* CUA mode uses the same BrowserEnv backend choices as evaluation. The current bundled CUA server template forwards `OPENAI_API_KEY` into Stagehand if it is present, but that is a server-template detail rather than a separate BrowserEnv training argument.
* Browserbase credentials belong in environment variables or training secrets, not in the TOML file.
* The same BrowserEnv args used with `prime eval run -a '{...}'` belong in the training config under `[[env]]` `args = { ... }`.
* BrowserEnv does not expose sandbox authentication as a constructor argument.
Both Hosted Training and self-managed `prime-rl` use the same BrowserEnv-side mode choice and environment args. The difference is in how the training job is orchestrated, not in how BrowserEnv behaves.
### Picking DOM vs CUA for Training
| Question | DOM Mode | CUA Mode |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| What does the model observe? | Semantic browser actions through Stagehand | Screenshots plus tool status text |
| What kind of model fits best? | Text models | Vision-language models |
| How do Stagehand calls behave during training? | Separate `stagehand_model` by default, or shared rollout routing with `proxy_model_to_stagehand=true` | Not applicable to the agent-side tool surface |
| What usually matters most? | Semantic reliability and cheaper context | Visual grounding and UI disambiguation |
### Performance Notes
* Browser training rollouts are slower than text-only environments because each rollout step interacts with a real browser session.
* CUA mode is heavier than DOM mode because screenshots must be rendered, carried in context, and consumed by the model.
* If your task does not need visual grounding, DOM mode is usually the faster and cheaper place to start.
***
## Quick Reference
### DOM Mode Arguments
| Argument | Default | Description |
| -------------------------- | ----------------------- | -------------------------------------------- |
| `project_id` | Required | Browserbase project ID |
| `browserbase_api_key_var` | `"BROWSERBASE_API_KEY"` | Env var for Browserbase API key |
| `stagehand_model` | `"openai/gpt-4o-mini"` | Model for Stagehand DOM operations |
| `model_api_key_var` | `"MODEL_API_KEY"` | Env var for Stagehand's API key |
| `max_turns` | `10` | Max conversation turns |
| `proxy_model_to_stagehand` | `False` | Route Stagehand LLM calls through eval model |
### CUA Mode Arguments
| Argument | Default | Description |
| ------------------------- | --------------------------------- | ----------------------------------------------- |
| `use_sandbox` | `True` | Auto-deploy CUA server to sandbox |
| `use_prebuilt_image` | `True` | Use pre-built Docker image (fastest) |
| `prebuilt_image` | `"deepdream19/cua-server:latest"` | Docker image for sandbox |
| `server_url` | `"http://localhost:3000"` | CUA server URL (when `use_sandbox=False`) |
| `viewport_width` | `1024` | Browser viewport width |
| `viewport_height` | `768` | Browser viewport height |
| `save_screenshots` | `True` | Persist screenshots to disk |
| `keep_recent_screenshots` | `2` | Screenshots in model context (`None` = all) |
| `max_turns` | `15` | Max conversation turns |
| `env` | `"BROWSERBASE"` | Browser provider (`"LOCAL"` or `"BROWSERBASE"`) |
| `proxies` | `False` | Enable Browserbase proxies |
| `advanced_stealth` | `False` | Enable anti-bot detection stealth mode |
| `cpu_cores` | `2` | Sandbox CPU cores |
| `memory_gb` | `4` | Sandbox memory |
### CUA Execution Modes
| Mode | Flag | Startup | Use Case |
| --------------- | -------------------------- | -------- | ------------- |
| Pre-built image | *(default)* | \~5-10s | Production |
| Binary upload | `use_prebuilt_image=false` | \~30-60s | Custom server |
| Manual server | `use_sandbox=false` | Instant | Local dev |
# Overview
Source: https://docs.primeintellect.ai/guides/index
Step-by-step Lab guides for training runs, environments, and agent workflows.
Practical walkthroughs for common Lab workflows. Start with training recipes for reusable patterns, or follow a full build-and-train guide for a concrete end-to-end project.
End-to-end walkthrough from workspace setup to Hosted Training and deployment.
Reusable RL patterns for math, code generation, tool use, and more.
Build document-search environments and train them end to end on Lab.
# Training Recipes
Source: https://docs.primeintellect.ai/guides/recipes
Practical RL training recipes for math reasoning, code generation, tool use, and more on Lab.
Each example here covers a common RL use case: what kind of environment to build, a minimal working implementation, a training config, and practical tips. Use these as starting points or drop-in templates for your own runs on [Lab](/hosted-training/what-is-lab).
If you haven't launched a training run yet, start with the [Getting Started](/hosted-training/getting-started) guide first.
***
## Math Reasoning
Train models to solve mathematical problems step-by-step, using symbolic verification to reward correct answers.
**Environment type:** `SingleTurnEnv` with `MathRubric`
**Why RL works here:** Models learn to produce correct final answers through trial and error. The reward signal is binary and cheap to compute — symbolic math verification checks whether the model's `\boxed{}` answer matches the ground truth, without needing an LLM judge.
**Example environment:**
```python theme={null}
import verifiers as vf
from datasets import load_dataset
def load_environment(split: str = "train", num_examples: int = -1) -> vf.Environment:
ds = load_dataset("openai/gsm8k", split=split)
dataset = vf.Dataset.from_hf(ds, question_col="question", answer_col="answer")
if num_examples > 0:
dataset = dataset.select(range(num_examples))
rubric = vf.MathRubric()
return vf.SingleTurnEnv(
dataset=dataset,
rubric=rubric,
system_prompt="Solve the problem step by step. Put your final answer in \\boxed{}.",
)
```
**Training config:**
```toml theme={null}
model = "Qwen/Qwen3-4B-Instruct-2507"
max_steps = 200
batch_size = 256
rollouts_per_example = 8
[sampling]
max_tokens = 1024
[[env]]
id = "your-username/gsm8k"
```
**Tips:**
* Start with GSM8K for validation — baseline models typically score 40–70%, leaving room for improvement.
* For harder tasks (AIME, competition math), use a larger model like `Qwen/Qwen3-235B-A22B-Thinking-2507` and increase `max_tokens`.
***
## Code Generation with Sandboxes
Train models to write correct code by executing their solutions in sandboxed environments and verifying outputs against test cases.
**Environment type:** `PythonEnv` or `SandboxEnv`
**Why RL works here:** The model gets a concrete pass/fail signal from running code. Unlike static checking, execution-based verification catches subtle bugs and rewards solutions that actually work. Multi-turn interaction lets the model iteratively debug when tests fail.
**Example environment:**
```python theme={null}
import verifiers as vf
from datasets import Dataset
def load_environment() -> vf.Environment:
dataset = Dataset.from_list([
{
"question": "Write a function `fibonacci(n)` that returns the nth Fibonacci number.",
"info": '{"test_code": "assert fibonacci(0) == 0\\nassert fibonacci(1) == 1\\nassert fibonacci(10) == 55"}'
},
# ... more examples
])
async def tests_pass(completion, info, state) -> float:
code = completion[-1]["content"]
test_code = info["test_code"]
try:
exec_result = state.get("exec_result", "")
return 1.0 if "PASSED" in exec_result else 0.0
except Exception:
return 0.0
rubric = vf.Rubric(funcs=[tests_pass])
return vf.PythonEnv(
dataset=dataset,
rubric=rubric,
max_turns=5,
)
```
**Training config:**
```toml theme={null}
model = "Qwen/Qwen3-30B-A3B-Instruct-2507"
max_steps = 300
batch_size = 256
rollouts_per_example = 16
[sampling]
max_tokens = 2048
[[env]]
id = "your-username/code-gen"
```
**Tips:**
* Use `PythonEnv` for Python-specific tasks — it provides a persistent REPL that the model can use across turns.
* Use `SandboxEnv` for multi-language tasks or when you need shell access.
* Set `max_turns` to 3–5 to let the model iterate on failing test cases.
* Consider a partial reward for passing some but not all tests, rather than all-or-nothing scoring.
***
## Multi-Turn Games and Puzzles
Train models on interactive tasks where they must take actions over multiple turns, receiving feedback after each move.
**Environment type:** Custom `MultiTurnEnv` subclass
**Why RL works here:** Games provide dense, structured reward signals. The model learns strategies through repeated play — each rollout is a complete game, and the final score becomes the reward. Multi-turn structure naturally teaches planning and sequential decision-making.
**Example environment (word guessing game):**
```python theme={null}
import verifiers as vf
from datasets import Dataset
import random
class WordGameEnv(vf.MultiTurnEnv):
async def setup_state(self, state, **kwargs):
state["target"] = state["info"]["target_word"]
state["guesses"] = []
return await super().setup_state(state, **kwargs)
async def env_response(self, messages, state):
guess = messages[-1]["content"].strip().lower()
target = state["target"]
state["guesses"].append(guess)
if guess == target:
state["won"] = True
return [{"role": "user", "content": "Correct! You found the word."}]
# Give hints: which letters are in the right position
hints = []
for i, (g, t) in enumerate(zip(guess, target)):
if g == t:
hints.append(f"Position {i+1}: correct")
elif g in target:
hints.append(f"Position {i+1}: wrong position, letter is in the word")
else:
hints.append(f"Position {i+1}: letter not in word")
return [{"role": "user", "content": "\n".join(hints) + "\nGuess again."}]
@vf.stop
async def game_won(self, state):
return state.get("won", False)
def load_environment() -> vf.Environment:
words = ["apple", "brain", "cloud", "dance", "eagle"]
dataset = Dataset.from_list([
{"question": "Guess the 5-letter word. I'll give you hints after each guess.",
"info": f'{{"target_word": "{w}"}}'} for w in words
])
async def win_reward(state) -> float:
if state.get("won"):
return max(0.2, 1.0 - 0.15 * len(state["guesses"]))
return 0.0
rubric = vf.Rubric(funcs=[win_reward])
return WordGameEnv(dataset=dataset, rubric=rubric, max_turns=8)
```
**Training config:**
```toml theme={null}
model = "Qwen/Qwen3-4B-Instruct-2507"
max_steps = 100
batch_size = 128
rollouts_per_example = 8
[sampling]
max_tokens = 256
[[env]]
id = "your-username/word-game"
```
**Tips:**
* Games are excellent for validating your setup since they tend to show clear reward improvements within a small number of steps.
* Shape rewards to be gradient-rich — instead of just 0/1 for win/loss, give partial credit (e.g., reward based on number of turns taken to win).
* The built-in `alphabet-sort` environment is a great starting point — install it with `prime env install primeintellect/alphabet-sort`.
***
## Tool Use and Agentic Tasks
Train models to use tools effectively — calling the right tool with the right arguments to accomplish a goal.
**Environment type:** `ToolEnv` or `MCPEnv`
**Why RL works here:** Tool use requires the model to reason about which tool to call, compose correct arguments, interpret results, and decide on next steps. RL training lets the model learn this decision-making loop through practice, improving both tool selection and argument construction.
**Example environment (research assistant with search):**
```python theme={null}
import verifiers as vf
from datasets import Dataset
async def web_search(query: str) -> str:
"""Search the web for information.
Args:
query: The search query to look up.
Returns:
Search results as text.
"""
# your search implementation
return await do_search(query)
async def calculate(expression: str) -> str:
"""Evaluate a mathematical expression.
Args:
expression: A math expression to evaluate (e.g. "2 + 2 * 3").
Returns:
The result of the evaluation.
"""
try:
return str(eval(expression))
except Exception as e:
return f"Error: {e}"
def load_environment() -> vf.Environment:
dataset = Dataset.from_list([
{
"question": "What is the population of France divided by the population of Switzerland?",
"answer": "approximately 8.3"
},
# ... more examples requiring tool use
])
async def answer_quality(completion, answer, judge) -> float:
verdict = await judge(completion, answer)
return 1.0 if "correct" in verdict.lower() else 0.0
rubric = vf.JudgeRubric(judge_model="gpt-4.1-mini")
rubric.add_reward_func(answer_quality)
return vf.ToolEnv(
dataset=dataset,
tools=[web_search, calculate],
rubric=rubric,
max_turns=10,
)
```
**Training config:**
```toml theme={null}
model = "Qwen/Qwen3-30B-A3B-Instruct-2507"
max_steps = 200
batch_size = 256
rollouts_per_example = 16
[sampling]
max_tokens = 1024
[[env]]
id = "your-username/research-assistant"
env_file = ["secrets.env"]
```
**Tips:**
* Use `JudgeRubric` with an LLM judge for open-ended tasks where exact matching isn't feasible.
* Store API keys for external services (judge models, search APIs) in a `secrets.env` file and reference it with `env_file`.
* Monitor tool call counts via the automatic metrics — if the model isn't calling tools, the task may need a clearer prompt.
* `MCPEnv` is useful when your tools are already implemented as MCP servers.
***
## Multi-Environment Training
Train a single model on multiple tasks simultaneously to improve generalization.
**Why RL works here:** Training on diverse tasks prevents the model from overfitting to a single task's reward surface. The model learns transferable skills (reasoning, tool use, instruction following) that improve performance across all tasks.
**Training config:**
```toml theme={null}
model = "Qwen/Qwen3-235B-A22B-Instruct-2507"
max_steps = 500
batch_size = 512
rollouts_per_example = 16
[sampling]
max_tokens = 2048
[[env]]
id = "primeintellect/gsm8k"
args = { split = "train" }
[[env]]
id = "your-username/code-gen"
[[env]]
id = "primeintellect/alphabet-sort"
args = { min_turns = 3, max_turns = 5 }
[wandb]
project = "multi-env-training"
name = "235b-multi-task"
[eval]
interval = 100
```
**Tips:**
* Run baseline evaluations on each environment before training to understand starting performance.
* Use W\&B logging to compare per-environment reward curves during training.
***
## Workflow Summary
Regardless of the use case, the typical Hosted Training workflow is:
Create an environment with a dataset, harness, and rubric using the [verifiers](/verifiers/overview) library.
Run `prime eval run` against your environment to measure where the model starts.
Write a `.toml` config and launch with `prime train run`.
Watch reward curves on the dashboard, adjust your environment or config, and re-run.
Download the trained LoRA adapter or [deploy it for inference](/inference/adapter-deployments).
Launch your first Hosted Training run in minutes.
Detailed walkthrough of a complete training run.
Learn how to build custom environments with verifiers.
Multi-environment training, evals, and more.
# Your First Model Training
Source: https://docs.primeintellect.ai/guides/rl-training
Go from idea to training run to deployed agent using Lab. No programming or RL experience required.
In this guide, we’ll walk you through setting up your Lab workspace, creating your first agent environment, using it to evaluate baseline performance, launching a Hosted Training run, and deploying your model for inference.
These instructions are intended for use on a Mac or Linux CPU development environment. No previous experience with RL is required. In fact, experience with *coding* isn’t even required — we’ll use agents for everything. We’ll just assume that you have Claude Code, Codex, Cursor, OpenCode, Amp, or some other similar coding agent installed on your computer.
## Setting Up Your Lab Workspace
Ensure you have `uv` installed for managing Python packages:
```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
```
Install the `prime` CLI:
```bash theme={null}
uv tool install prime
```
Choose a folder on your machine as your Lab workspace (e.g. `~/dev/my-lab`) and do:
```bash theme={null}
prime lab setup
```
This command prepares your workspace with:
Creates a Python project and installs `verifiers` for environment development.
Configures your workspace for coding-agent workflows.
Downloads agent instruction files like `AGENTS.md` and Agent Skills.
Downloads example training and evaluation configs.
```text theme={null}
~/dev/demo prime lab setup
Supported coding agents: codex, claude, cursor, opencode, amp
Primary coding agent [codex]:
Using multiple coding agents? [y/N]:
No pyproject.toml found, initializing uv project...
Running: uv init
Initialized project `demo`
Running: uv add verifiers
...
... # install + download outputs omitted
...
[................................................................................] 1371 / 1371
Downloaded configs/rl/wordle.toml from https://github.com/primeintellect-ai/verifiers
+------------------------------------------ get started -------------------------------------------+
| |
| idea -> environment -> eval -> training |
| |
| +-------------------------------- ask codex ---------------------------------+ |
| | | |
| | I want to train a model for . Propose an initial | |
| | environment scaffold including relevant tools, generate a small | |
| | synthetic dataset, run a quick eval baseline, inspect the results, | |
| | and decide how to iterate on refining the implementation. | |
| | | |
| +----------------------------------------------------------------------------+ |
| |
| +-------------------- quick commands --------------------+ |
| | | |
| | $ prime env init my-env | |
| | $ prime eval run my-env -m gpt-5-nano -n 5 | |
| | $ prime eval tui | |
| | $ prime train run configs/rl/wiki-search.toml | |
| | $ prime gepa run my-env -m gpt-5-nano | |
| | | |
| +--------------------------------------------------------+ |
| |
+--------------------------------------------------------------------------------------------------+
```
Use one Lab workspace per research project and version it with Git. A workspace can contain multiple environments, configs, scripts, data, and eval outputs.
## Prompting Your Coding Agent
For many low-to-medium complexity environments, we find that the latest coding agents are often capable of “one-shotting” them, when equipped with the provided context from `prime lab setup` and given a sufficiently detailed prompt.
Providing the prompt below to a frontier coding agent (OpenCode + Codex 5.3) resulted in a fully functional environment for a calendar scheduling agent:
Save this as `prompt.md` and pass it directly to your coding agent as your initial task prompt.
```md prompt.md wrap theme={null}
Make an environment for a calendar scheduling agent.
In each task, there should be a set of people with busy calendars, and individual + global constraints for scheduling the meeting.
Some constraints can be "hard" (not allowed to violate), others can be "soft", where violating a constraint incurs some utility cost for certain attendees.
Each attendee has a utility for the proposed meeting time between 0 and 1, and the task score will be the weighted average of attendee scores if an acceptable meeting time is found, and 0 otherwise.
Attendee importance weights should be normalized to 1 for each task.
We should be able to programmatically generate task problems, and deterministically validate that satisfying solutions exist (and what their best possible score would be).
We should have fine-grained controls for key degrees of freedom in task generation, with higher-level parameters ("easy" / "medium" / "hard") for the full task set, which then map into setting ranges for the more fine-grained controls.
Be creative, and use your judgment to design clean composition rules for converting meeting choices and conflicts into scores. Avoid complex branching/conditional logic where possible.
Think carefully about designing your system in a way which discourages "backdoor" strategies or reward hacks.
The best approach for an agent should be to make a good-faith effort to satisfy constraints as best as possible.
Experiment with sampling strategies to ensure that tasks are solvable most of the time (so that we can pre-filter any unsolvable tasks cheaply), and that they aren't too easy -- there shouldn't be an abundance of valid solutions, random proposal times should be a bad strategy.
Types of constraints we want to potentially account for:
- Conflicting schedules
- Time zones + early/late/day preferences
- Meeting length
- Room availability
- Back-to-back meeting preferences
- Desired-but-optional attendees
- Other related constraints which reflect real-world calendar challenges
Degrees of freedom:
- Number of attendees
- Window of consideration
- Types of constraints
- Tightness of constraints
Use the StatefulToolEnv pattern, and in-memory data structures for the calendar + attendee information. The agent should have tools for things like:
- Checking attendee calendars
- Viewing attendee constraints
- Checking score of a proposed window
- Submitting a window
The environment should have a max_turns parameter, and tool results should show the remaining turns to the agent.
Default limit should be enough to allow reasonable exploration, but not so high that the agent can brute-force search all times.
We should also have a nice standalone script in the environment which creates a TUI to visualize a "calendar problem" similar to typical meeting apps, including attendees, timeblocks, and constraints, but fully in the terminal, using Rich styling, similar design language to the `prime eval tui` viewer implemented within the `verifiers` library (inspect verifiers source for reference).
Create a detailed design doc and plan for testing (PLAN.md), implement in full, revise PLAN.md after major milestones to reflect accomplishments and updated TODOs, and run basic small evals throughout as needed.
You are welcome to use the PRIME_API_KEY set in my environment for inference tests (see configs/endpoints.toml for models).
Let me know when you're happy with your implementation.
```
You can view the created environment in the Environments Hub:
Environments Hub
We can use the visualizer script we asked our agent to make for viewing the environment task structure more directly:
```bash theme={null}
~/dev/demo (main*) uv run --project environments/calendar_scheduling calendar-scheduling-tui --show-oracle --difficulty medium --seed 5
Calendar Scheduling TUI
╭───────────────────────────────────────────────────── Calendar Scheduling Problem ─────────────────────────────────────────────────────╮
│ Task ID medium-209463 │
│ Difficulty medium │
│ Seed 209463 │
│ Window 5 days │
│ Candidate UTC hours 09:00 - 19:00 │
│ Meeting duration 90 minutes │
│ Score-check budget 6 │
│ Total candidates 90 │
│ Valid candidates 8 │
│ Oracle best score 0.7970 │
│ Random baseline 0.0630 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Attendees
┏━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ ID ┃ Name ┃ Type ┃ TZ ┃ Weight ┃ Preferred Day ┃ Preferred Local ┃ Hard Local ┃
┡━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ attendee_1 │ Dakota │ optional │ UTC-2 │ 0.073 │ Day 1 (Tue Jan 06) │ 11.7-17.5 │ none │
│ attendee_2 │ Morgan │ optional │ UTC-7 │ 0.155 │ Day 1 (Tue Jan 06) │ 8.8-14.4 │ none │
│ attendee_3 │ Elliot │ required │ UTC+5 │ 0.072 │ Day 4 (Fri Jan 09) │ 10.9-16.6 │ none │
│ attendee_4 │ Avery │ required │ UTC+0 │ 0.146 │ Day 0 (Mon Jan 05) │ 12.5-17.0 │ none │
│ attendee_5 │ Parker │ required │ UTC-7 │ 0.180 │ Day 0 (Mon Jan 05) │ 10.7-16.4 │ 8.3-18.8 │
│ attendee_6 │ Robin │ required │ UTC-6 │ 0.185 │ Day 3 (Thu Jan 08) │ 10.2-14.7 │ none │
│ attendee_7 │ Reese │ optional │ UTC-1 │ 0.189 │ Day 4 (Fri Jan 09) │ 9.3-14.2 │ none │
└────────────┴────────┴──────────┴───────┴────────┴────────────────────┴─────────────────┴────────────┘
╭───────────────────────────────────────────────────────────── Day Labels ──────────────────────────────────────────────────────────────╮
│ Day 0 (Mon Jan 05) | Day 1 (Tue Jan 06) | Day 2 (Wed Jan 07) | Day 3 (Thu Jan 08) | Day 4 (Fri Jan 09) │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Availability Timeline (X busy, . free, =/# best window overlay)
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Lane ┃ Timeline ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ attendee_1 │ ........X...........|X..XXXXX..XXX.===...|....................|..XXX...............|.........XXXXX...... │
│ attendee_2 │ ............XXXXX...|..............===...|....................|..XXX...............|XXXXX............... │
│ attendee_3 │ ....................|.X............===...|.........XXXX.......|......X.............|XXX..X.............. │
│ attendee_4 │ ....................|..............===...|.....X.........X....|...............XXXXX|.................... │
│ attendee_5 │ ................XX..|..............===...|.XXXX...XXXX........|....................|.....X.............. │
│ attendee_6 │ .XXX............XXXX|..............===...|....................|.........XXXXX......|.....XX............X │
│ attendee_7 │ .XX.................|..XX.......XXX###...|...............XXXX.|....................|............XX...... │
│ room_1 (room) │ ....................|..........XXX.===X..|....................|....................|.................... │
└───────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘
╭────────────────────────────────────────────────────────── Constraint Model ───────────────────────────────────────────────────────────╮
│ Hard constraints │
│ - Required attendees must be able to attend │
│ - Hard local-time bounds cannot be violated │
│ - Chosen room must be available │
│ - Duration must match task duration exactly │
│ │
│ Soft utility penalties │
│ - Early/late local-time penalties │
│ - Day-preference distance penalties │
│ - Back-to-back penalty near busy blocks │
│ - Optional attendee absence penalty │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Oracle Best Windows
┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Rank ┃ Window ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1 │ Day 1 (Tue Jan 06) 16:00-17:30 UTC in room_1 │
└──────┴──────────────────────────────────────────────┘
╭─────────────────────────────────────────────────────────── Oracle Summary ────────────────────────────────────────────────────────────╮
│ Best score: 0.7970 │
│ Valid windows: 8 / 90 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Legend: X busy, . free, = highlighted best free slot, # highlighted busy slot
```
## Hosted Training
After the environment is created, we prompt our agent to test performance more exhaustively, then to start a Hosted Training run using `Qwen/Qwen3-30B-A3B-Instruct-2507`, which is available for LoRA finetuning via Hosted Training.
```text wrap=true theme={null}
test with GPT-4.1 models, make sure we're seeing proper rollouts that succeed with non-zero scores. those models should be able to solve; if they can't we have a bug somewhere or need clearer instructions in the env. Once you feel good about the env, make a RL config and start a training run with bs 128, 8 rollouts per group, for the 30b instruct qwen model.
```
Available models can be viewed with:
```bash theme={null}
prime train models
```
Example training configs (in `configs/rl` after running `prime lab setup`) look like:
```bash theme={null}
# calendar-scheduling.toml
model = "Qwen/Qwen3-30B-A3B-Instruct-2507"
max_steps = 100
batch_size = 128
rollouts_per_example = 8
env_file = ["../../secrets.env"]
[sampling]
max_tokens = 768
[[env]]
id = "prime/calendar-scheduling"
args = { difficulty = "medium", num_train = 512, num_eval = 128, max_turns = 18 }
[wandb]
project = "calendar-scheduling"
name = "qwen3-30b-i-calendar-scheduling-bs128-r8"
entity = "primeintellect"
```
The inclusion of `secrets.env` is optional — here we use it to set our W\&B key for logging:
```bash theme={null}
# secrets.env
OPENAI_API_KEY="${OPENAI_API_KEY}"
WANDB_API_KEY="${WANDB_API_KEY}"
```
Runs can be started by running:
```bash theme={null}
prime train run configs/rl/calendar-scheduling.toml
```
From the platform, you can view training curves, rollouts, configs, logs, and checkpoints.
Training runs are shareable. You can view this example run here:
Hosted Training
## Deploying Your Model
Under [Deployments](https://app.primeintellect.ai/dashboard/deployments), you can deploy LoRA adapters for inference with a single click:
You can then share the Model Identifier with your coding agent and ask it to run some more evals if desired, or incorporate it into your application directly. And with that, you have now successfully deployed your first RL-trained model!
# Training Search Agents
Source: https://docs.primeintellect.ai/guides/search-agents
Build three progressive RL environments for document search and run Hosted Training on Lab.
Step-by-step walkthrough for training a model to search and reason over documents using RL. The example uses patents, but the same architecture applies to any document domain: legal filings, SEC documents, medical literature, enterprise knowledge bases.
**Stack:** verifiers, ChromaDB, BigQuery, Lab, Qwen3-4B, Llama-3.2-3B
***
## Environments Overview
| Level | Name | Reward type | What the agent does |
| ------- | ---------------------- | ------------------------------- | ----------------------------------------------------------- |
| Level 1 | Metadata Retrieval | Exact match | Single tool call, parse structured response |
| Level 2 | Multi-step Computation | Binary on deterministic answers | 2+ tool calls, arithmetic, cross-patent comparison |
| Level 3 | Open-ended Analysis | LLM judge, normalized | Full patent text reading, synthesis, cross-patent reasoning |
***
## Prerequisites
* Google Cloud account with BigQuery access. Free tier is sufficient; with that, you get 1 TB/month of queries. Downloading \~1,500 patents with full text uses \~3 GB.
* Prime Intellect account with Lab access. You'll push environments to the Environments Hub and launch training runs from the CLI.
* OpenAI API key, which is used for generating Level 3 Q\&A ground truth and the LLM judge at training time.
* Python 3.10+ with `verifiers`, `chromadb`, `openai`, and the `prime` CLI. Install the Prime CLI with `pip install prime-cli`.
***
## Step 1 - Get Patent Data
Use Google's `patents-public-data` BigQuery dataset. It has full patent text (abstract, claims, description) and supports SQL filtering by company.
### BigQuery query
```sql theme={null}
-- fetch_patents.sql
SELECT
publication_number,
title,
abstract,
claims,
description,
filing_date,
grant_date,
ARRAY_LENGTH(claims_localized) AS claim_count
FROM
`patents-public-data.patents.publications`
WHERE
assignee_harmonized.name IN (
'QUALCOMM INCORPORATED',
'ERICSSON',
'NOKIA',
'SAMSUNG',
'HUAWEI',
'SONY',
)
AND country_code = 'US'
AND grant_date IS NOT NULL
LIMIT 1500
```
Export the results to a GCS bucket or download directly as JSON. You can start with just \~1,500 patents; it produces meaningful training signal without excessive embedding costs.
### Format each patent
Structure each patent as a markdown document and store metadata separately. Preserving document structure matters for Level 3 because the agent needs to navigate named sections.
```python theme={null}
# format_patents.py
def format_patent(row):
doc = f"""# {row['title']}
## Abstract
{row['abstract']}
## Claims
{row['claims']}
## Description
{row['description']}
"""
metadata = {
"patent_id": row["publication_number"],
"title": row["title"],
"filing_date": row["filing_date"],
"grant_date": row["grant_date"],
"claim_count": row["claim_count"],
}
return doc, metadata
```
***
## Step 2 - Design Three RL Environments
The three levels impose a curriculum. For example, an agent that can't reliably call `get_metadata()` and parse a date string shouldn't attempt multi-patent technical comparisons. Each level introduces qualitatively harder capabilities.
### Level 1 - Single-tool metadata retrieval
Questions that require exactly one tool call with no computation or reasoning
| Question type | Example |
| ------------- | ------------------------------------- |
| Filing date | "When was patent X filed?" |
| Grant date | "When was patent X granted?" |
| Claim count | "How many claims does patent X have?" |
| Title | "What is the title of patent X?" |
**Dataset size:** 6,000 Q\&A pairs. Every answer is deterministically verifiable from the metadata.
**Reward:** Binary - 1.0 for exact string match, 0.0 otherwise.
**Tools available:** `search_patents(query)`, `get_metadata(patent_id)`
Level 1 is a pipeline validation step. If reward doesn't climb toward 1.0 within 15 steps, something is wrong with your tool definitions, reward function, or data formatting. It isn't a model problem.
### Level 2 - Multi-step computation and comparison
Questions requiring 2 or more tool calls followed by arithmetic or string comparison.
| Question type | Example | What the agent does |
| ----------------- | ------------------------------------------------------ | -------------------------------------------------- |
| Days to grant | "How many days between filing and grant for patent X?" | `get_metadata()` then parse two dates and subtract |
| Filed first | "Which was filed first: X or Y?" | `get_metadata()` twice, then compare |
| Claim difference | "How many more claims does X have than Y?" | `get_metadata()` twice, then subtract |
| Search + retrieve | "Find a patent about SSB and tell me its filing date" | `search_patents()` then `get_metadata()` |
| Abstract content | "Does the abstract of patent X mention '5G'?" | `get_abstract()` then string check |
**Dataset size:** 500 Q\&A pairs.
**Reward:** Binary on deterministic answers. For search questions with multiple valid answers, the reward function checks against a precomputed set of valid answers.
**New tool:** `get_abstract(patent_id)` returns abstract text.
The environment does not do the math. The agent receives raw dates and counts from tool calls and must compute the answer itself. This is intentional to train the reasoning.
### Level 3 - Open-ended technical analysis
Questions that require reading patent content, understanding technical concepts, and synthesizing answers that can't be verified by pure string matching.
| Question type | Example |
| ------------------------ | --------------------------------------------------------------------- |
| Technical summary | "Summarize the key technical innovation in patent X in 2-3 sentences" |
| Problem identification | "What problem does patent X solve?" |
| Cross-patent comparison | "What is the key technical difference between patent X and Y?" |
| Standards identification | "What wireless communication standards are referenced in patent X?" |
| Claim feature extraction | "What are the key limiting features in claim 1 of patent X?" |
| Abstract-claim mapping | "Which claims are most directly described in the abstract?" |
**Dataset size:** 500 Q\&A pairs (LLM-generated with ground truth).
**Reward:** LLM judge, normalized.
**New tools:** `view_sections(patent_id)`, `read_section(patent_id, section_name)`
***
## Step 3 - Generate Verifiable Q\&A Pairs
### Levels 1 and 2
Iterate over the patent dataset, select random patents (or pairs for comparison questions), compute the answer directly from structured data, and generate the pair. Every answer is verified against the source data.
For search-type questions, precompute all valid answers across the entire dataset. For example, if and question asks to identify "SSB" patents and three patents mention "SSB" in their title or abstract, all three filing dates are valid answers. Store this set, and the reward function will check against it at training time.
```python theme={null}
# generate_qa.py
import random
from datetime import datetime
def generate_date_question(patents: list[dict]) -> dict:
patent = random.choice(patents)
return {
"question": f"When was patent {patent['patent_id']} filed?",
"answer": patent["filing_date"], # e.g. "2019-03-15"
"patent_ids": [patent["patent_id"]],
"level": 1,
"type": "filing_date",
}
def generate_days_to_grant(patents: list[dict]) -> dict:
patent = random.choice([p for p in patents if p["grant_date"]])
filed = datetime.strptime(patent["filing_date"], "%Y-%m-%d")
granted = datetime.strptime(patent["grant_date"], "%Y-%m-%d")
days = (granted - filed).days
return {
"question": f"How many days elapsed between the filing and grant of patent {patent['patent_id']}?",
"answer": str(days),
"patent_ids": [patent["patent_id"]],
"level": 2,
"type": "days_to_grant",
}
```
### Level 3 - LLM-generated ground truth
Each Level 3 entry needs a structured reference that the judge can use at training time. Use the LLM to generate three components per question:
* **answer** - the reference answer in writing
* **key\_points** - specific factual claims the answer must contain
* **source\_quotes** - direct quotes from the patent text supporting each key point
```python theme={null}
# generate_l3_qa.py
import json
from openai import OpenAI
client = OpenAI()
GENERATION_PROMPT = """You are generating ground truth for a patent analysis training dataset.
Patent text:
{patent_text}
Question: {question}
Return ONLY valid JSON with no preamble:
{{
"answer": "prose answer to the question",
"key_points": ["specific factual claim 1", "specific factual claim 2"],
"source_quotes": ["exact quote from patent supporting point 1", "exact quote supporting point 2"]
}}"""
def generate_l3_ground_truth(patent_text: str, question: str) -> dict:
response = client.chat.completions.create(
model="model",
messages=[{
"role": "user",
"content": GENERATION_PROMPT.format(
patent_text=patent_text,
question=question
)
}],
temperature=0
)
return json.loads(response.choices[0].message.content)
```
### Validate your dataset
Run these checks on every generated pair before training:
* Every patent ID referenced in a question exists in the dataset
* Every Level 3 ground truth has at least one key point
* Source quotes from Level 3 ground truth actually appear in the patent text (substring match)
Manually review a stratified sample across all question types. This surfaces systematic prompt issues that automated checks miss such as questions conflating problem and solution, or rubric with contradictions, or answers with technical inaccuracies.
***
## Step 4 - Reward Design
### Levels 1 and 2
Level 1 and 2 reward designs were straightforward.
```python theme={null}
# rewards.py
def reward_exact(agent_answer: str, ground_truth: str) -> float:
"""Binary reward for single-answer questions."""
return 1.0 if normalize(agent_answer) == normalize(ground_truth) else 0.0
def reward_set(agent_answer: str, valid_set: set[str]) -> float:
"""For search questions with multiple valid answers."""
return 1.0 if normalize(agent_answer) in valid_set else 0.0
def normalize(s: str) -> str:
return s.strip().lower().replace(",", "").replace(".", "")
```
### Level 3 - LLM judge
Getting the judge right required three iterations. Here's what failed and what ended up working.
**Iteration 1 - didn't work:** Multi-dimensional weighted scoring (accuracy, completeness, reasoning, conciseness, 0-10 each). Failed because regex parsing of scores was fragile, and per-category weight tuning is a second optimization problem on top of the first.
**Iteration 2 - didn't work:** Per-question custom rubrics (5 criteria x 2 points each, LLM-generated per question). More principled, but LLM-generated rubrics introduced contradictions like asking for content that would actually weaken an otherwise correct answer.
**Iteration 3 - works:** Universal rubric with content-specific ground truth. The ground truth (answer, key\_points, source\_quotes) already provides all content specificity needed.
| Criterion | Points | What it catches |
| ------------------------------------- | ------ | ----------------------------------- |
| Factually accurate relative to patent | 3 | Wrong technical details |
| Free of hallucinated information | 3 | Made-up claims, features, standards |
| Covers key points from ground truth | 2 | Missing important content |
| Directly answers the question asked | 1 | Off-topic or evasive responses |
| References specific patent content | 1 | Unsupported assertions |
The judge receives the question, the agent's response, the reference answer, key points, and source quotes. It returns structured JSON, normalized to \[0, 1]:
```json theme={null}
// judge_output_schema.json
{
"key_points": [
{"point": "SSB spatial overloading concept", "covered": true},
{"point": "Spatially separated beams", "covered": true},
{"point": "Reduced bandwidth overhead", "covered": false}
],
"hallucination": false,
"factual_error": false,
"final_score": 7
}
```
```python theme={null}
# rewards.py
def reward_l3(judge_output: dict) -> float:
score = judge_output["final_score"] / 10.0 # normalize to [0, 1]
if judge_output.get("hallucination"):
score *= 0.2 # hallucination is catastrophic in patent context
elif judge_output.get("factual_error"):
score *= 0.5
return score
```
Note on the 0.2 multiplier: zeroing out hallucinated responses entirely creates a sharp gradient that can cause instability. A 0.2 multiplier still strongly penalizes hallucination while providing a small gradient signal. In a patent context, hallucinated technical claims have direct commercial and legal consequences so we err on the side of undertrained vs. positively trained on hallucinations.
***
## Step 5 - Build the Tool Environment
The environment is implemented using the [verifiers](https://github.com/willccbb/verifiers) library as a `ToolEnv`. The agent receives a system prompt describing its role as a patent analyst with access to a dataset of N patents, a question, and a set of tools.
### Tool definitions
```python theme={null}
# patent_env.py
import chromadb
from verifiers import ToolEnv
class PatentEnv(ToolEnv):
def __init__(self, patents: list[dict], level: int):
self.corpus = {p["patent_id"]: p for p in patents}
self.level = level
self._init_vector_store(patents)
super().__init__(tools=self._get_tools(level))
def _init_vector_store(self, patents):
self.chroma = chromadb.Client()
collection = self.chroma.create_collection("patents")
collection.add(
ids=[p["patent_id"] for p in patents],
documents=[f"{p['title']} {p['abstract']}" for p in patents],
)
self.collection = collection
def search_patents(self, query: str) -> list[dict]:
"""Returns patent IDs and titles matching a semantic query."""
results = self.collection.query(query_texts=[query], n_results=5)
return [
{"patent_id": id_, "title": self.corpus[id_]["title"]}
for id_ in results["ids"][0]
]
def get_metadata(self, patent_id: str) -> dict:
"""Returns title, filing_date, grant_date, claim_count."""
p = self.corpus[patent_id]
return {
"title": p["title"],
"filing_date": p["filing_date"],
"grant_date": p["grant_date"],
"claim_count": p["claim_count"],
}
def get_abstract(self, patent_id: str) -> str:
return self.corpus[patent_id]["abstract"]
def view_sections(self, patent_id: str) -> list[str]:
"""Lists available section names in the full patent text."""
doc = self.corpus[patent_id]["full_text"]
return [
line.lstrip("# ").strip()
for line in doc.split("\n")
if line.startswith("## ")
]
def read_section(self, patent_id: str, section_name: str) -> str:
"""Returns the full text of a named section."""
doc = self.corpus[patent_id]["full_text"]
start = doc.find(f"## {section_name}")
if start == -1:
return f"Section '{section_name}' not found."
end = doc.find("\n## ", start + 1)
return doc[start:end if end != -1 else len(doc)]
def _get_tools(self, level: int):
tools = [self.search_patents, self.get_metadata]
if level >= 2:
tools.append(self.get_abstract)
if level >= 3:
tools += [self.view_sections, self.read_section]
return tools
```
Keep tools stateless. The agent can call `get_metadata()` on the same patent multiple times with no side effects.
### Schema to tools generalization
Patents have standardized sections (Abstract, Claims, Description) and coded metadata. This makes them well-suited for tool-based environments. The same structure applies to any document domain with a known schema such as legal filings map to `get_case_metadata()`, SEC filings to `read_section("Risk Factors")`, and so on.
***
## Step 6 - Parsing Strategy
Avoid regex in the pipeline because when not implemented well, it can introduce reward hacking. Use simple string methods and structured JSON instead:
```python theme={null}
# parsing.py
import json
def extract_claims_text(full_text: str) -> str:
"""Extract the Claims section using string slicing, not regex."""
start = full_text.find("## Claims")
if start == -1:
return ""
end = full_text.find("\n## ", start + 1)
return full_text[start:end if end != -1 else len(full_text)]
def parse_patent_id(full_code: str) -> str:
"""'US10123456B2' -> split and handle deterministically."""
return full_code.split("-")[0] if "-" in full_code else full_code
def parse_judge_output(raw: str) -> dict:
"""Parse judge JSON with a robust fallback."""
try:
return json.loads(raw)
except json.JSONDecodeError:
# Fallback: find final_score by character iteration
key = '"final_score":'
idx = raw.find(key)
if idx == -1:
return {"final_score": 0, "hallucination": True, "key_points": []}
score_start = idx + len(key)
digits = ""
for ch in raw[score_start:].lstrip():
if ch.isdigit():
digits += ch
elif digits:
break
return {
"final_score": int(digits) if digits else 0,
"hallucination": "hallucin" in raw.lower(),
"key_points": [],
}
```
***
## Step 7 - Train on Lab
Prime Intellect's Lab platform handles GPU orchestration and multi-tenant LoRA deployments. You push your environment to the Environments Hub, define a training config, and launch the run from the CLI.
### Step 7.1 - Push environment to the Hub
```bash theme={null}
# Push each level as a separate named environment
prime env push --name basic-patent-q-and-a
prime env push --name advanced-patent-q-and-a
prime env push --name patent-technical-analysis
```
### Step 7.2 - Define a training config
```toml theme={null}
# config.toml
model = "meta-llama/Llama-3.2-3B-Instruct"
max_steps = 100
batch_size = 128
rollouts_per_example = 8
[sampling]
max_tokens = 1024
[[env]]
id = "primeintellect/advanced-patent-q-and-a"
```
### Step 7.3 - Calibrate model to environment difficulty
Before committing to a full training run, verify the base model finds the task challenging but not impossible. A model that starts too high has nothing to learn; one that starts too low can't generate useful gradient signal. Target starting reward is around 0.15 to 0.35.
Do a short 10-step run first and read the reward curve. It plots mean reward per step across all rollouts in the batch. If starting reward is outside that range, adjust difficulty or model size before launching the full run.
```bash theme={null}
# Quick calibration: 10 steps to measure starting reward
prime train run config.toml \
--env-var OPENAI_API_KEY=sk-... \
--override max_steps=10
```
If starting reward is above 0.7, make the questions harder or use a smaller base model. If it's below 0.1, the questions may be too hard or the tools too opaque.
### Step 7.4 - Launch the full run
```bash theme={null}
prime train run config.toml --env-var OPENAI_API_KEY=sk-...
```
You can pass secrets at runtime with `--env-var`, or set them in the environment settings — both work.
Once the run is live, the Lab dashboard shows per-step metrics including mean reward, reward standard deviation, response length, and tool call count. Tool call count is worth checking early. If the agent isn't calling tools in the first few steps, the system prompt isn't clear enough about what tools are available and when to use them. Below are some screenshots of what the Prime Intellect dashboard looks like.
### Config parameters to tune
| Parameter | What it controls | Starting point |
| ---------------------- | --------------------------------- | -------------------------------- |
| `max_steps` | Training duration | 50 for L1, 100 for L2/L3 |
| `batch_size` | Examples per step | 128 |
| `rollouts_per_example` | Trajectories sampled per question | 8, increase for noisy L3 rewards |
| `max_tokens` | Max agent response length | 1024, L3 may need more |
***
## Results
Trained Qwen3-4B-Instruct and Llama-3.2-3B-Instruct across all three levels. The rollout viewer in Lab lets you inspect individual trajectories turn by turn: each tool call, its response, and the final answer. It's the most direct way to see what the model is actually learning to do at each level and understand in depth why the reward curve is behaving as it is.
### Level 1
| Checkpoint | Reward |
| ---------- | ------ |
| Start | \~0.05 |
| Step 15 | \~1.0 |
Saturates within 15 steps and stays there. Confirms the pipeline works end-to-end. Once the agent learns the pattern of calling `get_metadata()` and reading the response, it gets nearly everything right.
### Level 2
| Checkpoint | Reward |
| ---------- | ------ |
| Start | \~0.25 |
| Step 50 | \~0.70 |
Clear upward trend with variance throughout. Some batches land on straightforward date subtraction, others require chaining a search into a metadata call and then doing a comparison. The curve hasn't plateaued at step 50, so longer runs will get more performance.
### Level 3
| Checkpoint | Reward |
| ---------- | ------ |
| Start | \~0.30 |
| Step 100 | \~0.50 |
Noisy, as expected from LLM-judged reward. Individual batches swing between 0.15 and 0.9. The upward trend is real but needs larger batches and more steps to converge. The noise has two sources: judge subjectivity, and question difficulty variation (ex: technical summary vs. cross-patent comparison are not the same task and vary in difficulty).
***
## Extending This to Other Domains
The core architecture (schema-derived tools, progressive difficulty levels, universal rubric with content-specific ground truth) is not patent-specific. To adapt it to the following categories, you can:
* **Legal case search:** Replace `read_section("Claims")` with `read_section("Holding")`. The judgment or ruling is your Level 3 answer target.
* **SEC filings:** 10-K documents have standardized sections (Risk Factors, MD\&A, Financial Statements).
* **Medical literature:** PubMed abstracts for Level 1, full-text PMC articles for Level 3. Use MeSH terms for structured metadata.
* **Enterprise knowledge bases:** Internal docs with known schemas. Level 3 judge needs domain-appropriate ground truth generation.
The main work in each new domain is: (1) acquiring the data with full text, (2) defining the question types that capture the actual analytic tasks, and (3) writing the Level 3 ground truth generation prompt with domain-specific few-shot examples.
***
All environments are published on the [Environments Hub](https://hub.primeintellect.ai/). The patent dataset is on [HuggingFace](https://huggingface.co/datasets/jessicafeiyali/wirelesspatents).
# Advanced Configurations
Source: https://docs.primeintellect.ai/hosted-training/advanced-configs
Full configuration reference for Hosted Training runs
Hosted Training runs are configured via a `.toml` file. This page covers all available configuration fields, from basic setup to advanced features like multi-environment training, online evaluation, and W\&B integration.
Hosted Training accepts both Verifiers environment config shapes:
* **Legacy (Verifiers v0) environments** use `id` and optional `args`.
* **Verifiers v1 (taskset/harness) environments** use `taskset = { ... }` and optional `harness = { ... }`.
The outer Hosted Training config still uses `[[env]]` and `[[eval.env]]`; the native `prime-rl` equivalent is `[[orchestrator.train.env]]` and `[[orchestrator.eval.env]]`. See [the environment model](/hosted-training/environment-model#legacy-and-verifiers-v1-environment-configs) for the conceptual difference between the two shapes.
## Full Config Reference
Below is a complete annotated config showing all available fields. Required fields are uncommented; optional fields are shown as comments with their defaults.
```toml theme={null}
# ============================================================
# Core Configuration (required)
# ============================================================
model = "Qwen/Qwen3-30B-A3B-Instruct-2507" # HuggingFace model ID
max_steps = 100 # Total training steps
batch_size = 256 # Rollouts per training batch
rollouts_per_example = 8 # Rollouts generated per dataset example
# ============================================================
# Training Hyperparameters (optional)
# ============================================================
# learning_rate = 1e-4 # Learning rate for LoRA
# lora_alpha = 16 # LoRA alpha scaling factor
# oversampling_factor = 2.0 # Oversample factor for rollout generation
# trajectory_strategy = "interleaved" # "interleaved" or "branching"
# ============================================================
# Secrets (optional)
# ============================================================
# env_file = ["secrets.env"] # File(s) containing environment secrets
# ============================================================
# Sampling Configuration (required)
# ============================================================
[sampling]
max_tokens = 512 # Max tokens per model response
# enable_thinking = false # Toggle thinking mode (Qwen3.5, Nemotron)
# reasoning_effort = "high" # Reasoning effort: "low" | "medium" | "high" (GPT-OSS)
# ============================================================
# Environment(s) (at least one required)
# ============================================================
[[env]]
id = "primeintellect/alphabet-sort" # Environments Hub ID (owner/name)
# args = { min_turns = 3, max_turns = 5 } # Arguments passed to load_environment()
# Verifiers v1 taskset/harness shape:
# [[env]]
# name = "alphabet-sort"
# taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, power_per_turn = false }
# harness = { id = "default", runtime = { type = "subprocess" } }
# Add multiple [[env]] sections for multi-environment training:
# [[env]]
# id = "primeintellect/another-env"
# args = { split = "train", max_examples = 1000 }
# ============================================================
# Weights & Biases Logging (optional)
# ============================================================
# [wandb]
# project = "my-project" # W&B project name
# name = "my-run-name" # W&B run name
# entity = "my-team" # W&B team/entity
# ============================================================
# Online Evaluation (optional)
# ============================================================
# [eval]
# interval = 100 # Run eval every N training steps
# num_examples = -1 # Number of eval examples (-1 = all)
# rollouts_per_example = 1 # Rollouts per eval example
# skip_first_step = false # Skip the pre-training eval of the base model
#
# [eval.sampling] # Eval-time sampling overrides
# max_tokens = 2048 # Max tokens per eval response
# temperature = 0.0 # Eval sampling temperature
# enable_thinking = false # Toggle thinking mode at eval time
# reasoning_effort = "high" # Reasoning effort at eval time
#
# [[eval.env]] # Environment-specific eval overrides
# id = "primeintellect/eval-env"
# args = { split = "test" }
# num_examples = 30
# rollouts_per_example = 4
#
# Verifiers v1 eval envs use the same taskset/harness shape:
# [[eval.env]]
# taskset = { id = "gsm8k-v1", split = "test" }
# harness = { id = "default", runtime = { type = "subprocess" } }
# num_examples = 256
# group_size = 4
# ============================================================
# Validation During Training (optional)
# ============================================================
# [val]
# num_examples = 64 # Validation examples per check
# rollouts_per_example = 1 # Rollouts per validation example
# interval = 5 # Validate every N steps
# ============================================================
# Rollout Filters (optional)
# ============================================================
# [[pre_batch_filters]] # Applied before rollouts fill a batch slot
# type = "zero_advantage" # "gibberish" | "repetition" | "zero_advantage"
# enforce = true # Drop flagged rollouts (false = metrics only)
#
# [[post_batch_filters]] # Applied after a batch is assembled
# type = "repetition"
# enforce = false
# ============================================================
# Warm-Start from Checkpoint (optional)
# ============================================================
# checkpoint_id = "..." # Resume training from an existing checkpoint
# ============================================================
# Checkpoints (optional)
# ============================================================
# [checkpoints]
# interval = 100 # Save checkpoint every N steps
# keep_cloud = 5 # Keep N checkpoints in cloud (-1 = keep all)
# ============================================================
# Adapters (optional)
# ============================================================
# [adapters]
# interval = 0 # Upload adapter every N steps (0 = only at run end)
# keep_last = 3 # Keep N adapters in cloud (-1 = keep all)
# ============================================================
# Infrastructure (optional)
# ============================================================
# [infrastructure]
# compute_size = "M" # CPU allocation: S, M (default), or L
```
## Field Reference
### Core Fields
| Field | Type | Required | Description |
| ---------------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model` | string | ✓ | HuggingFace model ID. Must be a [supported model](/hosted-training/models-and-pricing). Run `prime train models` to see available options. |
| `max_steps` | integer | ✓ | Total number of training steps. |
| `batch_size` | integer | ✓ | Number of rollouts consumed per training batch. Larger values improve stability. |
| `rollouts_per_example` | integer | ✓ | Number of rollouts generated per dataset example. Higher values give more reward signal diversity. |
| `checkpoint_id` | string | — | Checkpoint ID to warm-start from. The checkpoint must be in READY status, accessible to you, and from a run using the same model. See [Warm-Starting from a Checkpoint](#warm-starting-from-a-checkpoint). |
### Training Hyperparameters
| Field | Type | Default | Description |
| --------------------- | ---------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `learning_rate` | float | `1e-4` | Learning rate for the LoRA adapter. |
| `lora_alpha` | integer | `16` | LoRA alpha scaling factor. Controls the magnitude of LoRA updates. |
| `oversampling_factor` | float | `2.0` | Generate this many more rollouts than needed per batch to ensure sufficient data. |
| `trajectory_strategy` | string | `"interleaved"` | How multi-turn trajectories are generated. `"interleaved"` runs turns across examples concurrently. `"branching"` generates full trajectories per example before moving on. |
| `env_file` | array of strings | `[]` | Path(s) to `.env` files containing secrets (e.g., API keys). See [Secrets Management](#secrets-management). |
### Sampling
| Field | Type | Required | Description |
| ----------------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `[sampling].max_tokens` | integer | ✓ | Maximum number of tokens the model can generate per response turn. |
| `[sampling].enable_thinking` | boolean | — | Toggle thinking mode for supported models. Mutually exclusive with `reasoning_effort`. |
| `[sampling].reasoning_effort` | string | — | Reasoning effort for supported models. One of `"low"`, `"medium"`, `"high"`. Mutually exclusive with `enable_thinking`. |
### Eval Sampling
Overrides the inference server's default sampling for eval-time rollouts only. All fields are optional; when the whole `[eval.sampling]` block is omitted, eval uses the server defaults.
| Field | Type | Required | Description |
| ---------------------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `[eval.sampling].max_tokens` | integer | — | Maximum tokens generated per eval response turn. |
| `[eval.sampling].temperature` | float | — | Eval sampling temperature. `0.0` for deterministic eval scoring. |
| `[eval.sampling].extra_body` | table | — | Free-form extra parameters forwarded with each eval request to the inference server. |
| `[eval.sampling].enable_thinking` | boolean | — | Toggle thinking mode at eval time for supported models. Mutually exclusive with `reasoning_effort`. |
| `[eval.sampling].reasoning_effort` | string | — | Reasoning effort at eval time for supported models. One of `"low"`, `"medium"`, `"high"`. Mutually exclusive with `enable_thinking`. |
### Environment
Each `[[env]]` entry must provide either a legacy (Verifiers v0) `id` or a Verifiers v1 `taskset.id`.
| Field | Type | Required | Description |
| --------------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `[[env]].id` | string | Either `id` or `taskset.id` | Legacy environment ID. Use `owner/name` for an Environments Hub package, or a bare id for a runtime-installed/local environment. |
| `[[env]].args` | table | — | Legacy v0 arguments passed to `load_environment(**args)`. |
| `[[env]].name` | string | — | Display name for this environment instance. Defaults to `id` or `taskset.id`. Use this when the same env appears more than once. |
| `[[env]].taskset` | table | Either `id` or `taskset.id` | v1 taskset config. Must include `id`, plus taskset-specific fields such as `split`, `min_turns`, or dataset options. |
| `[[env]].harness` | table | — | v1 harness config. If omitted, the runtime default harness is used. Include `runtime` when the harness should run in a specific backend such as `subprocess` or `modal`. |
| `[[env]].ratio` | float | — | Relative sampling weight for multi-environment v1 runs. If one env sets `ratio`, set it on all envs in the group. |
| `[[env]].group_size` | integer | — | Per-environment rollout group size for v1 training. Defaults to the top-level `rollouts_per_example`. |
| `[[env]].pool` | table | — | v1 env-server worker pool config, for example `{ type = "static", num_workers = 4 }`. |
| `[[env]].timeout` | table | — | v1 rollout/setup/scoring timeout config. |
| `[[env]].retries` | table | — | v1 retry policy config. |
| `[[env]].sampling` | table | — | Per-environment v1 sampling overrides. |
| `[[env]].max_turns` | integer | — | Per-environment turn limit. |
| `[[env]].max_input_tokens` | integer | — | Per-environment input-token limit. |
| `[[env]].max_output_tokens` | integer | — | Per-environment output-token limit. |
| `[[env]].max_total_tokens` | integer | — | Per-environment total-token limit. |
| `[[env]].multiplex` | integer | — | v1 environment multiplexing setting. |
`taskset.id`, `harness.id`, and legacy `id` can be bare runtime ids or Hub refs. Bare ids such as `"alphabet-sort-v1"` or `"default"` are resolved by the installed runtime package. Slash-shaped ids such as `"team/my-taskset"` are resolved through the Environments Hub and can include `@version`.
### Legacy (Verifiers v0) envs
Use `id` and optional `args` for existing Verifiers environments that expose `load_environment(**args)`:
```toml theme={null}
[[env]]
id = "primeintellect/alphabet-sort"
args = { min_turns = 3, max_turns = 5 }
```
### Verifiers v1 (taskset/harness) envs
Use `taskset` and `harness` for Verifiers v1 environments:
```toml theme={null}
[[env]]
name = "alphabet-sort"
taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, power_per_turn = false }
harness = { id = "default", runtime = { type = "subprocess" } }
```
`taskset` owns task data, task controls, rewards, metrics, and task-owned tools. `harness` selects the program that drives the model — the built-in `default` harness, or an agent harness such as `bash` or `codex` — and its nested `runtime` field selects where that program runs: a local `subprocess`, or a `docker`/`prime`/`modal` sandbox.
## Multi-Environment Training
You can train on multiple environments simultaneously by adding multiple `[[env]]` sections:
```toml theme={null}
[[env]]
id = "primeintellect/alphabet-sort"
args = { min_turns = 3, max_turns = 5 }
[[env]]
id = "primeintellect/gsm8k"
args = { split = "train" }
```
For Verifiers v1 envs, put sampling weights directly on each `[[env]]` entry:
```toml theme={null}
[[env]]
name = "alphabet-sort"
taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5 }
harness = { id = "default", runtime = { type = "subprocess" } }
ratio = 0.75
[[env]]
name = "gsm8k"
taskset = { id = "gsm8k-v1", split = "train" }
harness = { id = "default", runtime = { type = "subprocess" } }
ratio = 0.25
```
## Online Evaluation
Enable periodic evaluation during training to track progress without interrupting the run:
```toml theme={null}
[eval]
interval = 100 # Evaluate every 100 steps
num_examples = -1 # Use all eval examples
rollouts_per_example = 1
skip_first_step = false # Evaluate the base model before training starts
[[eval.env]]
id = "primeintellect/alphabet-sort"
args = { split = "test" }
num_examples = 50
rollouts_per_example = 4
```
The `[eval]` section sets global defaults, and `[[eval.env]]` sections can override settings per environment.
Verifiers v1 eval entries use the same `taskset`/`harness` shape. `group_size` is the v1 name for per-env eval rollouts; `rollouts_per_example` is still accepted for compatibility.
```toml theme={null}
[[eval.env]]
taskset = { id = "alphabet-sort-v1", split = "test" }
harness = { id = "default", runtime = { type = "subprocess" } }
num_examples = 50
group_size = 4
```
### Eval Sampling
Eval rollouts use the inference server's default sampling unless overridden via `[eval.sampling]`. The fields mirror `[sampling]` and `[teacher.sampling]` so the same knobs work everywhere — most commonly, you'd turn thinking off at eval time to get deterministic, faster scoring on a model that uses chain-of-thought during training:
```toml theme={null}
[eval.sampling]
max_tokens = 2048
temperature = 0.0
enable_thinking = false # Disable thinking at eval time
# reasoning_effort = "high" # Or constrain reasoning effort
```
`enable_thinking` and `reasoning_effort` are mutually exclusive — set at most one. Both ride on `extra_body.chat_template_kwargs` under the hood; you can also set `extra_body` directly if you need other chat-template controls.
## Validation
Validation is a lightweight check that runs more frequently than full evaluation:
```toml theme={null}
[val]
num_examples = 64
rollouts_per_example = 1
interval = 5 # Validate every 5 steps
```
This uses the training environment's validation split (if available) and reports metrics to W\&B and the dashboard.
## Rollout Filters
prime-rl filters rollouts at two points in the training pipeline. `[[pre_batch_filters]]` run before a rollout enters the training batch, so flagged rollouts never consume a batch slot; `[[post_batch_filters]]` run after a batch is assembled, and flagged rollouts are recorded but not shipped to the trainer. Three filter types are available — `gibberish`, `repetition`, and `zero_advantage` — and each either records detection metrics only (`enforce = false`) or drops flagged rollouts (`enforce = true`).
By default, all three filters run in monitor mode pre-batch and `zero_advantage` is enforced post-batch. Setting either section replaces the default filter list for that slot.
To focus training compute on examples with useful reward signal — the successor to the removed difficulty buffer's `online_difficulty_filtering` — enforce the zero-advantage filter pre-batch:
```toml theme={null}
[[pre_batch_filters]]
type = "zero_advantage"
enforce = true
```
Type-specific tuning knobs (such as `repetition`'s `window` and `prob_threshold`) pass through to the trainer as written.
## Checkpoints
Control how often checkpoints are saved and how many are retained in cloud storage:
```toml theme={null}
[checkpoints]
interval = 100 # Save checkpoint every 100 steps
keep_cloud = 5 # Keep last 5 checkpoints in cloud
```
| Field | Type | Default | Description |
| ------------ | ------- | --------------- | -------------------------------------------------------------------------------------- |
| `interval` | integer | cluster default | Save a checkpoint every N training steps. |
| `keep_cloud` | integer | `5` | Number of checkpoints to retain in cloud storage. Set to `-1` to keep all checkpoints. |
Checkpoints enable resuming training from a specific step if a run is interrupted. They're automatically uploaded to cloud storage and can be used to create new runs from a saved state.
## Warm-Starting from a Checkpoint
Start a new run from an existing checkpoint by setting `checkpoint_id` at the top level of your config. The checkpoint must be READY, use the same model, and you need access to the original run.
```toml theme={null}
checkpoint_id = "cp_abc123"
```
List available checkpoints with `prime train checkpoints `.
## Adapters
Configure periodic adapter uploads during training. Adapters are LoRA weights that can be deployed for inference.
```toml theme={null}
[adapters]
interval = 100 # Upload adapter every 100 steps
keep_last = 3 # Keep last 3 adapters in cloud
```
| Field | Type | Default | Description |
| ----------- | ------- | ------- | ---------------------------------------------------------------------------------------------- |
| `interval` | integer | `0` | Upload adapter every N training steps. Set to `0` to only upload the final adapter at run end. |
| `keep_last` | integer | `3` | Number of adapters to retain in cloud storage. Set to `-1` to keep all adapters. |
Deployed adapters are protected from automatic cleanup. If you deploy an adapter for inference, it will not be deleted even if it exceeds the `keep_last` limit.
## Infrastructure
Control the CPU and memory resources allocated to your environment containers. This only affects the environments you provide — trainer and inference infrastructure is fully managed by us.
```toml theme={null}
[infrastructure]
compute_size = "L"
```
| Size | Description |
| ---- | ---------------------------------------------------------------------------------------------------------------------- |
| `S` | Lower CPU allocation. Suitable for lightweight environments. |
| `M` | Default. Balanced allocation for most workloads. |
| `L` | High CPU allocation. Use for environments that compile code or for vision-language models with heavy image processing. |
If not specified, runs default to `M`. Most users won't need to change this — use `L` if you notice slow CPU-bound operations during training.
## Tailscale Networking
Tailscale networking is an **enterprise-only** feature. Contact your account team to enable it on your organization.
When enabled, every env-server (training and eval) for the run joins your Tailscale tailnet via a sidecar. From inside your environment code you can then reach private services — internal APIs, MCP servers, datasets behind a VPN — by their Tailscale IP, MagicDNS hostname, or by native LAN IP if a [subnet router](https://tailscale.com/kb/1019/subnets) advertises it.
```toml theme={null}
[tailscale]
enabled = true
# auth_key = "tskey-auth-..." # preferably via TAILSCALE_AUTH_KEY env var
# hostname_prefix = "prime-hosted-training"
```
| Field | Type | Default | Description |
| ----------------------------- | ------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `[tailscale].enabled` | boolean | `false` | Toggle the per-run sidecar. |
| `[tailscale].auth_key` | string | — | Tailscale [pre-authenticated key](https://tailscale.com/kb/1085/auth-keys) (must start with `tskey-auth-`). OAuth client secrets are not supported. Prefer the `TAILSCALE_AUTH_KEY` environment variable so the secret is not committed to `rl.toml`. |
| `[tailscale].hostname_prefix` | string | `"prime-hosted-training"` | Prefix for the Tailscale node name. The full name is derived as `{prefix}-env-{idx}-{run_id}`. 1–30 lowercase alphanumeric chars or hyphens, must start with a letter. |
Use a **tagged**, ephemeral, reusable auth key. Tagged keys let you scope the env-servers in your tailnet ACL without granting them the same access as a user-owned device.
## Weights & Biases Integration
Log training metrics, reward curves, and rollout samples to W\&B:
```toml theme={null}
[wandb]
project = "my-rl-experiments"
name = "qwen3-30b-alphabet-sort"
entity = "my-team"
```
When W\&B is configured, all training metrics, evaluation results, and sample rollouts are logged automatically.
## Secrets Management
The recommended way to supply secrets to Hosted Training is via [environment secrets](/tutorials-environments/secrets). Secrets linked or added to your environment are automatically injected at runtime — no config changes needed.
If you prefer to supply secrets via a file, you can use `env_file` in your training config instead:
```toml theme={null}
env_file = ["secrets.env"]
```
The `secrets.env` file should contain key-value pairs:
```
OPENAI_API_KEY=sk-...
CUSTOM_API_KEY=...
```
You can also manage secrets via the CLI:
```bash theme={null}
prime secret list # list global secrets
prime env secret list my-env # list secrets for an environment
```
In your environment code, validate required keys early using `vf.ensure_keys()`:
```python theme={null}
def load_environment(api_key_var: str = "OPENAI_API_KEY") -> vf.Environment:
vf.ensure_keys([api_key_var])
# ...
```
Walk through a complete training run step by step.
Solutions for common issues with Hosted Training.
# End-to-End Training Run
Source: https://docs.primeintellect.ai/hosted-training/end-to-end-run
Walk through a complete Hosted Training run from environment setup to results
This guide walks you through a complete Hosted Training run — from setting up your workspace and choosing an environment to launching a run, monitoring progress, and reviewing results.
## Prerequisites
Make sure you've completed the initial setup:
```bash theme={null}
# Install and authenticate the CLI
uv tool install prime
prime login
# Set up a workspace
mkdir ~/dev/my-lab && cd ~/dev/my-lab
prime lab setup
```
See [Getting Started](/hosted-training/getting-started) if you need help with any of these steps.
## Step 1: Choose an Environment
You can use an existing environment from the [Environments Hub](https://app.primeintellect.ai/dashboard/environments) or create your own. For this walkthrough, we'll use the `alphabet-sort` environment — a multi-turn game where the model must sort letters into alphabetical order. If you're new to how environments work, see [The Environment Model](/hosted-training/environment-model).
Install it:
```bash theme={null}
prime env install primeintellect/alphabet-sort
```
## Step 2: Run a Baseline Evaluation
Before training, evaluate the base model to establish a baseline. This helps you confirm the environment works and understand where the model starts:
```bash theme={null}
prime eval run primeintellect/alphabet-sort \
-m Qwen/Qwen3-4B-Instruct-2507 \
-n 20 -r 1
```
A good training environment should have a baseline reward between roughly 10–80%. If the model scores 0% after many attempts, the task is too hard. If it's already at 80%+, consider harder examples or a different environment.
View the results:
```bash theme={null}
prime eval tui
```
## Step 3: Choose a Model
Check which models are available for Hosted Training:
```bash theme={null}
prime train models
```
For a first run, we recommend starting with a smaller model to validate your setup quickly:
| Use Case | Recommended Model |
| ---------------- | -------------------------------------------------------------------- |
| Quick validation | `Qwen/Qwen3-4B-Instruct-2507` |
| Experimentation | `Qwen/Qwen3-30B-A3B-Instruct-2507` |
| Production scale | `Qwen/Qwen3-235B-A22B-Instruct-2507` or `PrimeIntellect/INTELLECT-3` |
See [Models & Pricing](/hosted-training/models-and-pricing) for the full list.
## Step 4: Create a Training Config
Training runs are configured via a `.toml` file. Create one in your `configs/rl/` directory:
```toml theme={null}
# configs/rl/alphabet-sort.toml
model = "Qwen/Qwen3-4B-Instruct-2507"
max_steps = 50
batch_size = 128
rollouts_per_example = 8
[sampling]
max_tokens = 512
[[env]]
id = "primeintellect/alphabet-sort"
```
This is a minimal config suitable for a validation run. The key fields are:
* `model` — The Hugging Face model ID (must be a supported model)
* `max_steps` — Total number of training steps
* `batch_size` — Number of rollouts per training batch
* `rollouts_per_example` — How many rollouts to generate per dataset example
* `[sampling].max_tokens` — Maximum tokens the model can generate per response
* `[[env]].id` — Legacy environment selector, usually an Environments Hub ID
Hosted Training also supports Verifiers v1 taskset/harness environment configs in the same outer `[[env]]` block. The Verifiers v1 equivalent of the environment section above looks like:
```toml theme={null}
[[env]]
name = "alphabet-sort"
taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, power_per_turn = false }
harness = { id = "default", runtime = { type = "subprocess" } }
```
Use the legacy `id` shape for existing Verifiers environments that expose `load_environment(**args)`. Use the Verifiers v1 `taskset`/`harness` shape when the environment is packaged as a taskset (task data, controls, rewards, metrics) plus a harness (the program that drives the model). Bare ids such as `"alphabet-sort-v1"` are resolved by the runtime package; slash-shaped ids such as `"team/my-taskset"` are resolved through the Environments Hub.
## Step 5: Launch the Training Run
Start the run:
```bash theme={null}
prime train run configs/rl/alphabet-sort.toml
```
You'll see output confirming the configuration and a link to the dashboard:
```
Loading config from configs/rl/alphabet-sort.toml
Creating RL training run...
Configuration:
Model: Qwen/Qwen3-4B-Instruct-2507
Environments: primeintellect/alphabet-sort
Max Steps: 50
Batch Size: 128
Rollouts per Example: 8
Max Tokens: 512
✓ Run created successfully!
Monitor run at:
https://app.primeintellect.ai/dashboard/training/
```
## Step 6: Monitor the Run
You can monitor your run in two ways:
**In the terminal** — stream logs in real-time:
```bash theme={null}
prime train logs -f
```
**On the dashboard** — open the URL printed when the run started. The dashboard shows reward curves, rubric scores, reward distributions, and individual rollouts.
Key metrics to watch:
* **Reward** — The overall reward curve should trend upward over time
* **Rubric** — Individual rubric component scores
* **Reward Distribution** — Should shift from lower to higher values as training progresses
## Step 7: Review Results
Once the run completes, you can review the trained model's performance by running an evaluation with the trained adapter. Trained LoRA adapters can be downloaded from the dashboard.
You can also deploy your trained LoRA adapter for live inference — see [Deploying LoRA Adapters for Inference](/inference/adapter-deployments) for a step-by-step guide.
To compare against the baseline, re-run the same evaluation you ran in Step 2 and compare scores.
## Putting It All Together
Here's the complete workflow as a single script:
```bash theme={null}
# Setup
uv tool install prime
prime login
mkdir ~/dev/my-lab && cd ~/dev/my-lab
prime lab setup
# Install and evaluate baseline
prime env install primeintellect/alphabet-sort
prime eval run primeintellect/alphabet-sort \
-m Qwen/Qwen3-4B-Instruct-2507 -n 20 -r 1
# Launch training
prime train run configs/rl/alphabet-sort.toml
# Monitor
prime train logs -f
```
## Run Size Guidelines
Depending on your goals, here are some recommended configurations:
### Small Run (Validation)
Use this to verify your environment and config work correctly before committing to a longer run.
```toml theme={null}
model = "Qwen/Qwen3-4B-Instruct-2507"
max_steps = 50
batch_size = 128
rollouts_per_example = 8
[sampling]
max_tokens = 512
```
### Medium Run (Experimentation)
Good for iterating on environment design and hyperparameters.
```toml theme={null}
model = "Qwen/Qwen3-30B-A3B-Instruct-2507"
max_steps = 200
batch_size = 256
rollouts_per_example = 16
[sampling]
max_tokens = 512
[wandb]
project = "my-experiment"
name = "alphabet-sort-30b"
[eval]
interval = 50
```
### Large Run (Production)
For serious training with full monitoring and evaluation.
```toml theme={null}
model = "Qwen/Qwen3-235B-A22B-Instruct-2507"
max_steps = 500
batch_size = 512
rollouts_per_example = 16
[sampling]
max_tokens = 1024
[wandb]
project = "production"
name = "alphabet-sort-235b"
[eval]
interval = 100
num_examples = -1
rollouts_per_example = 1
skip_first_step = false
[val]
num_examples = 64
rollouts_per_example = 1
interval = 5
```
Explore all configuration options including multi-env training, evaluation, and checkpointing.
Solutions for common issues with Hosted Training runs.
# The Environment Model
Source: https://docs.primeintellect.ai/hosted-training/environment-model
Understand environment types, rubrics, and the RL training loop
## The Training Loop
When you launch an RL training run, three decoupled components coordinate:
* **Inference** generates model completions turn-by-turn via a vLLM server.
* **Orchestrator** samples prompts from the environment's dataset, dispatches rollouts to inference, drives the multi-turn interaction loop (calling the environment's response logic and tool execution between each model turn), scores completed rollouts via the rubric, packs training batches, and relays weight updates between trainer and inference.
* **Trainer** receives scored rollouts, computes advantages, and produces weight updates via GRPO.
One step of training:
```
Orchestrator samples prompts from the environment
-> Inference generates a rollout (multi-turn, driven by env logic)
-> Environment scores the rollout via its rubric
-> Orchestrator packs the batch and computes advantages
-> Trainer updates policy weights
-> Orchestrator broadcasts new weights to inference
-> Repeat
```
The full loop is asynchronous. Rollout generation at step N overlaps with training at step N-1, and a single trajectory may span multiple policy versions as weights update mid-rollout.
**The environment is the only part you write.** Everything else (batching, weight sync, GPU scheduling, async rollout management) is handled by the infrastructure. An environment plugs directly into this loop, whether you run it via Hosted Training or self-hosted with `prime-rl`.
***
## What is an Environment?
An environment is a self-contained Python module that packages three things:
1. **A dataset** of prompts (with optional ground-truth answers or metadata).
2. **A harness** that controls how the model interacts with the task: single-turn Q\&A, multi-turn tool calling, stateful sandbox sessions, etc.
3. **A rubric** of reward functions that score the model's output and produce scalar signals.
The same environment definition drives RL training, standalone evaluation (`prime eval run`), and synthetic data generation. RL environments and agent evals are the same abstraction: dataset + harness + scoring rules.
Environments are distributed as versioned Python wheels declared in `pyproject.toml`, installable and shareable via the Environments Hub.
### Legacy and Verifiers v1 environment configs
Verifiers ships two packaging/configuration styles, and Hosted Training supports both:
* **Legacy (Verifiers v0) environments** expose a `load_environment(**args)` entrypoint. In training TOML, these use `[[env]] id = "owner/name"` plus optional `args = { ... }`.
* **Verifiers v1 (taskset/harness) environments** split task data and scoring from model-facing execution. In training TOML, these use `[[env]] taskset = { id = "..." }` and optional `harness = { id = "..." }`.
In the Verifiers v1 model, the **taskset** owns train/eval tasks, task controls, rewards, metrics, and task-owned tools. The **harness** is the program that drives the model — the built-in `default` harness, or an agent harness such as `bash` or `codex`. Its `runtime` field selects *where* that program runs: a local `subprocess`, or a `docker`/`prime`/`modal` sandbox. The Verifiers v1 `Environment` composes a taskset and harness and plugs into the same evaluation and training loop as legacy environments (the `vf.Environment` subclasses such as `SingleTurnEnv` and `ToolEnv` described below).
Hosted Training keeps the public config wrapper as `[[env]]`; self-managed `prime-rl` uses the same inner environment config under `[[orchestrator.train.env]]`.
***
## The Type Hierarchy
All user-facing environment types descend from `Environment`, the abstract base class that defines the dataset, rubric, and rollout interface. `MultiTurnEnv` extends `Environment` and implements the core rollout loop — the `rollout()` method is finalized there. The concrete types (`SingleTurnEnv`, `ToolEnv`, `StatefulToolEnv`) each extend `MultiTurnEnv`, layering interaction complexity progressively.
### `SingleTurnEnv`
Prompt in, completion out, reward computed. This is `MultiTurnEnv` with `max_turns=1`. Use it for standard Q\&A benchmarks.
```python theme={null}
import verifiers as vf
dataset = vf.load_example_dataset("gsm8k")
async def correct_answer(completion, answer) -> float:
return 1.0 if completion[-1]["content"] == answer else 0.0
env = vf.SingleTurnEnv(
dataset=dataset,
rubric=vf.Rubric(funcs=[correct_answer]),
)
```
### `MultiTurnEnv`
The base class for anything conversational or interactive. Override two hooks:
* **`env_response(messages, state)`**: produce the environment's next message given the conversation so far. Game logic, simulation steps, or external calls go here.
* **`@vf.stop` methods**: async methods decorated with `@vf.stop` act as termination conditions. The rollout ends when any stop condition returns `True` or `max_turns` is reached.
The rollout loop alternates between model inference and `env_response` until a stop condition fires or `max_turns` is reached.
### `ToolEnv`
Adds native tool/function-calling. Pass a list of Python functions; `verifiers` extracts JSON schemas from type hints and docstrings, wires them into the OpenAI-compatible tool-calling protocol, and dispatches calls during rollouts.
```python theme={null}
async def calculate(expression: str) -> str:
"""Evaluate a mathematical expression.
Args:
expression: A math expression to evaluate (e.g. "2 + 2 * 3")
"""
return str(eval(expression))
env = vf.ToolEnv(
dataset=dataset,
tools=[calculate, search_tool],
rubric=rubric,
max_turns=10,
)
```
Tools in `ToolEnv` must be **stateless and idempotent**: each call is fully determined by its arguments. The environment terminates when the model responds without issuing any tool calls (or hits `max_turns`).
### `StatefulToolEnv`
When tools need per-rollout state (a sandbox handle, a database connection, a session token), `StatefulToolEnv` adds two hooks:
* **`setup_state(state)`**: called at rollout start to initialize a fresh state dict.
* **`update_tool_args(tool_name, args, messages, state)`**: intercepts each tool call to inject state into arguments before dispatch.
All mutable state lives in the state dict. Never store globals.
***
## Environment Structure
```
my_env/
├── my_env.py # Implementation
├── pyproject.toml # Dependencies + metadata
└── README.md # Documentation
```
The entry point is `load_environment()`, which returns an `Environment` instance. This is what the training and evaluation infrastructure calls.
```python theme={null}
def load_environment(split: str = "train") -> vf.Environment:
dataset = load_dataset("my-org/my-dataset", split=split)
rubric = vf.Rubric(
funcs=[correctness, format_score],
weights=[0.8, 0.2],
)
return vf.ToolEnv(
dataset=dataset,
tools=[search_tool, execute_tool],
rubric=rubric,
max_turns=5,
)
```
Expensive setup (loading datasets, building indices) goes in `load_environment()` or `__init__()`. Per-rollout setup goes in `setup_state()`.
***
## Rubrics
A `Rubric` is a set of scoring functions evaluated after each rollout. Functions can be sync or async, and receive keyword arguments like `prompt`, `completion`, `answer`, `state`, and `parser` depending on what they need.
Functions are weighted and combined into a scalar reward. You can also define non-reward metrics (token count, tool call frequency) that get logged without affecting gradients. For subjective criteria, `JudgeRubric` delegates scoring to an external LLM judge.
# Full Fine-Tuning (Beta)
Source: https://docs.primeintellect.ai/hosted-training/full-finetuning
Dedicated full-parameter RL training on Hosted Training
Full fine-tuning is in **closed beta**. Access is gated per-team — reach out to us to get enabled.
Full fine-tuning updates every parameter of the model on a dedicated cluster reserved for your run, instead of training a LoRA adapter on top of a shared deployment.
## Config
Full-FT runs use the native [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) config schema. Set `type = "full_finetune"` at the top of your TOML and size the run with `[deployment]` — `num_train_gpus` / `num_infer_gpus` for single-node, `num_train_nodes` / `num_infer_nodes` for multi-node.
Minimal single-node example (1 trainer GPU + 1 inference GPU):
```toml theme={null}
type = "full_finetune"
name = "reverse-text-full-ft"
max_steps = 100
seq_len = 2048
[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
[deployment]
num_train_gpus = 1
num_infer_gpus = 1
[trainer.optim]
lr = 3e-6
[orchestrator]
batch_size = 64
rollouts_per_example = 8
[orchestrator.train.sampling]
max_completion_tokens = 512
[[orchestrator.train.env]]
id = "primeintellect/reverse-text"
name = "reverse-text"
[orchestrator.renderer]
name = "default"
[inference]
```
Multi-node example (2 train nodes + 2 inference nodes, each a full 8-GPU node):
```toml theme={null}
type = "full_finetune"
name = "qwen30b-math"
seq_len = 32768
[model]
name = "Qwen/Qwen3-30B-A3B-Thinking-2507"
[deployment]
num_train_nodes = 2
num_infer_nodes = 2
[trainer.model]
impl = "custom"
attn = "flash_attention_3"
ep = 8 # expert parallel (MoE)
[trainer.optim]
type = "adamw"
lr = 1e-6
[orchestrator]
batch_size = 512
oversampling_factor = 2
max_off_policy_steps = 8
[orchestrator.train.sampling]
max_completion_tokens = 32768
[[orchestrator.train.env]]
id = "primeintellect/math-env"
name = "math"
[inference.parallel]
tp = 8 # tensor parallel inside each inference replica
```
Multi-node runs broadcast weights over NCCL by default and auto-discover the cluster's RDMA devices — no extra config needed.
See the [prime-rl docs](/prime-rl/configuration) and [config examples](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/configs) for the full schema.
## Launching a run
Same CLI as LoRA — `prime train` auto-detects the config shape:
```bash theme={null}
prime train run configs/full-ft.toml
```
On dispatch you get a run ID:
```
Dispatched hosted run wn2cjdrzdo6bmfqajoeuu30p
```
Runs use the `main` tag of the prime-rl image by default. Pin a specific build with `--image-tag v0.5.1` on the CLI or `image_tag = "v0.5.1"` in the TOML (CLI wins).
## Monitoring
A full-FT run has several distinct components. Pick which one to read with `-c / --component`:
```bash theme={null}
prime train logs # orchestrator (default)
prime train logs -c trainer # trainer (FSDP / torchrun)
prime train logs -c inference # vLLM inference server
prime train logs --env # env-server for a specific env
```
List the orchestrator and env-server components for a run:
```bash theme={null}
prime train components
```
Follow and filter the same way as LoRA — `-f`, `--search`, `--regex`, `--level`, `--since`. See [Monitoring](/hosted-training/end-to-end-run#step-6-monitor-the-run) for details.
The dashboard works as it does for LoRA runs: reward curves, rubric scores, and individual rollouts at `https://app.primeintellect.ai/dashboard/training/`.
LoRA walkthrough — most workflow steps apply identically.
Full reference for the underlying training framework config schema.
# Getting Started
Source: https://docs.primeintellect.ai/hosted-training/getting-started
Launch your first Hosted Training run in minutes
Train a model using reinforcement learning on Prime Intellect's infrastructure — no GPUs to manage.
## Prerequisites
* Python 3.10+
* A [Prime Intellect account](https://app.primeintellect.ai)
## 1. Install the CLI and log in
```bash theme={null}
# install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# install the prime CLI
uv tool install -U prime
# log in
prime login
```
## 2. Set up your workspace
```bash theme={null}
# ~/dev/my-lab
prime lab setup
```
This sets up your workspace with `AGENTS.md`, environments, and example training configs in `configs/rl/`.
## 3. Launch training
Pick one of the example configs and run it:
```bash theme={null}
prime train run configs/rl/alphabet-sort.toml
```
That's it. Prime Intellect handles the GPU cluster, rollouts, and optimization. You'll get a dashboard link in the output.
You can also launch your coding agent (Claude, Cursor, OpenCode, etc.) and use it to manage your workspace and research workflows.
## 4. Monitor progress
**Terminal:**
```bash theme={null}
prime train logs -f
```
**Dashboard:** Open the link from the previous step to see reward curves, rubric scores, and individual rollouts in real time.
## Next steps
Detailed walkthrough with baseline evals, model selection, and result review.
Multi-environment training, online evals, and W\&B integration.
Supported models and pricing details.
Understand the full platform and how everything fits together.
# Models & Pricing
Source: https://docs.primeintellect.ai/hosted-training/models-and-pricing
Supported models and pricing for Hosted Training
Hosted Training supports a range of open-weights models. This page lists the currently available models, their pricing, and guidance on choosing the right model for your use case.
## Available Models
Prices are per million tokens, billed separately for input, output, and training.
| Model | Input (\$ / 1M) | Output (\$ / 1M) | Train (\$ / 1M) |
| -------------------------------------------- | --------------- | ---------------- | --------------- |
| `Qwen/Qwen3.5-0.8B` | 0.02 | 0.06 | 0.06 |
| `Qwen/Qwen3.5-2B` | 0.05 | 0.15 | 0.15 |
| `Qwen/Qwen3.5-4B` | 0.10 | 0.30 | 0.30 |
| `Qwen/Qwen3.5-9B` | 0.20 | 0.60 | 0.60 |
| `Qwen/Qwen3.5-35B-A3B` | 0.25 | 0.75 | 1.00 |
| `Qwen/Qwen3.6-35B-A3B` | 0.25 | 0.75 | 1.00 |
| `meta-llama/Llama-3.2-1B-Instruct` | 0.02 | 0.06 | 0.06 |
| `meta-llama/Llama-3.2-3B-Instruct` | 0.05 | 0.15 | 0.15 |
| `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | 0.15 | 0.45 | 0.60 |
| `openai/gpt-oss-20b` | 0.10 | 0.30 | 0.40 |
| `poolside/Laguna-XS.2` | 0.00 | 0.00 | 0.00 |
| `sprints/Llama-3.2-1B-Instruct` | 0.00 | 0.00 | 0.00 |
## Choosing a Model
### For Validation and Debugging
Start with a small, fast model to verify your environment and config work correctly before committing compute to a larger run.
**Recommended:** `Qwen/Qwen3.5-0.8B` or `meta-llama/Llama-3.2-1B-Instruct`
```toml theme={null}
model = "Qwen/Qwen3.5-0.8B"
max_steps = 50
batch_size = 128
rollouts_per_example = 8
```
### For Experimentation
MoE models with small active parameter counts give strong performance per token.
**Recommended:** `Qwen/Qwen3.5-35B-A3B`
```toml theme={null}
model = "Qwen/Qwen3.5-35B-A3B"
max_steps = 200
batch_size = 256
rollouts_per_example = 16
```
### For Production Training
For serious training runs where you want the strongest results available, use a flagship MoE model.
**Recommended:** `Qwen/Qwen3.6-35B-A3B`
```toml theme={null}
model = "Qwen/Qwen3.6-35B-A3B"
max_steps = 500
batch_size = 512
rollouts_per_example = 16
```
### Thinking Mode
Qwen3.5 and Nemotron models support a thinking mode that produces extended chain-of-thought reasoning before the final answer. Toggle it via `[sampling].enable_thinking` in your config. Thinking mode tends to help on tasks that benefit from multi-step reasoning (math, code, logic) at the cost of longer outputs.
## Checking Available Models
Always use the CLI to check the current list of supported models:
```bash theme={null}
prime train models
```
Add `--output json` to get the live pricing alongside the model list. This may differ from this page as models are being added regularly.
# Troubleshooting
Source: https://docs.primeintellect.ai/hosted-training/troubleshooting
Common issues and solutions for Hosted Training and Lab
This page covers common issues you may encounter when using Lab and Hosted Training, along with their solutions.
## Environment Issues
The verifiers library is not installed in your project.
```bash theme={null}
uv add verifiers
```
Or if you're installing a specific environment:
```bash theme={null}
prime env install my-env
```
This usually means a module name collision — your environment name conflicts with an existing Python package, or the environment wasn't installed correctly.
**Solutions:**
* Rename your environment to avoid conflicts
* Reinstall: `prime env install my-env`
* Check that your environment module exposes a `load_environment` function at the top level
The environment ID doesn't match any published environment.
```bash theme={null}
# Check the exact ID
prime env info owner/environment-name
# Install a specific version
prime env install owner/environment-name@latest
```
Make sure you're using the correct `owner/name` format and that you have access if it's a private environment.
The environment requires API keys that aren't set. The error message will list which keys are missing.
**For local evaluation:**
```bash theme={null}
export OPENAI_API_KEY=sk-...
```
**For Hosted Training**, the recommended approach is to link secrets to your environment via the [Environments Hub](/tutorials-environments/secrets). Alternatively, supply them via your training config:
```toml theme={null}
env_file = ["secrets.env"]
```
Or use the CLI to manage secrets:
```bash theme={null}
prime secret list # global secrets
prime env secret list my-env # per-environment secrets
```
## Training Issues
The task is too hard for the model at its current capability level. The model can't solve any examples, so there's no reward signal to learn from.
**Solutions:**
* Try a larger or more capable model
* Use easier examples (filter your dataset or adjust environment args)
* Increase `max_tokens` in `[sampling]` — the model may need more space to reason
* Check your rubric implementation for bugs that might always return 0
* Run a baseline evaluation first: `prime eval run my-env -m -n 20 -r 1`
The task is too easy — the model already solves everything.
**Solutions:**
* Use harder examples or a more challenging dataset split
* Add more demanding rubric criteria
* Use a smaller model that has more room to improve
There could be several causes:
* **Low reward diversity:** If all rollouts for an example get the same reward, there's no contrast for the model to learn from. Increase `rollouts_per_example` (16–32) to get more variation.
* **Learning rate too low:** Try increasing `learning_rate` (e.g., from `1e-4` to `3e-4`).
* **Batch size too small:** Larger batches provide more stable gradient estimates. Try `batch_size = 512`.
* **Task mismatch:** The task may not be suitable for RL training. Ensure the reward function produces a meaningful gradient of scores, not just binary 0/1.
A field in your TOML config has the wrong type or an invalid value.
**Common causes:**
* String values not quoted: `model = Qwen/Qwen3-4B` → `model = "Qwen/Qwen3-4B"`
* Integer where float expected or vice versa
* Missing required sections like `[sampling]` or `[[env]]`
Double-check your config against the [config reference](/hosted-training/advanced-configs).
The model you specified isn't currently supported for Hosted Training.
```bash theme={null}
# Check available models
prime train models
```
The model list is subject to change during the beta period. See [Models & Pricing](/hosted-training/models-and-pricing) for the current list.
## CLI Issues
The CLI isn't installed or isn't on your PATH.
```bash theme={null}
# Install or reinstall
uv tool install prime
# If already installed, upgrade
uv tool install -U prime
# Verify
prime --version # Should be >= 0.5.15
```
If you installed with `uv tool install` and it's still not found, make sure `~/.local/bin` is in your PATH.
Your CLI session may have expired.
```bash theme={null}
prime login
```
This opens a browser window to re-authenticate.
Check the following:
* Your config file is valid TOML (no syntax errors)
* The model is available: `prime train models`
* The environment ID is correct and accessible
* You're authenticated: `prime login`
* Your CLI is up to date: `uv tool install -U prime`
## Evaluation Issues
Reduce concurrency when running evaluations:
```bash theme={null}
prime eval run my-env -m openai/gpt-4.1-mini -n 100 -c 8
```
The `-c` flag controls maximum concurrent requests. Lower it if you're hitting rate limits.
Qwen3 and DeepSeek-R1 models have chat templates that automatically remove `` tags from message history. This conflicts with `ThinkParser`.
**Solution:** Use `MaybeThinkParser` or `Parser` instead of `ThinkParser` in your environment:
```python theme={null}
# Instead of:
parser = vf.ThinkParser(extract_fn=my_fn)
# Use:
parser = vf.MaybeThinkParser(extract_fn=my_fn)
```
* **Check `rollouts_per_example`:** Low values (1–2) produce noisy results. Use at least 3–5 for reliable metrics.
* **Check `num_examples`:** Very small sample sizes can be misleading.
* **Check sampling temperature:** High temperatures produce more variation between runs.
* **Check your rubric:** Make sure reward functions handle edge cases (empty responses, malformed outputs, etc.).
## Environment Development Issues
Common causes:
* **Missing dependencies:** Make sure all required packages are listed in your environment's `pyproject.toml`
* **Missing secrets:** API keys available locally may not be set for hosted runs. Link them to your environment via the [Environments Hub](/tutorials-environments/secrets) or use `env_file` in your config as a fallback.
* **Hardcoded paths:** Avoid absolute file paths in your environment code
* **Network access:** Some external APIs may not be reachable from the hosted environment
Start with a local evaluation using verbose output:
```bash theme={null}
prime eval run my-env -m openai/gpt-4.1-mini -n 5 -r 1 -v
```
The `-v` flag enables verbose logging. You can also test your environment directly in Python:
```python theme={null}
from verifiers import load_environment
env = load_environment("my-env")
# Inspect the dataset, rubric, etc.
```
## Getting Help
If your issue isn't covered here:
* **Discord:** Join the [Prime Intellect Discord](https://discord.gg/ZTFydGWPKj) for community support and Q\&A
* **Research Support:** Fill out the [research support form](https://form.typeform.com/to/iYn9UliG) for hands-on assistance
* **Feedback:** Use the thumbs-up/down on any docs page to let us know what's helpful or missing
# What is Lab?
Source: https://docs.primeintellect.ai/hosted-training/what-is-lab
An overview of Lab, Prime Intellect's open research platform for post-training
Lab is Prime Intellect's open research platform for post-training. It unifies the [Environments Hub](/tutorials-environments/environments), Hosted Training, and Hosted Evaluations into a full-stack platform for reinforcement learning research and optimization.
The platform enables the entire lifecycle of post-training research — from large-scale agentic RL, to inference and evaluation — without needing to worry about the costs of massive GPU clusters or the details of low-level algorithm implementation.
## Core Concepts
### Environments
Lab is built around **environments**, which contain everything needed to run a model on a task:
* A **dataset** of tasks (input prompts with optional ground-truth answers)
* A **harness** for the model (tools, sandboxes, context management, multi-turn interactions)
* A **rubric** to score the model's performance (reward functions, metrics)
Environments are self-contained Python modules built with the [verifiers](https://github.com/PrimeIntellect-ai/verifiers) library. They can be used to train models with reinforcement learning, evaluate capabilities, generate synthetic data, optimize prompts, experiment with agent harnesses, and more. See [The Environment Model](/hosted-training/environment-model) for a detailed breakdown of environment types and the rollout loop.
### Environments Hub
The [Environments Hub](https://app.primeintellect.ai/dashboard/environments) is a community registry where you can browse and deploy open-source environments, or create and share your own. Environments support running evaluations with any OpenAI-compatible model endpoint, or training any supported open-weights model via Hosted Training.
The hub includes a range of pre-built environments spanning math, coding, games, search and tool use, multimodal tasks, and more — as well as validated implementations of popular benchmarks like AIME, MATH-500, and Humanity's Last Exam.
### Hosted Training
Hosted Training allows you to run large-scale RL training experiments in your own environments without managing infrastructure. Prime Intellect currently supports agentic RL with LoRA, built on the open-source [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) training library.
### Hosted Evaluations
Run evaluations against any environment directly from the web UI or CLI, using Prime Inference or your own OpenAI-compatible endpoints. Results are saved and viewable in the terminal UI or on the dashboard.
## Architecture
The platform is powered by the [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) training framework. The three key components of the architecture are:
* **Trainer** — Consumes batches and updates model weights using LoRA
* **Inference** — Serves the model via an OpenAI-compatible API with live weight updating
* **Orchestrator** — Manages environment logic, schedules rollout requests, and coordinates the training loop
For Hosted Training, each run is allocated a dedicated Orchestrator which manages environment logic, while Prime Intellect manages multi-tenant LoRA deployments for the Trainer and Inference components. This architecture allows shared hardware across runs, enabling high efficiency and per-token pricing.
## What You Can Do with Lab
| Capability | Description |
| ----------------------------- | --------------------------------------------------------------------------------- |
| **RL Training** | Train open-weights models with reinforcement learning on your custom environments |
| **Evaluation** | Benchmark any OpenAI-compatible model against environments on the Hub |
| **Environment Development** | Build, test, and share environments using the verifiers library |
| **Synthetic Data Generation** | Generate rollout data for downstream use |
## Key Libraries
Lab integrates with two core open-source libraries:
**[verifiers](https://github.com/PrimeIntellect-ai/verifiers)** — The environment framework. Provides building blocks for datasets, rubrics, tools, sandboxes, and multi-turn interactions. Environments built with verifiers are portable across local evaluation, hosted evaluation, and Hosted Training.
**[prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)** — The training framework. Supports fully asynchronous distributed RL at scale with FSDP2 training and vLLM inference. Used both for self-hosted training on your own GPUs and as the backend for Hosted Training.
## Getting Access
To get started with Lab:
1. [Create a Prime Intellect account](https://app.primeintellect.ai)
2. Install the Prime CLI: `uv tool install prime`
3. Authenticate: `prime login`
4. Set up a workspace: `prime lab setup`
Set up your workspace and run your first environment.
Walk through a complete Hosted Training run.
# Deploying LoRA Adapters for Inference
Source: https://docs.primeintellect.ai/inference/adapter-deployments
Deploy trained LoRA adapters from Hosted Training runs and query them via an OpenAI-compatible API
When a Hosted Training run completes, it produces a LoRA adapter — a lightweight set of model weights that captures what the model learned during training. You can deploy these adapters for live inference and query them through an OpenAI-compatible API, using the same tools and SDKs you already use.
## Prerequisites
* A completed training run with a **READY** LoRA adapter (see [End-to-End Training Run](/hosted-training/end-to-end-run))
* The Prime CLI installed and authenticated (`prime login`)
* A Prime API key with **Inference** permission (see [Inference Overview](/inference/overview))
## Step 1: List Your LoRA Adapters
View all LoRA adapters from your training runs and their current status:
```bash theme={null}
prime deployments list
```
You'll see a table like this:
| ID | Name | Base Model | Status | Deployed At |
| -------------------------- | -------- | --------------------------- | ------------- | ----------- |
| `gw3zytpj9den6zgp4w9xosnk` | my-model | Qwen/Qwen3-4B-Instruct-2507 | NOT\_DEPLOYED | - |
The **Status** column shows where each adapter is in the deployment lifecycle — see the [Status Reference](#status-reference) below for all possible states.
Use `prime deployments list -o json` for machine-readable output, or `--team ` to filter by team.
## Step 2: Deploy a LoRA Adapter
Deploy a LoRA adapter by its ID:
```bash theme={null}
prime deployments create
```
The CLI will show LoRA adapter details and ask for confirmation:
```
Deploying model:
ID: gw3zytpj9den6zgp4w9xosnk
Name: my-model
Base Model: Qwen/Qwen3-4B-Instruct-2507
Are you sure you want to deploy this model? [y/N]: y
Deployment initiated successfully!
Status: DEPLOYING
The model is being deployed. This may take a few minutes.
Use 'prime deployments list' to check deployment status.
```
Deployment typically takes a few minutes. Use `prime deployments list` to check when the status changes to **DEPLOYED**.
## Step 3: Run Inference
Once deployed, query your LoRA adapter through the OpenAI-compatible inference API. The model identifier uses the format `base_model:adapter_id`.
Set your API key if you haven't already:
```bash theme={null}
export PRIME_API_KEY="your-api-key-here"
```
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PRIME_API_KEY" \
-d '{
"model": "Qwen/Qwen3-4B-Instruct-2507:gw3zytpj9den6zgp4w9xosnk",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 100
}'
```
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
base_url="https://api.pinference.ai/api/v1",
api_key="your-api-key"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-4B-Instruct-2507:gw3zytpj9den6zgp4w9xosnk",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100
)
print(response.choices[0].message.content)
```
Replace `Qwen/Qwen3-4B-Instruct-2507:gw3zytpj9den6zgp4w9xosnk` with your actual base model and adapter ID from `prime deployments list`.
## Step 4: Unload a LoRA Adapter
When you no longer need the LoRA adapter for inference, unload it:
```bash theme={null}
prime deployments delete
```
```
Unload initiated successfully!
Status: UNLOADING
The model is being unloaded.
Use 'prime deployments list' to check status.
```
Unloading removes the LoRA adapter from serving but preserves the model files. You can redeploy the same adapter at any time with `prime deployments create`.
## Status Reference
| Status | Description |
| --------------- | ----------------------------------------------------- |
| `NOT_DEPLOYED` | Adapter is not loaded for inference |
| `DEPLOYING` | Adapter is being loaded onto inference infrastructure |
| `DEPLOYED` | Adapter is live and accepting inference requests |
| `UNLOADING` | Adapter is being removed from inference |
| `DEPLOY_FAILED` | Deployment failed — check error details |
| `UNLOAD_FAILED` | Unload failed — check error details |
## Troubleshooting
### Deployment stuck in DEPLOYING
Deployments typically complete within a few minutes. If the status remains **DEPLOYING** for an extended period, try running `prime deployments list` to check status. If the issue persists, contact support or post in the #inference channel in our [Discord](https://discord.gg/ZTFydGWPKj).
### DEPLOY\_FAILED or UNLOAD\_FAILED
These error states indicate an infrastructure issue. You can retry deployment with `prime deployments create `, or unload a failed deployment with `prime deployments delete `. If the error recurs, contact support or post in the #inference channel in our [Discord](https://discord.gg/ZTFydGWPKj).
Walk through a complete Hosted Training run from start to finish.
Learn more about the OpenAI-compatible inference API.
Solutions for common issues with Hosted Training runs.
# Inference Overview
Source: https://docs.primeintellect.ai/inference/overview
Access powerful language models through Prime Intellect Inference API
Prime Intellect Inference provides OpenAI-compatible API access to state-of-the-art language models. Our inference service routes requests to various model providers, offering flexible model selection made for running large scale evaluations.
## Getting Started
### 1. Get Your API Key
First, obtain your API key from the [Prime Intellect Platform](https://app.primeintellect.ai):
1. Navigate to your account settings
2. Go to the API Keys section
3. Generate a new API key with **Inference** permission enabled
Make sure to select the **Inference** permission when creating your API key. Without this permission, your requests will fail with authentication errors.
### 2. Set Up Authentication
Set your API key as an environment variable:
```bash theme={null}
export PRIME_API_KEY="your-api-key-here"
```
### 3. Access through the CLI or API
You can use Prime Inference in two ways:
#### Prime CLI (Recommended for Evaluations)
The Prime CLI provides easy access to inference models, especially useful for running evaluations:
```bash theme={null}
# List available models
prime inference models
# Use with environment evaluations (most common use case)
prime env eval gsm8k -m meta-llama/llama-3.1-70b-instruct -n 25
```
**For evaluations**: See [Environment Evaluations guide](/tutorials-environments/evaluating) for comprehensive examples and best practices regarding evaluations.
#### Direct API Access (OpenAI-Compatible)
**Team accounts**: Include the `X-Prime-Team-ID` header to use team credits instead of personal account. Find your team ID via `prime teams list` or on your [Team Profile page](https://app.primeintellect.ai/dashboard/team-profile).
```python Python theme={null}
import openai
import os
# Personal account
client = openai.OpenAI(
api_key=os.environ.get("PRIME_API_KEY"),
base_url="https://api.pinference.ai/api/v1"
)
# Team account (add X-Prime-Team-ID header)
client = openai.OpenAI(
api_key=os.environ.get("PRIME_API_KEY"),
base_url="https://api.pinference.ai/api/v1",
default_headers={
"X-Prime-Team-ID": "your-team-id-here"
}
)
# Make a chat completion request
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "user", "content": "What is Prime Intellect?"}
]
)
print(response.choices[0].message.content)
```
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $PRIME_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [
{"role": "user", "content": "What is Prime Intellect?"}
]
}'
# With team account (add X-Prime-Team-ID header)
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $PRIME_API_KEY" \
-H "X-Prime-Team-ID: your-team-id-here" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [
{"role": "user", "content": "What is Prime Intellect?"}
]
}'
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.PRIME_API_KEY}`,
'Content-Type': 'application/json',
// Add 'X-Prime-Team-ID': 'your-team-id-here' for team accounts
},
body: JSON.stringify({
model: 'meta-llama/llama-3.1-70b-instruct',
messages: [
{ role: 'user', content: 'What is Prime Intellect?' }
]
})
});
const data = await response.json();
console.log(data.choices[0].message.content);
```
## Available Models
Prime Inference provides access to various state-of-the-art language models. You can list all available models using the models endpoint:
### Get All Available Models
```python Prime CLI theme={null}
# List all available models
prime inference models
```
```python OpenAI Client theme={null}
# List all available models
models = client.models.list()
for model in models.data:
print(f"Model: {model.id}")
```
```bash cURL theme={null}
curl -H "Authorization: Bearer your-api-key-here" \
https://api.pinference.ai/api/v1/models
```
## Pricing and Billing
Prime Inference uses token-based pricing with competitive rates:
* **Input tokens**: Charged for tokens in your prompt
* **Output tokens**: Charged for tokens in the model's response
* **Billing**: Automatic deduction from your Prime Intellect account balance
Pricing varies by model. We will provide more details on pricing soon and make it available through the models API.
### Viewing Your Inference Usage
Track your inference usage and billing on the [Billing Dashboard](https://app.primeintellect.ai/dashboard/billing) under the **Inference** tab:
## Next Steps
Streaming responses, advanced parameters, and more examples
Using inference with team accounts and managing team billing
Fix common inference errors, including insufficient funds and team billing context
**Primary use case**: Learn how to run model evaluations using `prime env eval` with inference models
Detailed documentation for models and chat completion endpoints
# Using Team Accounts
Source: https://docs.primeintellect.ai/inference/team-accounts
How to use Prime Inference with team accounts
When using Prime Inference with a team account, you must explicitly pass your team ID to charge against team credits instead of your personal account.
## Finding Your Team ID
You can find your team ID in two ways:
**Via CLI:**
```bash theme={null}
prime teams list
```
**Via Web:**
Visit your [Team Profile page](https://app.primeintellect.ai/dashboard/team-profile)
## Setting Team ID
The `prime config set-team-id` command only affects CLI operations. For direct API usage (OpenAI client or HTTP requests), you must explicitly pass the `X-Prime-Team-ID` header.
### CLI Usage
When using the Prime CLI, configure your team ID once:
```bash theme={null}
# Set team ID for CLI operations
prime config set-team-id
# Verify configuration
prime config view
# Now CLI commands use team account
prime inference models
prime env eval gsm8k -m meta-llama/llama-3.1-70b-instruct -n 25
```
### API Usage
For direct API access, include the `X-Prime-Team-ID` header in your requests.
#### Option 1: Set as Default Header (Recommended)
Set the header once when creating the client:
```python Python theme={null}
import openai
import os
client = openai.OpenAI(
api_key=os.environ.get("PRIME_API_KEY"),
base_url="https://api.pinference.ai/api/v1",
default_headers={
"X-Prime-Team-ID": "your-team-id-here"
}
)
# All requests will now use team account
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Hello"}]
)
```
```bash cURL theme={null}
# Include header in every request
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $PRIME_API_KEY" \
-H "X-Prime-Team-ID: your-team-id-here" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
```javascript JavaScript theme={null}
const client = {
apiKey: process.env.PRIME_API_KEY,
baseUrl: 'https://api.pinference.ai/api/v1',
teamId: 'your-team-id-here'
};
const response = await fetch(`${client.baseUrl}/chat/completions`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${client.apiKey}`,
'X-Prime-Team-ID': client.teamId,
'Content-Type': 'application/json'
},
body: JSON.stringify({
model: 'meta-llama/llama-3.1-70b-instruct',
messages: [{ role: 'user', content: 'Hello' }]
})
});
```
#### Option 2: Set Per Request
For occasional team usage or switching between accounts:
```python Python theme={null}
# Create client without team ID
client = openai.OpenAI(
api_key=os.environ.get("PRIME_API_KEY"),
base_url="https://api.pinference.ai/api/v1"
)
# Add team ID to specific requests
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={
"X-Prime-Team-ID": "your-team-id-here"
}
)
```
# Troubleshooting
Source: https://docs.primeintellect.ai/inference/troubleshooting
Fix common Prime Inference errors
## Insufficient funds
Prime Inference returns an `insufficient_funds` error when the billing account selected for the request cannot pay for the model call.
The selected billing account is determined by:
1. The API key in `Authorization: Bearer $PRIME_API_KEY`
2. The optional team header `X-Prime-Team-ID`
If `X-Prime-Team-ID` is not included, the request bills your personal balance. If the header is included, the request bills that team's balance.
### Error example
```json theme={null}
{
"error": {
"message": "Insufficient balance (including overdraft). Please add funds to continue.",
"type": "insufficient_quota",
"code": "insufficient_funds"
}
}
```
### How to fix it
If you want to bill your personal account:
* Add funds in the [Billing Dashboard](https://app.primeintellect.ai/dashboard/billing)
* Make sure the API key belongs to the account with funds
If you want to bill a team:
* Add funds to the team billing balance
* Include `X-Prime-Team-ID` on every direct pinference or OpenAI-compatible API request
* Confirm the API key owner has access to the team
You can find your team ID with:
```bash theme={null}
prime teams list
```
Or from the [Team Profile page](https://app.primeintellect.ai/dashboard/team-profile).
`PRIME_API_KEY` and Prime CLI team config do not select a team balance for direct pinference calls. pinference bills a team only when the request contains `X-Prime-Team-ID`.
### Direct API examples
```python Python theme={null}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["PRIME_API_KEY"],
base_url="https://api.pinference.ai/api/v1",
default_headers={
"X-Prime-Team-ID": "your-team-id-here",
},
)
response = client.chat.completions.create(
model="openai/gpt-5.4-mini",
messages=[{"role": "user", "content": "Hello"}],
)
```
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $PRIME_API_KEY" \
-H "X-Prime-Team-ID: your-team-id-here" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-5.4-mini",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### Related pages
* [Using Team Accounts](/inference/team-accounts)
* [Hosted Evaluations](/tutorials-environments/hosted-evaluations)
* [Environment Evaluations](/tutorials-environments/evaluating)
# Advanced Usage
Source: https://docs.primeintellect.ai/inference/usage
Streaming, advanced parameters, and usage patterns
## Basic Chat Completion
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in simple terms."}
],
max_tokens=500,
temperature=0.7
)
```
## Streaming Responses
For real-time applications, use streaming to receive responses as they're generated:
```python Python theme={null}
stream = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "user", "content": "Write a short story about a robot."}
],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")
```
```bash cURL theme={null}
curl -X POST https://api.pinference.ai/api/v1/chat/completions \
-H "Authorization: Bearer $PRIME_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [{"role": "user", "content": "Tell me a story"}],
"stream": true
}'
```
```javascript JavaScript theme={null}
const response = await fetch('https://api.pinference.ai/api/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.PRIME_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
model: 'meta-llama/llama-3.1-70b-instruct',
messages: [{ role: 'user', content: 'Tell me a story' }],
stream: true
})
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
process.stdout.write(chunk);
}
```
## Usage Metadata
Include `"usage": {"include": true}` to get token counts and cost:
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Hello"}],
extra_body={"usage": {"include": True}}
)
```
Response includes:
```json theme={null}
{
"usage": {
"prompt_tokens": 10,
"completion_tokens": 25,
"total_tokens": 35,
"input_tokens": 10,
"output_tokens": 25,
"cost": 0.000123
}
}
```
## Advanced Parameters
Prime Inference supports all standard OpenAI API parameters:
```python theme={null}
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Your prompt here"}],
# Generation parameters
max_tokens=1000,
temperature=0.8,
top_p=0.9,
frequency_penalty=0.1,
presence_penalty=0.1,
# Advanced options
stream=False,
stop=["END", "\n\n"],
logprobs=True,
top_logprobs=3
)
```
## Next Steps
Using inference with team accounts
Complete API documentation
# Introduction
Source: https://docs.primeintellect.ai/introduction
Welcome to the Prime Intellect Documentation
## Lab
Train, evaluate, and deploy AI models with our frontier research infrastructure.
} href="/hosted-training/getting-started">
Run large-scale RL training without managing infrastructure
} href="/tutorials-environments/environments">
Create, share, and discover RL environments for training and evaluation
} href="/tutorials-environments/evaluating">
Run evaluations on your environments with hosted inference
} href="/sandboxes/overview">
Secure code execution environments for AI agents
} href="/inference/overview">
Access frontier models via our inference API
Step-by-step workflows for training, environments, and Lab
## Libraries
Open source tooling for environments, evaluations, and large-scale RL training.
Our library for environments and evals
Our large-scale async RL framework
## Compute
Access GPUs and infrastructure for AI workloads.
} href="/cli-reference/provision-gpu">
Deploy single GPU instances in under a minute
} href="/tutorials-multi-node-cluster/deploy-multi-node">
Scale to 64+ H100 clusters for distributed training
} href="/tutorials-storage/create-persistent-storage">
Persistent storage for datasets and checkpoints
} href="/tutorials-reserved-clusters/monitoring">
Dedicated GPU clusters with monitoring
## Resources
REST API documentation
Prime CLI commands
Common questions
# Advanced
Source: https://docs.primeintellect.ai/prime-rl/advanced
This page covers the specialized features layered on top of the core training stack: our custom model implementations (with EP for MoE families and CP for long-context training), multimodal training, LoRA training, multi-tenant training, and disaggregated prefill/decode inference. For developer-side workflows (adding new model architectures, debugging modeling code at small scale), see [Development](/prime-rl/development).
## Table of Contents
* [Custom Modeling](#custom-modeling)
* [Expert Parallelism Backends](#expert-parallelism-backends)
* [Multimodal Training](#multimodal-training)
* [Supported Families](#supported-families)
* [Enabling VLM Mode](#enabling-vlm-mode)
* [Limitations](#limitations)
* [LoRA Training](#lora-training)
* [Multi-Tenant Training](#multi-tenant-training)
* [Disaggregated Prefill/Decode Inference](#disaggregated-prefilldecode-inference)
## Custom Modeling
`prime-rl` ships custom optimized model implementations for several MoE families. With `model.impl = "auto"` (default) the trainer picks the custom path when the HF config type is registered, falling back to plain HF otherwise. To force one:
```toml theme={null}
[trainer.model]
impl = "custom" # or "hf" to force the HF path
```
| Family | HF config types | EP | CP |
| ------------------------------- | ------------------------------------------------------------------------------ | -------- | -- |
| GLM-5 / GLM-5.2 (`glm_moe_dsa`) | `zai-org/GLM-5`, `zai-org/GLM-5-FP8`, `zai-org/GLM-5.2`, `zai-org/GLM-5.2-FP8` | ✅ | ✅ |
| Qwen3 MoE | `Qwen/Qwen3-30B-A3B`, … | ✅ | ✅ |
| Qwen3.5 MoE | `Qwen/Qwen3.5-35B-A3B`, … | ✅ | ✅ |
| Qwen3 / Qwen3.5 VLMs | see [Multimodal training](#multimodal-training) | MoE only | ❌ |
| Laguna | `poolside/Laguna-XS.2` | ✅ | ✅ |
| MiniMax M2 | `MiniMax/MiniMax-M2` | ✅ | ✅ |
| Nemotron H | `nvidia/Nemotron-3-Nano-30B-A3B`, … | ✅ | ❌ |
| Trinity (AFMoE) | `arcee-ai/Trinity-Mini`, … | ✅ | ✅ |
| GLM-4 / GLM-4.5 / INTELLECT-3 | `THUDM/GLM-4-9B-0414`, `zai-org/GLM-4.5`, `PrimeIntellect/INTELLECT-3`, … | ✅ | ✅ |
| GPT-OSS (HF MoE) | `openai/gpt-oss-20b`, `openai/gpt-oss-120b` | ❌ | ✅ |
The custom path enables you to set EP, CP, selective activation checkpointing, low-precision training (`[trainer.model.quantization]`), and faster MoE kernels (`moe_use_grouped_mm = true`, default). Forcing `impl = "hf"` is mostly useful when debugging — it's slower and disables most MoE-specific knobs.
### Low-precision training
Set `[trainer.model.quantization]` to train dense linears and MoE expert GEMMs in low precision. Two backends are available via the `type` discriminator:
* `type = "fp8"` — DeepGEMM FP8 blockwise (requires SM90+ / Hopper). Options: `enable_grouped_gemm` (FP8 MoE expert GEMM). Both default on.
* `type = "mxfp8"` — torchao MXFP8 microscaling (requires SM100+ / Blackwell). Options: `enable_grouped_gemm`, `enable_a2a` (MXFP8 expert-parallel all-to-all), and `recipe` (`mxfp8_rceil` default or `mxfp8_rceil_wgrad_with_hp`).
```toml theme={null}
[trainer.model.quantization]
type = "mxfp8"
recipe = "mxfp8_rceil"
enable_a2a = true
```
GLM-5.2 adds IndexShare: the DSA sparse-attention indexer runs only on a subset of layers and the remaining layers reuse the cached top-k indices. The trainer reads this schedule from the model's `indexer_types` config field and enables the index cache automatically, so no extra config is needed. To override the schedule manually, set `[trainer.model.index_cache]` (`topk_freq` or `topk_pattern`).
### Expert Parallelism Backends
`model.ep_comm_backend` picks the all-to-all kernel used for EP dispatch/combine:
* **`torch`** (default): TorchTitan's all-to-all collective. Works everywhere, no extra install.
* **`deepep`**: Utilizes DeepEP's custom all-to-all collectives. This provides better performance if EP dimension spans multiple nodes. We provide pre-built binaries for H100/H200 with cuda runtime 12.9 installed, you can install them by running `uv sync --all-extras`.
DeepEP requires some careful tuning to achieve optimal performance, tuning parameters are `deepep_num_sms` and `deepep_token_chunk_size`.
With DeepEP, gradient clipping is currently not supported. (`optim.max_norm` is set to `None` automatically.)
## Multimodal Training
### Supported Families
The built-in VLM registry covers:
| Family | `model_type` | Vision attr | LM attr |
| ----------- | ------------- | -------------- | ---------------------- |
| Qwen3.5 | `qwen3_5` | `model.visual` | `model.language_model` |
| Qwen3.5-MoE | `qwen3_5_moe` | `model.visual` | `model.language_model` |
### Enabling VLM Mode
Add `[model.vlm]` and bfloat16 dtypes:
```toml theme={null}
[model]
name = "Qwen/Qwen3.5-4B"
impl = "custom"
optimization_dtype = "bfloat16"
reduce_dtype = "bfloat16"
[model.vlm]
vision_encoder_attr = "model.visual"
language_model_attr = "model.language_model"
# freeze_vision_encoder = true # default; set false to fine-tune the encoder
```
The weight-broadcast key prefix is derived as `{language_model_attr}.layers.` automatically.
VLM training requires a registered custom PrimeRL implementation.
### Limitations
* **Vision encoder frozen by default.** The default LoRA targets do not match Qwen3.5 vision modules. Set `freeze_vision_encoder = false` to fine-tune the encoder; this is incompatible with LoRA because LoRA freezes all non-adapter parameters.
* **bfloat16 mandatory.** The trainer config validator refuses any other `optimization_dtype` / `reduce_dtype` for VLMs — vLLM serves VLMs in bfloat16 and a mismatch breaks the importance ratio.
* **Higher KL mismatch with multi-image inputs.** Expect noisier `mismatch_kl` than text-only; this is from minor numerical differences between the trainer's and vLLM's image processing.
* **Images aren't logged to monitors.** Sample logging captures the prompt text but not the actual images.
## LoRA Training
LoRA is enabled by adding `[model.lora]`:
```toml theme={null}
[model.lora]
rank = 16
alpha = 32
dropout = 0.0
```
`target_modules` defaults to a reasonable cross-family set (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `experts`, plus a few latent-projection names for Nemotron). Unknown names are silently ignored, so the defaults work across architectures. Add architecture-specific names to extend coverage (e.g. `in_proj` / `out_proj` for Mamba).
LoRA is supported across SFT and RL. For RL, NCCL weight broadcast is **not** supported with LoRA — the default NCCL transport automatically falls back to filesystem when LoRA is enabled. To save the raw adapter alongside the merged HF weights:
```toml theme={null}
[ckpt.weights]
save_adapter_separately = true
```
LoRA pairs naturally with [multi-tenant training](#multi-tenant-training) — each tenant gets its own adapter and the backbone is shared across all of them in trainer memory.
## Multi-Tenant Training
Multi-tenant training lets a single trainer + inference deployment serve many concurrent LoRA "tenants" — each a fully isolated run with its own orchestrator, LoRA adapter, optimizer, scheduler, checkpoints, and progress tracking — sharing the same backbone weights and the same vLLM server. This is the topology behind hosted training on the [Prime Intellect platform (Lab)](https://app.primeintellect.ai). The trainer-side implementation is the `MultiRunManager` singleton, enabled by setting `trainer.max_concurrent_runs > 1`. For the full API surface, see [`src/prime_rl/trainer/runs.py`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/src/prime_rl/trainer/runs.py).
## Disaggregated Prefill/Decode Inference
For large MoE serving, splitting prefill and decode onto separate vLLM groups can substantially improve throughput. Pick the prefill:decode ratio based on workload shape:
| Workload | P:D ratio | Why |
| ------------------------ | --------- | ---------------------------------------------- |
| Agentic (SWE, Lean) | 3:1 | Long growing contexts → prefill-heavy |
| Non-agentic (math, chat) | 1:2 | Short prompts, long generations → decode-heavy |
Example config: [`examples/advanced/glm-5.2/swe.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/advanced/glm-5.2/swe.toml) — full RL run on `GLM-5` with P/D disaggregation behind a `vllm-router`, FP8 inference, and NCCL weight broadcast, paired with an inference config from [`examples/advanced/glm-5.2/infer/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.2/infer).
Monitor live queue depths to detect imbalance:
```bash theme={null}
curl -s http://:8100/metrics | grep num_requests_waiting
curl -s http://:8200/metrics | grep num_requests_waiting
```
If prefill queues and decode is idle, add prefill nodes (and vice versa).
**Required setup for disaggregated P/D (NIXL/UCX).** The pip-wheel NIXL's bundled UCX segfaults on the prefill→decode KV transfer (`signal 11: invalid permissions for mapped object` in `libucs.so`) — reproduced on vLLM 0.22 and 0.23, with/without mooncake, with/without llm-d. Building NIXL against UCX 1.19.x from source is therefore **required** (not optional) for disaggregated P/D.
```bash theme={null}
salloc -N 1 --gres=gpu:1 bash -c 'bash scripts/install_nixl_from_source.sh'
uv pip install --reinstall --no-deps deps/nixl_cu12-*.whl
```
The script writes UCX 1.19 to `third_party/ucx/`; the bundled sbatch templates prepend it to `LD_LIBRARY_PATH` so it overrides the system version. Re-run both commands after every `uv sync`, since the lock pins the wheel.
# Algorithms
Source: https://docs.primeintellect.ai/prime-rl/algorithms
This page covers the math and the configurable algorithmic components: the algorithm abstraction and its algorithms, how off-policy training works, the loss components and advantage functions, how to plug in your own, the filters applied between rollout and training, and how multi-turn rollouts get merged into training samples.
## Table of Contents
* [The Algorithm Abstraction](#the-algorithm-abstraction)
* [Model References](#model-references)
* [The Algorithms](#the-algorithms)
* [Customizing Components](#customizing-components)
* [Per-Env Algorithms](#per-env-algorithms)
* [The Algorithm Classes](#the-algorithm-classes)
* [Async / Off-Policy Training](#async--off-policy-training)
* [Loss](#loss)
* [Loss Components](#loss-components)
* [Default RL Loss](#default-rl-loss)
* [Custom Loss](#custom-loss)
* [Advantage](#advantage)
* [Default Advantage](#default-advantage)
* [Hierarchical GRPO](#hierarchical-grpo)
* [Self-Play Advantage (RAE)](#self-play-advantage-rae)
* [Authoring an Algorithm](#authoring-an-algorithm)
* [Reference Scoring](#reference-scoring)
* [Filters](#filters)
* [Multi-Turn Trajectories](#multi-turn-trajectories)
* [Extension Property](#extension-property)
* [Best-Effort Interleaving](#best-effort-interleaving)
* [Renderers](#renderers)
* [Discontinuous Trajectories](#discontinuous-trajectories)
## The Algorithm Abstraction
A training algorithm in `prime-rl` is configured under `[orchestrator.algo]`, where **`type` names the algorithm** (`grpo`, `opd`, `sft`, …) and the class defaults are its vetted setting. It has two parts:
1. **Sampling** (`algo.sampling`) — how train rollouts are produced: which model generates them. `source` is a [model reference](#model-references): `"policy"` (the live policy, the default) or an inline frozen hosted model. Group sizing stays on the env config (`group_size`).
2. **The per-token training signal** — credit assignment and loss routing, fused; the algorithm's own parameters sit directly on `algo`. One mapping from a finalized rollout to per-token *(loss component, weight)* pairs — the credit a token gets and the loss that consumes it are two coordinates of the same output. Group-relative algorithms compute credit on the orchestrator and ship per-token advantage streams; reference-KL algorithms query a reference model at batch-ship time (bounded concurrency) and ship its prefill logprobs for the trainer to evaluate against the live policy. The `type` determines which loss component consumes the action tokens (`rl` / `ce` / `ref_kl`) and what happens to env-provided observation tokens in multi-turn rollouts (masked out by default; `echo` trains on them with weighted CE).
The trainer is algorithm-blind: the loss is a sum of three components (rl, ce, ref\_kl), each normalized by its own global token count; per-token streams ship on the wire (the `rl_weights` / `ce_weights` / `ref_kl_weights` component weights plus the `advantages` stream on each training sample) and the trainer just executes them. Adding an algorithm never touches the dispatcher, packer, or trainer hot path.
### Model References
`prime-rl` hosts exactly one model: the trainable policy (`[orchestrator.model]`). Every other model an algorithm uses is an external OpenAI-compatible endpoint, declared *inline on the component that uses it*. A model reference is either the string `"policy"` (the live policy) or a frozen hosted model (`name` + `base_url`):
```toml theme={null}
[orchestrator.algo]
type = "opd"
[orchestrator.algo.teacher] # opd's teacher: the frozen model it scores against
name = "Qwen/Qwen3-32B"
base_url = ["http://localhost:8001/v1"]
```
Model *roles* are algorithm-local vocabulary — each algorithm names its reference on the field where the model is actually used, and there is no shared `teacher` slot. `opd` declares a `teacher` field (the frozen model whose reverse KL the policy distills toward); `sft`'s teacher *is* its `sampling.source` (the frozen model it imitates); `opsd` self-distills against the live policy and names no model at all. No role exists outside the algorithm that declares it: the dispatcher, sink, and trainer branch on liveness alone, never on what an algorithm calls a model.
So for `opd` set `[orchestrator.algo.teacher]`; for `sft` set `[orchestrator.algo.sampling.source]`; `opsd` needs neither. `opd`'s teacher must be a frozen endpoint — it is typed `FrozenModelConfig`, so `"policy"` isn't representable (the KL would be identically zero); `opsd`'s teacher *is* the live policy by definition (self-distillation conditioned on a demonstration), so it exposes no reference to configure.
Liveness is a property of the reference, not of any role: rollouts sampled from `"policy"` get version-salted prefix caches, carry sampling logprobs for importance ratios, and age off-policy as weights update; rollouts and scores from frozen models get a stable prefix cache and never go stale. Frozen models are externally hosted (`base_url` is required) — `prime-rl` never launches or updates them, and each env's algorithm builds its own client pool to the endpoints it declares.
### The Algorithms
The `algo.type` names the algorithm, and each type's class defaults are its vetted setting — picking a type with no other keys IS the algorithm:
```toml theme={null}
[orchestrator.algo]
type = "grpo" # the default
```
| `type` | Sampling | Loss | What it is |
| ------------------- | --------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `grpo` | policy | `rl` on actions | Standard group-relative RL. |
| `max_rl` | policy | `rl` on actions | MaxRL ([arXiv:2602.02710](https://arxiv.org/abs/2602.02710)): GRPO's centered reward normalized by the group **mean** instead of the standard deviation — the gradient is unbiased for the order-`group_size` truncation of the maximum-likelihood objective, upweighting hard examples like `1/p`. |
| `rae` | policy | `rl` on actions | RAE (SPIRAL, [arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): reward minus a per-agent EMA baseline of that agent's own rewards — the estimator for multi-agent self-play envs, where the group mean would mix the agents' opposite reward scales. See [Self-Play Advantage](#self-play-advantage-rae). |
| `hierarchical_grpo` | policy | `rl` on actions | GRPO for proposer-solver envs. Solvers are compared only with attempts on the same proposed problem; proposers are compared with the other proposals in the group. See [Hierarchical GRPO](#hierarchical-grpo). |
| `opd` | policy | `ref_kl` on actions | On-policy distillation ([Thinking Machines](https://thinkingmachines.ai/blog/on-policy-distillation/)): the policy samples, per-token reverse KL against a reference model as the gradient signal. Needs a `teacher`. |
| `sft` | *(the teacher)* | `ce` on actions | Hard distillation: a frozen model generates rollouts, the policy trains with CE on its tokens. Needs a frozen `sampling.source` (the teacher it samples from). |
| `opsd` | policy | `ref_kl` on actions | SDFT ([arXiv:2601.19897](https://arxiv.org/abs/2601.19897)): the model is its own reference, conditioned on an expert demonstration. The teacher *is* the live policy (the paper's setting, no extra deployment) — no model to configure. |
| `echo` | policy | `rl` on actions + weighted `ce` on observations | ECHO: standard GRPO plus a cross-entropy loss on env-provided tokens already present in the rollout, selected by message role (needs the renderer's role attribution). Defaults to tool-response bodies at `alpha = 0.1` (ECHO's λ); set `roles` to train other roles, each at its own weight. |
### Customizing Components
Every key beyond `type` is visibly your own assembly — there is no preset layer to diverge from. The vetted setting is the class defaults; what you set is what runs:
```toml theme={null}
# echo on tool AND user feedback tokens, each at its own weight.
# Setting any role replaces the whole table.
[orchestrator.algo]
type = "echo"
[orchestrator.algo.roles.tool]
alpha = 0.25
[orchestrator.algo.roles.user]
alpha = 0.05
```
A new algorithm is a named class in code, not a config that points at an import path — see [Authoring an Algorithm](#authoring-an-algorithm).
Echo also takes an optional user-supplied token filter that narrows the role selection per rollout — e.g. dropping warning lines from tool output, or tokens the sampler found unlikely:
```toml theme={null}
[orchestrator.algo.filter]
import_path = "my_module.drop_warnings"
[orchestrator.algo.filter.kwargs]
patterns = ["WARNING"]
```
```python theme={null}
# my_module.py — sees the raw rollout (message text, sampling logprobs);
# returns one keep-mask per trainable branch, spanning that branch's
# token_ids. False = never echo-trained.
def drop_warnings(rollout, *, patterns: list[str]) -> list[list[bool]]: ...
```
Component compatibility is validated at config time: frozen-model sampling can only feed the `ce` loss component — the `rl` and `ref_kl` components need the live policy's own sampling logprobs for importance ratios — `opd` pointed at `"policy"` is rejected as degenerate (zero KL), `sft` without a frozen source is rejected (CE on the policy's own tokens is not a distillation target). A group-relative algorithm with `group_size = 1` produces all-zero advantages; the resulting empty batch is caught at runtime (the orchestrator warns and aborts after repeated zero-trainable batches), not at config time.
### Per-Env Algorithms
Both components resolve per environment. Each env inherits `[orchestrator.algo]` unless it sets its own, so a single run can mix algorithms across envs — e.g. GRPO on math, ECHO on a terminal env:
```toml theme={null}
[orchestrator.algo]
type = "grpo"
[[orchestrator.train.source]]
name = "math" # inherits the top-level grpo
[orchestrator.train.source.env.taskset]
id = "math-v1"
[orchestrator.train.source.env.agent.harness]
id = "null"
[orchestrator.train.source.env.agent.runtime]
type = "subprocess"
[[orchestrator.train.source]]
name = "terminal"
[orchestrator.train.source.env.taskset]
id = "terminal-v1"
[orchestrator.train.source.env.agent.harness]
id = "bash"
[orchestrator.train.source.env.agent.runtime]
type = "subprocess"
# this env runs its own algorithm
[orchestrator.train.source.algo]
type = "echo"
```
### The Algorithm Classes
At runtime, each env's resolved config builds two objects: a `Sampler` (`prime_rl.orchestrator.sampler`) from the `sampling` component — the pool rollouts are generated from, and the home of future sampling strategies like replay buffers or branching — and one of the named algorithm classes in `prime_rl.orchestrator.algo` (one module per algorithm: `algo/grpo.py`, `algo/opd.py`, …) from the algorithm config. Algorithm dispatch is keyed on `algo.type` — it names the algorithm, and each config class's defaults are its vetted parameterization:
| `algo.type` | Class | hook(s) — stage |
| ------------------- | --------------------------- | ------------------------------------------------------------------------------------------------ |
| `grpo` | `GRPOAlgorithm` | `score_group`: group-norm credit (optional length penalty) |
| `echo` | `EchoAlgorithm` | `score_rollout`: weighted ce on observation tokens; `score_group`: group-norm credit (inherited) |
| `max_rl` | `MaxRLAlgorithm` | `score_group`: mean-normalized group credit |
| `rae` | `RAEAlgorithm` | `score_group`: per-agent EMA-baseline credit |
| `hierarchical_grpo` | `HierarchicalGRPOAlgorithm` | `score_group`: GRPO baseline per episode for solvers, per group for the proposer |
| `opd` | `OPDAlgorithm` | `score_rollout`: own-context prefill under the teacher |
| `opsd` | `OPSDAlgorithm` | `score_rollout`: demo-conditioned prefill under the live policy |
| `sft` | `SFTDistillAlgorithm` | `score_group`: group-norm credit (feeds filters) |
Each class owns its hooks outright — reading one top to bottom reads the algorithm, and everything on the class is an override point. The two hooks are one scope-and-timing ladder — the wider scope is unlocked by a later barrier, so the two axes coincide. Each is handed the `Rollout` directly — the env's typed trace (`reward`, `nodes`, `num_turns`, ...) with `samples` attached, plus `assign_advantages` to write credit:
* `async score_rollout(rollout)` — one rollout, **on arrival** (as it's tokenized, before its group is complete): rollout-local credit (`rollout.assign_advantages(...)`, scalar broadcast or per-token), observation ce weights, **or** model I/O — query a reference pool (e.g. `self.teacher_pool`, connected in `setup()` via `self.connect(...)`, or the live `self.policy_pool` for opsd) and attach per-token results (e.g. teacher logprobs) with bounded concurrency. No siblings. `echo` weights observation tokens here, identifying env-provided observation nodes by their non-sampled status and source step role attribution, applying the optional user filter, and writing the `ce_weights` stream. Model I/O runs *before* the pre-batch filters, so it pays compute on rollouts that may then be filtered out.
* `score_group(group)` — the cohort, **before filtering** (filters read the streams), synchronous: group-relative credit (GRPO/MaxRL baselines). `group` is a list of `Rollout`.
The pipeline drives the hooks through two non-virtual methods it never looks inside: `algorithm.finalize_rollout(rollout)` per arrival (rollout-local scoring + reference I/O) and `algorithm.finalize_group(rollouts)` per group (scoring + wire stamping; after this the records are frozen — groups die at stamping). Sample construction (interleaving) is pure pipeline — observation-token provenance is available through structural attribution (`node.sampled`, `node.is_content`) for any algorithm that trains on env-provided tokens.
Class-level declarations state what the algorithm needs: which loss component its action tokens feed (`action_loss_type`). Every class is constructed with its algorithm config plus the one host-owned resource it can't rebuild — the live policy pool (`self.policy_pool`). Everything else an algorithm needs it builds from its own config in `setup()`: `opd` connects its frozen `teacher`; `opsd` builds the renderer for its demonstration hint (tokenizer is always the live policy's — self-distillation has no separate model). The pipeline only ever calls the two `finalize_*` methods — writing your own algorithm is subclassing `Algorithm` and overriding the hooks its signal needs (see [Authoring an Algorithm](#authoring-an-algorithm)). Shared math (efficiency shaping, prefill alignment) lives as plain functions in `prime_rl.orchestrator.algo.advantage`.
## Async / Off-Policy Training
`prime-rl` is asynchronous by default. The trainer and inference always run one step overlapped: while the trainer is producing $\pi_n$ from rollouts at step $n$, inference is already generating the rollouts for step $n+1$ using $\pi_{n-1}$. With matched trainer and inference step times this produces fully-overlapped pipeline parallelism — neither side ever idles.
At step $n = 1, 2, 3, \dots$:
* **Trainer** produces policy $\pi_n$ with weights $\theta_n$ from rollouts $(x_n, y_n)$.
* **Inference** produces rollouts $(x_n, y_n)$ from policy $\pi_{\max(0,\,n-1)}$.
Step indices are 1-indexed; policy versions are 0-indexed, with $\pi_0$ the base model. At step 1 inference samples from $\pi_0$.
## Loss
### Loss Components
The training loss is a **sum of three components**, each with its own per-token weight stream and its own normalization:
$$
\mathcal{L} = \frac{\sum \mathcal{L}_{rl}}{N_{rl}} + \frac{\sum \mathcal{L}_{ce}}{N_{ce}} + \frac{\sum \mathcal{L}_{ref\_kl}}{N_{ref\_kl}}
$$
* `rl` — the configured RL loss (`[trainer.loss]`): DPPO + KL by default, or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, `hierarchical_grpo`, and `echo`'s action tokens).
* `ce` — masked NLL. Used for frozen-model tokens (`sft`) and env-observation tokens (`echo`).
* `ref_kl` — the per-token reverse KL to a reference model ($\log \pi_{\text{ref}} - \log \pi$) as the policy-gradient signal, importance-ratio corrected with a one-sided trust region (`opd`, `opsd`). Requires `ref_logprobs` from a [reference scoring](#reference-scoring); the scoring model must be a vLLM server (it's the only one that exposes `prompt_logprobs`).
The orchestrator stamps each sample's component membership as per-token weight streams (`rl_weights` / `ce_weights` / `ref_kl_weights` on the wire): a weight scales that component's per-token loss, `0.0` leaves the token out of the component entirely (mask *and* denominator), and components may overlap on the same token — their gradients sum. Each $N$ is the global (all-reduced) count of that component's member tokens, so the components don't dilute each other: adding echo observation tokens never changes the rl term's effective per-token learning rate, and an sft env packed next to a GRPO env doesn't soften its gradient. Tokens of different components pack freely into the same micro batch, and a plain GRPO run ships no weight streams at all (absent streams mean rl weight 1.0 on every trainable token — the unchanged hot path). Advantages always ship per token (`advantages` on the wire), assigned as per-token streams from the start — uniform group credit is broadcast over completion tokens at assignment; algorithms with no rl credit (opd, opsd) ship none.
### Default RL Loss
The default RL loss is a DPPO policy-gradient term combined with a KL regularizer similar to Kimi-K2.5. For each prompt $x_j$ we sample a group of $G$ rollouts $\{y_i\}_{i=1}^G$, score them to get $s_i$, then optimize:
$$
\mathcal{L}(\theta) = -\,\mathcal{J}_{\text{PG}}(\theta) \;+\; \tau_{KL}\,\mathcal{L}_{KL}(\theta)
$$
where the policy-gradient term is
$$
\mathcal{J}_{\text{PG}}(\theta)
= \frac{1}{\sum_{j,i} |y_i^{(j)}|}
\sum_{j,i,t}
\min\!\left(\frac{\pi(y_{i,t}^{(j)}\mid x_j, y_{i, LossOutputs:
ratio = torch.exp(inputs.trainer_logprobs - inputs.inference_logprobs)
clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps)
surr1 = ratio * inputs.advantages
surr2 = clipped * inputs.advantages
loss = -torch.min(surr1, surr2)[inputs.loss_mask].sum()
return LossOutputs(
loss=loss,
metrics={
"clip_frac": (ratio != clipped)[inputs.loss_mask].float().mean(),
},
)
```
Wire it up:
```toml theme={null}
[trainer.loss]
type = "custom"
import_path = "my_module.ppo_clip_loss"
[trainer.loss.kwargs]
clip_eps = 0.2
```
The dataclasses:
```python theme={null}
@dataclass
class LossInputs:
trainer_logprobs: Float[Tensor, "seq"] # current policy
inference_logprobs: Float[Tensor, "seq"] # rollout-time policy
ref_logprobs: Float[Tensor, "seq"] | None # set by reference-scoring algorithms
advantages: Float[Tensor, "seq"]
loss_mask: Bool[Tensor, "seq"] # this component's member tokens
loss_weights: Float[Tensor, "seq"] | None # the component's weight stream (None = 1.0)
@dataclass
class LossOutputs:
loss: Float[Tensor, ""]
metrics: dict[str, Tensor]
```
Anything you put in `metrics` is averaged across sequences and logged with the other trainer metrics.
## Advantage
The per-token training signal is set by `algo.type` and the [algorithm](#the-algorithm-abstraction)'s parameters — every signal is a per-token advantage stream, varying in evaluation site (orchestrator vs. trainer). The `algo.type` values:
| Type | Component | Effect |
| ------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `grpo` | `rl` | Group-norm: reward minus per-group baseline, optional length penalty. |
| `max_rl` | `rl` | Mean-normalized group credit (maximum-likelihood RL). |
| `rae` | `rl` | Reward minus a per-agent EMA baseline (SPIRAL's role-conditioned advantage estimation) — for multi-agent self-play envs. |
| `hierarchical_grpo` | `rl` | GRPO for proposer-solver envs: solvers are compared within one proposed problem, while proposers are compared across proposals. |
| `echo` | `rl` + `ce` | Group-norm on action tokens, plus weighted CE on env-provided tokens selected by message role (each role's `alpha` is its ECHO λ), optionally narrowed by a user filter. |
| `opd` | `ref_kl` | On-policy distillation: per-token reverse KL to a reference model (`teacher`, an inline frozen hosted model), evaluated in the trainer from shipped reference logprobs. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream; `group_size` only fans out sampling. |
| `opsd` | `ref_kl` | SDFT: per-token reverse KL to a demo-conditioned reference. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream. |
| `sft` | `ce` | Cross-entropy on the sampled tokens. Assigns no advantage — trains on every sampled token. |
### Default Advantage
The default advantage is per-group reward minus per-group baseline (DR-GRPO without std normalization). For each prompt's group of `group_size` rollouts, every token in rollout $i$ receives advantage $s_i - \bar{s}$ where $\bar{s}$ is the group mean.
This is intentionally simple — it does the right thing for most envs. Write a named algorithm class when you need group-aware shaping that depends on trajectory metadata (sub-agent rollouts, relative-rank shaping, …) — see [Authoring an Algorithm](#authoring-an-algorithm).
A **length penalty** (`length_penalty` on the `grpo`-family algorithms) can be layered on top to discourage rambling. The `linear` penalty subtracts a single `pass_rate`-scaled penalty from each reward before the GRPO baseline, combining output tokens (`num_output_tokens_weight`), input / context tokens (`num_input_tokens_weight`), and turns (`num_turns_weight`) — each normalized by the group's own max for that quantity, with `num_input_tokens_weight` and `num_turns_weight` defaulting to `0.1`.
```toml theme={null}
[orchestrator.algo]
type = "grpo"
[orchestrator.algo.length_penalty]
type = "linear"
```
### Hierarchical GRPO
GRPO gives each rollout its reward minus the average reward of comparable rollouts. In an ordinary single-agent group, every rollout answers the same task, so one group average is enough.
A proposer-solver env is different. Starting from one source task, it produces several proposed problems, then runs several solver attempts on each problem:
```text theme={null}
one source task
├── proposed problem A
│ ├── proposer trace
│ ├── solver attempt 1
│ └── solver attempt 2
└── proposed problem B
├── proposer trace
├── solver attempt 1
└── solver attempt 2
```
The solver attempts for A should not be compared with the solver attempts for B: the two problems may have very different difficulty. Proposer and solver rewards should not be compared either: they measure different jobs.
`hierarchical_grpo` therefore chooses the average separately for each role:
| Trace | Compared with | Why |
| -------- | -------------------------------------------------- | ----------------------------------------------------------------- |
| Solver | Other solver attempts on the same proposed problem | They attempted the same problem. |
| Proposer | Other proposer traces in the group | They started from the same source task and proposed alternatives. |
For example, if three solvers receive rewards `[1, 1, 0]` on one proposed problem, their average is `2/3` and their advantages are `[1/3, 1/3, -2/3]`. Solver rewards from other proposed problems do not affect those values. The proposers are scored separately according to how useful their problems were for the solvers, then compared with the other proposers in the group.
Configure which roles are compared within a single proposed problem with `episode_agents`. For `proposer-solver-v1`, that role is `solver`:
```toml theme={null}
[orchestrator.algo]
type = "hierarchical_grpo"
episode_agents = ["solver"]
[[orchestrator.train.source]]
name = "proposer-solver"
group_size = 4 # proposed problems per source task
env.n = 4 # solver attempts per proposed problem
[orchestrator.train.source.env.taskset]
id = "proposer-solver-v1"
[orchestrator.train.source.env.proposer.harness]
id = "null"
[orchestrator.train.source.env.proposer.runtime]
type = "subprocess"
[orchestrator.train.source.env.solver.harness]
id = "null"
[orchestrator.train.source.env.solver.runtime]
type = "subprocess"
```
`group_size` controls how many problems are proposed from each source task. `env.n` controls how many solvers attempt each proposed problem. If a comparison contains only one trace—for example, a solver when `env.n = 1`—its advantage is zero and the zero-advantage filter removes it.
This algorithm is accepted only for proposer-solver envs. Use the env's `train_proposer` and `train_solver` settings if you want to train only one role.
### Self-Play Advantage (RAE)
Group-relative baselines assume the group is exchangeable attempts by one agent. A multi-agent self-play env breaks that: one episode yields one trace per agent, all trainable, and in a zero-sum game the rewards sum to \~0 whatever the policy does — the group mean carries no information, and centering against it converts any structural asymmetry (a first-mover edge) into permanent credit for one agent.
`rae` implements SPIRAL's role-conditioned advantage estimation ([arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): each agent keeps an exponential-moving-average baseline of its own rewards, and every trace's advantage is its reward minus its agent's baseline — measured against the *pre-update* baseline (the unbiased order), then folded in at `decay` (SPIRAL's α, default 0.95). The algorithm instance is per-env, so baselines are keyed per (env, agent) — the paper's per (game, role). Advantages are not normalized, and `group_size` is free (RAE needs no sibling rollouts; `group_size = 1` is fine). Baselines live in orchestrator memory and re-warm from 0 over \~`1/(1 − decay)` traces per agent after a restart.
```toml theme={null}
[orchestrator.algo]
type = "rae"
decay = 0.95
[[orchestrator.train.source]]
name = "kuhn-poker"
[orchestrator.train.source.env.taskset]
id = "kuhn-poker-v1"
[orchestrator.train.source.env.player0.harness]
id = "null"
[orchestrator.train.source.env.player0.runtime]
type = "subprocess"
[orchestrator.train.source.env.player1.harness]
id = "null"
[orchestrator.train.source.env.player1.runtime]
type = "subprocess"
```
Both of `kuhn-poker-v1`'s agents late-bind to the run's own model — shared-policy self-play against a continuously improving opponent. Pin one agent to a frozen endpoint (`env.player1.model = ...`) for asymmetric play; its traces are marked untrainable by the env and never reach the advantage computation. A single-agent env under `rae` degrades to REINFORCE with an EMA baseline.
### Authoring an Algorithm
There is no config hook that points at user code — a new credit-assignment scheme is a new named algorithm in the repo. Subclass `Algorithm`, assign credit in the scoring hook whose timing fits your signal, and register the class. The hook receives the group's `Rollout`s (each the env's typed `verifiers.Trace` — turns, tool calls, metadata in `info` — with `samples` attached) and writes credit via `assign_advantages`:
```python theme={null}
# src/prime_rl/orchestrator/algo/my_algo.py
import torch
from prime_rl.orchestrator.algo.base import Algorithm
class MyAlgorithm(Algorithm):
async def score_group(self, group):
rewards = torch.tensor([rollout.reward for rollout in group], dtype=torch.float32)
advantages = ... # one value per rollout
for rollout, advantage in zip(group, advantages.tolist(), strict=True):
rollout.assign_advantages(advantage)
```
Add a typed `MyAlgoConfig` to `prime_rl.configs.algorithm` and its discriminated union, then register `"my_algo": MyAlgorithm` in `ALGORITHM_CLASSES`. Pick the hook by *when* your signal is ready: `score_rollout` for per-arrival credit or credit that needs a model call (it's `async`), `score_group` for group-relative credit (GRPO/MaxRL). `assign_advantages` takes a scalar (broadcast over the rollout's trainable tokens — the common case) or a full-length per-token list aligned to the concatenated sample token\_ids (process rewards, step-level credit; `0.0` off-mask).
Each per-token list must match the rollout's completion-token count exactly — validated loudly when the view writes it. Advantage-based filters and metrics derive from the streams (the zero-advantage filter checks for all-zero streams; logged distributions use per-rollout means). Signals that depend on the live policy's weights (like OPD's reverse KL) cannot be precomputed here; those are reference-scoring algorithms, evaluated in the trainer.
### Reference Scoring
`OPDAlgorithm` / `OPSDAlgorithm` do their model I/O in `score_rollout`: as each rollout arrives they query a reference (the sample's own context for `opd`, the demo-conditioned context for `opsd`) and attach per-token reference logprobs to each sample. Rollouts are consumed serially by the orchestrator's main loop and each carries only a handful of samples, so the in-flight request count is naturally bounded — no explicit concurrency cap:
* `opd` — score each sample's own context under the `teacher` (a frozen [model reference](#model-references)) via prefill; fills `ref_logprobs` for the `ref_kl` loss component (on-policy distillation). The `teacher` is typed `FrozenModelConfig`, so `"policy"` isn't representable (the KL would be identically zero).
* `opsd` — SDFT: prepend an expert demonstration as a leading system message (`template`, with a `{demonstration}` placeholder) and score the sample under that demo-conditioned context. The sample is scored verbatim (`hint_block + token_ids`, slicing the hint's logprobs back off), so the join is BPE-clean and it's robust to tool/multimodal prompts and any number of turns. The scoring reference *is* the live policy — self-distillation names no teacher. opsd builds its own renderer to tokenize the hint block: the tokenizer is always the live policy's (not configurable — there is no separate model), and only the `renderer` family is settable (defaults to `"auto"`, resolved from the policy tokenizer; set it to match a non-auto policy renderer). The demonstration is read from the example's `info[demo_key]`, falling back to a top-level rollout field of the same name (e.g. `answer`).
```toml theme={null}
[orchestrator.algo]
type = "opsd"
demo_key = "demonstration"
```
Scoring runs at arrival, *before* the pre-batch filters, so a rollout that is later filtered still cost its reference compute — accepted for the simpler one-rollout-at-a-time shape (advantage-based filters never fire for opd/opsd anyway, since neither assigns an advantage).
## Filters
Filters drop rollouts between scoring and training. Built-ins (composable):
| Filter | Effect |
| ---------------- | ------------------------------------------------------------------------------------------------ |
| `gibberish` | Drops rollouts whose mean log-prob fall below a threshold — usually a sign of degenerate output. |
| `repetition` | Drops rollouts with high n-gram repetition. |
| `zero_advantage` | Drops rollouts whose advantage is zero, so the trainer doesn't waste tokens on them. |
The default `[orchestrator]` config registers all three in both filter slots: `post_batch_filters` enforce by default (flagged rollouts are recorded but not shipped to the trainer), while `pre_batch_filters` run in monitor mode (`enforce = false`); flip `enforce = true` there to drop matching rollouts before they consume a slot in the batch. Setting a slot replaces its defaults wholesale:
```toml theme={null}
[[orchestrator.post_batch_filters]]
type = "zero_advantage"
[[orchestrator.post_batch_filters]]
type = "repetition"
threshold = 0.4
```
Filtered rollouts still appear in W\&B distributions, just not in the trainer batch — useful for spotting whether filtering is doing its job.
## Multi-Turn Trajectories
Multi-turn rollouts (tool use, browser environments, long conversations) used to be stitched into a single fake "single-turn" sample, which silently corrupted the importance ratio when chat templates didn't roundtrip. Since [`verifiers` v0.1.8](https://github.com/PrimeIntellect-ai/verifiers/releases/tag/v0.1.8), `prime-rl` records each LLM request/response as an independent **trajectory step** and merges them at training time using best-effort interleaving — with [renderers](#renderers) as the mechanism that keeps the merge safe by construction.
### Extension Property
A sequence of trajectory steps has the **extension property** when each successive step's prompt contains all previous prompts and completions as an exact prefix. The trainer relies on this property — when it holds:
* Multiple steps merge into one training sample.
* Compute scales as $O(T)$ in the trajectory length.
When it breaks (chat template strips past thinking, environment compacts context, an agent hands off to a sub-agent, etc.), the trainer starts a new training sample from that step:
* Graceful fallback to multiple samples — no corrupted data.
* Worst case (every step breaks extension) is $O(T^2)$.
### Best-Effort Interleaving
Concretely:
```
5-step trajectory where extension breaks at step 4:
steps 1–3: extension holds → merged into Sample 1
step 4: extension breaks (e.g. thinking stripped from history)
steps 4–5: extension holds → merged into Sample 2
result: 2 training samples instead of 5
```
The orchestrator enforces an **exact prefix invariant**: the prompt at turn $t$ must be the exact concatenation of prior messages exactly as the LLM originally generated them. If turn 2's prompt is `U1, A1', U2` while `A1' ≠ A1`, the orchestrator can't safely merge — either choice produces logprob drift between trainer and inference. Starting a fresh sample is the only correct behavior, so that's what happens.
### Renderers
Best-effort interleaving works because the renderer guarantees the exact-prefix invariant *by construction* — it never re-renders prior turns, so it can't lose tokens to chat-template normalization, BPE retokenization drift, or thinking stripping. A renderer turns a model's chat template into a Python object that can:
* `render_ids(messages)` — tokenize messages to ids the inference engine accepts.
* `parse_response(completion_ids)` — recover structured `(content, reasoning_content, tool_calls)` from sampled ids.
* `bridge_to_next_turn(prev_prompt_ids, prev_completion_ids, new_messages)` — extend the previous turn's tokens verbatim with the new environment turn, instead of re-rendering history.
When `bridge_to_next_turn` succeeds, the trainer sees the exact token stream the sampler produced; when it can't be proven safe (e.g. the renderer is `DefaultRenderer` and the template's stop sequence is unknown), it returns `None` and the orchestrator falls back to a full re-render — which triggers the new-sample fallback above.
A common source of breakage in the absence of a hand-coded renderer is models like Qwen3 whose chat templates strip past `` blocks across user turns:
```python theme={null}
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
messages = [
{"role": "user", "content": "U1"},
{"role": "assistant", "content": "R1A1"},
{"role": "user", "content": "U2"},
]
tok.apply_chat_template(messages[:1], tokenize=False)
# <|im_start|>user
# U1<|im_end|>
tok.apply_chat_template(messages, tokenize=False)
# <|im_start|>user\nU1<|im_end|>\n<|im_start|>assistant\nA1<|im_end|>\n<|im_start|>user\nU2<|im_end|>
# (the R1 from turn 2 is gone)
```
Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `glm-5`, `glm-4.5`, `minimax-m2`, `deepseek-v3`, `kimi-k2`, `kimi-k2.5`, `nemotron-3`, `gpt-oss`; anything else falls back to `DefaultRenderer` (a generic `apply_chat_template` wrapper). Pick one via:
```toml theme={null}
[orchestrator.renderer]
name = "auto" # detect from tokenizer; pass an explicit name for fine-tunes
```
For the full design rationale (failure modes ruled out, empirical token-identity comparison against `apply_chat_template`, when to write a hand-coded renderer), see [the renderers writeup on the Prime Intellect blog](https://www.primeintellect.ai/blog/renderers) — the canonical reference.
### Discontinuous Trajectories
Some envs are discontinuous by design — e.g. a main agent delegating to a sub-agent and getting back only a summarized result, not the sub-agent's whole conversation. Best-effort interleaving handles this naturally: each agent's contiguous turns merge, the handoff starts a new sample. The trainer never sees fabricated extension where there is none.
# Configuration
Source: https://docs.primeintellect.ai/prime-rl/configuration
Every `prime-rl` entrypoint uses [`pydantic-config`](https://github.com/PrimeIntellect-ai/pydantic-config): TOML files for reproducible base configs, CLI flags for one-off overrides.
> **AI agents working in this repo:** the equivalent runbook is at [`skills/configs/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/configs/SKILL.md), with extra runtime hints (where config classes live, validator conventions, the trainer-side `enable_token_export` flag) that aren't surfaced here.
## Table of Contents
* [Sources and Precedence](#sources-and-precedence)
* [TOML Composition](#toml-composition)
* [CLI Overrides](#cli-overrides)
* [Inspecting and Validating](#inspecting-and-validating)
* [Syntax](#syntax)
* [Booleans](#booleans)
* [Lists](#lists)
* [Dicts](#dicts)
* [Optional Sub-Configs](#optional-sub-configs)
* [None](#none)
* [Discriminated Unions](#discriminated-unions)
* [Environments](#environments)
* [Environment Variables](#environment-variables)
* [Examples](#examples)
## Sources and Precedence
Field values come from three sources — Pydantic defaults, TOML files (passed with `@`), and CLI flags. They're layered in this order, with later sources winning:
1. **Defaults** declared on the Pydantic model.
2. **TOML files** passed with `@`, left to right — later files override earlier ones.
3. **CLI flags** in dotted, kebab-case form (`--model.name`).
## TOML Composition
The `@` token introduces a TOML file. Multiple `@` arguments compose left-to-right, deep-merged — unset fields in an overlay keep the base value:
```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml # one file
uv run rl @ base.toml @ overlay.toml # left to right
uv run rl --trainer @ trainer.toml --orchestrator @ orch.toml # per-section
uv run rl @ base.toml --trainer @ trainer.toml # mixed
```
> Mind the space: `@ path/to/x.toml`, not `@path/to/x.toml`.
## CLI Overrides
CLI flags mirror the TOML tree using dots:
```bash theme={null}
--max-steps 50 # top-level
--model.name Qwen/Qwen3-4B # nested
--trainer.optim.lr 1e-5 # double-nested
--inference.parallel.tp 4
```
> Field names are snake\_case in TOML (`max_model_len`) and kebab-case on the CLI (`--max-model-len`).
> A renamed field keeps no alias for its old name: the old spelling fails as an unknown key rather than being silently translated.
## Inspecting and Validating
```bash theme={null}
uv run rl --help # full schema
uv run rl @ rl.toml --dry-run --output-dir /tmp/check # write resolved configs
```
## Syntax
### Booleans
CLI uses paired flags: bare `--flag` sets `True`, `--no-flag` sets `False`. TOML must be explicit:
```bash theme={null}
uv run rl @ rl.toml --clean-output-dir # True
uv run rl @ rl.toml --no-clean-output-dir # False
```
```toml theme={null}
clean_output_dir = true
```
### Lists
CLI accepts space-separated values or a JSON literal. TOML uses an array literal. Both forms target the same field:
```bash theme={null}
uv run rl @ rl.toml --trainer.model.lora.target-modules q_proj k_proj v_proj
uv run rl @ rl.toml --trainer.model.lora.target-modules '["q_proj", "k_proj", "v_proj"]'
```
```toml theme={null}
[trainer.model.lora]
target_modules = ["q_proj", "k_proj", "v_proj"]
```
Overlay TOMLs **replace** lists wholesale — an overlay that wants to add one item must still spell out the full list. For arrays of tables, see [Environments](#environments).
### Dicts
CLI takes a JSON literal. TOML uses a table. CLI dicts deep-merge with TOML dicts — CLI keys win on conflict but don't wipe the file's keys:
```bash theme={null}
uv run rl @ rl.toml --orchestrator.train.source.0.args \
'{"dataset_name": "openai/gsm8k", "dataset_subset": "main"}'
```
```toml theme={null}
[[orchestrator.train.source]]
[orchestrator.train.source.args]
dataset_name = "openai/gsm8k"
dataset_subset = "main"
```
### Optional Sub-Configs
Many sub-configs are typed `SomeConfig | None`. Two patterns enable them:
* **Bare flag with defaults**: `--model.compile` or, in TOML, an empty section `[model.compile]`. The sub-config materializes with all-default values.
* **Enable and set fields together**: `--model.compile.fullgraph` (CLI) or any populated `[model.compile]` table (TOML).
To **disable** a sub-config that's on by default, use `--no-` on the CLI or assign the string `"None"` in TOML (see [None](#none)). This is how `[ckpt]`, `[model.lora]`, `[model.compile]`, `[trainer.wandb]`, etc. are turned on and off.
### None
TOML has no `null`. Use the string `"None"`, which the loader coerces:
```toml theme={null}
[inference.model]
max_model_len = "None"
```
On the CLI: `--inference.model.max-model-len None`.
### Discriminated Unions
Loss, advantage, optimizer, scheduler, weight broadcast transport, and several others are discriminated unions. Set the `type` field to pick a variant:
```toml theme={null}
[trainer.optim]
type = "muon"
lr = 1e-5
mu = 0.95
```
Omit `type` to keep the default variant.
### Environments
Training and evaluation sources are arrays of tables. Set one source per environment; training sources can optionally carry sampling weights:
```toml theme={null}
[[orchestrator.train.source]]
name = "gsm8k"
ratio = 3 # 75% of batches
[orchestrator.train.source.env.taskset]
id = "gsm8k-v1"
split = "train"
[orchestrator.train.source.env.agent.harness]
id = "null"
[orchestrator.train.source.env.agent.runtime]
type = "subprocess"
[[orchestrator.train.source]]
name = "reverse-text"
ratio = 1 # default — 25% of batches
[orchestrator.train.source.env.taskset]
id = "reverse-text-v1"
[orchestrator.train.source.env.agent.harness]
id = "null"
[orchestrator.train.source.env.agent.runtime]
type = "subprocess"
[[orchestrator.eval.source]]
name = "gsm8k-eval"
[orchestrator.eval.source.env.taskset]
id = "gsm8k-v1"
split = "test"
[orchestrator.eval.source.env.agent.harness]
id = "null"
[orchestrator.eval.source.env.agent.runtime]
type = "subprocess"
```
`ratio` defaults to `1` (equal weight per env); values are relative weights normalized to probabilities across envs.
Everything environment lives under the `env` block (verifiers' `[env]` shape): `env.taskset` configures the v1 taskset, and each agent is a field on the env — `env.agent.harness` selects how the single-agent env's tasks are run, and per-run caps are per-agent (`env.agent.max_turns`, `env.agent.timeout`, `env.agent.max_output_tokens`). A multi-agent env declares its own seats (`env..*`).
The same taskset can appear multiple times across train and eval (or with different settings) — useful for evaluating on a held-out split or comparing two configurations side by side. When it is reused, set a distinct `name` on each entry; `name` defaults to the taskset id and must be unique across all envs in the same group.
### Environment Variables
OS environment variables exported into launched component process(es). In `rl` configs, top-level `[env_vars]` applies to trainer, inference, and orchestrator:
```toml theme={null}
[env_vars]
HF_HUB_OFFLINE = "1"
TOKENIZERS_PARALLELISM = "false"
```
Component-specific tables layer on top:
```toml theme={null}
[trainer.env_vars]
NCCL_DEBUG = "INFO"
PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:False"
[inference.env_vars]
VLLM_USE_DEEP_GEMM = "1"
[orchestrator.env_vars]
PI_USAGE_BASE_URL = "https://..."
```
The `rl` launcher applies these the same way in both single-node and multi-node (SLURM) runs. Precedence, low to high:
1. The launcher's own defaults — **your `env_vars` override these**.
2. Your top-level `[env_vars]`.
3. Your `[component.env_vars]`.
4. Orchestration-critical vars the launcher always sets last — `CUDA_VISIBLE_DEVICES` (GPU partitioning) and `WANDB_SHARED_*` (the single shared W\&B run) — **these cannot be overridden** from `env_vars`.
For standalone `sft` and `inference` configs, `[env_vars]` applies to that entrypoint's process(es). For disaggregated P/D inference, the role-specific [`deployment.{prefill,decode}_env_vars`](/prime-rl/inference) layer on top of any shared inference env vars.
## Examples
The shipped end-to-end examples in [`examples/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples) are the canonical, kept-up-to-date references — the rest of the repo's TOMLs (under `configs/`) are CI- and debug-internal and may drift. Each basic example directory has its own README with the full launch story; the advanced examples are config-only.
**Basic** (1–8 GPUs):
* [**Reverse Text**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/reverse-text) — `Qwen3-0.6B` reversing a chunk of text. Tiny single-turn SFT + RL; runs on a single consumer GPU in minutes.
* [**Wordle**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wordle) — `Qwen3-1.7B` playing Wordle. Multi-turn SFT + RL; 2–4 H100s.
* [**Alphabet Sort**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/alphabet-sort) — `Qwen3-4B-Instruct-2507` sorting names alphabetically. Multi-turn LoRA RL without SFT warmup; one H100.
* [**Wiki Search**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/wiki-search) — `Qwen3-4B-Instruct-2507` answering trivia by searching a Wikipedia corpus. Multi-turn with tool use.
* [**Hendrycks Sanity**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/basic/hendrycks-sanity) — `DeepSeek-R1-Distill-Qwen-1.5B` on a filtered MATH subset. Useful for algorithm ablations.
**Advanced** (32–2048 GPUs, SLURM):
* [**Qwen3-30B-A3B**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/qwen3-30b-a3b) — `Qwen3-30B-A3B` on math, SWE, and tool use.
* [**GLM-4.5-Air**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-4.5-air) — `GLM-4.5-Air` on search, SWE, and terminal.
* [**Nemotron-3-Super**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/nemotron-3-super) — `Nemotron-3-Super-120B` hybrid-Mamba MoE on SWE at 131k context.
* [**MiniMax-M2.5 SWE**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/minimax-m2.5) — `MiniMax-M2.5` on agentic SWE.
* [**INTELLECT-3.1**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/intellect-3.1) — reproduces our INTELLECT-3.1 training run.
* [**High-throughput GLM-5**](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/advanced/glm-5.2) — large-scale `GLM-5`/`GLM-5.2` inference with P/D disaggregation and FP8.
### Worked Example: Compose, Override, Dry-Run
Start from a shipped base config, override two fields on the CLI, and dry-run:
```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml \
--wandb.name my-experiment \
--trainer.optim.lr 5e-6 \
--output-dir /tmp/reverse-dry \
--dry-run
```
Then inspect the resolved config:
```bash theme={null}
ls /tmp/reverse-dry/configs/
# rl.toml trainer.toml orchestrator.toml inference.toml
```
Each per-process TOML reflects the final, validated configuration that the actual run would consume — exactly what each process sees when started standalone (`uv run trainer @ /tmp/reverse-dry/configs/trainer.toml`, etc.). This is the easiest way to bisect a misbehaving config: dry-run a known-good base, dry-run your overlay, diff the two.
# Development
Source: https://docs.primeintellect.ai/prime-rl/development
This page covers workflows for developing on `prime-rl` itself — running the test suite, contributing changes, and adding new model architectures with the small-scale tooling we use to iterate on MoE families without booting up a 100B+ run.
## Table of Contents
* [Test Suite](#test-suite)
* [Layout](#layout)
* [Running Tests Locally](#running-tests-locally)
* [CI Workflows](#ci-workflows)
* [Markers](#markers)
* [Pre-Commit Hooks](#pre-commit-hooks)
* [Adding a New Model](#adding-a-new-model)
* [Implement the Modeling Code](#implement-the-modeling-code)
* [Register a Mini Preset](#register-a-mini-preset)
* [Run the Smoke Test](#run-the-smoke-test)
* [Adding a Custom VLM Implementation](#adding-a-custom-vlm-implementation)
## Test Suite
The test suite is split into three tiers, each with its own CI workflow.
### Layout
* **`tests/unit/`** — fast-running, hermetic tests for isolated logic: config parsing and validation, advantage / loss / scheduler / packer math, individual dataset paths, model-conversion roundtrips, etc. Tests that need a GPU are tagged with the `gpu` marker.
* **`tests/integration/`** — full-stack RL/SFT runs on a tiny model end-to-end through inference + orchestrator + trainer.
* **`tests/nightly/`** — runs the configs in [`examples/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples) every night to catch regressions in the shipped examples.
### Running Tests Locally
```bash theme={null}
uv run pytest -v # everything
uv run pytest tests/unit -v # unit only
uv run pytest tests/integration -v # integration only
uv run pytest -v -m "not gpu" # CPU-only subset (mirrors CPU CI)
uv run pytest -v -m gpu # GPU-only subset
uv run pytest tests/integration/test_reverse_text.py -vvs # one specific scenario
```
### CI Workflows
| Workflow | Trigger | What runs | Where |
| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- |
| [`cpu_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/cpu_tests.yaml) | every PR + push to `main` | `pytest tests/unit -m "not gpu"`, plus a slim-wheel install check that `prime-rl-configs` imports cleanly without heavy deps (no torch / vllm / transformers / wandb / verifiers / datasets / liger / loguru in `sys.modules`) | `ubuntu-latest` |
| [`gpu_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/gpu_tests.yaml) | every non-draft PR + push to `main` | `pytest tests/unit -m gpu`, plus a matrix of named integration scenarios (`reverse_text`, `reverse_text_sft`, `reverse_text_lora`, `reverse_text_moe`, `reverse_text_multi_run`, `reverse_text_rl_opd`, `reverse_text_rl_sft`, `reverse_text_sft_lora`, `alphabet_sort`, `benchmark_regression`) | self-hosted GPU runners (`vm`, `4xa6000`) |
| [`nightly_tests.yaml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/.github/workflows/nightly_tests.yaml) | 03:00 PST daily + manual `workflow_dispatch` (single-file filter optional) | every file in `tests/nightly/`, one matrix job per file | `research-cluster` |
The GPU + Nightly workflows skip drafts — open the PR as **Draft** until you're ready to consume CI compute, then mark it ready for review to trigger the GPU matrix.
### Markers
Two pytest markers are declared in `pyproject.toml` (`addopts = "--strict-markers"`):
* `gpu` — gate a test that needs CUDA. CPU CI uses `-m "not gpu"`; the GPU unit job uses `-m gpu`.
* `slow` — gate a test that's expensive enough you'd usually skip it locally. Deselect with `-m "not slow"`.
## Pre-Commit Hooks
Install the [pre-commit](https://pre-commit.com) hooks before your first commit so ruff check + format run on staged Python files automatically:
```bash theme={null}
uv run pre-commit install
```
## Adding a New Model
Bringing up a new model family is three steps: implement the modeling code, register a mini preset, and run the smoke test. The preset and smoke test let you iterate on the modeling code at \~0.5B scale on 1–2 GPUs instead of paying the cost of the full-size model — useful for catching bugs in modeling code, state-dict conversions, and pipeline integration before scaling.
### Implement the Modeling Code
Drop the modeling code under `src/prime_rl/trainer/models//` (HF-compatible config, modeling, and weight conversion). Mirror the layout of an existing family — `glm4_moe/` or `qwen3_moe/` are good starting points.
### Register a Mini Preset
Add an entry to [`scripts/mini_moe.py`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/scripts/mini_moe.py) so the smoke-test workflow can build a \~0.5B test model in your architecture. The preset names the config class, picks small dimensions, and wires up the HF + prime-rl model classes plus a tokenizer source:
```python theme={null}
ARCH_PRESETS = {
"glm4_moe": {
"config_class": Glm4MoeConfig,
"config_kwargs": dict(hidden_size=1024, num_hidden_layers=24, n_routed_experts=8, ...),
"hf_model_class": HFGlm4MoeForCausalLM,
"prime_model_class": PrimeRLGlm4MoeForCausalLM,
"tokenizer_source": "THUDM/GLM-4-9B-0414",
},
# add your arch here
}
```
### Run the Smoke Test
Build the mini model. This creates a \~543M-parameter GLM-4 MoE (1024 hidden, 24 layers, 8 experts) with random weights, copies the tokenizer from the original GLM-4 model, and verifies the HF↔prime-rl roundtrip is lossless:
```bash theme={null}
uv run python scripts/mini_moe.py --arch glm4_moe --output-dir ./mini-glm-moe
```
To re-verify the roundtrip after a modeling-code change without re-creating the model:
```bash theme={null}
uv run python scripts/mini_moe.py --arch glm4_moe --output-dir ./mini-glm-moe --verify-only
```
Warm up the random-weight mini model with SFT on reverse-text so KL divergence becomes meaningful in the RL phase. Loss drops from \~12 to \~2.5 — the output won't be coherent, but the distribution is non-trivial. A pre-built SFT'd checkpoint lives at [samsja/mini-glm-moe](https://huggingface.co/samsja/mini-glm-moe) if you want to skip this step:
```bash theme={null}
uv run sft \
--model.name ./mini-glm-moe \
--data.name PrimeIntellect/Reverse-Text-SFT \
--max_steps 200 \
--optim.lr 1e-4 \
--ckpt.weights
```
Then run the full RL stack on reverse-text:
```bash theme={null}
uv run rl @ configs/ci/integration/reverse-text-moe/start.toml \
--model.name samsja/mini-glm-moe \
--trainer.model.impl custom \
--inference.gpu-memory-utilization 0.7 \
--inference.model.max-model-len 2048
```
What to look for:
* **No crashes.** Validates the full inference + orchestrator + trainer pipeline end-to-end.
* **Finite, non-zero KL.** Confirms the reference distribution is meaningful.
* **Loss reasonable.** Not NaN, not stuck.
Don't expect reward to climb meaningfully in 20 steps on a random model.
### Requirements for merging a new model
Before merging a new model, you need to ensure the following:
* The model is correctly registered and defines and all the required methods - such as `convert_hf_layer_to_tt` and `convert_tt_layer_to_hf`.
* The small smoke test passes.
In the PR that adds the new model, you also need to provide a table covering the KL mismatch across 20 steps on `math` environment with `batch_size=64`. All the entries in the table must lower than 0.015. If this is not met, the PR will not be merged (unless reasonable justification is provided). This is to ensure all our models are consistent and their implementations match the implementations in the inference framework.
## Adding a Custom VLM Implementation
VLM training (any run with `[model.vlm]` set, SFT or RL) is custom-implementation-only: `get_model` rejects models without a custom PrimeRL VLM class at load time. To make a new VLM family trainable, extend a custom text model with a composite VLM body — the Qwen3.5 dense (`models/qwen3_5/`) and MoE (`models/qwen3_5_moe/`) implementations are the reference. The pieces, in dependency order:
1. **Custom text model first.** The VLM body wraps a custom `*ForCausalLM` (see [Adding a New Model](#adding-a-new-model)), so the text side — including its state-dict conversion and KL-mismatch table — comes first.
2. **Composite VLM body.** A `*VLMModel` that holds the HF vision encoder and the custom text model, with a `prepare_inputs_embeds_and_position_ids` step: embed tokens, run the vision encoder, scatter image embeddings over placeholder tokens, and build MRoPE 3D positions from `mm_token_type_ids` (the renderer owns the token→modality mapping). The unified `*ForCausalLM` dispatches on the config: composite config → VLM path, text config → text path.
3. **Always run the vision encoder.** Text-only micro-batches must feed the encoder dummy pixels and graft the result into the graph with zero contribution (`inputs_embeds + image_embeds.sum() * 0.0`) so FSDP/EP collectives stay symmetric across ranks when the encoder is trainable.
4. **Packed-boundary consumption.** Samples pack into shared rows with per-document boundaries in `seq_lens`; every custom model's `forward()` declares the typed `seq_lens`/`seq_lens_are_pre_shard` parameters (the trainer passes them unconditionally) and must honor the boundaries — varlen flash `cu_seqlens`, linear-attention state resets per document, and a loud rejection on attention paths that can't (see the packed-batch guard in any modeling file). Set `supports_packed_multimodal_training` on the VLM model once packed rows are handled — RL fails loudly at startup for VLM models without it.
5. **Registration.** Register the composite `model_type` in `_CUSTOM_VLM_MAPPING` (`models/__init__.py`) so `get_model` dispatches to the custom class, and describe the family in `VLM_REGISTRY` (`utils/vlm.py`).
6. **Context parallelism (optional).** CP-capable VLMs implement `set_context_parallel_attributes` and shard embeds/positions inside the model after the vision merge; the trainers defer sharding to the model for MRoPE batches under ulysses.
7. **Validation.** Same bar as text models: the KL-mismatch table for the text path, plus an SFT run and an RL run on a real multimodal dataset (the `color-codeword` environment is the reference task).
# Inference
Source: https://docs.primeintellect.ai/prime-rl/inference
This page covers the inference configuration and the supported features/deployment shapes. It covers how to scale the inference server from a single GPU to 1000s of GPUs that run agentic workloads at the speed of light with all the bells and whistles configured.
## Table of Contents
* [Overview](#overview)
* [Single-Node](#single-node)
* [Multi-Node](#multi-node)
* [Multi-replica](#multi-replica)
* [Wide-EP](#wide-ep)
* [P/D Disaggregation](#pd-disaggregation)
* [Router](#router)
* [Routing policies](#routing-policies)
* [Advanced Configuration](#advanced-configuration)
* [KV Cache Offload](#kv-cache-offload)
* [Optimized P/D disaggregation deployment](#optimized-pd-disaggregation-deployment)
* [Other vLLM features](#other-vllm-features)
* [Router Replay](#router-replay)
## Overview
`prime-rl` chooses to use `vLLM` as the inference engine. We aim to stay up-to-date with the latest vLLM features, being at-most 1 version behind the latest stable release. This allows us to use the latest features from vLLM as soon as they are released - such as router replay, CPU KV cache offload, and more.
We support 3 distinct deployment shapes:
* [Single-Node](#single-node) - Runs the inference server on a single node. Useful for debugging, small scale experiments or smaller models. The default deployment shape.
* [Multi-Node](#multi-node) - Runs the inference server on multiple nodes. Useful for large scale experiments or larger models, where latency is not a concern - i.e. single turn inference, long context inference, etc.
* [Disaggregated](#pd-disaggregation) - Runs the inference server on multiple nodes, but disaggregates the prefill and decode stages. Useful for large scale experiments or larger models, where latency is a concern and multi-node deployment creates very high E2E rollout latency, such as agentic workflows.
Most of the features are supported for all deployment shapes, with few exceptions. These exceptions are rejected on validation.
Every deployment shape has the same client-facing layout: a single global router listens on `inference.server.port` and fronts all vLLM engines, which listen on `inference.backend_port` (+ rank offset). Clients always talk to one URL, regardless of how many engines run behind it.
You can select the deployment shape with `InferenceDeploymentConfig` in your config file. This is a config-field that allows you to set the deployment shape and topology knobs such as `num_nodes` and `num_replicas`.
```toml theme={null}
[inference.deployment]
type = "single_node" # or "multi_node" or "disaggregated"
```
To configure the inference server, you can use the `InferenceConfig` field. This is a config-field that allows you to set the inference server-specific knobs. Most of these are supported for all deployment shapes, with few exceptions. These exceptions are rejected on validation.
```toml theme={null}
[inference]
model = "PrimeIntellect/INTELLECT-3"
...
```
We will now walk through the supported features and deployment shapes in detail, starting with the single-node deployment.
## Single-Node
The single-node deployment is the default deployment shape. It runs the inference server on a single node. It is useful for debugging, small scale experiments or smaller models. You can configure the single-node deployment with the `SingleNodeInferenceDeploymentConfig` config-field.
```toml theme={null}
[inference.deployment]
type = "single_node"
```
The launcher starts a `vllm-router` on `inference.server.port` (default `8000`) fronting the vLLM engine on `inference.backend_port` (default `8100`). Clients connect to the router URL; admin operations (weight updates, health checks) bypass the router and hit the engine port directly — the RL entrypoint wires `orchestrator.model.client.admin_base_url` accordingly.
This deployment shape runs the inference server on a single node, if configured with NVLink enabled, it allows you more freedom in terms of parallelism configurations.
```toml theme={null}
[inference]
enable_expert_parallel = true # defaults to False
[inference.parallel]
tp = 2
dp = 4
[inference.deployment]
type = "single_node"
```
We reccomend choosing your parallelism based on the expected throughput and latency requirements. High `dp` might create high latency, however it will also give you the highest throughput. This is a tradeoff you need to make based on your use case and required `orchestrator.max_inflight_requests`. Setting `tp` to a higher value will usually give you lower latency, but the inference server also will become saturated faster with lower number of requests.
Another thing to consider, is the memory usage. You need to make sure that the model will fit into the available GPU memory. We will not go into the details on how to do this in this document. Related thing to consider, is the space for the KV cache. This will heavily affect the amount of requests your inference server can handle. You want to shard your model, either using `inference.enable_expert_parallel` or `inference.parallel.tp` to maximize the available GPU memory.
You can also increase the available KV cache memory by enabling `inference.kv_cache_offload`. More details in the [Advanced Configuration](#advanced-configuration) section.
## Multi-Node
This deployment shape branches into 2 sub-shapes:
* [Multi-replica](#multi-replica) - Runs the inference server on multiple nodes, but each node runs an independent vLLM replica. You can think of this as a for-loop over single-node deployments.
* [Wide-EP](#wide-ep) - This option is gated behind `inference.enable_expert_parallel = true`. It allows you to run the inference server on multiple nodes, allowing you to use multi-node expert parallelism. This is a more advanced feature that is suitable for high-throughput, high-concurrency workloads.
### Multi-replica
This deployment shape runs the inference server on multiple nodes, but each node runs an independent vLLM replica.
Parallelism configuration is the same as the single-node deployment. The shape is defined by setting `inference.deployment.type = "multi_node"` and `inference.deployment.num_nodes` to the number of nodes you want to run the inference server on.
```toml theme={null}
[inference.deployment]
type = "multi_node"
num_nodes = 2
[inference]
model = "PrimeIntellect/INTELLECT-3"
[inference.parallel]
tp = 2
dp = 4
```
This configuration will run 2 independent vLLM replicas, each with `tp=2` and `dp=4`. Routing is handled by a single global router running on the first inference node, fronting the per-rank endpoints of all replicas — either `vllm-router` (default) or the upstream `llm-d` EPP+Envoy, selected via the `[inference.router]` block. You can read more about the supported routing options in the [router](#router) section.
### Wide-EP
For huge, 200B+ scale models, you might want to use multi-node expert parallelism to maximize the KV-cache space. This deployment shape is defined by setting `inference.deployment.type = "multi_node"` and `inference.enable_expert_parallel = true`.
```toml theme={null}
[inference.deployment]
type = "multi_node"
num_nodes = 2
[inference]
model = "PrimeIntellect/INTELLECT-3"
enable_expert_parallel = true
[inference.parallel]
tp = 2
dp = 8
```
This configuration will run 2 vLLM processes, each with `data_parallel_size_local = 4` and `tp = 2` and expert parallelism spanning 2 nodes. The requests are again routed to these processes via the `vllm-router`.
## P/D Disaggregation
This is the most advanced deployment shape. It allows you to disaggregate the prefill and decode stages, with KV cache flowing between them. This is useful for large scale deployments, where there are high requirements on latency, such as agentic workflows spanning 100s of turns.
This deployment shape is defined by setting `inference.deployment.type = "disaggregated"` and choosing how many nodes each prefill and decode replica spans.
```toml theme={null}
[inference.deployment]
type = "disaggregated"
prefill_nodes_per_replica = 2
decode_nodes_per_replica = 2
```
Sometimes, you may want to run multiple independent vLLM instances within the prefill and decode stages. You can do this by setting `inference.deployment.num_prefill_replicas` and `inference.deployment.num_decode_replicas` to the number of role replicas you want to run.
```toml theme={null}
[inference.deployment]
type = "disaggregated"
prefill_nodes_per_replica = 2
num_prefill_replicas = 2
decode_nodes_per_replica = 2
num_decode_replicas = 1
```
Now each prefill replica spans 2 nodes and each decode replica spans 2 nodes. With 2 prefill replicas and 1 decode replica, one inference island spans 6 nodes.
For RL runs, the top-level deployment can multiply that whole inference island by setting `deployment.num_infer_replicas`. `deployment.num_infer_nodes` is inferred from the nested inference deployment when you omit it.
```toml theme={null}
[deployment] # this is a top-level RL deployment, not inference.deployment!!
type = "multi_node"
num_train_nodes = 4
num_infer_replicas = 3
```
This will run 3 inference islands, each running on 6 nodes. The total inference deployment will span 18 nodes, fronted by the single global router.
## Router
Every deployment fronts its vLLM engines with a single global router — it listens on `inference.server.port` and is the one URL clients connect to. The backend is configured via a discriminated `[inference.router]` block (`type = "vllm-router" | "llm-d"`):
```toml theme={null}
[inference.router] # or [router] for the standalone inference entrypoint
type = "llm-d" # "vllm-router" (default) or "llm-d"
non_cached_tokens = 16 # llm-d only: below this many non-cached prompt tokens, skip remote prefill (P/D)
# llm-d only: base scorer weights, applied to every profile
[inference.router.scorers]
"prefix-cache-scorer" = 3.0
"active-request-scorer" = 2.0
# llm-d only: merged onto the P/D prefill profile (decode_scorer_overrides for decode)
[inference.router.prefill_scorer_overrides]
"queue-scorer" = 2.0
"kv-cache-utilization-scorer" = 2.0
```
* **`vllm-router`** (default) — our fork of [vllm-router](https://github.com/PrimeIntellect-ai/router). Knob: `policy`. The only backend supported for single-node (local) deployments.
* **`llm-d`** — the upstream [llm-d](https://llm-d.ai) Endpoint Picker (EPP) + Envoy proxy (multi-node / disaggregated SLURM deployments only). Routing combines **prefix-cache affinity** (grouped rollouts reuse a cached prefix and skip prefill) with the **`active-request-scorer`** — an in-flight load balancer that spreads requests across ranks immediately, unlike the metrics-scraped `queue-scorer` / `kv-cache-utilization-scorer` / `load-aware-scorer` (which lag and concentrate bursts of same-prefix requests). The scorer weights follow the upstream llm-d P/D guide; tune via `scorers` (base) + `prefill_scorer_overrides` / `decode_scorer_overrides` (per-profile, P/D). Does not support `enable_return_routed_experts` (router replay).
Both backends support the 2 most important things:
* Request routing - KV cache re-use and balanced routing
* P/D disaggregation - handling the prefill and decode stages separately
### Routing policies
The 2 policies you might want to configure are:
* `consistent_hash` - this is the default policy that optimizes for KV cache re-use across turns - this works by hashing a request header to determine where to route the request to. You can configure what to hash by setting
`orchestrator.model.client.extra_headers_from_state` to the header the `router` expects to be set.
We set it to a sensible default, that works with all verifiers environments.
```toml theme={null}
[orchestrator.model.client.extra_headers_from_state]
X-Session-ID = "trajectory_id" # this is the default - each rollout has a unique trajectory_id and router expects X-Session-ID
```
* `round_robin` - this policy will round-robin the requests between the available replicas. This is useful if you want to balance the load between the replicas. This might give you better results if you don't have enough rollouts to make `consistent_hash` hashing saturated.
## Advanced Configuration
### KV Cache Offload
Maximizing KV-Cache space is crucial to support high-concurrency workloads. You can offload the KV cache to CPU memory (and, behind it, disk) by setting `inference.kv_cache_offload`. It is a discriminated config with two composable tiers, `cpu` and `disk`: a `cpu` tier is always required, and an optional `disk` tier is layered behind it (GPU → DRAM → disk). Disk-only is not supported.
The `type` field selects the backend:
* `native` — vLLM's built-in offloading. CPU-only uses `OffloadingConnector`; CPU+disk uses `TieringOffloadingSpec` (a CPU primary tier with a filesystem secondary tier). Fully self-contained — no extra processes.
* `mooncake` — a [Mooncake](https://github.com/kvcache-ai/Mooncake) **shared distributed store** (SLURM only). One `mooncake_master` + metadata server runs on the head inference node; every inference node runs a `mooncake_client` that contributes its DRAM (and, with `disk`, SSD) segment to that *single* pool. Because blocks are keyed by model + parallel rank + content hash (no instance id), a prefix cached by one node/replica is reusable by all of them over RDMA — pooling every node's CPU RAM into one KV cache. Use `native` for local/single-process runs.
```toml theme={null}
# Native CPU offload (reserves 128GB of CPU KV cache for this instance)
[inference.kv_cache_offload]
type = "native"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000 # 128GB
# Native CPU + disk tiering (self-contained)
[inference.kv_cache_offload]
type = "native"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000
[inference.kv_cache_offload.disk]
path = "/scratch/kv" # disk capacity is bounded by the filesystem
# Mooncake CPU + disk (per-node distributed store, RDMA)
[inference.kv_cache_offload]
type = "mooncake"
[inference.kv_cache_offload.cpu]
num_bytes = 128_000_000_000
[inference.kv_cache_offload.disk]
path = "/scratch/kv"
```
For `native`, `cpu.num_bytes` is the aggregate CPU KV pool for the instance (vLLM shards it across workers). For `mooncake`, `cpu.num_bytes` is the DRAM each node contributes to the shared pool (so the total pool ≈ `num_bytes × #inference-nodes`); the store uses RDMA, so it requires an RDMA-capable fabric. Enabling offload automatically enables prefix caching.
### Optimized P/D disaggregation deployment
For optimal P/D disaggregation deployment, we automatically set the decode `all2all_backend` to `deepep_low_latency` and the prefill `all2all_backend` to `deepep_high_throughput`. We currently don't support customizing all2all backends for P/D disaggragation out of the box. You can do this by overriding the slurm template only.
For KV cache transfer, we utilize the NIXL connector. This is the default and only currently supported connector. We aim to support more advanced options, such as D->P transfer, or Mooncake Connector in the future.
> **Required:** The pip-wheel NIXL's bundled UCX segfaults on the prefill→decode KV transfer. You must build NIXL against UCX 1.19.x from source — see [Disaggregated Prefill/Decode Inference](/prime-rl/advanced#disaggregated-prefilldecode-inference) in the Advanced docs for the full setup.
For configuring various knobs with environment variables, we enable you to configure prefill and decode environment variables separately. This is useful if you want to configure different environment variables for the prefill and decode stages.
```toml theme={null}
[inference.deployment]
type = "disaggregated"
[inference.deployment.prefill_env_vars]
"VLLM_ENABLE_MOE_DP_CHUNK" = "0"
"VLLM_DEEP_GEMM_WARMUP" = "skip"
[inference.deployment.decode_env_vars]
"VLLM_DEEP_GEMM_WARMUP" = "skip"
```
These are role-specific and layer on top of [`env_vars`](/prime-rl/configuration#environment-variables) shared by all inference processes regardless of role.
### Other vLLM features
We support various other vLLM features. Some of those, such as `enable_dbo`, `enable_eplb` are exposed as a top-level config fields. For those that are not, you can configure them by setting `inference.vllm_extra` to the desired value.
```toml theme={null}
[inference.vllm_extra]
headless = true
```
### Router Replay
Router replay works by capturing the expert routing decisions into a buffer. This buffer then gets sent to the trainer, which can use it instead of re-computing the routing. This lowers the trainer↔inference mismatch by an order of magnitude, resulting in more stable training.
To enable router replay, you can set `inference.enable_return_routed_experts = true`.
```toml theme={null}
[trainer]
enable_router_replay = true # this will also auto-set the inference.enable_return_routed_experts = true
[inference]
enable_return_routed_experts = true
```
This however is not free, it adds a significant overhead to the HTTP requests as this payload can grow quite large. We reccomend sizing up the env server pool (`orchestrator.*.source.serve.pool`) to allow for more parallelization on the verifiers side.
Currently this feature is also not supported with CPU KV cache offload, which can have negative impact on the inference throughput.
# Overview
Source: https://docs.primeintellect.ai/prime-rl/overview
`prime-rl` is a framework for large-scale, asynchronous reinforcement learning of large language models. It is designed to be easy to use and hackable, yet capable of training 1T+-parameter MoE models on 1000+ GPU clusters.
## Architecture
A `prime-rl` RL run is three cooperating processes:
* **Inference** — vLLM-backed server (or fleet) holding the current policy. The orchestrator drives rollouts through the token-in `/inference/v1/generate` route via the [`renderers`](https://github.com/PrimeIntellect-ai/renderers) package (OpenAI-compatible chat/completions routes are also exposed for external clients). We are trying to stay up-to-date with the latest vLLM features, you can read more about the supported features and deployment options in the dedicated [inference documentation](/prime-rl/inference).
* **Orchestrator** — Lightweight CPU process that owns the data plane across many [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) training and eval environments. Each env runs in an isolated subprocess with a variable-size pool of env workers for scalability. The orchestrator drives multi-turn rollouts against the inference fleet (tool use, browsers, sandboxes, long horizons) without re-tokenizing across turns, computes advantages, packs the rollouts into training batches, and relays new weights from trainer to inference.
* **Trainer** — FSDP2 process group that consumes packed rollouts and steps the optimizer. We ship optimized custom modeling code for many MoE / dense / VLM families that unlocks advanced trainer parallelism — expert parallelism (EP, with DeepEP kernels) and context parallelism (CP) for long-sequence training — plus selective activation checkpointing, FP8 training on Hopper+, LoRA, and multi-tenant training (many concurrent LoRA tenants sharing one trainer + inference deployment). You can read more in the dedicated [training documentation](/prime-rl/training).
The three processes communicate through configurable transports — by default the trainer↔orchestrator rollout link uses ZMQ, and weight broadcast uses NCCL for synchronous in-memory transfer (falling back to filesystem when LoRA is enabled or no inference server is configured). Swap the rollout link to the local filesystem (`rollout_transport.type = "filesystem"`) if you want rollouts persisted to disk. See [Scaling](/prime-rl/scaling) for the deployment options.
## Installation
```bash theme={null}
curl -sSL https://raw.githubusercontent.com/PrimeIntellect-ai/prime-rl/main/scripts/install.sh | bash
```
The script clones the repo, initializes the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) / [`renderers`](https://github.com/PrimeIntellect-ai/renderers) / [`research-environments`](https://github.com/PrimeIntellect-ai/research-environments) submodules, installs `uv`, and runs `uv sync --all-extras`. For manual setup, or troubleshooting, see the [README](https://github.com/PrimeIntellect-ai/prime-rl#setup).
You need at least one NVIDIA GPU (RTX 3090/4090/5090, A100, H100, H200, or B200). Single-GPU runs are supported for debugging; production RL is typically 1× inference node + 1+ trainer nodes.
## Quick Run
Train an SFT-warmed `Qwen3-0.6B` on the `reverse-text` task — the env is bundled with the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) submodule so no separate install is needed. This config ships in the repo and runs on two GPUs (one for inference, one for the trainer):
```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml
```
The `rl` entrypoint reads `examples/basic/reverse-text/rl.toml`, splits it into per-process sub-configs, picks GPU 0 for inference and GPU 1 for the trainer, launches all three processes, and tees their stdout into `outputs/logs/{trainer,orchestrator,inference}.log`. Within a minute the trainer should log `step 1` and a reward sample; after 20 steps the run completes and final HF-compatible weights land at `outputs/weights/step_20`.
## Documentation
* **[Configuration](/prime-rl/configuration)** — TOML composition, CLI overrides, dry-run.
* **[Training](/prime-rl/training)** — Launch and observe RL and SFT runs.
* **[Inference](/prime-rl/inference)** — vLLM-backed server (or fleet) holding the current policy.
* **[Scaling](/prime-rl/scaling)** — Single-GPU through multi-node clusters via FSDP / EP / CP and SLURM.
* **[Algorithms](/prime-rl/algorithms)** — Async semantics, loss / advantage / filter plugins, trajectory merging.
* **[Advanced](/prime-rl/advanced)** — Custom modeling, multimodal, LoRA, multi-tenant, P/D inference.
* **[Development](/prime-rl/development)** — Test suite, pre-commit hooks, adding a new model.
# Scaling
Source: https://docs.primeintellect.ai/prime-rl/scaling
This page covers how to scale `prime-rl` from a single GPU to a 1000-GPU cluster: single-node and multi-node deployments, FSDP / expert parallelism / context parallelism, and throughput benchmarking. See [Training](/prime-rl/training) for detailed documentation of the trainer configuration and [Inference](/prime-rl/inference) for the inference configuration.
## Table of Contents
* [Single-Node vs. Multi-Node Deployment](#single-node-vs-multi-node-deployment)
* [Single-Node](#single-node)
* [RL Placement](#rl-placement)
* [SFT and Torchrun](#sft-and-torchrun)
* [Multi-Node](#multi-node)
* [Parallelism Knobs](#parallelism-knobs)
* [FSDP](#fsdp)
* [Expert Parallelism](#expert-parallelism)
* [Context Parallelism](#context-parallelism)
* [Activation Checkpointing and Offloading](#activation-checkpointing-and-offloading)
* [Optimizer Offloading](#optimizer-offloading)
* [LM Head Chunking](#lm-head-chunking)
* [Memory-Tight Recipe](#memory-tight-recipe)
* [SLURM](#slurm)
* [Activation](#activation)
* [`[deployment]` Block](#deployment-block)
* [Examples](#examples)
* [Custom Templates](#custom-templates)
* [Benchmarking](#benchmarking)
## Single-Node vs. Multi-Node Deployment
The `rl`, `sft`, and `inference` entrypoints all accept a `[deployment]` block (`type = "single_node"` or `"multi_node"`) that picks how the trainer / orchestrator / inference processes are placed across hardware. **Single-node** runs locally; **multi-node** currently goes through [SLURM](#slurm) — the launcher writes an sbatch script that places inference replicas, the orchestrator, and the trainer with the right rendezvous endpoints, IPs, ports, and shared-filesystem paths wired in.
### Single-Node
#### RL Placement
`rl` defaults to 1 trainer GPU and 1 inference GPU. To give inference 6 GPUs with data parallelism and the trainer the remaining 2 on an 8-GPU node:
```bash theme={null}
uv run rl @ rl.toml \
--deployment.num-infer-gpus 6 \
--deployment.num-train-gpus 2 \
--inference.parallel.dp 6
```
The launcher allocates GPUs in order from `CUDA_VISIBLE_DEVICES` (or all visible GPUs): inference first, trainer next, teacher last. To target a specific physical subset, pin `CUDA_VISIBLE_DEVICES` before launching.
For quick A/B ablations on the same node, run two RL instances side-by-side in separate tmux sessions, each pinned to half the GPUs and a separate inference port:
```bash theme={null}
# session 1, GPUs 0–1, default port 8000
bash scripts/tmux.sh -s exp1 -o outputs/exp1
CUDA_VISIBLE_DEVICES=0,1 uv run rl @ rl.toml --output-dir outputs/exp1
# session 2, GPUs 2–3, port 8001
bash scripts/tmux.sh -s exp2 -o outputs/exp2
CUDA_VISIBLE_DEVICES=2,3 uv run rl @ rl.toml \
--inference.server.port 8001 \
--orchestrator.client.base-url http://localhost:8001/v1 \
--output-dir outputs/exp2
```
#### SFT and Torchrun
`uv run sft` handles distributed launch internally. To scale from 1 to N GPUs, set the deployment GPU count (or just let it pick up `WORLD_SIZE`). For non-default layouts, the manual equivalent is:
```bash theme={null}
uv run torchrun \
--nproc-per-node 8 \
--local-ranks-filter 0 \
src/prime_rl/trainer/sft/train.py @ sft.toml
```
`--local-ranks-filter 0` keeps console output to rank 0 only; per-rank stdout/stderr is still captured in `/logs/trainer/torchrun/`.
### Multi-Node
Multi-node deployments (RL or SFT) are launched via [SLURM](#slurm) — set `[deployment] type = "multi_node"` plus the matching `[slurm]` block, and the launcher writes the sbatch script that places inference, orchestrator, and trainer across the requested nodes with the inter-process wiring set up correctly. See [SLURM § Examples](#examples) for full configs.
## Parallelism Knobs
### FSDP
FSDP2 is the default model sharding strategy. By default the trainer fully shards parameters, gradients, and optimizer state across the data-parallel mesh. Tweakable knobs:
| Knob | Effect |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `trainer.model.dp_replicate` | Number of dimensions to **replicate** instead of shard. Set to 2 to run 2-way DP replication × FSDP sharding within each replica — useful for very large clusters where pure FSDP communication dominates. |
| `trainer.model.reshard_after_forward` | If `true` (default), parameters are resharded after the forward pass to free memory; the backward pass re-gathers. Set `false` to keep params resident — faster but more memory. |
| `trainer.model.fsdp_cpu_offload` | Offload params + grads + optimizer state to CPU. Big memory win, large throughput hit. |
| `trainer.model.optim_cpu_offload` | Offload only optimizer state. Mid-ground — small throughput cost, decent memory savings, especially at low GPU count. |
### Expert Parallelism
EP shards MoE expert weights across the EP mesh, dramatically reducing the FSDP communication volume per layer and improving the training throughput. EP is only available with the custom model implementation (`model.impl = "custom"` or `"auto"` for supported families).
```toml theme={null}
[trainer.model]
impl = "custom"
ep = 8 # EP degree; must divide num_experts
ep_comm_backend = "torch" # or "deepep"
```
`ep_comm_backend = "deepep"` uses DeepEP's custom dispatch/combine kernels for speed, with two extra knobs (`deepep_num_sms`, `deepep_token_chunk_size`) — tune on your hardware.
### Context Parallelism
CP shards a single sequence across multiple GPUs along the token dimension — for long-context sequences. We reccomend using `ulysses` style CP for most of the models to get the most throughput. Some models (e.g. GLM-5) only support `ring` style CP. Wrong setting will be rejected on validation.
```toml theme={null}
[trainer.model]
impl = "custom"
attn = "flash_attention_2" # or fa3 / fa4
cp = 2 # CP degree
cp_style = "ulysses" # "ring"
```
### Activation Checkpointing and Offloading
| Knob | Memory ↓ | Throughput ↓ |
| ------------------------------------- | -------- | ------------ |
| `trainer.model.ac` | large | \~25% |
| `trainer.model.ac.mode = "selective"` | medium | small |
| `trainer.model.ac_offloading` | extra | a bit more |
Enable selective AC (custom impl only) for the best memory/throughput tradeoff:
```toml theme={null}
[trainer.model.ac]
mode = "selective"
targets = ["norm", "attn_proj"] # see Reference for the full list per architecture
```
We reccomend also using `ac_offloading` and `ac_offloading.max_inflight_activations = 5` to further reduce the memory footprint in tradeoff for some throughput. We've observed this feature to be very effective, lowering the peak memory usage by 30-40% in some cases, while only lossing \~3-5% of throughput:
```toml theme={null}
[trainer.model.ac_offloading]
max_inflight_activations = 5
```
### Optimizer Offloading
Offloading optimizer states to CPU is a near-free memory win at low GPU counts:
```toml theme={null}
[trainer.optim]
# any optimizer type
type = "adamw"
[trainer.model]
optim_cpu_offload = true
```
Mutually exclusive with `fsdp_cpu_offload`. Also incompatible with `trainer.max_concurrent_runs > 1` (multi-tenant training). Muon doesn't support `fsdp_cpu_offload` but does support `optim_cpu_offload`.
### LM Head Chunking
The vanilla LM head materializes a `[batch * seq, vocab]` logits tensor on every step — a major memory tax when the vocabulary is large (often >100K). `fused_lm_head_token_chunk_size` swaps in a custom fused linear + logprob/entropy kernel that streams through `chunk_size` tokens at a time, avoiding the materialization:
```toml theme={null}
[trainer.model]
fused_lm_head_token_chunk_size = "auto" # picks 8192 for RL
# or explicit:
# fused_lm_head_token_chunk_size = 1024 # smaller = lower memory, more launches
# fused_lm_head_token_chunk_size = "disabled" # default; vanilla LM head
```
`auto` is a safe starting point for RL. Drop the chunk size further when peak memory is still tight (e.g. with very long sequences); raise it to amortize kernel-launch overhead. Only available with `model.impl = "custom"`, and currently RL-only — the SFT trainer rejects integer values.
## Memory-Tight Recipe
The kitchen-sink config for fitting large MoE on limited GPUs at acceptable throughput:
```toml theme={null}
[trainer.model]
impl = "custom"
fused_lm_head_token_chunk_size = 1024
ep = 8
cp = 2
optim_cpu_offload = true
[trainer.model.compile]
[trainer.model.ac]
freq = 1
[trainer.model.ac_offloading]
max_inflight_activations = 1
```
Walks through every memory lever in order: FSDP+EP shard the weights, CP shards the activations along the token dim, AC + AC offloading shrink the activation footprint, fused LM head chunks the loss, `torch.compile` reduces fragmentation, optim offload moves Adam state off GPU. Apply selectively — each knob has a throughput cost.
## SLURM
The `rl`, `sft`, and `inference` entrypoints all submit to SLURM when a `[slurm]` table is present — there's no separate entrypoint.
### Activation
A SLURM config is usually a thin overlay that adds `[slurm]` (and `[deployment]` for multi-node) on top of a base config. Configs are composed left-to-right via the `@` CLI syntax — see [Configuration § TOML Composition](/prime-rl/configuration#toml-composition):
```toml theme={null}
# my_slurm.toml
output_dir = "/shared/outputs/my-rl"
[slurm]
job_name = "my-rl-run"
```
Launch:
```bash theme={null}
uv run rl @ base_rl.toml @ my_slurm.toml # submits via sbatch
uv run rl @ base_rl.toml @ my_slurm.toml --dry-run # writes the sbatch script + resolved config, exits
```
### `[deployment]` Block
`[deployment]` is a discriminated union picked by `type` — `single_node` or `multi_node` for RL/SFT, with an extra disaggregated variant for inference. RL multi-node:
```toml theme={null}
[deployment]
type = "multi_node"
num_train_nodes = 2
num_infer_nodes = 1
gpus_per_node = 8 # default
nodes_per_fsdp_group = 1 # optional — controls FSDP island size
```
SFT multi-node:
```toml theme={null}
[deployment]
type = "multi_node"
num_nodes = 2
gpus_per_node = 8
```
### Examples
Full multi-node configs ship in [`examples/multinode/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/examples/multinode):
* [`rl.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/multinode/rl.toml) — two-node RL run with NCCL weight broadcast on a 30B MoE student.
* [`sft.toml`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/multinode/sft.toml) — two-node SFT against the same model.
For inference-only multi-node, set `[deployment] type = "multi_node"` on an inference TOML — each node runs an independent vLLM replica (TP and DP must fit within one node), and the launcher prints one URL per node. Front the URLs with a router or point clients at any of them.
### Custom Templates
For unusual partitions, module loads, or environment setup, supply your own Jinja2 template:
```bash theme={null}
uv run rl @ my_config.toml --slurm.template-path path/to/my_template.sbatch.j2
```
The default templates live under [`src/prime_rl/templates/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/src/prime_rl/templates) — copy one as a starting point.
## Benchmarking
Every entrypoint supports a `--bench` flag that runs a few warm-up + measurement steps with fake data and prints a rich-formatted throughput / MFU table:
```bash theme={null}
# SFT trainer alone
uv run sft @ sft.toml --bench
uv run sft ... --data.type fake --data.length variable --bench # variable-length fake data
# RL trainer alone (no inference involved)
uv run trainer @ train.toml --data.fake --bench
# Inference alone — start the server normally, then bench the orchestrator
uv run inference @ infer.toml
uv run orchestrator @ orch.toml --bench
# Full RL stack (trainer with fake data, inference with real data from orchestrator)
uv run rl @ rl.toml --bench
```
Persist results with `--bench.output-json`. Use this to compare parallelism configs before committing a multi-day run.
# Training
Source: https://docs.primeintellect.ai/prime-rl/training
This page covers everything you need to launch, observe, checkpoint, and recover a `prime-rl` training run — the RL trainer (and the distillation algorithms that run through it) and the SFT trainer. For multi-node and cluster layouts, see [Scaling](/prime-rl/scaling). For the loss math and algorithm knobs, see [Algorithms](/prime-rl/algorithms).
> **AI agents working in this repo:** the equivalent runbooks are at [`skills/training/`](https://github.com/PrimeIntellect-ai/prime-rl/tree/main/skills/training) — top-level routing in [`skills/training/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/SKILL.md), launch details in [`skills/training/start-run/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/start-run/SKILL.md), and check-in / restart procedures in [`skills/training/monitor-run/SKILL.md`](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/skills/training/monitor-run/SKILL.md).
## Table of Contents
* [Entrypoints](#entrypoints)
* [RL Trainer](#rl-trainer)
* [Launch](#launch)
* [Useful Knobs](#useful-knobs)
* [Algorithms](#algorithms)
* [Important Metrics](#important-metrics)
* [SFT Trainer](#sft-trainer)
* [Dataset Format](#dataset-format)
* [Launch](#launch-1)
* [SFT-Specific Knobs](#sft-specific-knobs)
* [Important Metrics](#important-metrics-1)
* [Checkpointing](#checkpointing)
* [Enabling Checkpoints](#enabling-checkpoints)
* [Resuming a Run](#resuming-a-run)
* [Serving Checkpoints](#serving-checkpoints)
* [Observability](#observability)
* [Log Files](#log-files)
* [Console Output](#console-output)
* [Weights & Biases](#weights--biases)
* [Platform Monitoring](#platform-monitoring)
* [Rules of Thumb](#rules-of-thumb)
## Entrypoints
| Command | Purpose | Notes |
| --------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `uv run rl` | Wraps the trainer, orchestrator, and inference server in one launch from a merged TOML. | The default for any RL run. Runs locally for single-node experiments; submits to SLURM for single- or multi-node when `[slurm]` is set (see [Scaling § SLURM](/prime-rl/scaling#slurm)). |
| `uv run sft` | Supervised fine-tuning on a HF dataset. | Launches torchrun internally; never call torchrun directly. |
| `uv run inference` | vLLM server. | Always use this entrypoint over `vllm serve` — it adds `/update_weights`, `/load_lora_adapter`, and `/init_broadcaster`. |
| `uv run trainer` | Standalone trainer process group. | Use only when launching the trainer separately from the orchestrator (e.g. multi-node RL without the `rl` wrapper). |
| `uv run orchestrator` | Standalone orchestrator process. | Pair with a separately-launched trainer, inference, and one `env-server` per source. |
| `uv run env-server` | Standalone env server for one environment. | The `rl` launcher starts these automatically (one per train/eval source, at the source's derived `serve.address`); only needed when running the orchestrator standalone. |
## RL Trainer
### Launch
The minimal RL run trains an SFT-warmed `Qwen3-0.6B` on the `reverse-text` task — the env is bundled with the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) submodule, so nothing else needs to be installed:
```bash theme={null}
uv run rl @ examples/basic/reverse-text/rl.toml
```
### Useful Knobs
A condensed view of the knobs you'll most often tune. For trainer-side parallelism, sampling, optimizer, and loss knobs see [Scaling](/prime-rl/scaling) and [Algorithms](/prime-rl/algorithms).
**Data and algorithm:**
| Knob | What it does |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `orchestrator.batch_size` | Tasks per trainer step. |
| `orchestrator.group_size` | Rollouts generated per task. |
| `orchestrator.max_off_policy_steps` | How many distinct policies may have contributed to one rollout before it's discarded (default 8). The main off-policy dial on long agentic rollouts — bump for throughput, lower for tighter on-policyness. Watch `errored_rollouts` and `mismatch_kl/all/mean` when tuning. |
| `[orchestrator.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `hierarchical_grpo`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). |
| `[[orchestrator.train.source]]` | Training sources. List multiple tables for multi-env training; weight them via `ratio`. See [Configuration § Training sources](/prime-rl/configuration#training-sources-orchestratortrainsource). |
| `[[orchestrator.eval.source]]` + `orchestrator.eval.interval` | Eval environments and cadence (default every 100 steps). |
**Monitoring:**
| Knob | What it does |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `log.level` | Process log level for trainer + orchestrator (`info` default; falls back to `$PRIME_LOG_LEVEL`). Set per-process via `trainer.log.level` / `orchestrator.log.level`, or globally on the `rl` entrypoint to propagate to both. |
| `orchestrator.log.vf_level` | Env-worker / [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) log level (`info` default; `debug` is noisy but useful for env debugging). |
| `--wandb` (+ `--wandb.project`, `--wandb.name`) | Enable Weights & Biases logging. See [Weights & Biases](#weights--biases). |
| `--orchestrator.prime-monitor` | Stream metrics to the Prime Intellect platform (Prime Lab). See [Platform monitoring](#platform-monitoring). |
**Run management:**
| Knob | What it does |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--clean-output-dir` | Wipe `` before starting. Useful when re-running an experiment with the same name during iteration. |
| `--output-dir outputs/` | Per-run output directory. Always set this when running more than one experiment in parallel. |
| `--max-steps N` | Stop after `N` trainer steps. Overrides the config value. |
| `--dry-run` | Resolve + validate the full config, write per-process TOMLs to `/configs/`, and exit without launching. The fastest way to debug a misbehaving config. |
### Algorithms
The RL entrypoint supports several training algorithms, switched via `[orchestrator.algo]`'s `type` (see [Algorithms](/prime-rl/algorithms#the-algorithm-abstraction) for the full reference, model references, and per-algorithm customization):
| `algo.type` | Frozen model | Use case |
| ------------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `grpo` (default) | None | Standard group-relative RL |
| `max_rl` | None | [MaxRL](https://arxiv.org/abs/2602.02710): GRPO with mean-normalized advantages (maximum-likelihood RL) |
| `rae` | None | [SPIRAL](https://arxiv.org/abs/2506.24119)'s role-conditioned advantage estimation: reward minus a per-agent EMA baseline, for multi-agent self-play envs (e.g. `kuhn-poker-v1`) |
| `hierarchical_grpo` | None | GRPO for proposer-solver envs: compare solvers only with attempts on the same proposed problem, and compare proposers with the other proposals in the group |
| `opd` | Required, must be vLLM (needs `prompt_logprobs`) | [On-policy distillation](https://thinkingmachines.ai/blog/on-policy-distillation/): the policy generates rollouts, the trainer minimizes per-token reverse KL to a reference model |
| `sft` | Required, any OpenAI-compatible endpoint | Hard-distill: a frozen model generates rollouts, the policy trains on its tokens |
| `opsd` | None — the live policy is its own reference (no deployment) | [SDFT](https://arxiv.org/abs/2601.19897): the model is its own reference conditioned on expert demonstrations |
| `echo` | None | GRPO plus cross-entropy on env-observation tokens |
A new algorithm is a named class in code, not a config — see [Algorithms § Authoring an Algorithm](/prime-rl/algorithms#authoring-an-algorithm).
Frozen models are declared inline on the algorithm, named where the model is used — `[orchestrator.algo.teacher]` for `opd` (the frozen model scored against), `[orchestrator.algo.sampling.source]` for `sft` (the model it samples from) — each with `name` + `base_url`. `opsd` declares no frozen model: it self-distills against the live policy. The `rl` entrypoint only manages policy inference — start frozen-model servers yourself and point `base_url` at them:
```bash theme={null}
CUDA_VISIBLE_DEVICES=1 uv run inference \
--model.name --server.port 8001
```
The standalone `uv run sft` entrypoint is the more traditional SFT path — pure dataset-based, no orchestrator. Use the `sft` algorithm only when you want a frozen model to generate the supervision on the fly.
### Important Metrics
Pulled from the console logs and mirrored to W\&B.
**Progress** (orchestrator):
* `reward/{all,env}/mean` — main signal. Should trend upward over hundreds of steps.
* `seq_len/{all,env}/mean` and `is_truncated/{all,env}/mean` — rollout length and truncation rate.
* `num_turns/{all,env}/mean` — for multi-turn envs.
* `empty_rollouts/{all,env}`, `errored_rollouts/{all,env}` — non-zero is fine in small numbers; sustained > 5% is a smell.
* `eval/{env}/{avg@k,pass@k}` — eval scores when `[orchestrator.eval]` is set.
**Stability** (trainer):
* `mismatch_kl/{all,env}/{mean,std,max}` — KL between trainer's current policy and the (older) inference policy that generated the rollouts. A sustained, growing mean is the early-warning sign for off-policy collapse.
* `entropy/{all,env}/mean` — too low means mode-collapse; too high means the model isn't committing.
* `masked_advantage_{positive,negative}/mean` — fraction of DPPO-masked tokens, split by sign.
* `optim/grad_norm` — spikes precede divergence; check the loss config or lower the LR.
**Performance** (trainer + orchestrator step independently):
| Source | Metric | Reading |
| ------------ | --------------------- | ---------------------------------- |
| trainer | `time/wait_for_batch` | **high → orchestrator bottleneck** |
| orchestrator | `time/wait_for_ckpt` | **high → trainer bottleneck** |
## SFT Trainer
`uv run sft` runs supervised fine-tuning from a HF dataset. It shares model loaders, FSDP setup, checkpointing, and the chat-template plumbing with the RL trainer, so a typical workflow is *SFT → RL → SFT → …* without any reformatting.
### Dataset Format
Two accepted layouts:
* **Prompt-completion**: a HF dataset with `prompt` and `completion` columns ([TRL format](https://huggingface.co/docs/trl/en/dataset_formats#prompt-completion)). The trainer masks out the prompt and computes loss only over the completion.
* **Messages**: a HF dataset with a single `messages` column containing a list of chat turns. The trainer interprets the whole conversation as one sample, applies role-based loss masking, and trains over all assistant turns.
If both columns are present, `messages` takes precedence.
**Tool definitions and renderer controls.** For tool-use SFT, add a `tools` column (OpenAI function-calling format) or `tool_defs` ([`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) rollout format). Each row's value can be either a list of dicts or a JSON-encoded string of a list — both are accepted, and `tool_defs` rows are auto-converted to OAI shape before being passed into the renderer.
Renderer-backed SFT reads template controls from the typed `[renderer]` config in the SFT TOML. For example:
```toml theme={null}
[renderer]
name = "qwen3"
enable_thinking = false
```
If a model needs another template control, add it to that model's renderer config in `renderers` (for example a new field on the relevant `*RendererConfig`) and consume it in the renderer implementation.
**Renderer-backed tokenization.** SFT tokenization is renderer-only. The [`renderers`](/prime-rl/algorithms#renderers) package owns message-to-token conversion and loss attribution end-to-end, so position-dependent chat templates (for example templates that strip past `` blocks across user turns) do not corrupt the loss mask. `[renderer]` defaults to `name = "auto"`; set a typed renderer config only when you need model-specific template controls. Hand-coded renderers ship for Qwen3, Qwen3.5, GLM-5, GLM-4.5, Kimi K2/K2.5, MiniMax M2, DeepSeek V3, Nemotron 3, GPT-OSS, and VLM families such as Qwen3-VL/Qwen3.5.
**VLM training requires a custom PrimeRL implementation.** Training a model with `[model.vlm]` set (SFT or RL) requires `model.impl = "custom"` and only works for models with a registered PrimeRL VLM class (currently Qwen3.5 dense and MoE).
See [Algorithms § Multi-Turn Trajectories](/prime-rl/algorithms#multi-turn-trajectories) for the full picture.
### Launch
The minimal SFT run trains `Qwen3-0.6B` on the `reverse-text` SFT dataset:
```bash theme={null}
uv run sft @ examples/basic/reverse-text/sft.toml --wandb
```
Multi-GPU and multi-node use torchrun under the hood (the `sft` entrypoint manages this for you — see [Scaling § SFT and Torchrun](/prime-rl/scaling#sft-and-torchrun) for non-default layouts; multi-node SFT goes through [SLURM](/prime-rl/scaling#slurm)).
### SFT-Specific Knobs
| Knob | What it controls |
| ----------------- | ------------------------------------------------------------------ |
| `data.name` | HF dataset name or local path |
| `data.batch_size` | Tokens per trainer step (packed) |
| `data.seq_len` | Per-sample sequence length |
| `loss_mask.*` | Which roles contribute to loss (system / user / assistant / tool). |
| `val.interval` | Run validation every N steps; `val.data` mirrors `data` |
### Important Metrics
Pulled from the console log and mirrored to W\&B.
**Progress and loss:**
* `loss/mean` — main signal. Should decrease through the run.
* `val/loss` — validation loss when `[val]` is set, logged every `val.interval` steps.
* `progress/epoch`, `progress/num_samples`, `progress/num_tokens` — dataset progress.
* `progress//ratio_{samples,tokens}` — when training on multiple HF subsets/splits, the realized mixing ratio.
**Stability and optimization:**
* `optim/grad_norm` — spikes precede divergence.
* `optim/lr`, `optim/zero_grad_ratio` — LR schedule and the fraction of params that received zero gradients (high → dead path or wrong loss masking).
* For MoE: `max_vio/mean` (load-balancing violation), `routing_confidence/mean` — both are logged when non-zero.
**Performance:**
| Metric | Reading |
| ------------------------------------------------------ | ---------------------------- |
| `perf/throughput`, `perf/throughput_per_gpu` | tokens/s overall and per GPU |
| `perf/mfu` | MFU |
| `perf/peak_memory` | peak GPU memory (GiB) |
| `time/step`, `time/forward_backward`, `time/save_ckpt` | step breakdown |
## Checkpointing
Checkpointing is split across processes because the orchestrator and trainer can be on different machines and on different steps at any given time. Inference is stateless.
| Process | What's saved | Where |
| -------------------- | ----------------------------------------------------------- | ----------------------------------------------- |
| Trainer | FSDP-sharded model (DCP), optimizer, scheduler, progress | `/checkpoints/step_N/trainer/` |
| Orchestrator | Progress, per-env data state | `/checkpoints/step_N/orchestrator/` |
| Inference | *nothing* — re-pushed from the latest checkpoint on restart | n/a |
| Trainer (HF weights) | HF-compatible weight snapshot for serving | `/weights/step_N/` |
### Enabling Checkpoints
Checkpointing is **off by default** to save disk. Enable it with `--ckpt`:
```bash theme={null}
uv run rl @ rl.toml --ckpt # default: end-of-training only
uv run rl @ rl.toml --ckpt.interval 25 # every 25 steps
uv run rl @ rl.toml --ckpt.interval 25 --ckpt.keep-last 3 # rolling window of 3
uv run rl @ rl.toml --ckpt.interval 25 --ckpt.keep-interval 100 # …plus permanent every 100
```
### Resuming a Run
Re-run the same launch command and pass `--ckpt.resume-step ` (or `-1` for "latest"). Make sure `--max-steps` is at least the target final step, not the remaining delta:
```bash theme={null}
# First run: steps 1–10
uv run rl @ rl.toml --max-steps 10 --ckpt
# Resume: continue to step 20
uv run rl @ rl.toml --max-steps 20 --ckpt.resume-step 10
```
### Serving Checkpoints
HF-compatible weight snapshots are written under `/weights/step_N/` whenever a full checkpoint runs (or you can write weights-only via `--ckpt.weights-only` for cheaper snapshots). Upload directly:
```bash theme={null}
uv run hf upload /-RL outputs/weights/step_100
```
For LoRA runs, set `ckpt.weights.save_adapter_separately = true` to also write the raw adapter alongside the merged weights — useful when serving the adapter through a separate `/load_lora_adapter` call.
## Observability
### Log Files
The launcher tees every process's stdout/stderr into `/logs/`. The full layout (single-node runs skip the `node_*.log` and `router.log` files — there the router logs into `inference.log`):
```
/logs/
├── trainer.log # rank 0 only; symlink → trainer/node_0.log on multi-node
├── orchestrator.log # single instance, single file
├── inference.log # symlink → inference/node_0.log on multi-node
├── trainer/
│ ├── node_*.log # per-node trainer stdout (multi-node only)
│ └── torchrun//attempt_0//{stdout,stderr}.log # per-rank
├── inference/
│ ├── node_*.log # per-node inference stdout (multi-node only)
│ └── router.log # the single global router (multi-node only)
└── envs/{train,eval}/.log # one env server process per source (broker + its workers)
```
Env logs are the first place to look for env-side errors (most user code lives there). Verbosity is controlled by `orchestrator.log.vf_level`. For multi-rank trainer debugging, drop into `logs/trainer/torchrun//attempt_0//{stdout,stderr}.log` — verbose and per-rank.
Live tailing from a single point (works on the head node for multi-node runs over a shared filesystem):
```bash theme={null}
tail -F /logs/{trainer,orchestrator,inference}.log
tail -F /logs/trainer/node_*.log # multi-node only
tail -F /logs/inference/router.log # multi-node only
```
### Console Output
`scripts/tmux.sh` opens a 4-pane tmux session that follows `trainer.log`, `orchestrator.log`, `inference.log`, and the union of env worker logs. Start it before launching:
```bash theme={null}
bash scripts/tmux.sh
# then in the Launcher window:
uv run rl @ ... --output-dir outputs/my-run
```
Pass `-s ` and `-o ` to run multiple parallel experiments side-by-side in different sessions. The helper also works on a SLURM head node — `bash scripts/tmux.sh my-rl-job /shared/outputs/my-rl-job`.
### Weights & Biases
W\&B is off by default. Enable with `--wandb`:
```bash theme={null}
uv run rl @ rl.toml --wandb # default project, random name
uv run rl @ rl.toml --wandb.project my-proj --wandb.name run-42
uv run rl @ rl.toml --no-wandb # force-disable even if the TOML enables it
```
The trainer and orchestrator log into a **single shared W\&B run**, so all metrics from both processes land in one place. Shared mode requires the W\&B SDK ≥ 0.19.9 and is incompatible with `wandb.offline = true`.
By default, every 10 steps each process also logs a sample of prompts/completions (with rewards and advantages) and reward/advantage/entropy distributions as W\&B tables. Tune via `--wandb.log-extras.interval` and `--wandb.log-extras.sample-ratio`, or disable subsets:
```bash theme={null}
uv run rl @ rl.toml --wandb \
--orchestrator.wandb.log-extras.interval 50 \
--no-trainer.wandb.log-extras.distributions
```
prime-rl deliberately logs a **large number of metrics** for maximum observability: every rollout metric is emitted per subset (`all`/`effective`), per statistic (`mean`/`max`/`min`/`p10`/`p90`), and per environment alongside a cross-env aggregate, so a multi-env run can emit thousands of series. To keep that navigable, W\&B mode **auto-creates an `overview` saved view** on the first run into a project — curating the handful of metrics that matter into `train`, `eval`, `stability`, and `performance` sections (with per-env breakdowns). The view is created once per project and adapts to the run's environments; if a later run uses a different set of environments, a new versioned view (`overview-v2`, …) is created instead of overwriting the first.
### Platform Monitoring
Register a run on the Prime Intellect platform (Prime Lab) and stream training metrics, samples, and distributions to the platform dashboard. Bare flag uses defaults:
```bash theme={null}
uv run rl @ rl.toml --orchestrator.prime-monitor
```
Or set it in TOML:
```toml theme={null}
[orchestrator.prime_monitor]
run_name = "my-experiment"
```
Requires `PRIME_API_KEY` (set via `prime login` or env var) and an allowlisted team. Currently internal-only.
## Rules of Thumb
* **Start small.** Run `examples/basic/reverse-text/rl.toml` end-to-end on 2 GPUs before scaling. If the smoke run finishes cleanly, your install is good.
* **Batch size ≥ 64.** Smaller batches give noisy gradient estimates and the trainer's overhead-per-step dominates throughput. 64 is the practical floor; 128–512 is the range for quick ablations; production RL often runs at 1024+.
* **Group size ≥ 8.** Bigger groups (`orchestrator.group_size`) make it more likely that a task produces a mix of high- and low-reward rollouts, which is what gives the trainer a usable signal — if all rollouts in a group succeed or all fail, the within-group advantage collapses to zero and the trainer learns nothing from that task. Bigger groups also tighten advantage normalization. 8 is the floor; 16–32 is common.
* **Pin `output_dir` per run.** Sharing a directory across runs will mix rollouts and break resumes. `--output-dir outputs/` is the simplest discipline.
* **Use `--dry-run` before SLURM.** Validators (e.g. CP needs flash-attention) fail fast in dry-run and slow in queue.
# Sandbox CLI Guide
Source: https://docs.primeintellect.ai/sandboxes/cli
Command-line workflows for managing sandboxes
## Spin Up a Workspace
```bash theme={null}
prime sandbox create python:3.11-slim \
--name analytics-lab \
--cpu-cores 2 \
--memory-gb 4 \
--disk-size-gb 20 \
--timeout-minutes 240 \
--env PROFILE=production \
--secret DB_PASSWORD=hunter2
```
Why it's nice:
* Omit `--name` to auto-generate a slug.
* Pass `--team-id` if you need to charge a different workspace.
* Add `--yes` to skip the confirmation prompt in automation.
## Idle Timeout
`--timeout-minutes` caps the total lifetime. `--idle-timeout-minutes` terminates the sandbox sooner when nothing is using it — useful for agent workflows that might forget to clean up.
```bash theme={null}
prime sandbox create python:3.11-slim \
--timeout-minutes 240 \
--idle-timeout-minutes 15
```
The sandbox shuts itself down if no `exec`, `upload`, `download`, or file-read request lands within the idle window. Each new request resets the clock; long-running execs stay pinned for their full duration.
* Disabled by default — set `--idle-timeout-minutes` to opt in.
* Must satisfy `1 ≤ idle ≤ timeout` and `idle ≤ 1440`.
* Not supported for VM-backed sandboxes (`--vm`).
* SSH sessions do **not** count as activity yet, so a sandbox with an active SSH connection can still be reaped if you set an idle timeout.
When the sandbox terminates this way, its termination reason in `prime sandbox get` and on the dashboard reads **Idle Timeout**.
## Environment Variables & Secrets
Use `--env` for general configuration and `--secret` for sensitive values. Both accept `KEY=VALUE` format and can be specified multiple times.
```bash theme={null}
prime sandbox create python:3.11-slim \
--env APP_ENV=staging \
--env LOG_LEVEL=debug \
--secret API_KEY=sk-abc123 \
--secret DB_PASSWORD=hunter2
```
**Environment variables** are stored in plain text and visible when you inspect the sandbox.
**Secrets** are encrypted at rest and obfuscated in CLI output. Use them for API keys, passwords, database credentials, and anything else you wouldn't want to appear in logs. Both are injected into the sandbox container and accessible as standard environment variables at runtime.
## Start Command
By default, sandboxes run `tail -f /dev/null` to keep the container alive for interactive use. Use `--start-command` to override this with your own entrypoint:
```bash theme={null}
prime sandbox create python:3.11-slim \
--start-command "python serve.py --port 8000"
```
This replaces the image's `ENTRYPOINT`, so make sure your command is the full process you want running. If you omit `--start-command`, the default keeps the sandbox idle and ready for `prime sandbox run` commands.
## Network Access
Sandboxes have outbound internet access enabled by default. For isolated environments (e.g., running untrusted code), disable it:
```bash theme={null}
# Create a sandbox without internet access
prime sandbox create python:3.11-slim --no-network-access
# Create with internet access (default behavior)
prime sandbox create python:3.11-slim --network-access
```
When network access is disabled:
* Outbound connections to the internet are blocked
* DNS resolution for internal services still works
* Communication within the sandbox (e.g., sidecar) is allowed
## Custom Docker Images
You can build and push custom Docker images to use in sandboxes. See the [Prime Images](/sandboxes/images) guide for details.
## Check In on Sandboxes
```bash theme={null}
# Overview at a glance
prime sandbox list --status RUNNING --output table
# Rich details for one sandbox
prime sandbox get sbx_123 --output json
# Quick command to verify the runtime
prime sandbox run sbx_123 --working-dir /workspace "python -c 'print(42)'"
# Run as a specific user (username or UID, optionally USER:GROUP), like `docker exec -u`
prime sandbox run sbx_123 --user agent "whoami"
# Capture logs for later debugging
prime sandbox logs sbx_123 > logs.txt
```
## Organize with Labels
Labels help you tag and manage groups of sandboxes:
```bash theme={null}
# Create sandboxes with labels
prime sandbox create python:3.11-slim \
--label experiment \
--label ml-pipeline \
--label team-research
# List sandboxes with specific labels (must have ALL labels)
prime sandbox list --label experiment --label ml-pipeline
# Delete all sandboxes with specific labels
prime sandbox delete --label experiment --yes
```
Labels are useful for:
* Grouping related experiments or workflows
* Tracking which team or project owns a sandbox
* Bulk cleanup by category (dev, staging, test, etc.)
## Move Files Around
```bash theme={null}
# Push local assets into the sandbox
prime sandbox upload sbx_123 notebooks/analysis.ipynb /workspace/
# Pull results back home
prime sandbox download sbx_123 /workspace/report.csv reports/latest.csv
```
**Note:** File uploads are limited to 200MB per file.
If a transfer complains about auth, run `prime sandbox reset-cache` and retry—the CLI refreshes the gateway token for you.
## Expose Ports
Make services running inside your sandbox accessible from the internet. Both HTTP and TCP protocols are supported.
Ports must be in the range **22–9000**. Ports **8080**, **2222**, and **8081** cannot be exposed.
### HTTP
Expose an HTTP service and get a public HTTPS URL:
```bash theme={null}
prime sandbox create python:3.11-slim --name web-server
# Returns sandbox ID, e.g. sbx_abc123
prime sandbox run "nohup python -m http.server 8000 --bind 0.0.0.0 > /dev/null 2>&1 &"
prime sandbox expose 8000 --name web-server
prime sandbox list-ports
prime sandbox unexpose --yes
```
### TCP
Expose a raw TCP service and get a public `host:port` endpoint:
```bash theme={null}
prime sandbox expose 9000 --name tcp-server --protocol TCP
```
TCP exposures return an `external_endpoint` (host:port) and `external_port` instead of a URL. Connect using any TCP client:
Use `prime sandbox list-ports ` to see both HTTP and TCP exposures along with their protocols and external ports.
## SSH
Connect to a running sandbox with an interactive shell:
```bash theme={null}
prime sandbox ssh
```
The CLI generates an ephemeral key pair, creates a session, and connects automatically. The session is cleaned up on disconnect.
By default, the best available shell is auto-detected (bash > zsh > sh). To use a specific shell:
```bash theme={null}
prime sandbox ssh --shell zsh
```
You can also forward ports through the SSH connection:
```bash theme={null}
prime sandbox ssh -- -L 3000:localhost:3000
```
## Clean Up in Bulk
```bash theme={null}
# Delete specific sandboxes by ID (space or comma-separated)
prime sandbox delete sbx_123 sbx_456 sbx_789
# Delete all sandboxes matching specific labels
prime sandbox delete --label experiment --label staging --yes
# Wipe every active sandbox (careful!)
prime sandbox delete --all --yes
# --all only deletes your own sandboxes by default.
# Use --all-users to include sandboxes from all team members.
prime sandbox delete --all --all-users --yes
```
You must use exactly one of: sandbox IDs, `--label`, or `--all`. Deletes are batched behind the scenes, and the CLI prints success/failure per sandbox so you can re-run failed IDs.
Need more ideas? Check the runnable scripts in `prime-cli/examples/` for CLI walkthroughs you can customize.
## Quick Troubleshooting
* Sandbox stuck in `PROVISIONING`? Wait a minute, then rerun `prime sandbox list --status RUNNING`. If it stays pending, delete and recreate from a known-good image.
* Hitting auth issues? `prime sandbox reset-cache` refreshes the gateway token after you rotate API keys.
# Prime Images
Source: https://docs.primeintellect.ai/sandboxes/images
Build and manage custom Docker images for sandboxes
## What are Prime Images?
Prime Images lets you push custom Docker images to Prime's registry and use them in sandboxes. Builds happen in the cloud, so you don't need Docker running locally.
## Push an Image
```bash theme={null}
# Basic push (uses ./Dockerfile in current directory)
prime images push myapp:v1.0.0
# Make the image public when the build completes
prime images push myapp:v1.0.0 --public
# Specify a different Dockerfile
prime images push myapp:v1.0.0 --dockerfile custom.Dockerfile
# Use a different build context
prime images push myapp:v1.0.0 --context ./app
# Copy an existing public image into Prime's registry
prime images push --source-image ubuntu:22.04
# Copy and rename the destination tag
prime images push myubuntu:22.04 --source-image ubuntu:22.04
```
The CLI packages your build context, uploads it, and kicks off a remote build. You'll get a build ID to track progress.
Use `--source-image` to transfer a public image from Docker Hub, GHCR, Quay, public ECR, `registry.k8s.io`, or MCR without building from a Dockerfile.
When the push completes, the CLI prints the full image reference to use when creating a sandbox. Prime Image references always start with `prime/`. By default this uses your unique slug — your username for personal images, or the team slug when you're in a team context:
```text theme={null}
prime//myapp:v1.0.0 # personal
prime//myapp:v1.0.0 # team
```
If you haven't set a username (or your team has no slug yet), the reference falls back to the id form — `prime//myapp:v1.0.0` or `prime/team-/myapp:v1.0.0`. Run `prime images list` to see the exact reference for each image.
## Check Build Status
```bash theme={null}
# See all your images and their build status
prime images list
```
Status meanings:
* **Ready** – Build succeeded, image is usable
* **Building** – Build in progress
* **Pending** – Queued for build
* **Failed** – Build failed (check your Dockerfile)
## Use Your Image
Once the status shows **Ready**, create a sandbox with it:
```bash theme={null}
# Personal image
prime sandbox create prime//myapp:v1.0.0 --cpu-cores 2 --memory-gb 4
# Team image
prime sandbox create prime//myapp:v1.0.0 --cpu-cores 2 --memory-gb 4
```
## Publish or Unpublish an Image
Images are private by default. Publishing an image lets other authenticated Prime users use it in sandboxes with the `prime/` reference shown by `prime images list`.
```bash theme={null}
prime images publish myapp:v1.0.0
prime images unpublish myapp:v1.0.0
```
## Delete an Image
```bash theme={null}
prime images delete myapp:v1.0.0
# Skip confirmation
prime images delete myapp:v1.0.0 --yes
```
# Sandboxes Overview
Source: https://docs.primeintellect.ai/sandboxes/overview
Why sandboxes exist, how to launch one, and what it costs
## Why Sandboxes
Prime Sandboxes are disposable Docker environments for AI-assisted coding, benchmarking, and quick experiments. They give agents and humans an isolated workspace without touching production infrastructure.
### Popular moves
* **Prototype quickly** – launch a clean runtime and iterate without worrying about teardown.
* **Benchmark safely** – run untrusted code or model evaluations in a locked-down environment.
* **Run AI agents securely** – disable network access to prevent code from reaching the internet.
* **Inject secrets safely** – pass API keys, passwords, and tokens as encrypted secrets that never appear in logs or API responses.
* **Teach & demo** – hand teammates a reproducible workspace that mirrors your setup.
* **Use custom images** – push your own Docker images and use them in sandboxes with all your dependencies pre-installed.
* **Organize with labels** – tag sandboxes by project, team, or environment for easy filtering and bulk cleanup.
## First Run
Make sure you have an API key with the right permissions on hand (`prime login`) before running these commands.
```bash theme={null}
# Create
prime sandbox create python:3.11-slim --timeout-minutes 120
# See what is active
prime sandbox list
# Try a quick command
prime sandbox run "python --version"
# Clean up when you're done
prime sandbox delete
```
Track usage limits and billing inside the [Prime Billing dashboard](https://app.primeintellect.ai/dashboard/billing).
## Pricing
Sandboxes are billed while running:
* **CPU**: \$0.05 per core per hour
* **Memory**: \$0.01 per GB per hour
* **Disk**: \$0.001 per GB per hour
Example: 1 CPU core, 2 GB RAM, 10 GB disk → \$0.08/hour.
## Limits
### Per-sandbox
| Resource | Min | Max | Default |
| ------------ | ------ | ----------------- | -------- |
| CPU cores | 0.1 | 16 | 1 |
| Memory | 0.1 GB | 64 GB | 2 GB |
| Disk | 0.1 GB | 1,000 GB | 10 GB |
| Timeout | 1 min | 1,440 min (24 hr) | 60 min |
| Idle timeout | 1 min | 1,440 min (24 hr) | Disabled |
### Per-account
| Resource | Default limit |
| ------------------- | ------------- |
| Active sandboxes | 512 |
| Total CPU cores | 512 |
| Total memory | 1,024 GB |
| Total storage | 5,120 GB |
| HTTP port exposures | 128 |
| TCP port exposures | 32 |
Account limits apply across all running sandboxes. If you need higher limits, contact support.
## Container Images
Sandboxes support images from any Docker-compatible registry. When you pass an image reference to `prime sandbox create`, it resolves in this order:
1. **Fully-qualified references** go to the registry you name — `docker.io/pytorch/pytorch:latest`, `ghcr.io/org/image:tag`, `quay.io/org/image:tag`.
2. **Prime Images** — references that start with `prime/`, like `prime//:`, `prime//:`, or the id forms (`prime//:`, `prime/team-/:`), resolve to a [Prime Image](./images) you can access: your own, your team's, or one that's been published.
3. **Docker Hub** — any other reference with no registry host and no `prime/` prefix resolves as a Docker Hub image, so official images like `ubuntu:22.04` or `pytorch/pytorch:latest` work as usual.
For private images, add registry credentials via the [dashboard](https://app.primeintellect.ai/dashboard/instances?tab=templates), then switch to `Settings` in the toggle. Then pass `--registry-credentials-id` when creating a sandbox. Use `prime registry list` to view saved credentials and `prime registry check-image` to verify an image is accessible.
## GPU Support
GPU-enabled sandboxes are on the roadmap. For now, sandboxes run on CPU-only machines and setting `--gpu-count` to a value greater than 0 will return an error. We'll announce as soon as GPU tiers open up.
## Next Steps
* Use the [Sandbox CLI Guide](./cli) for day-to-day commands.
* Build automation with the [Sandbox SDK Guide](./sdk).
* Explore runnable demos in [prime-cli/examples](https://github.com/PrimeIntellect-ai/prime-cli/tree/main/examples).
# Sandbox SDK Guide
Source: https://docs.primeintellect.ai/sandboxes/sdk
Automate sandbox lifecycles with the Python SDK
The SDK ships two clients with identical methods: `SandboxClient` for synchronous scripts and `AsyncSandboxClient` for concurrent workloads. Both are importable from `prime_sandboxes`.
## Sync Client
`SandboxClient` is the simplest way to get started — no `async`/`await` needed.
```python theme={null}
from prime_sandboxes import SandboxClient, CreateSandboxRequest, APIClient
client = SandboxClient(APIClient())
# Create and wait
sandbox = client.create(
CreateSandboxRequest(
name="sdk-demo",
docker_image="python:3.11-slim",
labels=["experiment"],
timeout_minutes=120,
environment_vars={"LOG_LEVEL": "debug"},
secrets={"API_KEY": "sk-abc123"},
)
)
client.wait_for_creation(sandbox.id)
# Run a command
result = client.execute_command(sandbox.id, "python -c 'print(42)'")
print(result.stdout.strip())
# Upload / download files
client.upload_file(sandbox.id, "/workspace/data.csv", "./data.csv")
client.download_file(sandbox.id, "/workspace/output.csv", "./output.csv")
# Expose a port
exposed = client.expose(sandbox.id, port=8000, name="web")
print(exposed.url)
# Clean up
client.delete(sandbox.id)
```
Every method shown in the async sections below has an identical synchronous counterpart on `SandboxClient`.
## Async Client
Most sandbox automations spin up more than one environment. The async client lets you fan out creates, waits, commands, and teardown without juggling threads.
```python theme={null}
import asyncio
from prime_sandboxes import AsyncSandboxClient, CreateSandboxRequest
async def launch_demo() -> None:
async with AsyncSandboxClient() as sandboxes:
request = CreateSandboxRequest(
name="sdk-demo",
docker_image="python:3.11-slim",
labels=["experiment", "ml-pipeline", "team-research"],
timeout_minutes=120,
)
sandbox = await sandboxes.create(request)
await sandboxes.wait_for_creation(sandbox.id)
result = await sandboxes.execute_command(sandbox.id, "python -c 'print(42)'")
print(result.stdout.strip())
await sandboxes.delete(sandbox.id)
asyncio.run(launch_demo())
```
## Launch a Fleet
```python theme={null}
async def create_many(images: list[str]) -> None:
async with AsyncSandboxClient() as sandboxes:
requests = [
CreateSandboxRequest(name=f"batch-{i}", docker_image=image)
for i, image in enumerate(images, start=1)
]
created = await asyncio.gather(*[sandboxes.create(req) for req in requests])
await sandboxes.bulk_wait_for_creation([sbx.id for sbx in created])
print("Ready:", ", ".join(sbx.name for sbx in created))
# asyncio.run(create_many(["python:3.11-slim", "node:20-slim"]))
```
`bulk_wait_for_creation` polls via the list endpoint, backing off automatically if the API throttles you.
## Run Commands & Collect Logs
```python theme={null}
async def smoke_test(sandbox_id: str) -> None:
async with AsyncSandboxClient() as sandboxes:
results = await sandboxes.execute_command(
sandbox_id,
"python -c 'import platform; print(platform.python_version())'",
)
print("stdout:", results.stdout.strip())
logs = await sandboxes.get_logs(sandbox_id)
print("logs snippet:", logs[:120])
```
Command responses include stdout, stderr, and exit code so you can short-circuit pipelines when something breaks.
## Move Data In and Out
```python theme={null}
async def sync_artifacts(sandbox_id: str) -> None:
async with AsyncSandboxClient() as sandboxes:
await sandboxes.upload_file(sandbox_id, "/workspace/model.bin", "./artifacts/model.bin")
await sandboxes.download_file(sandbox_id, "/workspace/report.csv", "./reports/report.csv")
```
**Note:** File uploads are limited to 200MB per file.
Uploads/downloads use short-lived gateway tokens stored in a local cache. Call `sandboxes.clear_auth_cache()` if you rotate credentials or hit 401s.
## Expose Ports
Make services inside your sandbox accessible over the internet. Both HTTP and TCP protocols are supported.
Ports must be in the range **22–9000**. Ports **8080**, **2222**, and **8081** cannot be exposed.
### HTTP
Expose an HTTP service and get a public HTTPS URL:
```python theme={null}
async def run_web_server() -> None:
async with AsyncSandboxClient() as sandboxes:
sandbox = await sandboxes.create(
CreateSandboxRequest(name="web-server", docker_image="python:3.11-slim")
)
await sandboxes.wait_for_creation(sandbox.id)
# Start the server as a background job so it keeps running
await sandboxes.start_background_job(
sandbox.id, "python -m http.server 8000 --bind 0.0.0.0"
)
exposed = await sandboxes.expose(sandbox.id, port=8000, name="web-server")
await asyncio.sleep(10)
print(f"Server available at: {exposed.url}")
```
### TCP
Expose a raw TCP service and get a public `host:port` endpoint:
```python theme={null}
async def run_tcp_server() -> None:
async with AsyncSandboxClient() as sandboxes:
sandbox = await sandboxes.create(
CreateSandboxRequest(name="tcp-server", docker_image="python:3.11-slim")
)
await sandboxes.wait_for_creation(sandbox.id)
# Start a TCP echo server as a background job
await sandboxes.start_background_job(
sandbox.id,
"python -c \"import socketserver; "
"socketserver.TCPServer(('0.0.0.0', 9000), socketserver.StreamRequestHandler).serve_forever()\"",
)
exposed = await sandboxes.expose(sandbox.id, port=9000, name="tcp-server", protocol="TCP")
print(f"TCP endpoint: {exposed.external_endpoint}")
print(f"External port: {exposed.external_port}")
```
TCP exposures return an `external_endpoint` (host:port) and `external_port` instead of a URL. Connect using any TCP client, for example Python's `socket.create_connection()`.
## Start Command
By default, sandboxes run `tail -f /dev/null` to keep the container alive for interactive use. Pass `start_command` to override the image's `ENTRYPOINT` with your own process:
```python theme={null}
sandbox = await sandboxes.create(
CreateSandboxRequest(
name="api-server",
docker_image="python:3.11-slim",
start_command="python serve.py --port 8000",
)
)
```
If you omit `start_command`, the default keeps the sandbox idle and ready for `execute_command` calls.
## Environment Variables & Secrets
Pass configuration and credentials when creating a sandbox:
```python theme={null}
sandbox = await sandboxes.create(
CreateSandboxRequest(
name="configured-runner",
docker_image="python:3.11-slim",
environment_vars={
"APP_ENV": "staging",
"LOG_LEVEL": "debug",
},
secrets={
"DB_PASSWORD": "hunter2",
"API_KEY": "sk-abc123",
},
)
)
```
**Environment variables** are stored in plain text. **Secrets** are encrypted at rest and never returned in API responses — use them for API keys, passwords, and other sensitive values. Both are injected into the container as standard environment variables.
You can also pass per-command environment variables to `execute_command`:
```python theme={null}
result = await sandboxes.execute_command(
sandbox.id,
"echo $CUSTOM_VAR",
env={"CUSTOM_VAR": "hello"},
)
```
Use `user` to run a command as a specific user, like `docker exec -u`. It accepts a
username or numeric UID, optionally with a group (`"agent"`, `"1000"`, `"agent:agent"`,
`"1000:1000"`). The same `user` argument works on `start_background_job`.
```python theme={null}
result = await sandboxes.execute_command(
sandbox.id,
"whoami",
user="agent",
)
```
## Network Isolation
For running untrusted code, create sandboxes without internet access:
```python theme={null}
async def isolated_sandbox() -> None:
async with AsyncSandboxClient() as sandboxes:
# Create a sandbox without outbound internet access
request = CreateSandboxRequest(
name="isolated-runner",
docker_image="python:3.11-slim",
network_access=False, # Disable outbound internet
)
sandbox = await sandboxes.create(request)
await sandboxes.wait_for_creation(sandbox.id)
# Code runs in isolation - no external network calls possible
result = await sandboxes.execute_command(
sandbox.id,
"python -c 'import urllib.request; urllib.request.urlopen(\"https://example.com\")'",
)
# This will fail with a network error
await sandboxes.delete(sandbox.id)
```
When `network_access=False`:
* Outbound connections to the internet are blocked
* DNS resolution for internal services still works
By default, `network_access=True` and sandboxes have full internet access.
## Long-Running Commands
Commands can run up to 15 minutes using the `timeout` parameter:
```python theme={null}
async def run_long_command(sandbox_id: str) -> None:
async with AsyncSandboxClient() as sandboxes:
# Run a command that takes up to 15 minutes
result = await sandboxes.execute_command(
sandbox_id,
"python preprocessing.py --dataset large",
timeout=900, # 15 minutes max
)
print(f"Exit code: {result.exit_code}")
```
For tasks longer than 15 minutes, use background jobs instead. They're more
reliable and won't tie up your connection.
## Background Jobs
Use `start_background_job` for tasks that run longer than 15 minutes. The job continues running in the sandbox while you poll for completion.
```python theme={null}
async def run_training_job() -> None:
async with AsyncSandboxClient() as sandboxes:
sandbox = await sandboxes.create(
CreateSandboxRequest(
name="training-job",
docker_image="python:3.11-slim",
timeout_minutes=1440, # 24 hours
cpu_cores=4,
memory_gb=16,
)
)
await sandboxes.wait_for_creation(sandbox.id)
# Start a long-running job in the background
job = await sandboxes.start_background_job(
sandbox.id,
"python train.py --epochs 100"
)
print(f"Job started: {job.job_id}")
# Poll for completion
while True:
status = await sandboxes.get_background_job(sandbox.id, job)
if status.completed:
print(f"Exit code: {status.exit_code}")
print(status.stdout)
break
print("Still running...")
await asyncio.sleep(30)
# Download results
await sandboxes.download_file(sandbox.id, "/app/model.pt", "./model.pt")
await sandboxes.delete(sandbox.id)
```
The `timeout_minutes` parameter controls how long the sandbox stays alive. Background jobs persist across API calls until completion or sandbox termination.
## Idle Timeout
Set `idle_timeout_minutes` on `CreateSandboxRequest` (or `UpdateSandboxRequest`) to have the sandbox terminate itself when nothing has touched it for a while:
```python theme={null}
from prime_sandboxes import SandboxClient, CreateSandboxRequest, APIClient
client = SandboxClient(APIClient())
sandbox = client.create(
CreateSandboxRequest(
name="idle-aware",
docker_image="python:3.11-slim",
timeout_minutes=240,
idle_timeout_minutes=15,
)
)
```
Any `execute_command`, `upload_file`, `download_file`, or file-read call resets the idle clock. Long-running execs stay pinned for their full duration.
* Disabled by default — omit the field to keep the legacy lifetime-only behavior.
* Validated client-side: `1 ≤ idle_timeout_minutes ≤ timeout_minutes` and `idle_timeout_minutes ≤ 1440`.
* Not supported for VM-backed sandboxes.
* SSH sessions are **not** counted as activity yet.
When a sandbox shuts down for this reason, the response object exposes `termination_reason="idle_timeout"`.
## Error Handling
The SDK raises typed exceptions so you can handle specific failure modes. All exceptions are importable directly from `prime_sandboxes`.
```python theme={null}
from prime_sandboxes import (
# Sandbox lifecycle errors
SandboxNotRunningError,
SandboxOOMError,
SandboxTimeoutError,
SandboxImagePullError,
# Operation errors
CommandTimeoutError,
UploadTimeoutError,
DownloadTimeoutError,
# API errors
APIError,
UnauthorizedError,
PaymentRequiredError,
APITimeoutError,
)
```
### Sandbox Lifecycle Errors
These are raised when a sandbox is no longer in RUNNING state. They form a hierarchy — catch the base class for broad handling, or specific subclasses for targeted recovery.
| Exception | Cause | Typical Fix |
| ------------------------ | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `SandboxNotRunningError` | Operation attempted on a non-running sandbox (terminated, errored, or timed out) | Check sandbox status before operating on it |
| `SandboxOOMError` | Sandbox killed due to out-of-memory | Increase `memory_gb` in `CreateSandboxRequest` or optimize memory usage |
| `SandboxTimeoutError` | Sandbox exceeded its `timeout_minutes` and was terminated | Increase the timeout or split work into smaller tasks |
| `SandboxImagePullError` | Docker image could not be pulled | Verify image name, tag, and registry credentials |
`SandboxOOMError`, `SandboxTimeoutError`, and `SandboxImagePullError` are all subclasses of `SandboxNotRunningError`.
### Operation Errors
Raised during specific operations when the sandbox is still running but the operation itself fails.
| Exception | Cause | Typical Fix |
| ---------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------ |
| `CommandTimeoutError` | `execute_command` exceeded its `timeout` parameter | Increase the per-command timeout, or use `start_background_job` for long tasks |
| `UploadTimeoutError` | File upload timed out | Check file size (200MB limit) and network conditions |
| `DownloadTimeoutError` | File download timed out | Check file size and network conditions |
### API Errors
Raised for HTTP-level failures when communicating with the platform API.
| Exception | Cause | Typical Fix |
| ---------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------- |
| `APIError` | Base class for all API errors (non-2xx response, malformed response, network failure) | Inspect the error message for details |
| `UnauthorizedError` | Invalid or expired API key (HTTP 401) | Check `PRIME_API_KEY` or re-run `prime login` |
| `PaymentRequiredError` | Insufficient balance (HTTP 402) | Top up your account balance |
| `APITimeoutError` | API request timed out before receiving a response | Retry the request; check network connectivity |
## Clean Exit
Delete a single sandbox or use `bulk_delete` to tear down many at once by IDs or labels:
```python theme={null}
async def teardown() -> None:
async with AsyncSandboxClient() as sandboxes:
# Delete one sandbox
await sandboxes.delete("sbx_123")
# Bulk delete by IDs
result = await sandboxes.bulk_delete(sandbox_ids=["sbx_456", "sbx_789"])
print(f"Deleted: {result.succeeded}, Failed: {result.failed}")
# Bulk delete by labels — removes all sandboxes matching ALL given labels
result = await sandboxes.bulk_delete(labels=["experiment", "staging"])
print(f"Deleted: {result.succeeded}, Failed: {result.failed}")
```
You must pass either `sandbox_ids` or `labels`, not both.
For a full script, see `prime-cli/examples/sandbox_async_demo.py`, which covers create → wait → run → logs → delete.
# Prime Tunnel
Source: https://docs.primeintellect.ai/sandboxes/tunnel
Expose local services to the internet through secure reverse proxy
## What is Prime Tunnel?
Prime Tunnel creates secure, public URLs for services running on your local machine. You start a local server, create a tunnel, and get an HTTPS URL that anyone can access.
**Common use cases:**
* Expose a local development server to the internet
* Receive webhooks from external services during development
* Share a work-in-progress app with teammates
Hosted evaluations can use Prime Tunnel too. Launch the run with
`--allow-tunnel-access`, then create the tunnel from inside the hosted
sandbox using the same API or CLI flow described here. The flag only grants
tunnel permissions to the temporary `PRIME_API_KEY`. See
[Hosted Evaluations](/tutorials-environments/hosted-evaluations).
## Complete Example
Here's a full working example: start a local HTTP server and expose it to the internet.
**1. Start a local server**
```bash theme={null}
# Create a simple HTML file
echo '
Hello from Prime Tunnel!
' > index.html
# Start Python's built-in HTTP server on port 8000
python -m http.server 8000
```
**2. Create a tunnel (in another terminal)**
```python Python SDK theme={null}
import asyncio
from prime_tunnel import Tunnel
async def main():
tunnel = Tunnel(local_port=8000)
await tunnel.start()
print(f"Public URL: {tunnel.url}")
print("Press Ctrl+C to stop")
# Keep running until interrupted
try:
while True:
await asyncio.sleep(1)
except KeyboardInterrupt:
pass
finally:
await tunnel.stop()
asyncio.run(main())
```
```bash CLI theme={null}
prime tunnel start --port 8000
```
**3. Access your server**
Open the printed URL (e.g., `https://t-0-abc123def456.tunnel.pinfra.io`) in any browser. Your local server is now accessible from anywhere.
## Context Manager
For automatic cleanup, use the async context manager:
```python theme={null}
async with Tunnel(local_port=8000) as tunnel:
print(f"URL: {tunnel.url}")
# Tunnel is active here
# Tunnel automatically stopped
```
## Tunnel Properties
```python theme={null}
tunnel = Tunnel(local_port=8000)
await tunnel.start()
tunnel.tunnel_id # "t-0-abc123def456"
tunnel.url # "https://t-0-abc123def456.tunnel.pinfra.io"
```
## Protecting a Tunnel with Basic Auth
By default, anyone with the URL can reach your tunnel. To require credentials, provide a username when creating the tunnel — a strong password is **auto-generated server-side** and shown exactly once. All public traffic must then include HTTP basic auth; requests without valid credentials get a `401 Unauthorized`.
```python Python SDK theme={null}
tunnel = Tunnel(local_port=8000, http_user="alice")
await tunnel.start()
print(tunnel.url)
print(tunnel.http_password) # auto-generated
```
```bash CLI theme={null}
prime tunnel start --port 8000 --auth alice
```
The CLI prints the generated password when the tunnel starts:
```
Tunnel started successfully!
URL: https://t-0-abc123def456.tunnel.pinfra.io
Tunnel ID: t-0-abc123def456
Basic auth user: alice
Basic auth password: vBI6ioyAM3_X4KWaEug4TPSbzovK8eYb
Save this password - it is shown only once and cannot be retrieved later
```
Browsers prompt for the credentials automatically. For programmatic access, send the `Authorization` header:
```bash theme={null}
curl -u alice:vBI6ioyAM3_X4KWaEug4TPSbzovK8eYb https://t-0-abc123def456.tunnel.pinfra.io
```
## CLI Commands
```bash theme={null}
# Start a tunnel
prime tunnel start --port 8000
# Start a tunnel protected by HTTP basic auth (password is auto-generated)
prime tunnel start --port 8000 --auth alice
# List active tunnels
prime tunnel list
# Stop a tunnel
prime tunnel stop t-0-abc123def456
```
## Rate Limits
The tunnel API enforces per-user rate limits over a 60-second window.
| Endpoint | Method | Limit |
| --------------------- | -------- | ----------------------------- |
| `/api/v1/tunnel` | `POST` | 600 requests / 60s |
| `/api/v1/tunnel` | `GET` | 300 requests / 60s |
| `/api/v1/tunnel/{id}` | `GET` | 300 requests / 60s per tunnel |
| `/api/v1/tunnel/{id}` | `DELETE` | 100 requests / 60s |
| `/api/v1/tunnel` | `DELETE` | 100 requests / 60s (bulk) |
Requests over the limit return `429 Too Many Requests`. There is also a per-account cap on the number of concurrent tunnels (`tunnel_limit` on the wallet) - delete unused tunnels before creating new ones.
# Create & Upload Environment
Source: https://docs.primeintellect.ai/tutorials-environments/create
Learn how to create and upload environments to Prime Intellect's Environments Hub
## Prerequisites
Ensure you have:
1. **Prime CLI** installed and configured
* See [CLI Overview](/cli-reference/introduction) for setup instructions.
2. **Username** set on your [profile](https://app.primeintellect.ai/dashboard/profile)
3. **Authenticate** the CLI:
```bash theme={null}
prime login
```
## Creating a New Environment
### Initialize Environment
Create a new environment with our starter template:
```bash theme={null}
prime env init
```
This creates a template for a Python module with:
* A [README.md](http://README.md) file (displayed on the Environments Hub)
* A `pyproject.toml` file for managing dependencies, versioning, tags, description, etc.
* A Python file containing stub code for a `load_environment` function which returns a `vf.Environment` object — this will be the entrypoint for downstream applications to use your Environment, and should be used encapsulate any necessary preprocessing, resource provisioning, exposing configurable args, etc.
### Develop Your Environment
After initialization, you can modify and test your environment.
If your environment needs API keys or credentials, see [Secrets](/tutorials-environments/secrets) for the recommended setup.
To install your environment locally, you can run:
```bash theme={null}
uv pip install -e .
```
To test/evaluate the environment:
```bash theme={null}
uv run vf-eval my-environment
```
Make sure to follow the [verifiers library patterns](https://github.com/PrimeIntellect-ai/verifiers) when implementing your environment. Your environment should inherit from appropriate base classes and implement required methods.
### Upload Your Environment
Once you've developed and tested your environment, push it to the Environments Hub:
```bash theme={null}
# Inside the environments// directory
prime env push
```
This will upload your environment under your user account. You can also upload it to a team using:
```bash theme={null}
prime env push --team
```
Once uploaded, your environment will be available for installation by others using `prime env install owner/environment-name`, unless you use the `--visibility=PRIVATE` flag.
## URL Dependencies
If your environment depends on packages from Git repositories (not published to PyPI), you need to use the PEP 440/508 format directly in your `dependencies` list.
### Correct Format
```toml theme={null}
[project]
dependencies = [
"verifiers",
"tau2 @ git+https://github.com/sierra-research/tau2-bench.git",
]
```
### Incorrect Format
Do **not** use `[tool.uv.sources]` for URL dependencies that need to work with the Environments Hub:
```toml theme={null}
# ❌ This won't work - uv.sources is not embedded in wheel metadata
[tool.uv.sources]
tau2 = { git = "https://github.com/sierra-research/tau2-bench.git" }
```
The `[tool.uv.sources]` section is uv-specific and not included in the wheel package metadata. When users install your environment, the URL dependency information would be lost.
The Environments Hub automatically extracts URL dependencies from your wheel and passes them as direct requirements during installation. This works around uv's security restriction on transitive URL dependencies from registry packages.
## Version Management
### Updating Your Environment
When you make changes to your environment:
1. Update the version in `pyproject.toml`
2. Push the updated environment:
```bash theme={null}
prime env push
```
The system will automatically create a new version while keeping previous versions available.
You can use `prime env push --auto-bump` to automatically increment the version number for you, so you don't need to manually update `pyproject.toml` each time.
## Getting Help
* Check the [verifiers GitHub repository](https://github.com/PrimeIntellect-ai/verifiers) for examples
* Review the [verifiers documentation](/verifiers/overview) for detailed guides
* Join our [Discord community](https://discord.gg/ZTFydGWPKj) for support
# Environment Actions
Source: https://docs.primeintellect.ai/tutorials-environments/environment-actions
Quick look at the automated checks that run on every environment push
## What Happens
Every `prime env push` kicks off an automatic action. Prime builds the environment in a fresh container, installs your package, and runs its tests. The run is linked to that pushed version so teammates can see whether it passed before using it.
## Where To Check
* Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments) and switch to **Actions**.
* Each entry shows status, runtime, and logs. Click any run to inspect failures and rerun commands locally if needed.
## Current Limits
Actions always trigger on push and run the default test suite in the managed container. Custom triggers and advanced workflows are coming soon.
# Environment Variables
Source: https://docs.primeintellect.ai/tutorials-environments/environment-variables
Configure plain-text key-value pairs injected into your environment at runtime
Environment variables are plain-text key-value pairs injected into your environment at runtime. They are suited for non-sensitive configuration — things like model names, difficulty levels, dataset paths, or feature flags.
For sensitive values such as API keys, use [Secrets](/tutorials-environments/secrets) instead.
## Managing Variables
Variables are managed per-environment under the **Secrets** tab.
1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab and select **Variables**
3. Click **Add Variable**
4. Set a **Name**, **Value**, and optional description
Variables can be edited or deleted at any time by anyone with write access to the environment.
## Managing via CLI
```bash theme={null}
prime env var list owner/my-env # list all variables
prime env var create owner/my-env --name MY_VAR --value hello # create a variable
prime env var update owner/my-env --value new-value # update a variable
prime env var delete owner/my-env # delete a variable
```
## Naming Rules
Variable names follow env-var conventions:
* Must start with an uppercase letter
* Can contain uppercase letters, digits, and underscores only
* Example: `MODEL_NAME`, `DIFFICULTY`, `MAX_EXAMPLES`
## Where Variables Are Available
Variables are injected automatically for all hosted services:
* Environment Actions
* Hosted Evaluations
* Hosted Training
## Variables vs Secrets
| | Environment Variables | Secrets |
| -------------------------- | --------------------- | ------------------------- |
| **Stored as** | Plain text | Encrypted |
| **Visible after creation** | Yes | No |
| **Use for** | Non-sensitive config | API keys, credentials |
| **Scope** | Per-environment | Per-environment or global |
When both a variable and a secret share the same name, the secret takes priority. See [Secrets](/tutorials-environments/secrets) for the full precedence rules.
## Conflict Rules
A variable name cannot conflict with an existing linked secret on the same environment. If you try to create a variable with a name already used by a linked secret, the platform will reject it. Rename or unlink the secret first.
# Overview
Source: https://docs.primeintellect.ai/tutorials-environments/environments
Create, manage and share environments for reinforcement learning and evaluation
## About
Environments Hub is a community-powered platform for aggregating and showcasing environments, both for RL training and downstream evaluation.
You can view all available environments on the [Environments Hub](https://app.primeintellect.ai/dashboard/environments).
### Motivation
There are a few inter-related issues we see with the current ecosystem for both evals and RL environments which we're aiming to address with the Environments Hub:
* Despite the rapidly growing interest in training LLMs with RL, there is currently **no established community platform** for exploring and sharing train-ready environments.
* Environment implementations are often tied to a specific training RL stack and can be difficult to adapt to a new trainer.
* Popular evaluation suites (lm\_eval, lighteval, openbench, simple-evals, HELM) offer convenient entrypoints into many single-turn Q\&A evals, but these suites generally **lack support for tasks which are agentic in nature** or require complex infrastructure setups (TAU-bench, TerminalBench, SWE-bench), resulting in a proliferation of independent eval repos without shared entrypoints or specs.
* RL environments and agent evals are **basically the same thing** (dataset + harness + scoring rules), but current open-source efforts generally treat them as fundamentally separate.
* Realistic agent environments can be complex pieces of software requiring **dependencies and versioning**, and are ill-served by monorepo structures for environment collections which can quickly become unmaintainable.
With the Environments Hub, we’ve built a community platform that doubles as a proper Python package registry. Environments are modules which declare dependencies in a `pyproject.toml` and are distributed as wheels. By adopting the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) spec, development efforts can focus on task-specific components (datasets, tools or harnesses, reward functions) and automatically leverage existing infrastructure for running evaluations or [training models with RL](https://github.com/primeintellect-ai/prime-rl).
### Resources
* **Prime CLI**: Command-line tool to install, upload and manage environments
* Github: [https://github.com/PrimeIntellect-ai/prime-cli](https://github.com/PrimeIntellect-ai/prime-cli)
* Documentation: [CLI Overview](/cli-reference/introduction)
* **Verifiers**: A library of modular components for creating RL environments and training LLM agents.
* Github: [https://github.com/PrimeIntellect-ai/verifiers](https://github.com/PrimeIntellect-ai/verifiers)
* Documentation: [Verifiers Docs](/verifiers/overview)
* **Prime RL**: A library for large-scale RL training with FSDP.
* Github: [https://github.com/PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)
### Support
Join the [Prime Intellect discord](https://discord.gg/primeintellect) to discuss, share feedback, and ask any questions.
Build and upload custom environments to the registry
Install and use environments in your projects
# Evaluating Environments
Source: https://docs.primeintellect.ai/tutorials-environments/evaluating
Guide to running evaluations with Prime CLI using Prime Inference or custom model endpoints
The `prime eval` command provides powerful evaluation capabilities for testing environments against various language models through Prime Inference or other OpenAI-compatible providers. You can run evaluations locally or launch them as hosted evaluations on the platform with `--hosted`. This guide covers both workflows, model selection, and best practices.
## Quick Start: Running Your First Evaluation
### Prerequisites
1. **Python 3.10–3.13** — Required for the Prime CLI and verifiers
2. **Install Prime CLI** — Follow the [installation guide](/cli-reference/introduction)
3. **Set up API keys** — Configure your Prime API key via `prime login`; if you plan to use another provider, also export the provider key you will reference with `--api-key-var`
4. **Install an environment** — Use `prime env install owner/environment`
### Basic Evaluation
```bash theme={null}
# List available environments
prime env list --owner primeintellect
# Install an environment
prime env install primeintellect/gsm8k@latest
# Run a basic evaluation
prime eval gsm8k
```
Local `prime eval` runs are automatically uploaded to the platform after each run. Use `--skip-upload` to disable this.
### Hosted Evaluation Quick Start
If your environment is already published to the Environments Hub, you can run it remotely on Prime-managed infrastructure:
```bash theme={null}
prime eval run primeintellect/gsm8k --hosted
```
Use `--follow` to stream hosted logs until completion:
```bash theme={null}
prime eval run primeintellect/gsm8k --hosted --follow
```
See [Hosted Evaluations](/tutorials-environments/hosted-evaluations) for the full dashboard and CLI workflow.
## Using the prime eval Command
### Basic Syntax
```bash theme={null}
prime eval ENVIRONMENT [OPTIONS]
```
This is a shorthand for `prime eval run ENVIRONMENT`. Both forms work identically.
### Available Models
To see all available models for evaluation:
```bash theme={null}
prime inference models
```
**Example models:**
| Model | Notes |
| ------------------------------------ | ------------------------------ |
| `openai/gpt-4.1-mini` | Fast, cost-effective (default) |
| `openai/gpt-4.1` | Higher quality |
| `anthropic/claude-sonnet-4.5` | Strong reasoning |
| `meta-llama/llama-3.3-70b-instruct` | Open-weight, balanced |
| `deepseek/deepseek-r1-0528` | Advanced reasoning |
| `qwen/qwen3-235b-a22b-instruct-2507` | Large MoE model |
| `google/gemini-2.5-flash` | Fast multimodal |
Model availability and pricing may change. Always run `prime inference models` to get the current list with pricing.
### Core Parameters
Environment to evaluate. Supported forms:
* **Full slug** (e.g., `primeintellect/gsm8k`) — recommended for hosted runs and Hub environments
* **Short name** (e.g., `gsm8k`) — local-first resolution for installed environments
* **TOML config path** (e.g., `configs/eval/gsm8k.toml`) — for config-driven runs
Model to use for evaluation. Default: `openai/gpt-4.1-mini`
See `prime inference models` for all available models.
Number of examples to evaluate. Default: 5
Number of rollouts per example for statistical significance. Default: 3
### Advanced Options
Maximum concurrent requests to the inference API. Default: 32
Maximum number of automatic retries with exponential backoff when rollouts fail due to transient infrastructure errors (e.g., sandbox timeouts, API failures).
Maximum tokens to generate per request. If unset, uses model default.
Sampling temperature (0.0–2.0). Higher values = more randomness.
JSON string with additional sampling arguments.
Example: `'{"enable_thinking": false, "max_tokens": 256}'`
Environment-specific arguments as JSON.
Example: `'{"difficulty": "hard"}'`
### Output and Storage Options
Enable verbose output for detailed logging.
Save evaluation results to disk. Default: true
Save dataset every N rollouts. Useful for checkpointing. Default: 1
Save results to Hugging Face Hub.
Specify Hugging Face Hub dataset name.
Skip uploading results for local evaluations (local runs upload by default).
### Hosted Evaluation Options
Run the evaluation on the platform instead of locally. Requires a published environment.
Follow hosted evaluation status and stream logs until completion. Only valid with `--hosted`.
Polling interval in seconds for hosted status and log streaming. Only valid with `--hosted`.
Optional timeout in minutes for a hosted evaluation. Default: 1440 (24 hours). Min: 120. Max: 1440.
Allow sandbox read/write access for hosted evaluations.
Allow hosted evaluations to create and manage instances.
Allow hosted evaluations to create and manage tunnels from inside the
sandbox. This adds tunnel scopes to the temporary `PRIME_API_KEY`.
JSON object of additional secrets to inject into a hosted run.
Custom display name for a hosted evaluation.
`--api-base-url` and `--api-key-var` from the model configuration section also work with `--hosted`. When you use a custom endpoint for a hosted run, provide the referenced API key inside the remote sandbox via an environment secret or `--custom-secrets`.
## End-to-End Example
Here's a complete workflow from installation to viewing results:
```bash theme={null}
# 1. Install an environment
prime env install primeintellect/gsm8k@latest
# 2. Run evaluation with a fast model (5 examples, 1 rollout each)
prime eval gsm8k -m openai/gpt-4.1-mini -n 20 -r 2
# 3. List your evaluation runs
prime eval list
# 4. Get details of a specific evaluation
prime eval get
# 5. View samples from an evaluation
prime eval samples
```
**Example output:**
```
Running evaluation: gsm8k
Model: openai/gpt-4.1-mini
Examples: 5 | Rollouts: 1 | Concurrency: 32
Progress: 100%|████████████████████████████| 5/5 [00:12<00:00, 2.45s/it]
Results:
Average Score: 0.80
Total Samples: 5
Successful: 4
Failed: 1
Results saved to: outputs/evals/gsm8k--openai-gpt-4.1-mini/...
Uploaded to platform: https://app.primeintellect.ai/...
```
## Using TOML Configs for Multi-Environment Evals
For reproducible evals, you can pass a TOML config file instead of individual CLI flags:
```bash theme={null}
prime eval run configs/eval/my-benchmark.toml
```
Local TOML configs can define one or more `[[eval]]` entries, which makes them useful for benchmark suites and multi-environment comparisons:
```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20
[[eval]]
env_id = "primeintellect/gsm8k"
rollouts_per_example = 2
[[eval]]
env_id = "primeintellect/alphabet-sort"
rollouts_per_example = 2
```
When you use a TOML config, per-eval settings in `[[eval]]` override global defaults at the top of the file.
For the full config schema, precedence rules, and advanced options like ablations and endpoint registries, see [Verifiers Evaluation](/verifiers/evaluation).
## Hosted Evaluations from the CLI
Hosted eval runs are useful when you want the platform to execute the environment remotely and keep logs on the platform.
```bash theme={null}
prime eval run primeintellect/gsm8k \
--hosted \
```
Hosted runs can also target a custom OpenAI-compatible endpoint:
```bash theme={null}
prime eval run primeintellect/gsm8k \
--hosted \
-m openai/gpt-4.1-mini \
--api-base-url https://api.openai.com/v1 \
--api-key-var OPENAI_API_KEY \
--custom-secrets '{"OPENAI_API_KEY":"..."}'
```
When `--api-base-url` is set on a hosted run, Prime still hosts the evaluation sandbox, but model billing comes from your external provider instead of Prime Inference. See [Hosted Evaluations](/tutorials-environments/hosted-evaluations) for the full hosted billing details.
You can also launch a hosted run from a TOML file:
```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20
rollouts_per_example = 2
[[eval]]
env_id = "primeintellect/gsm8k"
env_args = { split = "test" }
```
```bash theme={null}
prime eval run configs/eval/gsm8k-hosted.toml --hosted
```
Hosted run management commands:
```bash theme={null}
prime eval logs -f
prime eval stop
```
## Managing Evaluation Results
### List Evaluations
```bash theme={null}
# List all your evaluations
prime eval list
# Filter by environment
prime eval list --env gsm8k
# Output as JSON
prime eval list --output json
# Paginate results
prime eval list --num 20 --page 2
```
### Get Evaluation Details
```bash theme={null}
# Get full details of an evaluation
prime eval get
# Pretty-print output
prime eval get --output pretty
```
### View Samples
```bash theme={null}
# Get samples from an evaluation
prime eval samples
# Paginate samples
prime eval samples --page 2 --num 50
```
### Push Local Results
If you ran evaluations offline or with `--skip-upload`, you can push results later:
```bash theme={null}
# Auto-discover and push from outputs/evals/
prime eval push
# Push a specific directory
prime eval push outputs/evals/gsm8k--gpt-4/abc123
# Push with environment context
prime eval push --env gsm8k
```
## Model Selection Guide
When choosing models for evaluation, consider:
* **Task complexity** — Harder tasks benefit from larger, reasoning-capable models
* **Cost** — Smaller models are significantly cheaper for large-scale evals
* **Throughput** — Some models handle high concurrency better than others
### Self-hosting with vLLM
For most users, we recommend Prime Inference for easier setup. Consider self-hosting only for specialized requirements or very large-scale evaluations.
Self-hosting makes sense when you:
* Need specific model variants or custom fine-tuned models
* Require maximum cost efficiency for very large evaluations (1M+ examples)
* Are testing smaller models not available via API
#### Configuring for Self-Hosted Models
```bash theme={null}
# Point to your vLLM instance
prime eval gsm8k \
-m Qwen/Qwen3-4B-Instruct-2507 \
--api-base-url http://localhost:8000/v1 \
--api-key-var CUSTOM_API_KEY \
```
#### Recommended Self-Hosted Models
**High Performance (MoE with small active parameters):**
```bash theme={null}
prime eval gsm8k \
-m Qwen/Qwen3-30B-A3B-Instruct-2507 \
--api-base-url http://localhost:8000/v1 \
-n 2000 -c 96
```
**Balanced Performance:**
```bash theme={null}
prime eval wordle \
-m Qwen/Qwen3-4B-Instruct-2507 \
--api-base-url http://localhost:8000/v1 \
-n 5000 -c 128
```
#### vLLM Configuration Example
```bash theme={null}
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen3-4B-Instruct-2507 \
--max-model-len 32768 \
--max-num-seqs 256 \
--tensor-parallel-size 2 \
--quantization fp8
```
## Environment-Specific Dependencies
Some environments may require additional dependencies beyond the base installation. Check the environment's documentation or use `prime env info` to see requirements:
```bash theme={null}
prime env info primeintellect/math_python
```
If an environment needs specific packages (e.g., sympy for math verification), install them before running evaluations.
## Troubleshooting
### Common Issues
**Error:** "Rate limit exceeded" or 429 responses
**Solution:** Reduce concurrency
```bash theme={null}
prime eval gsm8k -n 100 -c 8
```
**Error:** "Insufficient balance" or payment required
**Solution:** Add funds to your Prime Intellect account or use cheaper models
```bash theme={null}
prime eval gsm8k -m openai/gpt-4.1-mini -n 1000
```
**Error:** Environment not found or installation failures
**Solution:** Verify environment exists and reinstall
```bash theme={null}
prime env list --owner primeintellect
prime env uninstall gsm8k
prime env install primeintellect/gsm8k@latest
```
**Error:** Installation failures or import errors
**Solution:** Ensure you're using Python 3.10–3.13
```bash theme={null}
python --version # Should be 3.10, 3.11, 3.12, or 3.13
```
### Performance Tips
1. **Start Small:** Begin with `-n 5` to test your setup
2. **Monitor Costs:** Check token usage before large evaluations
3. **Use Appropriate Models:** Match model capability to task complexity
4. **Optimize Concurrency:** Balance speed vs. rate limits (default 32 is usually good)
5. **Save Results:** Results auto-save and upload by default
### Integration with Other APIs
You can use other OpenAI-compatible API providers:
```bash theme={null}
# DeepSeek API
prime eval math \
-m deepseek-reasoner \
--api-base-url https://api.deepseek.com/v1 \
--api-key-var DEEPSEEK_API_KEY
# OpenRouter API
prime eval gsm8k \
-m meta-llama/llama-3.1-405b-instruct \
--api-base-url https://openrouter.ai/api/v1 \
--api-key-var OPENROUTER_API_KEY
```
## Next Steps
Learn more about creating and managing environments
Detailed API documentation for inference endpoints
Complete CLI command reference
Build your own evaluation environments
# Getting Started
Source: https://docs.primeintellect.ai/tutorials-environments/getting-started
Create, manage and share environments for reinforcement learning and evaluation
To interact with environments on the Environments Hub, you'll need to set up your account on Prime Intellect, and install the Prime CLI.
### Prerequisites
Ensure you have:
1. **Prime CLI** installed and configured
* See [CLI Overview](/cli-reference/introduction) for setup instructions.
2. **Username** set on your [profile](https://app.primeintellect.ai/dashboard/profile)
3. **Authenticate** the CLI:
```bash theme={null}
prime login
```
### Quick Overview
```bash theme={null}
# List available environments
prime env list
# Get information about an environment
prime env info owner/environment-name
# Install an environment
prime env install owner/environment-name
# Create a new environment
prime env init my-new-environment
```
Build and upload custom environments to the registry
Install and use environments in your projects
# Hosted Evaluations
Source: https://docs.primeintellect.ai/tutorials-environments/hosted-evaluations
Run evaluations on the platform from the dashboard or the Prime CLI
Hosted Evaluations run your environment on Prime-managed infrastructure and store the run in Prime Evals. You can launch them either from the Environments Hub UI or directly from the CLI with `prime eval run --hosted`.
## What hosted evaluations are for
Use hosted evaluations when you want Prime to handle the execution environment for you:
* Run a published environment without setting up local Python dependencies
* Evaluate large jobs against a Hub environment slug
* Monitor logs remotely and share runs through the platform
* Grant temporary sandbox, instance, or tunnel permissions for tool-using environments
Hosted evaluations require an environment that is already published to the Environments Hub. If you only have a local environment, push it first with `prime env push`.
## Prerequisites
Before running a hosted evaluation, make sure you have:
1. **A published environment** on the Environments Hub
```bash theme={null}
prime env push
```
2. **Write access** to that environment
3. **Prime CLI installed and authenticated** if you plan to use the CLI flow
4. **Billing configured for your chosen inference path**
* Prime account balance if you are using Prime Inference
* Or an external provider API key if you are using `--api-base-url` with a custom OpenAI-compatible endpoint
## Quick start with the CLI
The new hosted eval flow is built into `prime eval run`.
```bash theme={null}
prime eval run primeintellect/gsm8k --hosted
```
This creates a hosted run on the platform instead of executing the evaluation locally.
### Follow logs until completion
```bash theme={null}
prime eval run primeintellect/gsm8k --hosted --follow
```
With `--follow`, the CLI keeps polling the run, streams hosted logs, and exits when the evaluation reaches a terminal state.
### Run from a TOML config
Hosted evals also support TOML configs.
```toml theme={null}
model = "openai/gpt-4.1-mini"
num_examples = 20
rollouts_per_example = 2
[[eval]]
env_id = "primeintellect/gsm8k"
env_args = { split = "test" }
[[eval]]
env_id = "primeintellect/alphabet-sort"
```
Run it with:
```bash theme={null}
prime eval run configs/eval/benchmark-hosted.toml --hosted
```
### Use a custom OpenAI-compatible endpoint
Hosted evaluations can also run on Prime-managed infrastructure while sending model requests to your own OpenAI-compatible endpoint.
```bash theme={null}
prime eval run primeintellect/gsm8k \
--hosted \
-m openai/gpt-4.1-mini \
--api-base-url https://api.openai.com/v1 \
--api-key-var OPENAI_API_KEY \
--custom-secrets '{"OPENAI_API_KEY":"..."}' \
--follow
```
Use `--api-key-var` to name the environment variable that contains your provider key inside the hosted sandbox. Then provide that secret either through the environment's stored secrets or with `--custom-secrets` for a one-off run.
`--api-base-url` only changes the inference endpoint. The environment still runs inside a Prime-hosted sandbox.
## Hosted-only CLI options
These flags only apply when you pass `--hosted`:
| Flag | Description |
| -------------------------- | ---------------------------------------------------------------------------------------------- |
| `--follow` | Stream hosted logs and wait for completion |
| `--poll-interval` | Polling interval for hosted status/log streaming |
| `--timeout-minutes` | Optional timeout in minutes for the hosted run. Default: 1440 (24 hours). Min: 120. Max: 1440. |
| `--allow-sandbox-access` | Allow sandbox read/write access |
| `--allow-instances-access` | Allow instance creation and management |
| `--allow-tunnel-access` | Allow hosted evaluations to create and manage tunnels from inside the sandbox |
| `--custom-secrets` | JSON object of secrets injected for the hosted run |
| `--eval-name` | Custom display name for the hosted evaluation |
### Example: Environment Args and Custom Secrets
```bash theme={null}
prime eval run my-team/browser-agent \
--hosted \
-m anthropic/claude-sonnet-4.5 \
-a '{"task":"checkout"}' \
--custom-secrets '{"SHOP_API_KEY":"..."}' \
--allow-sandbox-access \
--timeout-minutes 45
```
Use `--custom-secrets` for run-specific values. Secrets already configured on the environment continue to work as usual.
Hosted eval `env_args` are passed to `load_environment()`, similar to training `[[env]].args`. Use them for custom environment settings such as `split`, `difficulty`, tool configuration, or other environment-specific overrides supported by your environment.
## Monitoring and managing hosted runs
After starting a hosted evaluation, the CLI prints the evaluation id and platform URL.
`--follow` only streams logs and waits for completion. Hosted evaluations do not yet have a training-style checkpoint or restart workflow.
### List evaluations
```bash theme={null}
prime eval list
```
The list output includes a **Type** column so you can distinguish `HOSTED` and `LOCAL` evaluations.
### Inspect one run
```bash theme={null}
prime eval get
prime eval samples
```
### Stream logs for an existing hosted run
```bash theme={null}
prime eval logs -f
```
### Stop a running hosted evaluation
```bash theme={null}
prime eval stop
```
## Running a hosted evaluation from the dashboard
You can still launch the same workflow from the Environments Hub UI.
### Step 1: Open the environment
1. Go to the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Open your environment
3. Go to the **Evaluations** tab
4. Click **Run Hosted Evaluation**
### Step 2: Choose a model
Select an inference model for the run.
### Step 3: Configure the run
Set the number of examples, rollouts per example, and any environment arguments.
If the environment needs to expose a service during the run, expand **Permissions** and enable **Allow tunnel access for this evaluation**.
Environment secrets linked in the Hub are exposed automatically during hosted evaluation runs. You only need `--custom-secrets` when launching a CLI run with additional per-run secrets.
### Step 4: Monitor progress
You will be redirected to the evaluations list where you can watch the run status.
### Step 5: Review results
Completed runs show aggregate metrics and per-sample outputs in Prime Evals.
## Failure modes
When a hosted evaluation fails, the platform surfaces the error message and logs.
Common causes:
1. **Environment code errors** — import failures, dependency issues, invalid verifier logic
2. **Missing permissions** — the run needs sandbox, instance, or tunnel access but those flags were not enabled
3. **Missing secrets** — environment-linked or custom secrets were not available
4. **Timeouts** — the run exceeded the configured or platform timeout
5. **Inference issues** — temporary provider or model errors
Start with a small hosted run first, such as `-n 5 -r 1`, then scale up once logs and scores look correct.
## Pricing
Hosted evaluations support two billing modes:
* **Prime Inference (default)** — If you do not pass `--api-base-url`, the run uses Prime Inference pricing for the selected model. The hosted evaluation sandbox runtime is not billed separately.
* **Custom OpenAI-compatible endpoint (CLI/API)** — If you pass `--api-base-url`, the hosted evaluation is billed as sandbox compute on Prime while your external provider bills model tokens directly. In this mode you must provide the provider key through an environment secret or `--custom-secrets`.
In either mode, total cost still depends on:
* The selected model or endpoint pricing
* Prompt and completion token usage
* `num_examples × rollouts_per_example`
* Any extra tool usage triggered by the environment
## When to use dashboard vs CLI
* Use the **dashboard** when you want the simplest point-and-click flow
* Use the **CLI** when you want reproducible commands, TOML configs, log following, or automation in scripts/CI
# Install & Use Environment
Source: https://docs.primeintellect.ai/tutorials-environments/install
Learn how to install and use environments from the Prime Intellect Environments Hub
### Basic Installation
Install an environment from the Environments Hub:
```bash theme={null}
prime env install /
```
### Examples
```bash theme={null}
# Install latest version
prime env install will/wordle
# Install specific version
prime env install will/wordle@0.1.3
```
### Alternative Installation Methods
Use `prime env info /` to see all available installation methods and usage examples for a specific environment.
E.g. for [will/wordle](https://app.primeintellect.ai/dashboard/environments/will/wordle):
```bash theme={null}
# Direct pip installation from wheel URL
pip install https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl
# UV pip installation
uv pip install https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl
# Add to existing project with UV
uv add wordle@https://hub.primeintellect.ai/will/wordle/@latest/wordle-0.1.4-py2.py3-none-any.whl
```
## Using Installed Environments
### In Python
Once installed, you can use the environment in your Python code:
```python theme={null}
from verifiers import load_environment
# Load the environment
env = load_environment("will/wordle")
# Use the environment for evaluation or training
results = env.evaluate(examples=100, rollouts_per_example=1)
```
More examples and detailed usage documentation can be found in the [verifiers documentation](https://github.com/PrimeIntellect-ai/verifiers).
## Version Management
### Working with Versions
Install specific versions:
```bash theme={null}
# Install specific version
prime env install will/wordle@0.1.3
# Install latest (default)
prime env install will/wordle
```
### Upgrading Environments
To upgrade an already installed environment, simply run `prime env install` again:
```bash theme={null}
# Upgrade to latest version
prime env install will/wordle
# Upgrade to specific version
prime env install will/wordle@0.2.0
```
## Download Source Code
If you want to inspect, modify, or contribute to an environment, you can download its source code:
```bash theme={null}
# Download the latest version
prime env pull owner/environment-name
```
**Example:**
```bash theme={null}
prime env pull will/wordle
```
## Getting Help
For additional support and examples:
* **Verifiers Documentation**: [Verifiers Docs](/verifiers/overview)
* **GitHub Repository**: [https://github.com/PrimeIntellect-ai/verifiers](https://github.com/PrimeIntellect-ai/verifiers)
* **Discord Community**: Join our [Discord](https://discord.gg/ZTFydGWPKj) for community support
# Manage Collaborators
Source: https://docs.primeintellect.ai/tutorials-environments/manage-collaborators
Add collaborators to private environments on the Environments Hub.
## Overview
You can now share private environments directly with collaborators. Collaborators have read-only access today and can browse metadata, artifacts, and version history while we finish support for writer permissions.
Access collaborator management from each environment's **Settings → Collaborators** section.
## Add a collaborator
1. Open the environment in the Environments Hub and switch to the **Collaborators** tab.
2. Select **Add collaborator**.
3. Enter a Prime username or team name. Each owner and team can find these values in their profile page.
4. Confirm to grant read-only access.
Once a collaborator is listed, they can run `prime env info ` in the Prime CLI to review environment details and pull instructions.
## Manage collaborator access
* Remove collaborators at any time from the same tab.
* Collaborators currently inherit read-only permissions; writer access is coming soon.
## Troubleshooting
* **Can't find a collaborator?** Ask them to copy the username or team name exactly as it appears in their profile.
* **Need to share broadly?** Create a team that includes everyone who needs access and add the team identifier once.
* **Seeing an error?** Reach out via our support chat with the environment slug and the identifier you attempted to add.
# Secrets
Source: https://docs.primeintellect.ai/tutorials-environments/secrets
Create and link secrets across your environments
Secrets are encrypted values injected into your environment at runtime. They are the recommended way to supply API keys and other credentials to environments.
For non-sensitive configuration, use [Environment Variables](/tutorials-environments/environment-variables) instead.
## Secret Types
There are two ways to add a secret to an environment:
1. **Direct environment secrets**
* Created directly on a specific environment
* Only available to that environment
2. **Linked global secrets**
* Created once under [**Keys & Secrets**](https://app.primeintellect.ai/dashboard/tokens?tab=secrets)
* Can be linked to one or more environments
* Useful for credentials shared across several environments, such as judge-model keys
## Where Secrets Are Used
Secrets are injected automatically for all hosted services:
* Environment Actions
* Hosted Evaluations
* Hosted Training
You do **not** need to pass them via `--env-args`.
## Adding a Direct Secret
1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab
3. Click **Add Secret**
4. Set a **Name**, **Value**, and optional description
## Creating and Linking a Global Secret
### 1) Create a global secret
1. Go to [**Dashboard → Keys & Secrets**](https://app.primeintellect.ai/dashboard/tokens?tab=secrets)
2. Click **Add Secret**
3. Set a **Name**, **Value**, and optional description
### 2) Link it to an environment
1. Open your environment in the [Environments Hub](https://app.primeintellect.ai/dashboard/environments)
2. Go to the **Secrets** tab
3. Click **Link Global Secret**
4. Select the secret you want to link
Once linked, the secret is available to that environment at runtime.
## Managing via CLI
### Global secrets
```bash theme={null}
prime secret list # list your global secrets
prime secret create --name MY_KEY --value sk-... # create a global secret
prime secret update --value sk-new # update value
prime secret delete # delete
```
### Environment secrets
```bash theme={null}
prime env secret list owner/my-env # list all secrets (direct + linked)
prime env secret create owner/my-env --name MY_KEY --value sk-... # add a direct secret
prime env secret update owner/my-env --id --value sk-new # update a direct secret
prime env secret delete owner/my-env --id # delete a direct secret
prime env secret link owner/my-env # link a global secret
prime env secret unlink owner/my-env # unlink a global secret
```
## Precedence and Conflict Rules
Each environment has three types of values injected at runtime:
| Type | Encrypted | Description |
| -------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------- |
| [**Environment variables**](/tutorials-environments/environment-variables) | No | Plain-text key-value pairs, configured on the **Variables** tab under **Secrets** |
| **Linked global secrets** | Yes | Encrypted values linked from your global secrets |
| **Direct environment secrets** | Yes | Encrypted values set directly on the environment |
### Runtime precedence
If the same name appears in more than one place, the highest-priority value wins:
1. **Environment variables** (lowest)
2. **Linked global secrets**
3. **Direct environment secrets** (highest)
### Name validation
All three types share the same naming format:
* Must start with an uppercase letter
* Can contain uppercase letters, digits, and underscores only
* Example: `MY_API_KEY`
### Conflict checks
The platform prevents name collisions. When adding or linking a secret, it will be rejected if the name is already taken by:
* An existing direct secret
* Another linked secret
* An environment variable
## Scope and Access
* **Personal secrets** can be linked to your personal environments
* **Team secrets** can be linked to environments within the same team
## Troubleshooting
Make sure the secret exists in the same scope (personal vs team) as the environment.
Rename or remove the existing direct secret first, then link the global secret.
Rename or remove that environment variable before adding or linking the secret.
Confirm the secret appears on the environment **Secrets** tab and that your environment code reads the correct variable name.
# Deploy Multi-Node Cluster
Source: https://docs.primeintellect.ai/tutorials-multi-node-cluster/deploy-multi-node
Deploy a multi-node cluster on the Prime Intellect Platform.
You can spin up up to 64+ Multi-Node H100 GPUs on Prime Intellect On-demand.
# Step-by-Step Guide
Multi-Node Cluster
You can use these public IPs to SSH into your nodes and start running your multi-node use case.
To see how to run Megatron-Deepspeed, Huggingface Accelerate, Torch FSDP, and other multi-node use cases, refer to our Tutorials:
Deploy and manage clusters with Slurm workload orchestration
# Slurm Orchestration
Source: https://docs.primeintellect.ai/tutorials-multi-node-cluster/slurm-orchestration
Deploy and manage multi-node clusters with Slurm workload orchestration on Prime Intellect Platform.
Slurm is a powerful, open-source workload manager and job scheduler designed for high-performance computing clusters. When you deploy a multi-node cluster with Slurm on Prime Intellect, you get a fully configured orchestration system with shared storage for seamless distributed computing.
# Deploy a Slurm Cluster
Navigate to the Multi-Node Cluster tab and select a cluster configuration with shared storage attached. Choose Slurm as your orchestrator during the deployment process.
Once the cluster is deployed, the UI displays the controller IP address. Always connect to the controller node to issue Slurm commands - this is your main management interface for the entire cluster.
```bash theme={null}
ssh ubuntu@
```
After connecting to the controller, verify your Slurm cluster is properly configured and all nodes are available.
# Essential Slurm Commands
Once connected to the controller node, you can use these Slurm commands to manage your cluster:
## View Cluster Information
```bash theme={null}
# Display information about nodes and partitions
sinfo
# Show detailed node information
sinfo -Nel
# Display partition summary
sinfo -s
# Example output:
# PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
# gpu* up infinite 2 idle node[001-002]
```
## GPU Resource Allocation
Prime Intellect clusters use the Generic Resource (GRES) system for GPU allocation. Understanding the correct syntax is crucial for successful job submission.
### Interactive GPU Sessions
```bash theme={null}
# Request an interactive session with GPUs (use --gres for batch scripts)
srun --gpus=1 --pty bash
# Request multiple GPUs across nodes
srun --nodes=2 --gpus-per-node=8 nvidia-smi
# Request specific node with GPUs
srun --gpus=4 --nodelist=node001 --pty bash
# Alternative GRES syntax (required for batch scripts)
srun --nodes=2 --gres=gpu:8 --ntasks-per-node=1 nvidia-smi
# Once in the session, verify GPU access
nvidia-smi
# Check available GPUs in your session
nvidia-smi -L
```
When writing batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N` to avoid InvalidAccount errors.
## Batch Job Submission
Batch jobs allow you to queue work that runs without manual intervention. Create a script file (e.g., `job.sh`) with SBATCH directives:
### Basic GPU Job Script
```bash theme={null}
#!/bin/bash
#SBATCH --job-name=gpu-test
#SBATCH --nodes=2
#SBATCH --gres=gpu:8 # IMPORTANT: Use gres for GPUs in batch scripts
#SBATCH --ntasks-per-node=1
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out # %x=job-name, %j=job-id
#SBATCH --error=%x-%j.err
echo "Job started on $(date)"
echo "Running on nodes: $SLURM_JOB_NODELIST"
# Run nvidia-smi on all allocated nodes
srun -l nvidia-smi
# Your training or compute commands here
# srun python train.py
```
### Submit and Manage Jobs
```bash theme={null}
# Submit a batch job
sbatch job.sh
# View queued and running jobs
squeue
# View your jobs only
squeue -u $USER
# Cancel a job
scancel
# Cancel all your jobs
scancel -u $USER
# View detailed job information
scontrol show job
```
### Job Output Location
Job output files are written to the directory where `sbatch` was executed, which may be on the compute node's local filesystem if you're not in shared storage. Always submit jobs from the shared storage directory to ensure output accessibility.
## Example: Quick Cluster Test
Verify your Slurm cluster with these simple tests:
```bash theme={null}
# Test 1: Check all GPUs across nodes (interactive)
srun --nodes=2 --gpus-per-node=8 nvidia-smi
# Test 2: Verify task distribution across nodes
srun --nodes=2 --ntasks-per-node=2 hostname
# Test 3: Check task distribution
srun --nodes=2 --ntasks-per-node=2 bash -c 'echo "Task $SLURM_PROCID running on node $(hostname)"'
```
# Troubleshooting Common Issues
## InvalidAccount Error
If you encounter `InvalidAccount` errors when submitting batch jobs:
1. **Use correct GPU syntax**: In batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N`
2. **No accounting plugin**: Prime Intellect clusters intentionally don't use Slurm's accounting plugin since clusters are single-tenant with dedicated resources. Remove any `#SBATCH --account=` directives from your scripts
3. **Check partition availability**: Ensure the partition you're requesting exists with `sinfo`
Prime Intellect clusters run without the Slurm accounting plugin because each cluster is single-tenant with dedicated resources. This simplifies configuration and eliminates account-based resource restrictions, giving you full access to all allocated resources without quota management overhead.
### Incorrect (causes InvalidAccount)
```bash theme={null}
#!/bin/bash
#SBATCH --gpus-per-node=8 # Wrong for batch scripts
```
### Correct
```bash theme={null}
#!/bin/bash
#SBATCH --gres=gpu:8 # Correct for batch scripts
```
## Job Output Not Found
If you can't find your job output files:
* **Check working directory**: Output files are created where `sbatch` was run
* **Use absolute paths**: Specify full paths in `--output` and `--error` directives
* **Check other nodes**: If submitted from a compute node, outputs may be on that node's local storage
* **Always submit from shared storage**: Change to the shared storage directory before running `sbatch`
## Node Communication Issues
If nodes can't communicate or jobs hang:
* Verify all nodes are in `idle` state with `sinfo`
* Check node connectivity: `srun --nodelist= hostname`
* Ensure shared storage is mounted on all nodes
* Consider restarting your cluster if issues persist
# Advanced Slurm Commands
## Direct Node Access
Access specific compute nodes directly for debugging or monitoring:
```bash theme={null}
# SSH into a specific node via srun
srun --nodelist=computeinstance-abc123 --pty bash
# Run commands on specific nodes
srun --nodelist=node001,node002 hostname
# Allocate resources without running a command
salloc --nodes=2 --gres=gpu:8 --time=01:00:00
# Then use srun within the allocation
srun nvidia-smi
```
## Resource Monitoring
```bash theme={null}
# View detailed node status
scontrol show node
# Check GPU allocation
scontrol show node | grep -E "NodeName|Gres"
# Monitor job efficiency
seff
# View job accounting information
sacct -j --format=JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode
# Check cluster utilization
sreport cluster utilization
```
## Job Arrays for Parameter Sweeps
Run multiple similar jobs with different parameters:
```bash theme={null}
#!/bin/bash
#SBATCH --job-name=param-sweep
#SBATCH --array=1-10
#SBATCH --gres=gpu:1
#SBATCH --output=sweep_%A_%a.out # %A=array job ID, %a=array task ID
# Use SLURM_ARRAY_TASK_ID for different parameters
python train.py --seed=$SLURM_ARRAY_TASK_ID --lr=$(echo "0.001 * $SLURM_ARRAY_TASK_ID" | bc)
```
## Environment Variables
Useful Slurm environment variables available in jobs:
```bash theme={null}
echo "Job ID: $SLURM_JOB_ID"
echo "Job Name: $SLURM_JOB_NAME"
echo "Node List: $SLURM_JOB_NODELIST"
echo "Number of Nodes: $SLURM_JOB_NUM_NODES"
echo "Tasks per Node: $SLURM_NTASKS_PER_NODE"
echo "CPUs per Task: $SLURM_CPUS_PER_TASK"
echo "Submit Directory: $SLURM_SUBMIT_DIR"
echo "Task ID: $SLURM_PROCID"
echo "Node ID: $SLURM_NODEID"
```
# Shared Storage Integration
Your Slurm cluster comes with shared storage automatically mounted on all nodes. The UI displays the mount path for your shared storage directory. This ensures:
* Consistent file access across all compute nodes
* No need to manually copy data between nodes
* Simplified job submission and management
* Persistent storage for checkpoints and results
## Best Practices
Always submit jobs and run Slurm commands from the controller node, not compute nodes
Store your code, data, and outputs in the shared storage directory shown in the UI for seamless access across nodes
Regularly check cluster utilization with `sinfo` and `squeue` to optimize job scheduling
For parameter sweeps or similar tasks, use Slurm job arrays for efficient scheduling
# Cluster Monitoring
Source: https://docs.primeintellect.ai/tutorials-reserved-clusters/monitoring
Detect issues early and minimize downtime with production-ready monitoring
## Overview
Reserved clusters come with production-ready monitoring out of the box—no setup required. Our monitoring is built to meet the [ClusterMAX™](https://www.clustermax.ai/monitoring) standard, the industry benchmark for GPU cloud infrastructure developed by SemiAnalysis.
Detect hardware issues early, minimize downtime, and keep your training runs on track with:
* **Preconfigured Grafana dashboards** covering critical GPU, node, and network metrics
* **Proactive alerting** delivered to your preferred channels the moment issues arise
* **GPU health monitoring** including XID error detection and hardware anomaly tracking
## Accessing the Dashboard
Your cluster includes a Grafana dashboard for visualizing metrics. Access is provided during the cluster handover process—once your reserved cluster is ready, you will receive the dashboard URL and credentials from the Prime Intellect team.
## Available Metrics
The monitoring dashboard includes preconfigured graphs for critical metrics across your cluster:
| Category | Metrics | Source |
| ---------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| GPU | Utilization, memory, temperature, power usage, interconnect throughput (PCIe, NvLink) | [DCGM Exporter](https://github.com/NVIDIA/dcgm-exporter) |
| Node | CPU, memory, disk, and network usage | [Node Exporter](https://github.com/prometheus/node_exporter) |
| InfiniBand | Throughput, port errors, and discards | [Node Exporter](https://github.com/prometheus/node_exporter) |
| Slurm | Node states, cluster utilization, and running jobs | [Slurm Exporter](https://github.com/PrimeIntellect-ai/slurm-exporter) |
## Troubleshooting with the Dashboard
When a training job fails, use the dashboard to quickly identify the root cause. In large clusters, alerts eliminate the need to manually hunt for the bad node—they tell you exactly which node and device has the issue.
### 1. Check XID Alerts First
Start with the **Alert list** in the Cluster Overview. XID alerts identify the exact node and GPU device experiencing issues, so you can pinpoint problems immediately without searching through hundreds of nodes.
XID errors are NVIDIA's GPU error codes reported by the kernel driver. Each code indicates a specific hardware or software issue—from memory errors to thermal shutdowns. Use the [NVIDIA XID Error Catalog](https://docs.nvidia.com/deploy/xid-errors/index.html) to look up the error code and understand the recommended action.
### 2. Check InfiniBand Performance
Next, expand the **InfiniBand Metrics** section and look for:
* **Port Errors** - Non-zero values indicate connectivity problems
* **Throughput drops** - May explain slow collective operations during distributed training
### 3. Review Secondary Metrics
For additional context, check:
* **GPU Temperature** - Spikes above 80°C may indicate cooling issues
* **GPU Utilization** - Sudden drops to 0% during training suggest failures
* **Slurm Nodes State** - See if any nodes are marked as "down"
### Adjust Time Range
Use the time picker (top right) to zoom into the incident window or compare with historical baseline.
## Alerting
Get notified immediately when something needs attention. Alerts are triggered when metrics exceed critical thresholds, so you can respond before issues impact your workloads.
### Notification Channels
Alerts can be delivered to your preferred channels:
* **Slack** - Receive alerts directly in your Slack workspace
* **Microsoft Teams** - Get notifications in your Teams channels
* **Webhooks** - Integrate with any provider that accepts webhooks
To configure your notification channel, contact the Prime Intellect team with your preferred destination.
Your cluster comes with battle-tested alert rules based on our experience operating large-scale GPU clusters.
### Creating Custom Alerts
You can create custom alert rules directly in Grafana for your specific workloads. Here's how:
In your Grafana dashboard, navigate to **Alerting** → **Alert rules** → **+ New alert rule**.
Select your data source and write a query for the metric you want to monitor. Then set the condition that triggers the alert (e.g., `GPU temperature > 80°C`).
Configure how often the rule is evaluated and how long the condition must be true before firing. This helps prevent false alarms from brief spikes.
Choose where to send alerts—use an existing contact point (Slack, Teams, webhook) or create a new one.
Click **Save rule** to activate your custom alert.
For detailed documentation, see the [Grafana Alerting guide](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/).
# Cluster storage
Source: https://docs.primeintellect.ai/tutorials-storage/cluster-storage
Configure persistent and ephemeral shared storage for multi-node clusters
Multi-node clusters support two types of shared storage: persistent disks that survive beyond cluster lifecycles, and ephemeral shared storage that exists only during the cluster's lifetime.
## Storage Options
### Persistent Storage
* **Persists** after cluster termination
* **Shareable** across multiple clusters
* **Requires** pre-created disk with `Active` status
* **Use case**: Long-term data, models, and datasets
### Ephemeral Shared Storage
* **Temporary** storage for cluster lifecycle only
* **Automatic** creation during cluster provisioning
* **Shared** between all nodes in the cluster
* **Use case**: Temporary files, intermediate processing data
# Step-by-Step Guide
This disk will remain available after cluster termination and can be reattached to new clusters.Ephemeral storage is deleted when the cluster terminates. Do not store critical data here without backups.
# Create persistent storage
Source: https://docs.primeintellect.ai/tutorials-storage/create-persistent-storage
Create storage that can be shared between instances
Persistent storage allows you to create data volumes that persist beyond the lifecycle of individual instances and can be attached to different instances as needed.
# Step-by-Step Guide
Go to the `Storage` tab in your Instances page, and click on `Create Disk`.
Enter your desired disk size and click `Create Disk`.
The disk will begin provisioning. You can only attach disks that have an `Active` status.
# Use persistent storage with instances
Source: https://docs.primeintellect.ai/tutorials-storage/use-persistent-storage-with-instances
How to attach and use persistent storage with instances and clusters
Attach persistent storage volumes to your instances to provide durable, shareable storage that persists beyond instance lifecycles.
# Step-by-Step Guide
* An active persistent disk with `Active` status
* Available instances in the same provider and location as your disk
Go to `Deploy GPU Instance` or `Multi-Node Cluster`. Use the `Filter by your existing disks` toggle to show only clusters in locations where you have persistent disks available.
Look for providers with the `Add Shared Filesystem` button, indicating disk compatibility.
Persistent disks are location-specific. Ensure you're selecting instances in the same provider and datacenter as your disk.
# Overview
Source: https://docs.primeintellect.ai/verifiers/overview
Verifiers Repo
verifiers is a framework for defining tasks, running agents and harnesses, scoring them on set tasks, and using those for evaluations and reinforcement learning.
## Documentation
The documentation is split into two sections:
* **[verifiers v1](/verifiers/v1/overview)** — The current documentation for verifiers v1 (`verifiers.v1`), built around tasksets, harnesses, and traces. This is what new environments should target.
* **[Legacy (v0)](/verifiers/v0/overview)** — The documentation for legacy verifiers v0 environments (`import verifiers as vf`). v0 is considered deprecated and will be fully removed in a future release.
# Development
Source: https://docs.primeintellect.ai/verifiers/v0/development
v0 is considered deprecated and will be fully removed in a future release.
This guide covers setup, testing, and contributing to the verifiers package.
## Table of Contents
* [Setup](#setup)
* [Project Structure](#project-structure)
* [Prime CLI Plugin Export](#prime-cli-plugin-export)
* [Running Tests](#running-tests)
* [Writing Tests](#writing-tests)
* [Contributing](#contributing)
* [Contributor Practices](#contributor-practices)
* [Common Issues](#common-issues)
* [Environment Development](#environment-development)
* [Quick Reference](#quick-reference)
## Setup
### Prerequisites
* Python 3.13 recommended for CI parity with Ty checks
* [uv](https://docs.astral.sh/uv/) package manager
### Installation
```bash theme={null}
# Clone and install for development
git clone https://github.com/PrimeIntellect-ai/verifiers.git
cd verifiers
# CPU-only development:
uv sync
# GPU-based trainer development:
uv sync --all-extras
# Install pre-commit hooks (including pre-push Ty gate):
uv run pre-commit install
```
## Project Structure
```text theme={null}
verifiers/
├── verifiers/ # Main package
│ ├── envs/ # Environment classes
│ │ ├── integrations/ # Third-party wrappers (TextArena, ReasoningGym)
│ │ └── experimental/ # Newer environments (MCP, Harbor, etc.)
│ ├── parsers/ # Parser classes
│ ├── rubrics/ # Rubric classes
│ ├── rl/ # Training infrastructure
│ │ ├── inference/ # vLLM server utilities
│ │ └── trainer/ # Trainer implementation
│ ├── cli/ # Prime-facing CLI modules and plugin exports
│ ├── scripts/ # Compatibility wrappers around verifiers/cli commands
│ └── utils/ # Utilities
├── environments/ # Installable environment modules
├── configs/ # Example training configurations
├── tests/ # Test suite
└── docs/ # Documentation
```
## Prime CLI Plugin Export
Verifiers exports a plugin consumed by `prime` so command behavior is sourced from verifiers modules.
Entry point:
```python theme={null}
from verifiers.cli.plugins.prime import get_plugin
plugin = get_plugin()
```
The plugin exposes:
* `api_version` (current: `1`)
* command modules:
* `eval_module` (`verifiers.cli.commands.eval`)
* `gepa_module` (`verifiers.cli.commands.gepa`)
* `install_module` (`verifiers.cli.commands.install`)
* `init_module` (`verifiers.cli.commands.init`)
* `setup_module` (`verifiers.cli.commands.setup`)
* `build_module` (`verifiers.cli.commands.build`)
* `build_module_command(module_name, args)` to construct subprocess invocation for a command module
Contributor guidance:
* Add new prime-facing command logic under `verifiers/cli/commands/`.
* Export new command modules through `PrimeCLIPlugin` in `verifiers/cli/plugins/prime.py`.
* Keep `verifiers/scripts/*` as thin compatibility wrappers that call into `verifiers/cli`.
## Running Tests
```bash theme={null}
# Run all tests
uv run pytest tests/
# Run with coverage
uv run pytest tests/ --cov=verifiers --cov-report=html
# Run specific test file
uv run pytest tests/test_parser.py
# Stop on first failure with verbose output
uv run pytest tests/ -xvs
# Run tests matching a pattern
uv run pytest tests/ -k "xml_parser"
# Run V1 environment tests
uv run pytest tests/v1/test_envs.py -vv
# Run environment tests across all CPU cores
uv run pytest -n auto tests/v1/test_envs.py -vv
# Run specific environment tests
uv run pytest tests/v1/test_envs.py -k gsm8k_v1
```
The test suite includes 380+ tests covering parsers, rubrics, environments, and utilities.
## Writing Tests
### Test Structure
```python theme={null}
class TestFeature:
"""Test the feature functionality."""
def test_basic_functionality(self):
"""Test normal operation."""
# Arrange
feature = Feature()
# Act
result = feature.process("input")
# Assert
assert result == "expected"
def test_error_handling(self):
"""Test error cases."""
with pytest.raises(ValueError):
Feature().process(invalid_input)
```
### Using Mocks
The test suite provides a `MockClient` in `conftest.py` that implements the `Client` interface:
```python theme={null}
def test_with_mock(mock_client):
mock_client.set_default_responses(chat_response="test answer")
env = vf.SingleTurnEnv(client=mock_client, model="test", ...)
# Test without real API calls
```
### Guidelines
1. **Test both success and failure cases**
2. **Use descriptive test names** that explain what's being tested
3. **Leverage existing fixtures** from `conftest.py`
4. **Group related tests** in test classes
5. **Keep tests fast** - use mocks instead of real API calls
## Contributing
### Workflow
1. **Fork** the repository
2. **Create a feature branch**: `git checkout -b feature-name`
3. **Make changes** following existing patterns
4. **Add tests** for new functionality
5. **Run tests**: `uv run pytest tests/`
6. **Install hooks once per clone**: `uv run pre-commit install`
7. **Commit and push** (hooks run automatically on each commit/push)
8. **Update docs** if adding/changing public APIs
9. **Submit PR** with clear description
### Code Style
* Strict `ruff` enforcement via pre-commit hooks
* `ty` runs in the pre-push hook via `uv run --python 3.13 ty check verifiers`
* Use type hints for function parameters and returns
* Write docstrings for public functions/classes
* Keep functions focused and modular
* Fail fast, fail loud - no defensive programming or silent fallbacks
### PR Checklist
* [ ] Tests pass locally (`uv run pytest tests/`)
* [ ] Pre-commit and pre-push hooks pass on latest commit/push
* [ ] Added tests for new functionality
* [ ] Updated documentation if needed
## Contributor Practices
### Public Surface
Treat public config, docs, starter examples, skills, and generated agent guidance as one surface. If a behavior changes for users, update all matching surfaces in the same patch.
For TOML config, keep one shape across eval, GEPA, RL, and Hosted Training. Normalize old or alternate inputs at the loader boundary, then keep examples on the current golden path.
### Validation By Change Type
* Core runtime or shared config parsing: run the focused unit tests plus `uv run pre-commit run --all-files`.
* Example environment behavior: run the focused tests and a real `prime eval run` smoke when credentials and endpoint access are available.
* Environment packaging: exercise `tests/v1/test_envs.py` for the changed environment so a fresh venv installs the environment package and its dependencies.
* Docs or agent guidance (`AGENTS.md`): edit the Markdown directly and keep it minimal.
* Release prep: verify the version source, release notes commit range, `uv build`, and final worktree status.
* PR/CI follow-up: inspect the live review thread, check run, or log before patching, then rerun the smallest check that proves the fix.
### Downstream Checks
Before changing dependencies, optional extras, lockfiles, exported config fields, or upload/eval metadata, trace the consumers in `prime-cli`, `prime-rl`, Hosted Training, and public docs when they are in scope. Update the consumer or document the compatibility boundary rather than assuming transitive behavior remains safe.
## Common Issues
### Import Errors
```bash theme={null}
# Ensure package is installed in development mode
uv sync
```
### Integration Tests
```bash theme={null}
# Install optional dependencies for specific integrations
uv sync --extra ta # for TextArenaEnv
uv sync --extra rg # for ReasoningGymEnv
uv sync --extra modal # for the v1 Modal runtime
uv sync --extra notebook # for generate_sync() in Jupyter
uv sync --python 3.12 --extra harbor # for the Harbor Python package and CLI
```
### Test Failures
```bash theme={null}
# Debug specific test
uv run pytest tests/test_file.py::test_name -vvs --pdb
```
## Environment Development
### Creating a New Environment Module
```bash theme={null}
# Initialize a v0 environment stub
prime env init my-environment
# Test your environment
prime eval run my-environment -m openai/gpt-4.1-mini -n 5
```
### Environment Module Structure
```python theme={null}
# my_environment.py
import verifiers as vf
def load_environment(**kwargs):
"""Load the environment."""
dataset = vf.load_example_dataset("dataset_name")
parser = vf.XMLParser(fields=["reasoning", "answer"])
def reward_func(parser, completion, answer, **kwargs):
return 1.0 if parser.parse_answer(completion) == answer else 0.0
rubric = vf.Rubric(
funcs=[reward_func, parser.get_format_reward_func()],
weights=[1.0, 0.2],
parser=parser,
)
return vf.SingleTurnEnv(dataset=dataset, parser=parser, rubric=rubric, **kwargs)
```
## Quick Reference
### Essential Commands
```bash theme={null}
# Development setup
uv sync # CPU-only
uv sync --all-extras # With RL/training extras
uv run pre-commit install # One-time per clone (installs pre-commit + pre-push)
# Run tests
uv run pytest tests/ # All tests
uv run pytest tests/ -xvs # Debug mode
uv run pytest tests/ --cov=verifiers # With coverage
# Run environment tests
uv run pytest tests/v1/test_envs.py -vv # All environments
uv run pytest tests/v1/test_envs.py -k gsm8k_v1 # Specific environment
# Linting
uv run ruff check --fix . # Fix lint errors
uv run ruff format --check verifiers tests # Verify Python formatting
uv run ty check verifiers # Type check (matches CI Ty target)
# Environment tools
prime env init new-env # Create v0 environment stub
prime eval run new-env -m openai/gpt-4.1-mini -n 5 # Test environment
prime eval view # Browse evals in the tree browser
```
### CLI Tools
| Command | Description |
| ------------------- | -------------------------------------------------- |
| `prime eval run` | Run evaluations on environments |
| `prime env init` | Initialize new environment from template |
| `prime env install` | Install environment module |
| `prime lab setup` | Set up training workspace |
| `prime eval view` | Terminal UI for browsing evals and rollout details |
| `prime rl run` | Launch Hosted Training |
### Project Guidelines
* **Environments**: Installable modules with `load_environment()` function
* **Parsers**: Extract structured data from model outputs
* **Rubrics**: Define multi-criteria evaluation functions
* **Tests**: Comprehensive coverage with mocks for external dependencies
# Environments
Source: https://docs.primeintellect.ai/verifiers/v0/environments
v0 is considered deprecated and will be fully removed in a future release.
This guide walks through building environments in Verifiers, from simple single-turn tasks to complex multi-turn agents with tools. See [Overview](/verifiers/v0/overview) for how to initialize a new environment template.
## Table of Contents
* [Your First Environment](#your-first-environment)
* [Datasets](#datasets)
* [Building the Prompt](#building-the-prompt)
* [Evaluation Datasets](#evaluation-datasets)
* [Lazy Loading with DatasetBuilder](#lazy-loading-with-datasetbuilder)
* [Rubrics](#rubrics)
* [Reward Functions](#reward-functions)
* [Multiple Reward Functions](#multiple-reward-functions)
* [Execution Order and State](#execution-order-and-state)
* [Group-Based Reward Functions](#group-based-reward-functions)
* [Shared Objects](#shared-objects)
* [Rubric Groups](#rubric-groups)
* [Metrics and Monitor Rubrics](#metrics-and-monitor-rubrics)
* [Tool Environments](#tool-environments)
* [MCP Tool Environments](#mcp-tool-environments)
* [Stateful Tool Environments](#stateful-tool-environments)
* [Custom Multi-Turn Environments](#custom-multi-turn-environments)
* [The Rollout Loop](#the-rollout-loop)
* [Stop Conditions](#stop-conditions)
* [Error Handling](#error-handling)
* [State Initialization](#state-initialization)
* [Cleanup and Teardown](#cleanup-and-teardown)
* [Signaling Early Termination](#signaling-early-termination)
* [Developing Environments](#developing-environments)
* [pyproject.toml](#pyprojecttoml)
* [Managing Dependencies](#managing-dependencies)
* [Installation](#installation)
* [Environment Groups](#environment-groups)
* [Performance](#performance)
* [Avoiding Sync Operations](#avoiding-sync-operations)
* [Executor Autoscaling](#executor-autoscaling)
* [Integrations and Experimental Environments](#integrations-and-experimental-environments)
## Your First Environment
The simplest single-turn environments need only a dataset of tasks and a reward function for scoring responses:
```python theme={null}
import verifiers as vf
from datasets import Dataset
def load_environment():
# Your task data
dataset = Dataset.from_list(
[
{"prompt": [{"role": "user", "content": "What is 2+2?"}], "answer": "4"},
{"prompt": [{"role": "user", "content": "What is 3*5?"}], "answer": "15"},
]
)
# Your reward function
async def correct_answer(completion, answer) -> float:
response = completion[-1]["content"]
return 1.0 if answer in response else 0.0
rubric = vf.Rubric(funcs=[correct_answer])
return vf.SingleTurnEnv(dataset=dataset, rubric=rubric)
```
When running this environment, each row in the dataset becomes a **rollout**:
1. The `prompt` is sent to the model
2. The model generates a response, which becomes the `completion`
3. The reward function scores the result
In `SingleTurnEnv`, the simplest environment type, just a single model response occurs per rollout. More complex environment types will allow us to add tool use or other custom interaction protocols.
## Datasets
Environments use the `datasets` library from Hugging Face for loading and manipulating datasets. Each row typically has a `prompt` column, containing a list of initial messages to send to the model. Additionally, there are optional columns for scoring:
* `answer` — a simple string for ground truth comparisons
* `info` — structured metadata (dict or JSON string)
Depending on what your environment needs, you can include `answer`, `info`, both, or neither.
When using `info`, prefer using JSON strings if rows may have different schemas, e.g. different fields or nested structures:
```python theme={null}
dataset = Dataset.from_list(
[
{"prompt": [...], "info": '{"type": "math", "difficulty": 3}'},
{"prompt": [...], "info": '{"type": "code", "language": "python"}'},
]
)
```
These are parsed into a `dict` by the environment when running rollouts.
### Building the Prompt
The examples above use `prompt` directly, providing a list of messages ready to send to the model. Alternatively, you can provide a `question` column containing a string, and the environment will wrap it in a user message:
```python theme={null}
dataset = Dataset.from_list(
[
{"question": "What is 2+2?", "answer": "4"},
]
)
```
You can also pass a `system_prompt` to the environment, which prepends a system message:
```python theme={null}
return vf.SingleTurnEnv(
dataset=dataset,
system_prompt="You are a helpful math tutor.",
rubric=rubric,
)
```
Together, these construct the full prompt:
```python theme={null}
[
{"role": "system", "content": "You are a helpful math tutor."},
{"role": "user", "content": "What is 2+2?"},
]
```
If your dataset already has a `prompt` column, `question` is ignored. However, if a `system_prompt` is provided, it will be prepended to existing prompts that don't already start with a system message.
### Evaluation Datasets
Environments can be initialized with a separate `eval_dataset` for evaluation, distinct from the training dataset:
```python theme={null}
return vf.SingleTurnEnv(
dataset=train_dataset,
eval_dataset=eval_dataset,
rubric=rubric,
)
```
When running `prime eval run`, the evaluation dataset is used by default. If no `eval_dataset` is provided, evaluation falls back to the training dataset.
### Lazy Loading with DatasetBuilder
For large datasets or when running multiple environment replicas, you can defer dataset loading using a `DatasetBuilder`—a callable that returns a `Dataset` when invoked:
```python theme={null}
def get_dataset_builder(split: str = "train", seed: int = 42) -> vf.DatasetBuilder:
"""Returns a builder that lazily loads the dataset."""
def build() -> Dataset:
ds = load_dataset("my-dataset", split=split)
ds = ds.shuffle(seed=seed)
return ds
return build
def load_environment():
dataset_builder = get_dataset_builder(split="train")
eval_builder = get_dataset_builder(split="test")
return vf.SingleTurnEnv(
dataset=dataset_builder, # built on first access
eval_dataset=eval_builder, # built on first access
rubric=rubric,
)
```
The builder pattern is useful when:
* Dataset loading is expensive (e.g., downloading from Hugging Face)
* Multiple environment replicas don't all need to own the dataset
* You want to parameterize dataset creation without loading it immediately
When a raw `Dataset` is passed directly (the default pattern), it is loaded eagerly during environment initialization for backwards compatibility.
## Rubrics
Each environment has a `Rubric` that manages scoring. The rubric holds reward functions, combines their outputs into a final reward score, and tracks metrics for observability.
### Reward Functions
Reward functions evaluate rollouts and return floats, typically between 0.0 and 1.0. They can request data from the rollout by naming arguments directly:
```python theme={null}
async def correct_answer(completion, answer) -> float:
response = completion[-1]["content"]
return 1.0 if answer in response else 0.0
```
The basic available arguments, if present, are:
* `completion` — the model's output (list of messages)
* `prompt` — the input messages
* `answer` — from dataset
* `info` — from dataset
* `state` — the full rollout state (used in more complex environments)
This reference pattern extends to additional objects that the rubric provides in more advanced use cases.
### Multiple Reward Functions
Rubrics can combine multiple reward functions with custom weights:
```python theme={null}
async def check_keywords(completion, info) -> float:
response = completion[-1]["content"]
keywords = info["required_keywords"]
found = sum(1 for kw in keywords if kw.lower() in response.lower())
return found / len(keywords)
async def length_reward(completion) -> float:
response = completion[-1]["content"]
return 1.0 if len(response) < 500 else 0.5
rubric = vf.Rubric(funcs=[check_keywords, length_reward], weights=[1.0, 0.1])
```
The final rollout reward is computed as the weighted sum of all reward function scores.
Reward functions can also be added to a rubric after initialization:
```python theme={null}
rubric = vf.Rubric()
rubric.add_reward_func(check_keywords, weight=1.0)
rubric.add_reward_func(length_reward, weight=0.1)
```
Beyond the final score, reward functions can be used to track metrics for observability by setting `weight=0`:
```python theme={null}
async def response_length(completion) -> float:
return float(len(completion[-1]["content"]))
rubric.add_metric(response_length) # shorthand for weight=0
```
All reward functions (weighted or not) appear in the rollout metrics.
### Execution Order and State
Reward functions execute in the order they are added to the rubric. Since `state` is mutable and shared across all reward functions, earlier functions can store computed values for later functions to use:
```python theme={null}
async def similarity_score(completion, answer, state) -> float:
response = completion[-1]["content"]
score = compute_similarity(response, answer) # continuous 0-1
state["similarity"] = score
return score
async def similarity_threshold(state) -> float:
return 1.0 if state["similarity"] > 0.8 else 0.0
rubric = vf.Rubric(
funcs=[similarity_score, similarity_threshold],
weights=[0.0, 1.0], # log similarity, but only reward threshold
)
```
This avoids redundant computation when multiple reward functions need access to the same derived value.
### Group-Based Reward Functions
During evaluation and RL training, rollouts are organized into **groups** of rollouts from the same input example. When evaluating, group structure enables per-example aggregate statistics (e.g., pass\@k). When training with RL, groups are used for advantage computation relative to other rollouts for the same example. For a dataset with 100 example rows, running 4 rollouts per example yields 100 groups of 4 rollouts each.
In some cases, it is useful for reward functions to operate at the group level, such as to measure diversity or compute relative rankings. To define a group reward function, use plural argument names (`completions`, `prompts`, `answers`, `infos`) and return a list of scores:
```python theme={null}
async def diversity_bonus(completions) -> list[float]:
"""Reward unique responses within a group."""
responses = [c[-1]["content"] for c in completions]
unique = set(responses)
# Higher reward if this response is unique
return [0.2 if responses.count(r) == 1 else 0.0 for r in responses]
rubric = vf.Rubric(funcs=[correct_answer, diversity_bonus])
```
### Shared Objects
In rubric environments, reward functions can request static helper objects that live within the Rubric class. These are stored in the Rubric's `class_objects` dictionary, and can be added after initialization via `add_class_object()`:
```python theme={null}
rubric = vf.Rubric(funcs=[my_reward_func])
rubric.add_class_object("my_helper", some_helper_object)
async def my_reward_func(completion, my_helper) -> float:
# my_helper is now available by name
return await my_helper.score(completion)
```
For taskset/harness environments, keep shared dependencies behind the taskset or harness that owns them. Bindings are the canonical way to inject shared resources into rewards, updates, tools, and programs. Configured binding objects should use serializable loader paths when they cross a TOML or CLI boundary; Python-only construction may use factory callables directly when a resource cannot be serialized. Required Taskset and Toolset factory parameters must be supplied through bindings.
Judges are used for tasks where deterministic evaluation is impractical, and an LLM is used to score responses. **JudgeRubric** stores an LLM client inside the rubric, and provides a `judge` callable to reward functions for scoring responses:
```python theme={null}
judge_rubric = vf.JudgeRubric(
judge_model="gpt-4.1-mini",
)
async def judge_correctness(prompt, completion, answer, judge) -> float:
verdict = await judge(prompt, completion, answer)
return 1.0 if "yes" in verdict.lower() else 0.0
judge_rubric.add_reward_func(judge_correctness)
```
The `judge` callable formats a prompt comparing the model's response to the ground truth and returns the judge model's verdict.
For more control, JudgeRubric accepts a custom `judge_prompt` template and exposes its internals (`judge_client`, `judge_model`, `judge_prompt`, `judge_sampling_args`) as class objects:
```python theme={null}
judge_rubric = vf.JudgeRubric(
judge_model="gpt-4.1-mini",
judge_prompt="""Rate the writing quality of this response from 0-10.
Response: {response}
Score:""",
)
async def quality_score(
completion, judge_client, judge_model, judge_prompt, parser
) -> float:
response = parser.parse_answer(completion)
filled_prompt = judge_prompt.format(response=response)
result = await judge_client.chat.completions.create(
model=judge_model,
messages=[{"role": "user", "content": filled_prompt}],
)
# parse numeric score from result
...
return score
```
### Rubric Groups
Environments can include multiple rubrics by combining them into a `RubricGroup` (which itself behaves as a single rubric), aggregating all rewards and metrics from constituent rubrics. This is particularly useful for conjoining multiple rubrics of different types.
For example, `MathRubric` is a built-in rubric that uses symbolic verification to check mathematical correctness:
```python theme={null}
math_rubric = vf.MathRubric()
```
MathRubric includes a `correct_answer` reward function that parses `\boxed{}` answers and uses the `math-verify` library for symbolic equivalence checking. To add LLM-based evaluation alongside it:
```python theme={null}
math_rubric = vf.MathRubric()
judge_rubric = vf.JudgeRubric(judge_model="gpt-4.1-mini")
judge_rubric.add_reward_func(judge_correctness, weight=0.5)
rubric = vf.RubricGroup([math_rubric, judge_rubric])
```
All rubrics in a group are executed in parallel, and the final reward is the sum of all rubric rewards. Metrics from all rubrics are collected together.
### Metrics and Monitor Rubrics
For simple cases, metrics can be added directly to a rubric via `add_metric()` as shown above. Monitor rubrics extend this pattern by packaging metrics into separate rubrics that are combined via `add_rubric()`. This allows each environment type in a class hierarchy to contribute its own metrics automatically.
Many environment types automatically include a monitor rubric that tracks metrics specific to their level of the environment class hierarchy:
| Environment | Tracked Metrics |
| -------------- | ----------------------------------------------------------- |
| `MultiTurnEnv` | `num_turns` |
| `ToolEnv` | `total_tool_calls`, per-tool counts |
| `SandboxEnv` | `sandbox_ready_wait_time`, `sandbox_command_execution_time` |
| `PythonEnv` | `python_ready_wait_time` |
These metrics appear automatically in rollout results alongside any custom reward functions.
To add custom metrics to an environment, define a monitor rubric class and add it via `add_rubric()`:
```python theme={null}
class MyMonitorRubric(vf.Rubric):
def __init__(self):
super().__init__()
self.add_metric(self.custom_metric)
async def custom_metric(self, state: vf.State) -> float:
return len(state["trajectory"])
env = vf.ToolEnv(dataset=dataset, tools=tools, rubric=rubric)
env.add_rubric(MyMonitorRubric())
```
The environment automatically wraps rubrics in a `RubricGroup` as needed, so monitor rubrics stack up the class hierarchy—`PythonEnv` inherits metrics from both `SandboxEnv` and `ToolEnv`.
## Tool Environments
All currently-supported environment types in Verifiers are built on `MultiTurnEnv`, which implements the core single-agent rollout loop (even `SingleTurnEnv` is simply a `MultiTurnEnv` with `max_turns=1` and a placeholder `env_response` method). `ToolEnv` adds tool calling to this foundation.
Tools are defined as Python functions. Verifiers extracts tool schemas from function signatures and docstrings for use with OpenAI-compatible tool calling:
```python theme={null}
async def calculate(expression: str) -> str:
"""Evaluate a mathematical expression.
Args:
expression: A mathematical expression to evaluate (e.g. "2 + 2 * 3")
Returns:
The result of the evaluation.
"""
try:
result = eval(expression)
return str(result)
except Exception as e:
return f"Error: {e}"
async def lookup(term: str) -> str:
"""Look up a term in the knowledge base.
Args:
term: The term to search for.
Returns:
Information about the term.
"""
# your lookup logic here
...
```
The function name becomes the tool name, type hints define the parameter types, and the docstring provides both the tool description and individual parameter descriptions (via the Args section). Tools can be sync or async, though we always recommend using async for performance to avoid blocking the main thread.
To create a tool environment, pass the tools to `ToolEnv` directly:
```python theme={null}
vf_env = vf.ToolEnv(
dataset=dataset,
tools=[calculate, lookup],
rubric=rubric,
max_turns=10,
)
```
During rollouts, the model can call tools, receive results, and continue reasoning until it produces a response without tool calls (or hits `max_turns`). Each turn consists of a model response followed by the environment's tool execution. Tool call counts are tracked automatically via monitor rubrics (see above).
### MCP Tool Environments
For tools implemented as MCP (Model Context Protocol) servers, `MCPEnv` extends `ToolEnv` to provide an integration that automatically connects to MCP servers and exposes their tools to the model:
```python theme={null}
mcp_servers = [
{
"name": "fetch",
"command": "uvx",
"args": ["mcp-server-fetch"],
},
]
vf_env = vf.MCPEnv(
mcp_servers=mcp_servers,
dataset=dataset,
rubric=rubric,
)
```
### Stateful Tool Environments
`ToolEnv` and `MCPEnv` are designed for stateless, read-only tools where no session state needs to persist across calls within a rollout. For tools that require per-rollout state—such as a sandbox container, database connection, or session ID—use `StatefulToolEnv`.
The `setup_state` method is called at the beginning of each rollout for all environments which extend `MultiTurnEnv`, but is a no-op by default (including in `ToolEnv`).
`StatefulToolEnv` overrides this to initialize per-rollout resources, and introduces two additional concepts:
1. **Hidden arguments**: Tool functions can have parameters that are injected by the environment but hidden from the model's tool schema (via `args_to_skip`)
2. **`update_tool_args`**: An abstract method you implement to inject state into tool calls at runtime
```python theme={null}
class MySandboxEnv(vf.StatefulToolEnv):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.add_tool(self.run_code, args_to_skip=["session_id"])
async def setup_state(self, state, **kwargs):
state["session_id"] = await create_session()
await super().setup_state(state, **kwargs)
def update_tool_args(self, tool_name, tool_args, messages, state, **kwargs):
if tool_name == "run_code":
tool_args["session_id"] = state["session_id"]
return tool_args
async def run_code(self, code: str, session_id: str) -> str:
"""Execute code in the sandbox."""
return await execute_in_session(session_id, code)
```
The model sees `run_code(code: str)` in its tool schema, but the environment injects `session_id` from rollout state before each call.
Verifiers includes several built-in stateful environment classes: `SandboxEnv` provides a containerized bash shell, and `PythonEnv` extends it with a persistent Python REPL (both of which are configured for use with Prime Intellect's [Sandboxes](https://docs.primeintellect.ai/sandboxes/overview)). These handle sandbox lifecycle management automatically.
Both `SandboxEnv` and `CliAgentEnv` accept a `labels` parameter for tagging sandboxes:
```python theme={null}
env = vf.SandboxEnv(
dataset=dataset,
rubric=rubric,
labels=["experiment-1", "math-tasks"], # optional labels for sandbox categorization
)
```
Labels are passed to the Prime Sandboxes API and can be used for organizing, filtering, and managing sandboxes across experiments or training runs.
Stateful environments often define methods decorated with `@vf.cleanup` (called after each rollout) or `@vf.teardown` (called once at environment shutdown) for resource management. These decorators, along with `@vf.stop` for custom stop conditions (boolean functions checked after each turn), are powerful tools for rollout lifecycle control in custom `MultiTurnEnv` subclasses.
## Custom Multi-Turn Environments
For interaction patterns beyond tool calling—games, simulations, or other custom protocols—`MultiTurnEnv` can be subclassed directly, exposing full control over the rollout loop's behavior.
### The Rollout Loop
Each rollout follows this structure:
1. **Initialize state** — `setup_state(state)` is called to prepare per-rollout resources
2. **Loop until done:**
* Get prompt messages (initial prompt, or previous conversation + environment response)
* Get model response
* Check stop conditions — if any `@vf.stop` method returns `True`, exit loop
3. **Render completion** — final conversation is assembled into `state["completion"]`
4. **Cleanup** — all `@vf.cleanup` methods are called
The `env_response` method is an abstract method that must be overridden by all `MultiTurnEnv` subclasses, and defines how the environment responds after each model turn:
```python theme={null}
class MyGameEnv(vf.MultiTurnEnv):
def __init__(self, dataset, rubric, extract_action):
super().__init__(dataset=dataset, rubric=rubric)
self.extract_action = extract_action
async def env_response(self, messages: vf.Messages, state: vf.State) -> vf.Messages:
"""Generate the environment's response after each model turn."""
action = self.extract_action(messages)
feedback = process_action(action)
return [{"role": "user", "content": feedback}]
class ActionExtractor:
def __call__(self, messages: vf.Messages) -> str:
text = messages[-1]["content"] if messages else ""
return str(text).strip()
async def correct_action(extract_action, completion, answer) -> float:
return 1.0 if extract_action(completion) == answer else 0.0
def load_environment():
extract_action = ActionExtractor()
rubric = vf.Rubric(funcs=[correct_action])
rubric.add_class_object("extract_action", extract_action)
return MyGameEnv(dataset=dataset, rubric=rubric, extract_action=extract_action)
```
`env_response` receives the full conversation history thus far (and `state`) and returns a list of *new* messages to append. For tool environments, `env_response` typically executes tool calls and returns results. For games or other custom protocols, this might involve extracting structured output and returning state updates or feedback.
Several other methods can optionally be overridden for more control in complex custom environments:
* `setup_state(state)` — add environment-specific state fields at rollout start
* `get_prompt_messages(state)` — customize how messages are assembled (e.g. for non-linear conversations)
* `render_completion(state)` — customize how the final completion is assembled
* `add_trajectory_step(state, step)` — set intermediate rewards, advantages, or extra metadata per turn
### Stop Conditions
Rollouts continue until a stop condition is met, checked after each model response. Custom stop conditions are defined with the `@vf.stop` decorator:
```python theme={null}
class MyGameEnv(vf.MultiTurnEnv):
@vf.stop
async def game_won(self, state: vf.State) -> bool:
return state.get("won", False)
@vf.stop
async def game_lost(self, state: vf.State) -> bool:
return state.get("lives", 1) <= 0
```
`MultiTurnEnv` includes built-in stop conditions for errors, prompt length limits, `max_turns`, and `max_total_completion_tokens` by default. Per-rollout wall-clock timeouts are configured via the [`--timeout` flag](evaluation#evaluation-options) at evaluation time.
Execution order can be controlled with `priority` (higher runs first). This is useful for checking cheap conditions before expensive ones:
```python theme={null}
@vf.stop(priority=10) # cheap keyword check runs first
async def answer_submitted(self, state: vf.State) -> bool:
completion = state.get("completion", [])
if not completion:
return False
return "FINAL ANSWER:" in completion[-1].get("content", "")
@vf.stop(priority=-10) # expensive validation runs last
async def answer_detected(self, state: vf.State) -> bool:
# only runs if cheap checks didn't already stop
return await self.validator_client.check_for_answer(state)
```
### Error Handling
Verifiers defines a hierarchy of error types under `vf.Error`:
* `vf.ModelError` — errors from model interactions (e.g., `vf.EmptyModelResponseError`)
* `vf.OverlongPromptError` — prompt exceeds model context length
* `vf.ToolError` — tool-related errors (`vf.ToolParseError`, `vf.ToolCallError`)
* `vf.InfraError` — infrastructure errors (e.g., `vf.SandboxError`, `vf.TunnelError`)
When a `vf.Error` is raised during a rollout, it is automatically caught and stored in `state["error"]`, triggering the built-in `has_error` stop condition at the next check. This allows rollouts to terminate gracefully rather than crashing.
For tool environments, you can configure which errors should stop the rollout immediately via `stop_errors`:
```python theme={null}
vf_env = vf.ToolEnv(
tools=[my_tool],
stop_errors=[vf.ToolParseError], # stop on parse errors, but continue on other tool errors
...
)
```
Errors not in `stop_errors` are caught and returned as tool response messages, providing the model a chance to recover.
### State Initialization
Override `setup_state` to initialize per-rollout state:
```python theme={null}
class MyGameEnv(vf.MultiTurnEnv):
async def setup_state(self, state: vf.State) -> None:
state["board"] = initialize_board()
state["score"] = 0
await super().setup_state(state)
```
### Cleanup and Teardown
For resource management, use `@vf.cleanup` (per-rollout) and `@vf.teardown` (at environment shutdown):
```python theme={null}
class MyGameEnv(vf.MultiTurnEnv):
@vf.cleanup
async def save_game_log(self, state: vf.State):
await log_game_result(state["game_id"], state["score"])
@vf.teardown
async def close_connections(self):
await self.db_connection.close()
```
> **Important:** Cleanup methods should be **idempotent**—safe to call multiple times—and handle errors gracefully. This ensures correct behavior when rollouts are cancelled or interrupted, and that cleanup completes even when resources are in unexpected states.
### Signaling Early Termination
To end a rollout from within `env_response` (e.g., when the game ends), set `state["final_env_response"]`:
```python theme={null}
async def env_response(self, messages: vf.Messages, state: vf.State) -> vf.Messages:
if check_game_over(state):
final_message = [
{
"role": "user",
"content": "Game over! Final score: " + str(state["score"]),
}
]
state["final_env_response"] = final_message
return final_message
# ... normal response logic
```
This bypasses the normal model response loop and immediately terminates the rollout, which is useful when the environment response itself signals completion (e.g. a game is won, an answer is submitted) or is required for reward computation (e.g. final feedback or tool results).
## Developing Environments
Environments are packaged as installable Python projects. We recommend developing environments in a workspace with `environments/` and `configs/` folders. The `prime lab setup` command initializes this structure:
```bash theme={null}
prime lab setup
```
The `prime env init` command initializes a new environment project:
```bash theme={null}
prime env init my-env # v0 stub
```
This creates the following structure:
```text theme={null}
environments/my_env/
├── my_env.py # environment implementation
├── pyproject.toml # package metadata and dependencies
└── README.md # documentation template
```
### pyproject.toml
The `pyproject.toml` defines package metadata, dependencies, and evaluation defaults:
```toml theme={null}
[project]
name = "my-env"
description = "My custom environment"
tags = ["single-turn", "math", "train", "eval"]
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
"verifiers>=0.1.8",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build]
include = ["my_env.py", "pyproject.toml"]
[tool.verifiers.eval]
num_examples = 20
rollouts_per_example = 5
```
Key `pyproject.toml` sections:
* **`[project]`** — Package name (used by `prime env install` and `prime eval run`), description, version, and dependencies. The `tags` field is optional metadata for categorizing environments.
* **`[build-system]`** — Hatchling is used as the build backend for the Environments Hub.
* **`[tool.hatch.build]`** — Lists files to include in the package. Always include `pyproject.toml` alongside your environment file to ensure that environment metadata is available when the environment is installed. Add any additional source files here.
* **`[tool.verifiers.eval]`** — Default parameters for `prime eval run` when flags aren't provided.
### Managing Dependencies
All packages your environment needs must be declared in the `dependencies` array. Always include `verifiers` with a minimum version. If your environment uses additional libraries, add them here—they will be installed automatically when the environment is installed:
```toml theme={null}
dependencies = [
"verifiers>=0.1.8",
"chromadb",
"nltk>=3.9.2",
]
```
### Required API Keys
Environments that require external API keys (e.g., for judge models or external services) should validate them early in `load_environment()` using `vf.ensure_keys()`:
```python theme={null}
import verifiers as vf
def load_environment(api_key_var: str = "OPENAI_API_KEY") -> vf.Environment:
vf.ensure_keys([api_key_var])
# now safe to use os.environ[api_key_var]
...
```
This raises `MissingKeyError` with a clear message listing all missing keys and instructions for setting them:
* **Environments Hub**: Add secrets (or link global secrets) on the environment's **Secrets** tab
* **Hosted Training**: Set `env_file` in your config (e.g., `env_file = ["secrets.env"]`)
* **Local**: Export in your shell (e.g., `export OPENAI_API_KEY=...`)
Document required variables in your README under a "Required Environment Variables" section.
### Installation
Install a local environment with `prime env install`:
```bash theme={null}
prime env install my-env # from ./environments/my_env
prime env install my-env -p /path/to/environments # custom path
```
This runs `uv pip install -e` for local environments when you want an explicit editable install for non-eval tooling. Evaluations do not require this separate step because environment resolution happens inside `prime eval run`.
## Environment Groups
`EnvGroup` combines multiple environments into a single environment class, enabling multi-task evaluation and training across heterogeneous environments from a unified entrypoint. Each sub-environment maintains its own dataset, rubric, and rollout logic, while the group handles routing and metric aggregation:
```python theme={null}
math_env = load_math_environment()
code_env = load_code_environment()
reasoning_env = load_reasoning_environment()
combined = vf.EnvGroup(
envs=[math_env, code_env, reasoning_env],
env_names=["math", "code", "reasoning"],
)
```
The group concatenates all sub-environment datasets and injects `info["env_id"]` as internal routing metadata. It is not a top-level input, state, or output field. Metrics from all environments are tracked together.
## Performance
Verifiers runs rollouts concurrently on a single `asyncio` event loop. Any synchronous operation in environment code blocks **all** concurrent rollouts for its duration. At scale this adds up quickly — a 10ms sync call in at 2,000 concurrent rollouts serializes into 20 seconds of wall-clock blocking where no other rollout can make progress. The most impactful optimization is eliminating sync operations on the hot path rollout execution code, i.e. any method that runs *for each rollout* (e.g. `setup_state`, `env_response`, or reward functions).
### Avoiding Sync Operations
Common offenders include `time.sleep`, sync HTTP/LLM clients (`httpx.Client`, `OpenAI`), `deepcopy`, serialization, and file I/O. These should be **avoided at all costs**. Instead, use an async-native alternatives (e.g. `asyncio.sleep`, `httpx.AsyncClient`, `AsyncOpenAI`, `aiofiles`) or offload to the default thread pool with `asyncio.to_thread()`:
```python theme={null}
# ❌ time.sleep blocks the event loop
time.sleep(1)
# ✅ asyncio.sleep yields control
await asyncio.sleep(1)
# ❌ sync HTTP clients
requests.get(url)
# ✅ async HTTP clients
async with httpx.AsyncClient() as client:
await client.get(url)
# ❌ sync LLM clients
sync_client = OpenAI()
sync_client.chat.completions.create(...)
# ✅ use built-in async LLM calls
async_client = AsyncOpenAI()
await async_client.chat.completions.create(...)
# ❌ deepcopy blocks for large objects
copy.deepcopy(large_obj)
# ✅ offload to thread pool
await asyncio.to_thread(copy.deepcopy, large_obj)
# ❌ serialization blocks for large payloads
data_str = json.dumps(data)
# ✅ offload to thread pool (+use faster lib)
await asyncio.to_thread(orjson.dumps, data)
# ❌ sync file I/O
with open(file, "w") as f:
f.write(data)
# ✅ use the built-in helper
from verifiers.utils.path_utils import write_temp_file
tmp_path = await asyncio.to_thread(write_temp_file, data, ".txt")
```
Note that `asyncio.to_thread()` releases the event loop but still holds the GIL. For truly CPU-bound operations (heavy computation, compilation, large data transforms >50ms), use a process pool instead:
```python theme={null}
from concurrent.futures import ProcessPoolExecutor
executor = ProcessPoolExecutor(max_workers=4)
async def heavy_reward(data):
loop = asyncio.get_event_loop()
return await loop.run_in_executor(executor, cpu_bound_fn, data)
```
### Executor Autoscaling
`asyncio.to_thread()` dispatches work to a thread pool executor. By default Python's executor is small, but environments can scale it via `set_concurrency()`:
```python theme={null}
env.set_concurrency(256)
```
This resizes both the default event-loop executor (used by `asyncio.to_thread()`) and all registered executors in one call. If your environment creates its own `ThreadPoolExecutor` or `ProcessPoolExecutor` (e.g. for a custom client), register it so it scales automatically:
```python theme={null}
from concurrent.futures import ThreadPoolExecutor # or ProcessPoolExecutor
from verifiers.utils.thread_utils import register_executor, unregister_executor
# register during init — if set_concurrency() was already called,
# the executor is immediately resized to match
self.my_executor = ThreadPoolExecutor(max_workers=4)
register_executor("my-env-client", self.my_executor)
# unregister during teardown (does not shut down the executor)
unregister_executor("my-env-client")
self.my_executor.shutdown()
```
In practice, you rarely need to call `set_concurrency()` yourself. Both `prime eval run` and `prime-rl` automatically compute the right worker count from the concurrency level. If you wish to override the automatic value during evaluation, you can do so with the `--extra-env-kwargs` flag:
```bash theme={null}
prime eval run my-env -x '{"concurrency": 256}'
```
## Integrations and Experimental Environments
Beyond the core environment types, Verifiers includes integrations with several third-party environment libraries, as well as a few newer and more experimental environment classes (which are less stable and more subject to frequent changes).
Supported third-party environment integrations include:
* **`TextArenaEnv`** — wraps [TextArena](https://github.com/LeonGuertler/TextArena) text-based game environments
* **`ReasoningGymEnv`** — wraps [reasoning-gym](https://github.com/open-thought/reasoning-gym) procedural datasets
* **`BrowserEnv`** — unified browser automation via [Browserbase](https://browserbase.com) with DOM and CUA modes
* **`OpenEnvEnv`** — wraps OpenEnv gym and MCP contracts using Prime Sandboxes with prebuilt images referenced from `.build.json`
These require additional dependencies installed via extras (e.g., `uv add 'verifiers[ta]'` for TextArena, `uv add 'verifiers[browser]'` for BrowserEnv, `uv add 'verifiers[openenv]'` for OpenEnvEnv). The bundled OpenEnv project under `proj/` owns its server dependencies and must be built with `uv run vf-build ` before evaluation or training.
Newer and more experimental environment classes include:
* **`GymEnv`** — universal runner for Gym-compatible environments (OpenAI Gym / Gymnasium API)
* **`CliAgentEnv`** — runs agent code inside remote sandboxes, intercepting API requests through the `MultiTurnEnv` rollout loop. Accepts sandbox configuration parameters including `docker_image`, `cpu_cores`, `memory_gb`, `disk_size_gb`, `gpu_count`, `gpu_type`, `timeout_minutes`, `environment_vars`, and `labels` for sandbox categorization. Also accepts retry tuning (like `max_retries`) and connection pooling (like `sandbox_client_max_workers`) parameters via `SandboxMixin`. Subclasses can override `get_sandbox_resources(state)` for per-instance resource allocation and `build_env_vars(state)` for custom environment variables (`PROTECTED_ENV_VARS` cannot be overridden). VMs are auto-enabled when `gpu_count > 0`
* **`SandboxTimeouts`** — frozen dataclass of per-operation HTTP timeouts (seconds) applied to sandbox client calls, exported from `verifiers.envs.experimental.sandbox_mixin`. Fields (with defaults that preserve prior behavior): `read_file=10.0`, `extract=60.0`, `poll=60.0`, `mkdir=10.0`. These are request-level (httpx) timeouts, distinct from `SandboxSpec.timeout_minutes` (container lifetime) and the per-rollout wall-clock cap configured via the `--timeout` CLI flag. Override via the `timeouts` kwarg on `CliAgentEnv.__init__` (which flows through `SandboxMixin.init_sandbox_client`) when the sandbox gateway is slow or geographically distant:
```python theme={null}
from verifiers.envs.experimental.sandbox_mixin import SandboxTimeouts
env = MyCliAgentEnv(
dataset=dataset,
rubric=rubric,
timeouts=SandboxTimeouts(read_file=30.0, extract=180.0, poll=120.0),
)
```
* **`vf.Env` / `vf.Taskset` / `vf.Harness`** — preferred taskset/harness pattern for composing task data and program execution without subclassing. Use this for environments that need reusable tasksets, reusable harnesses, config-driven metrics, rewards, toolsets, users, endpoint interception, or sandboxed Python/command programs. `vf.Taskset` owns train/eval tasks, prompt shaping, setup/update/reward hooks, and toolsets. `vf.Harness` owns the framework program, endpoint proxy, model controls, sandbox options, and runtime hooks. `vf.Env` wires them into the standard evaluation and training surface.
* **`SandboxDebugEnv`** — no-agent debugger for sandbox-backed `SandboxTaskSet` instances. It creates the task sandbox, optionally runs `taskset.setup(state)`, performs one debug step (`none`, `gold_patch`, `command`, or `script`), and optionally runs the task tests and scorer. It records setup, sandbox creation, gold patch, debug command, and test timings in state for validation and timing investigations. `SWEDebugEnv` is a deprecated compatibility wrapper.
* **`HarborEnv`** — loads Harbor-format agent benchmark tasks
* **`OpenCodeEnv`** — runs [OpenCode](https://opencode.ai) CLI agents inside sandboxes with API call interception
* **`OpenCodeRLMEnv`** — extends `OpenCodeEnv` with concurrent sub-LLM handling via the [OC plugin](https://github.com/snimu/oc), routing `subagent`/`llm-subcall` requests through the interception proxy
# Evaluation
Source: https://docs.primeintellect.ai/verifiers/v0/evaluation
v0 is considered deprecated and will be fully removed in a future release.
This section explains how to run evaluations with Verifiers environments. See [Environments](/verifiers/v0/environments) for information on building your own environments.
## Table of Contents
* [Basic Usage](#basic-usage)
* [Hosted Evaluations](#hosted-evaluations)
* [Command Reference](#command-reference)
* [Environment Selection](#environment-selection)
* [Model Configuration](#model-configuration)
* [Sampling Parameters](#sampling-parameters)
* [Evaluation Scope](#evaluation-scope)
* [Concurrency](#concurrency)
* [Output and Saving](#output-and-saving)
* [Resuming Evaluations](#resuming-evaluations)
* [Environment Defaults](#environment-defaults)
* [Multi-Environment Evaluation](#multi-environment-evaluation)
* [TOML Configuration](#toml-configuration)
* [Ablation Sweeps](#ablation-sweeps)
* [Configuration Precedence](#configuration-precedence)
Use `prime eval` to execute rollouts against any supported model provider and report aggregate metrics. Supported providers include OpenAI-compatible APIs (the default) and the Anthropic Messages API (via `--api-client-type anthropic_messages`).
## Basic Usage
Run evaluations directly against a local or Hub environment:
```bash theme={null}
prime eval run my-env -m openai/gpt-4.1-mini -n 10
```
`prime eval` resolves and installs the environment when needed, imports the environment module using Python's import system, calls its `load_environment()` function, runs 5 examples with 3 rollouts each (the default), scores them using the environment's rubric, and prints aggregate metrics.
## Hosted Evaluations
You can also run evaluations on Prime-managed infrastructure with `prime eval run --hosted`. Hosted evaluations require an environment that has already been published to the Environments Hub, and they are useful when you want Prime to manage execution, monitor logs remotely, or run against a shared Hub environment slug instead of a local package.
```bash theme={null}
prime env push my-env
prime eval run my-env --hosted
prime eval run my-env --hosted --follow
```
Hosted runs also support TOML configs:
```bash theme={null}
prime eval run configs/eval/benchmark-hosted.toml --hosted
```
For the full hosted workflow and hosted-only flags such as `--follow`, `--timeout-minutes`, `--allow-sandbox-access`, and `--custom-secrets`, see the official [Hosted Evaluations](https://docs.primeintellect.ai/tutorials-environments/hosted-evaluations) guide.
## Command Reference
### Environment Selection
| Flag | Short | Default | Description |
| -------------------- | ------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `env_id_or_path` | (positional) | — | Environment ID(s) or path to TOML config |
| `--env-args` | `-a` | `{}` | JSON object passed to `load_environment()` |
| `--extra-env-kwargs` | `-x` | `{}` | JSON object passed to environment constructor |
| `--timeout` | — | `None` | Per-rollout wall-clock timeout in seconds. Wins over equivalent values in `--extra-env-kwargs` or TOML `[eval.extra_env_kwargs]`. Bounds generation only — scoring is not bounded. |
| `--env-dir-path` | `-p` | `./environments` | Base path for saving output files |
The positional argument accepts two formats:
* **Single environment**: `gsm8k` — evaluates one environment
* **TOML config path**: `configs/eval/benchmark.toml` — evaluates multiple environments defined in the config file
Environment IDs are converted to Python module names (`my-env` → `my_env`) and imported after `prime eval run` resolves the environment package.
For legacy or direct-constructor environments, the `--env-args` flag passes arguments to your `load_environment()` function:
```bash theme={null}
prime eval run my-env -a '{"difficulty": "hard", "num_examples": 100}'
```
The `--extra-env-kwargs` flag passes arguments directly to the environment constructor, useful for overriding defaults like `max_turns` which may not be exposed via `load_environment()`:
```bash theme={null}
prime eval run my-env -x '{"max_turns": 20}'
```
For per-rollout wall-clock timeouts, use the dedicated `--timeout` flag. It **wins over** equivalent values set via `--extra-env-kwargs` or TOML's `[eval.extra_env_kwargs]`:
```bash theme={null}
prime eval run my-env --timeout 600
```
When the timeout fires, the rollout is marked `timed_out=True` with `stop_condition="timeout_reached"` and ends cleanly through the same finalize path as a normal completion (timing, completion rendering, and cleanup handlers all run). `is_truncated` is **not** set on timeout — that field tracks model-output truncation (token caps), not wall-clock pressure. Note that `--timeout` bounds the **generation** phase only — scoring (reward function execution) is not bounded by this flag.
The same key works in TOML configs as a top-level entry of an `[[eval]]` table:
```toml theme={null}
[[eval]]
id = "my-env"
timeout = 600
```
#### Executor autoscaling
Thread-pool executors are automatically sized to match the evaluation concurrency. During `prime eval run`, if `concurrency` is not explicitly provided via `--extra-env-kwargs`, it is computed from the concurrency level (`max_concurrent`, or `num_examples * rollouts_per_example` when unlimited) using `recommended_max_workers()`. This value is passed to `Environment.set_concurrency()`, which resizes both the default event-loop executor and all registered executors.
To override the automatic value:
```bash theme={null}
prime eval run my-env -x '{"concurrency": 256}'
```
You can also call `set_concurrency()` directly at runtime:
```python theme={null}
env.set_concurrency(256)
```
### Model Configuration
| Flag | Short | Default | Description |
| --------------------- | ----- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--model` | `-m` | `openai/gpt-4.1-mini` | Model name or endpoint alias |
| `--api-base-url` | `-b` | `https://api.pinference.ai/api/v1` | API base URL |
| `--api-key-var` | `-k` | `PRIME_API_KEY` | Environment variable containing API key |
| `--api-client-type` | — | `openai_chat_completions` | Client type: `openai_completions`, `openai_chat_completions`, `openai_chat_completions_token`, `openai_responses`, `renderer`, `anthropic_messages`, or `nemorl_chat_completions` |
| `--endpoints-path` | `-e` | `./configs/endpoints.toml` | Path to TOML endpoints registry |
| `--header` | — | — | Extra HTTP header (`Name: Value`), repeatable |
| `--header-from-state` | — | framework session id | Per-request header whose value is read from rollout state (`Name: state_key`), repeatable |
For convenience, define model endpoints in `./configs/endpoints.toml` to avoid repeating URL and key flags.
```toml theme={null}
[[endpoint]]
endpoint_id = "gpt-4.1-mini"
model = "gpt-4.1-mini"
url = "https://api.openai.com/v1"
key = "OPENAI_API_KEY"
[[endpoint]]
endpoint_id = "qwen3-235b-i"
model = "qwen/qwen3-235b-a22b-instruct-2507"
url = "https://api.pinference.ai/api/v1"
key = "PRIME_API_KEY"
[[endpoint]]
endpoint_id = "claude-sonnet"
model = "claude-sonnet-4-5-20250929"
url = "https://api.anthropic.com"
key = "ANTHROPIC_API_KEY"
api_client_type = "anthropic_messages"
```
Each endpoint entry supports an optional `api_client_type` field to select the client implementation (defaults to `"openai_chat_completions"`). Use `"anthropic_messages"` for Anthropic models when calling the Anthropic API directly, and `"openai_responses"` for OpenAI-compatible Responses endpoints.
Optional HTTP headers for inference requests use a short TOML key `headers` (inline table). The alias `extra_headers` is accepted with the same shape; do not set both on one row.
```toml theme={null}
[[endpoint]]
endpoint_id = "my-proxy"
model = "gpt-4.1-mini"
url = "https://api.example/v1"
key = "OPENAI_API_KEY"
headers = { "X-Custom-Header" = "value" }
```
In `[[eval]]` TOML configs you can set extra headers as `headers = { ... }` and/or as a list `header = ["Name: Value", ...]` (same form as repeated `--header`). Merge order is: registry row, then the `headers` table, then each `header` / `--header` line, with later entries overriding the same name.
For per-request headers that need to vary per rollout, use `headers_from_state = { "X-Name" = "state_key" }` and/or `header_from_state = ["X-Name: state_key", ...]` (same form as repeated `--header-from-state`). The value for each request is resolved at send time as `state[state_key]`. If unset, Verifiers supplies a framework-managed `X-Session-ID`.
To define equivalent replicas, add multiple `[[endpoint]]` entries with the same `endpoint_id`.
Then use the alias directly:
```bash theme={null}
prime eval run my-env -m qwen3-235b-i
```
If the model name is in the registry, those values are used by default, but you can override them with `--api-base-url` and/or `--api-key-var`. If the model name isn't found, the CLI flags are used (falling back to defaults when omitted).
In other words, `-m/--model` is treated as an endpoint alias lookup when present in the registry, and otherwise treated as a literal model id.
When using eval TOML configs, you can set `endpoint_id` in `[[eval]]` sections to resolve from the endpoint registry. `endpoint_id` is only supported when `endpoints_path` points to a TOML registry file.
### Sampling Parameters
| Flag | Short | Default | Description |
| ----------------- | ----- | ------------- | ---------------------------------------------- |
| `--max-tokens` | `-t` | model default | Maximum tokens to generate |
| `--temperature` | `-T` | model default | Sampling temperature |
| `--sampling-args` | `-S` | — | JSON object for additional sampling parameters |
The `--sampling-args` flag accepts any parameters supported by the model's API:
```bash theme={null}
prime eval run my-env -S '{"temperature": 0.7, "top_p": 0.9}'
```
In eval TOML configs, put generation parameters under `[sampling]`:
```toml theme={null}
[sampling]
max_tokens = 1024
temperature = 0.7
reasoning_effort = "medium"
enable_thinking = true
[[eval]]
id = "my-env"
```
`reasoning_effort` and `enable_thinking` stay in `sampling_args` and are also mirrored into `extra_body.chat_template_kwargs` for OpenAI-compatible servers that read chat template options there. Keeping the top-level values lets the client translate them for the selected provider.
### Evaluation Scope
| Flag | Short | Default | Description |
| ------------------------ | ----- | --------------- | -------------------------------------------------------- |
| `--num-examples` | `-n` | 5 | Number of dataset examples to evaluate |
| `--rollouts-per-example` | `-r` | 3 | Rollouts per example (for pass\@k, variance) |
| `--shuffle` | — | false | Shuffle the evaluation dataset before selecting examples |
| `--shuffle-seed` | — | 0 when shuffled | Seed used with `--shuffle` |
Multiple rollouts per example enable metrics like pass\@k and help measure variance. The total number of rollouts is `num_examples × rollouts_per_example`.
When `--shuffle` is enabled, Verifiers shuffles the full evaluation dataset first, then selects `--num-examples`, then repeats each selected example according to `--rollouts-per-example`. This applies to standard environments and composable tasksets because it happens in the shared evaluation input-selection layer.
### Concurrency
| Flag | Short | Default | Description |
| ----------------------------- | ----- | ------------ | ----------------------------------------------------------------------------- |
| `--max-concurrent` | `-c` | 32 | Maximum concurrent requests |
| `--max-concurrent-generation` | — | same as `-c` | Concurrent generation requests |
| `--max-concurrent-scoring` | — | same as `-c` | Concurrent scoring requests |
| `--no-interleave-scoring` | `-N` | false | Disable interleaved scoring |
| `--independent-scoring` | `-i` | false | Score each rollout individually instead of by group |
| `--max-retries` | — | 0 | Retries per rollout on transient `InfraError` |
| `--num-workers` | `-w` | `auto` | Number of env server worker processes (`auto` = concurrency ÷ 256, minimum 1) |
By default, scoring runs interleaved with generation. Use `--no-interleave-scoring` to score all rollouts after generation completes.
The `--max-retries` flag enables automatic retry with exponential backoff when rollouts fail due to transient infrastructure errors (e.g., sandbox timeouts, API failures).
The `--num-workers` flag controls how many worker processes the env server spawns. Each worker owns its own environment instance and runs rollouts independently. The default `auto` scales with concurrency.
### Display
When evaluating multiple environments, the display shows an overview panel at the top with a compact status line per environment, and a detail panel below with full progress, metrics, and logs for one environment at a time. Use the **left/right arrow keys** to switch between environments. The overview scrolls to keep the selected environment visible and is capped at half the terminal height.
When an eval runs against a Prime Inference endpoint and the model id has pricing available from `prime inference models`, token usage rows also show the estimated total USD cost as `cost (all)`. Cost appears after the final input/output token metrics in the live display and in the final summary. If pricing or token usage is unavailable, the cost field is omitted.
### Output and Saving
| Flag | Short | Default | Description |
| ----------------------- | ----- | ------- | --------------------------------------------------------------------------------------- |
| `--verbose` | `-v` | false | Enable debug logging |
| `--fullscreen` | `-f` | false | Use alternate screen buffer (fullscreen) for the Rich display |
| `--disable-tui` | `-d` | false | Disable Rich display; use normal logging and tqdm progress |
| `--abbreviated-summary` | `-A` | false | Abbreviated summary: show settings and stats, skip example prompts |
| `--output-dir` | `-o` | — | Custom output directory for evaluation results and logs |
| `--save-results` | `-s` | false | Save results to disk |
| `--resume [PATH]` | `-R` | — | Resume from a previous run (auto-detect latest matching incomplete run if PATH omitted) |
| `--state-columns` | `-C` | — | Extra state columns to save (comma-separated) |
| `--save-to-hf-hub` | `-H` | false | Push results to Hugging Face Hub |
| `--hf-hub-dataset-name` | `-D` | — | Dataset name for HF Hub |
| `--heartbeat-url` | — | — | Heartbeat URL for uptime monitoring |
By default, results are saved to `./outputs/evals/{env_id}--{model}/{run_id}/`. Use `--output-dir` to override the base output directory — when set, results (and logs) are saved under `{output_dir}/evals/{env_id}--{model}/{run_id}/` instead. The directory contains:
* `results.jsonl` — rollout outputs, one per line
* `metadata.json` — evaluation configuration and aggregate metrics
When Prime Inference pricing is available for the evaluated model, `metadata.json` includes a `cost` object with total-run `input_usd`, `output_usd`, and `total_usd`. The field is omitted for unpriced models, third-party providers, unavailable pricing, or missing usage.
### Resuming Evaluations
Long-running evaluations can be interrupted and resumed using checkpointing. When `--save-results` is enabled, results are saved incrementally after each completed group of rollouts. Use `--resume` to continue from where you left off. Pass a path to resume a specific run, or omit the path to auto-detect the latest incomplete matching run.
**Running with checkpoints:**
```bash theme={null}
prime eval run my-env -n 1000 -s
```
With `-s` (save results) enabled, partial results are written to disk after each group completes. If the evaluation is interrupted, the output directory will contain all completed rollouts up until the interruption.
**Resuming from a checkpoint:**
```bash theme={null}
prime eval run my-env -n 1000 -s --resume ./environments/my_env/outputs/evals/my-env--openai--gpt-4.1-mini/abc12345
```
When a resume path is provided, it must point to a valid evaluation results directory containing both `results.jsonl` and `metadata.json`. With `--resume` and no path, verifiers scans the environment/model output directory and picks the most recent incomplete run matching `env_id`, `model`, `rollouts_per_example`, `shuffle`, and `shuffle_seed` where saved `num_examples` is less than or equal to the current run. When resuming:
1. Existing completed rollouts are loaded from the checkpoint
2. Remaining rollouts are computed based on the example ids and group size
3. Only incomplete rollouts are executed
4. New results are appended to the existing checkpoint
If all rollouts are already complete, the evaluation returns immediately with the existing results.
**Configuration compatibility:**
When resuming, the current run configuration should match the original run. Mismatches in parameters like `--model`, `--env-args`, `--rollouts-per-example`, `--shuffle`, or `--shuffle-seed` can lead to undefined behavior. For reliable results, resume with the same configuration used to create the checkpoint, only increasing `--num-examples` if you need additional rollouts beyond the original target.
**Example workflow:**
```bash theme={null}
# Start a large evaluation with checkpointing
prime eval run my-env -n 500 -r 3 -s
# If interrupted, find the run directory
ls ./environments/my_env/outputs/evals/my-env--openai--gpt-4.1-mini/
# Resume from the checkpoint
prime eval run my-env -n 500 -r 3 -s \
--resume ./environments/my_env/outputs/evals/my-env--openai--gpt-4.1-mini/abc12345
```
The `--state-columns` flag allows saving environment-specific state fields that your environment stores during rollouts:
```bash theme={null}
prime eval run my-env -s -C "judge_response,parsed_answer"
```
## Environment Defaults
Environments can specify default evaluation parameters in their `pyproject.toml` (See [Developing Environments](/verifiers/v0/environments#developing-environments)):
```toml theme={null}
[tool.verifiers.eval]
num_examples = 100
rollouts_per_example = 5
```
These defaults are used when higher-priority sources don't specify a value. The full priority order is:
1. TOML per-environment settings (when using a config file)
2. CLI flags
3. Environment defaults (from `pyproject.toml`)
4. Global defaults
See [Configuration Precedence](#configuration-precedence) for more details on multi-environment evaluation.
## Multi-Environment Evaluation
You can evaluate multiple environments using `prime eval` with a TOML configuration file. This is useful for running comprehensive benchmark suites.
### TOML Configuration
For multi-environment evals or fine-grained control over settings, use a TOML configuration file. When using a config file, CLI arguments are ignored.
```bash theme={null}
prime eval run configs/eval/my-benchmark.toml
```
The TOML file uses `[[eval]]` sections to define each evaluation. You can also specify global defaults at the top:
```toml theme={null}
# configs/eval/my-benchmark.toml
# Global defaults (optional)
model = "openai/gpt-4.1-mini"
num_examples = 50
[[eval]]
id = "gsm8k"
name = "gsm8k-baseline"
num_examples = 100 # overrides global default
rollouts_per_example = 5
[[eval]]
id = "alphabet-sort"
# Uses global num_examples (50)
rollouts_per_example = 3
[[eval]]
id = "math-python"
# Uses global defaults and built-in defaults for unspecified values
```
A minimal config requires only a single `[[eval]]` section:
```toml theme={null}
[[eval]]
id = "gsm8k"
```
Each `[[eval]]` section usually contains an `id` field. `env_id` is accepted as a legacy alias and normalizes to the same internal field. All other fields are optional:
| Field | Type | Description |
| ---------------------- | ------- | -------------------------------------------------------- |
| `id` | string | Environment module name |
| `name` | string | Optional eval label for display and saved result paths |
| `args` | table | Arguments passed to `load_environment()` |
| `num_examples` | integer | Number of dataset examples to evaluate |
| `rollouts_per_example` | integer | Rollouts per example |
| `shuffle` | boolean | Shuffle the evaluation dataset before selecting examples |
| `shuffle_seed` | integer | Seed used when `shuffle = true` |
| `extra_env_kwargs` | table | Arguments passed to environment constructor |
| `model` | string | Model to evaluate |
| `endpoint_id` | string | Endpoint registry id (requires TOML `endpoints_path`) |
| `sampling` | table | Shorthand for `sampling_args` generation parameters |
Use `name` to run the same environment more than once with different args:
```toml theme={null}
[[eval]]
id = "reverse-text"
name = "reverse-text-short"
[eval.args]
max_length = 32
[[eval]]
id = "reverse-text"
name = "reverse-text-long"
[eval.args]
max_length = 256
```
Example with legacy environment args:
```toml theme={null}
[[eval]]
id = "math-python"
num_examples = 50
[eval.args]
difficulty = "hard"
split = "test"
```
The legacy inline `sampling_args = { ... }` spelling is still accepted and normalizes the same way as `[sampling]` / `[eval.sampling]`.
### Ablation Sweeps
Use `[[ablation]]` blocks to automatically generate eval configs from a cartesian product of parameter values. This is useful for hyperparameter sweeps and ablation studies without manually writing each combination.
```toml theme={null}
# Global defaults apply to all evals and ablations
model = "openai/gpt-4.1-mini"
num_examples = 50
# Sweep temperature × difficulty → 6 eval configs
# split is fixed across all combinations
[[ablation]]
id = "my-env"
args = {split = "test"}
[ablation.sweep]
temperature = [0.0, 0.5, 1.0]
[ablation.sweep.args]
difficulty = ["easy", "hard"]
```
* **Fixed fields** in the `[[ablation]]` block (like `id`) apply to all expanded configs
* **`[ablation.sweep]`** keys are lists of values crossed as a cartesian product
* **`[ablation.sweep.args]`** keys are swept and merged into environment args
* **Fixed `args`** can be set alongside swept ones (e.g. `args = {split = "test"}` keeps `split` fixed while sweeping other env args). The same key cannot appear in both fixed and swept args.
* Multiple `[[ablation]]` blocks are independent (no cross-product between blocks)
* `[[ablation]]` and `[[eval]]` blocks can coexist in the same config file
* `id` can be a fixed field or a sweep key (e.g. `id = ["env-a", "env-b"]`), but note that all swept envs must accept the same `args` — use separate `[[ablation]]` blocks for envs with different argument schemas
Use `--abbreviated-summary` (`-A`) to get a compact summary focused on settings and stats, which is useful when comparing many ablation runs.
### Configuration Precedence
When using a **config file**, CLI arguments are ignored. Settings are resolved as:
1. **TOML per-eval settings** — Values specified in `[[eval]]` sections
2. **TOML global settings** — Values at the top of the config file
3. **Environment defaults** — Values from the environment's `pyproject.toml`
4. **Built-in defaults** — (`num_examples=5`, `rollouts_per_example=3`)
When using **CLI only** (no config file), settings are resolved as:
1. **CLI arguments** — Flags passed on the command line
2. **Environment defaults** — Values from the environment's `pyproject.toml`
3. **Built-in defaults** — (`num_examples=5`, `rollouts_per_example=3`)
# Faqs
Source: https://docs.primeintellect.ai/verifiers/v0/faqs
v0 is considered deprecated and will be fully removed in a future release.
## Getting Started
### How do I quickly test my environment?
Use `prime eval run` with a small sample:
```bash theme={null}
prime eval run my-environment -m openai/gpt-4.1-mini -n 5
```
The `-s` flag prints sample outputs so you can see what's happening.
### How do I see what the model is outputting?
**If using `prime eval run`**: Results are saved automatically. Browse them interactively with:
```bash theme={null}
prime eval view
```
The TUI opens a single run browser (`environment -> model -> run`). Press `Enter` on a run to open rollout details, `b` to go back, `tab` to cycle panes, `e` and `x` to expand or collapse history, `pageup` and `pagedown` to scroll history, and `c` for Copy Mode.
**If using the Python API** (`env.generate()` / `env.evaluate()`):
```python theme={null}
vf.print_prompt_completions_sample(outputs, n=3)
```
### How do I enable debug logging?
Set the `VF_LOG_LEVEL` environment variable:
```bash theme={null}
VF_LOG_LEVEL=DEBUG prime eval run my-environment -m openai/gpt-4.1-mini -n 5
```
## Environments
### Which environment class should I use?
* **SingleTurnEnv**: One prompt, one response (Q\&A, classification)
* **MultiTurnEnv**: Custom back-and-forth interaction (games, simulations)
* **ToolEnv**: Model calls Python functions (search, calculator)
* **StatefulToolEnv**: Tools that need per-rollout state (sandbox IDs, sessions)
### What does `max_turns=-1` mean?
Unlimited turns. The rollout continues until a stop condition is triggered (e.g., model stops calling tools, or a custom condition you define).
### How do I add a custom stop condition?
Use the `@vf.stop` decorator on a method that returns `True` to end the rollout:
```python theme={null}
@vf.stop
async def task_completed(self, state: State) -> bool:
return "DONE" in state["completion"][-1]["content"]
```
### How do I handle tool call errors gracefully?
In `ToolEnv`, customize error handling:
```python theme={null}
env = ToolEnv(
tools=[my_tool],
error_formatter=lambda e: f"Error: {type(e).__name__}: {e}",
stop_errors=[CriticalError], # These errors end the rollout
)
```
Non-critical errors are returned to the model as tool responses so it can retry.
## Reward Functions
### What arguments can my reward function receive?
Reward functions receive any of these via `**kwargs`:
* `completion` - the model's response
* `answer` - ground truth from dataset
* `prompt` - the input prompt
* `state` - full rollout state
* `parser` - the rubric's parser (if set)
* `task` - `vf.Task` object for taskset-backed environments
* `info` - metadata dict from dataset
Just include the ones you need in your function signature.
### How do group reward functions work?
Group reward functions receive plural arguments (`completions`, `answers`, `states`) and return a list of floats. They're detected automatically by parameter names:
```python theme={null}
def relative_reward(completions: list, answers: list, **kwargs) -> list[float]:
# Score all completions for an example together
scores = [compute_score(c, a) for c, a in zip(completions, answers)]
# Normalize relative to group
max_score = max(scores) if scores else 1.0
return [s / max_score for s in scores]
```
## Training
### How do I use a local vLLM server?
Point the client to your local server:
```python theme={null}
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed"
)
outputs = await env.evaluate(client, model="your-model-name", ...)
```
### Which `client_type` should I use for RL training?
Three options trade off control vs simplicity:
* **`openai_chat_completions`** (MITO) — server-side templating, text only. Standard OpenAI path. The trainer re-tokenizes for training, which can drift across multi-turn rollouts and fragment them into multiple samples.
* **`openai_chat_completions_token`** (TITO) — server-side templating, returns token IDs alongside text. The trainer doesn't re-tokenize. Use when the server's chat template is stable across turns.
* **`renderer`** — client-side tokenization via a per-model renderer in the [`renderers` package](https://github.com/PrimeIntellect-ai/verifiers/tree/main/packages/renderers). Stronger token-preservation in theory: `bridge_to_next_turn` keeps multi-turn rollouts merged into one sample and survives mid-completion truncation cleanly. Hand-coded renderers exist only for a subset of models and corner cases are still being shaken out.
For production training, use `openai_chat_completions_token` — it's the tried-and-tested path. Try `renderer` if you want the stronger guarantees and your model has a hand-coded renderer. See [Inference Client Types](/verifiers/v0/training#inference-client-types) for the full breakdown.
# Overview
Source: https://docs.primeintellect.ai/verifiers/v0/overview
v0 is considered deprecated and will be fully removed in a future release.
Verifiers is our library for creating environments to train and evaluate LLMs.
Environments contain everything required to run and evaluate a model on a particular task:
* A *dataset* of task inputs
* A *harness* for the model (tools, sandboxes, context management, etc.)
* A reward function or *rubric* to score the model's performance
Environments can be used for training models with reinforcement learning (RL), evaluating capabilities, generating synthetic data, experimenting with agent harnesses, and more.
Verifiers is tightly integrated with the [Environments Hub](https://app.primeintellect.ai/dashboard/environments?ex_sort=most_stars), as well as our training framework [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) and our [Hosted Training](https://app.primeintellect.ai/dashboard/training) platform.
## Getting Started
Ensure you have `uv` installed, as well as the `prime` [CLI](https://docs.primeintellect.ai/cli-reference/introduction) tool:
```bash theme={null}
# install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# install the prime CLI
uv tool install prime
# log in to the Prime Intellect platform
prime login
```
To set up a new workspace for developing environments, do:
```bash theme={null}
# ~/dev/my-lab
prime lab setup
```
This sets up a Python project if needed (with `uv init`), installs `verifiers` (with `uv add verifiers`), creates the recommended workspace structure, and downloads useful starter files:
```text theme={null}
configs/
├── endpoints.toml # OpenAI-compatible API endpoint configuration
├── rl/ # Example configs for Hosted Training
├── eval/ # Example multi-environment eval configs
└── gepa/ # Example configs for prompt optimization
.prime/
└── skills/ # Bundled workflow skills for create/browse/review/eval/GEPA/train/brainstorm
environments/
└── AGENTS.md # Documentation for AI coding agents
AGENTS.md # Top-level documentation for AI coding agents
CLAUDE.md # Claude-specific pointer to AGENTS.md
```
Alternatively, add `verifiers` to an existing project:
```bash theme={null}
uv add verifiers && prime lab setup --skip-install
```
Optional features are installed with extras:
| Extra | Install | Enables |
| ---------- | ------------------------------ | ------------------------------------------------------------------------- |
| `modal` | `uv add "verifiers[modal]"` | The v1 Modal sandbox runtime |
| `notebook` | `uv add "verifiers[notebook]"` | `Environment.generate_sync()` inside Jupyter or another active event loop |
Environments built with Verifiers are self-contained Python modules. To initialize a fresh environment template, do:
```bash theme={null}
prime env init my-env # creates a v0 stub in ./environments/my_env
```
This will create a new module called `my_env` with a runnable environment template.
```text theme={null}
environments/my_env/
├── my_env.py # Main implementation
├── pyproject.toml # Dependencies and metadata
└── README.md # Documentation
```
Environment modules should expose a `load_environment` function which returns an environment object. For simple legacy environments, this can still be a direct constructor:
```python theme={null}
# my_env.py
import verifiers as vf
def load_environment(dataset_name: str = "gsm8k") -> vf.Environment:
dataset = vf.load_example_dataset(dataset_name) # 'question'
async def correct_answer(completion, answer) -> float:
completion_ans = completion[-1]["content"]
return 1.0 if completion_ans == answer else 0.0
rubric = vf.Rubric(funcs=[correct_answer])
env = vf.SingleTurnEnv(dataset=dataset, rubric=rubric)
return env
```
To run a local evaluation with any OpenAI-compatible model, do:
```bash theme={null}
prime eval run my-env -m openai/gpt-5-nano # run and save eval results locally
```
Evaluations use [Prime Inference](https://docs.primeintellect.ai/inference/overview) by default; configure your own API endpoints in `./configs/endpoints.toml`.
View local evaluation results in the terminal UI:
```bash theme={null}
prime eval view
```
The TUI opens a single run browser (`environment -> model -> run`). Press `Enter` on a run to open rollout details, `b` to go back, `tab` to cycle panes, `e` and `x` to expand or collapse history, `pageup` and `pagedown` to scroll history, and `c` for Copy Mode.
To publish the environment to the [Environments Hub](https://app.primeintellect.ai/dashboard/environments?ex_sort=most_stars), do:
```bash theme={null}
prime env push my-env # equivalent to --path ./environments/my_env
```
To run an evaluation directly from the Environments Hub, do:
```bash theme={null}
prime eval run primeintellect/math-python
```
## Documentation
**[Environments](/verifiers/v0/environments)** — Create datasets, rubrics, and custom multi-turn interaction protocols.
**[Evaluation](/verifiers/v0/evaluation)** - Evaluate models using your environments.
**[Training](/verifiers/v0/training)** — Train models in your environments with reinforcement learning.
**[Development](/verifiers/v0/development)** — Contributing to verifiers
**[API Reference](/verifiers/v0/reference)** — Understanding the API and data structures
**[FAQs](/verifiers/v0/faqs)** - Other frequently asked questions.
# Reference
Source: https://docs.primeintellect.ai/verifiers/v0/reference
v0 is considered deprecated and will be fully removed in a future release.
## Table of Contents
* [Type Aliases](#type-aliases)
* [Data Types](#data-types)
* [Classes](#classes)
* [Environment Classes](#environment-classes)
* [Parser Classes](#parser-classes)
* [Rubric Classes](#rubric-classes)
* [Client Classes](#client-classes)
* [Configuration Types](#configuration-types)
* [Prime CLI Plugin](#prime-cli-plugin)
* [Decorators](#decorators)
* [Utility Functions](#utility-functions)
***
## Type Aliases
### Messages
```python theme={null}
Messages = str | list[ChatMessage]
```
The primary message type. Either a plain string (completion mode) or a list of chat messages (chat mode).
### ChatMessage
```python theme={null}
ChatMessage = ChatCompletionMessageParam # from openai.types.chat
```
OpenAI's chat message type with `role`, `content`, and optional `tool_calls` / `tool_call_id` fields.
### SystemMessage
```python theme={null}
class SystemMessage:
role: Literal["system"] = "system"
content: MessageContent
@classmethod
def from_path(cls, path: str | Path) -> "SystemMessage": ...
```
Provider-agnostic system message type. Use `vf.SystemMessage.from_path(...)` to load a system prompt from a UTF-8 text file while preserving the file contents verbatim.
### Info
```python theme={null}
Info = dict[str, Any]
```
Arbitrary metadata dictionary from dataset rows.
### SamplingArgs
```python theme={null}
SamplingArgs = dict[str, Any]
```
Generation parameters passed to the inference server (e.g., `temperature`, `top_p`, `max_tokens`).
### SystemPrompt
```python theme={null}
SystemPrompt = PromptInput | SystemPromptConfig | None
SystemPromptStrategy = Literal["HT", "TH", "H_OR_T", "T_OR_H", "H", "T", "REJECT"]
class SystemPromptConfig:
path: str | None = None
messages: list[JsonData] = []
```
v1 system prompt type. Plain strings are prompt text. Use `vf.SystemPromptConfig(path="system_prompt.txt")` for file-backed prompts, or override `load_system_prompt(config)` when prompt construction belongs to the class. System prompt resolution is per task: task prompt overrides taskset prompt for the taskset side, then the harness applies `system_prompt_strategy`. The default strategy is `HT`.
### RewardFunc
```python theme={null}
IndividualRewardFunc = Callable[..., float | Awaitable[float]]
GroupRewardFunc = Callable[..., list[float] | Awaitable[list[float]]]
RewardFunc = IndividualRewardFunc | GroupRewardFunc
```
Individual reward functions operate on single rollouts. Group reward functions operate on all rollouts for an example together (useful for relative scoring).
### ClientType
```python theme={null}
ClientType = Literal[
"openai_completions",
"openai_chat_completions",
"openai_chat_completions_token",
"openai_responses",
"renderer",
"anthropic_messages",
"nemorl_chat_completions",
]
```
Selects which `Client` implementation to use. Set via `ClientConfig.client_type`.
***
## Data Types
### State
```python theme={null}
class State(dict):
INPUT_FIELDS = ["prompt", "answer", "info"]
```
A `dict` subclass that tracks rollout information. Accessing keys in `INPUT_FIELDS` automatically forwards to the nested `input` object.
**Fields set during initialization:**
| Field | Type | Description |
| --------------- | ---------------------- | -------------------------------- |
| `input` | `RolloutInput` | Nested input data |
| `client` | `Client` | Client instance |
| `model` | `str` | Model name |
| `sampling_args` | `SamplingArgs \| None` | Generation parameters |
| `is_completed` | `bool` | Whether rollout has ended |
| `is_truncated` | `bool` | Whether generation was truncated |
| `tool_defs` | `list[Tool] \| None` | Available tool definitions |
| `trajectory` | `list[TrajectoryStep]` | Multi-turn trajectory |
| `trajectory_id` | `str` | UUID for this rollout |
| `timing` | `RolloutTiming` | Timing information |
**Fields set after scoring:**
| Field | Type | Description |
| ---------------- | -------------------------- | -------------------------------- |
| `completion` | `Messages \| None` | Final completion |
| `reward` | `float \| None` | Final reward |
| `advantage` | `float \| None` | Advantage over group mean |
| `metrics` | `dict[str, float] \| None` | Per-function metrics |
| `stop_condition` | `str \| None` | Name of triggered stop condition |
| `error` | `Error \| None` | Error if rollout failed |
### RolloutInput
```python theme={null}
class RolloutInput(TypedDict):
prompt: Messages # Required
answer: str # Optional
info: Info # Optional
```
### RolloutOutput
```python theme={null}
class RolloutOutput(dict):
# Required fields
prompt: Messages | None
completion: Messages | None
reward: float
timing: RolloutTiming
is_completed: bool
is_truncated: bool
metrics: dict[str, float]
# Optional fields
answer: str
info: Info
error: str | None
stop_condition: str | None
token_usage: TokenUsage
trajectory: list[TrajectoryStep]
tool_defs: list[Tool] | None
```
Serialized output from a rollout. This is a `dict` subclass that provides typed access to known fields while supporting arbitrary additional fields from `state_columns`. All values must be JSON-serializable. Used in `GenerateOutputs` and for saving results to disk.
### TrajectoryStep
```python theme={null}
class TrajectoryStep(TypedDict):
prompt: Messages
completion: Messages
response: Response
tokens: TrajectoryStepTokens | None
reward: float | None
advantage: float | None
is_truncated: bool
trajectory_id: str
extras: dict[str, Any]
```
A single turn in a multi-turn rollout.
### RoutedExpertsPayload
```python theme={null}
class RoutedExpertsPayload(TypedDict):
data: Any # actually memoryview; kept opaque so Pydantic skips schema validation
shape: list[int]
start: int
```
### TrajectoryStepTokens
```python theme={null}
class TrajectoryStepTokens(TypedDict):
prompt_ids: list[int]
prompt_mask: list[int]
completion_ids: list[int]
completion_mask: list[int]
completion_logprobs: list[float]
overlong_prompt: bool
is_truncated: bool
routed_experts: RoutedExpertsPayload | None
multi_modal_data: NotRequired[
Any
] # renderers.MultiModalData sidecar (pixel_values, placeholder ranges) — set only on multimodal rollouts
prompt_attribution: NotRequired[
Any
] # renderers.RenderedTokens fields as a dict (per-token is_content / sampled_mask / message_indices / message_roles) — set only on RendererClient rollouts
```
Token-level data for training.
### TimeSpan
```python theme={null}
class TimeSpan(CustomBaseModel):
"""A timed span. duration = end - start."""
start: float = 0.0 # Unix timestamp (seconds since epoch)
end: float = 0.0 # Unix timestamp (seconds since epoch)
# duration: float (computed_field)
```
### TimeSpans
```python theme={null}
class TimeSpans(CustomBaseModel):
"""A list of TimeSpan with aggregate duration (sum)."""
spans: list[TimeSpan] = []
# duration: float (computed_field)
```
### RolloutTiming
```python theme={null}
class RolloutTiming(CustomBaseModel):
"""Rollout-level timing. All values in seconds."""
start_time: float # wall-clock at rollout start
setup: TimeSpan = TimeSpan() # setup_state() span
generation: TimeSpan = TimeSpan() # full generation phase
scoring: TimeSpan = TimeSpan() # rubric.score_*() span
model: TimeSpans = TimeSpans() # all model-call spans
env: TimeSpans = TimeSpans() # all env-response spans
# total, overhead: float (computed_fields)
```
Derivations:
* `total = scoring.end - generation.start`
* `overhead = total - setup.duration - model.duration - env.duration - scoring.duration`
`generation.start` is stamped at the top of the rollout (before `setup_state`), so `total` covers the entire rollout including setup, generation loop, finalize, and scoring. `overhead` captures any time not attributed to the named phases.
### TokenUsage
```python theme={null}
class TokenUsage(TypedDict, total=False):
input_tokens: float
output_tokens: float
final_input_tokens: float
final_output_tokens: float
```
| Field | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------ |
| `input_tokens` | Sum of prompt tokens across all turns. Shared context is counted each time it appears in a prompt. |
| `output_tokens` | Sum of completion tokens across all turns. |
| `final_input_tokens` | Non-completion tokens in the final turn's context (system prompts, user messages, tool results, etc.). |
| `final_output_tokens` | Completion tokens in the final turn's context. Equals `output_tokens` for single-turn rollouts. |
In a single-turn rollout, `input_tokens == final_input_tokens` and `output_tokens == final_output_tokens`. In a multi-turn rollout, `input_tokens > final_input_tokens` because earlier turns' prompts are counted again.
The `final_*` metrics assume a single, continuously extended trajectory. Non-linear trajectories (multi-agent, context summarization, history rewriting) are not accounted for.
### GenerateOutputs
```python theme={null}
class GenerateOutputs(TypedDict):
outputs: list[RolloutOutput]
metadata: GenerateMetadata
```
Output from `Environment.generate()`. Contains a list of `RolloutOutput` objects (one per rollout) and generation metadata. Each `RolloutOutput` is a serialized, JSON-compatible dict containing the rollout's prompt, completion, answer, reward, metrics, timing, and other per-rollout data.
### GenerateMetadata
```python theme={null}
class VersionInfo(TypedDict):
vf_version: str
vf_commit: str | None
env_version: str | None
env_commit: str | None
class GenerateMetadata(TypedDict):
env_id: str
name: NotRequired[str]
env_args: dict
model: str
base_url: str
num_examples: int
rollouts_per_example: int
shuffle: NotRequired[bool]
shuffle_seed: NotRequired[int | None]
sampling_args: SamplingArgs
date: str
time_ms: float
avg_reward: float
avg_metrics: dict[str, float]
avg_error: float
pass_at_k: dict[str, float]
pass_all_k: dict[str, float]
pass_threshold: float
usage: TokenUsage | None
version_info: VersionInfo
state_columns: list[str]
path_to_save: Path
tools: list[Tool] | None
```
`base_url` is always serialized as a string. For multi-endpoint runs (e.g., using `ClientConfig.endpoint_configs`), it is stored as a comma-separated list of URLs.
`shuffle` records whether evaluation inputs were shuffled before selecting examples. `shuffle_seed` records the seed used for that shuffle; when shuffle is enabled without an explicit seed, the saved value is `0`.
`version_info` captures the verifiers framework version/commit and the environment package version/commit at generation time. Populated automatically by `GenerateOutputsBuilder`.
### RolloutScore / RolloutScores
```python theme={null}
class RolloutScore(TypedDict):
reward: float
metrics: dict[str, float]
class RolloutScores(TypedDict):
reward: list[float]
metrics: dict[str, list[float]]
```
***
## Classes
### Environment Classes
#### Environment
```python theme={null}
class Environment(ABC):
def __init__(
self,
dataset: Dataset | None = None,
eval_dataset: Dataset | None = None,
system_prompt: str | None = None,
few_shot: list[ChatMessage] | None = None,
parser: Parser | None = None,
rubric: Rubric | None = None,
sampling_args: SamplingArgs | None = None,
message_type: MessageType = "chat",
max_workers: int = 512,
env_id: str | None = None,
env_args: dict | None = None,
max_seq_len: int | None = None,
score_rollouts: bool = True,
pass_threshold: float = 0.5,
**kwargs,
): ...
```
Abstract base class for all environments.
**Generation methods:**
| Method | Returns | Description |
| -------------------------------------- | ----------------- | ------------------------------------------------------------------------------- |
| `generate(inputs, client, model, ...)` | `GenerateOutputs` | Run rollouts asynchronously. `client` accepts `Client \| ClientConfig`. |
| `generate_sync(inputs, client, ...)` | `GenerateOutputs` | Synchronous wrapper; inside an active event loop, install `verifiers[notebook]` |
| `evaluate(client, model, ...)` | `GenerateOutputs` | Evaluate on eval\_dataset |
| `evaluate_sync(client, model, ...)` | `GenerateOutputs` | Synchronous evaluation |
**Dataset methods:**
| Method | Returns | Description |
| ----------------------------------- | --------- | --------------------------------------------------- |
| `get_dataset(n=-1, seed=None)` | `Dataset` | Get training dataset (optionally first n, shuffled) |
| `get_eval_dataset(n=-1, seed=None)` | `Dataset` | Get evaluation dataset |
| `make_dataset(...)` | `Dataset` | Static method to create dataset from inputs |
**Rollout methods (used internally or by subclasses):**
| Method | Returns | Description |
| ------------------------------------------------------- | ------------- | ------------------------------- |
| `rollout(input, client, model, sampling_args)` | `State` | Abstract: run single rollout |
| `init_state(input, client, model, sampling_args)` | `State` | Create initial state from input |
| `get_model_response(state, prompt, ...)` | `Response` | Get model response for prompt |
| `is_completed(state)` | `bool` | Check all stop conditions |
| `run_rollout(sem, input, client, model, sampling_args)` | `State` | Run rollout with semaphore |
| `run_group(group_inputs, client, model, ...)` | `list[State]` | Generate and score one group |
Calling `generate_sync()` from Jupyter or another active event loop requires `uv add "verifiers[notebook]"`.
**Configuration methods:**
| Method | Description |
| ------------------------------ | ------------------------------------------------------------------------- |
| `set_kwargs(**kwargs)` | Set attributes using setter methods when available |
| `set_concurrency(concurrency)` | Set `concurrency` and scale all registered thread-pool executors to match |
| `add_rubric(rubric)` | Add or merge rubric |
| `set_max_seq_len(max_seq_len)` | Set maximum sequence length |
| `set_score_rollouts(bool)` | Enable/disable scoring |
#### SingleTurnEnv
Single-response Q\&A tasks. Inherits from `Environment`.
#### MultiTurnEnv
```python theme={null}
class MultiTurnEnv(Environment):
def __init__(
self,
max_turns: int = -1,
timeout_seconds: float | None = None,
**kwargs,
): ...
```
Multi-turn interactions. Subclasses must implement `env_response`.
**Abstract method:**
```python theme={null}
async def env_response(self, messages: Messages, state: State, **kwargs) -> Messages:
"""Generate environment feedback after model turn."""
```
**Built-in stop conditions:** `has_error`, `prompt_too_long`, `max_turns_reached`, `timeout_reached`, `max_total_completion_tokens_reached`, `has_final_env_response`
**Hooks:**
| Method | Description |
| -------------------------------------- | ----------------------------------- |
| `setup_state(state)` | Initialize per-rollout state |
| `get_prompt_messages(state)` | Customize prompt construction |
| `render_completion(state)` | Customize completion rendering |
| `add_trajectory_step(state, step)` | Customize trajectory handling |
| `set_max_total_completion_tokens(int)` | Set maximum total completion tokens |
#### ToolEnv
```python theme={null}
class ToolEnv(MultiTurnEnv):
def __init__(
self,
tools: list[Callable] | None = None,
max_turns: int = 10,
error_formatter: Callable[[Exception], str] = lambda e: f"{e}",
stop_errors: list[type[Exception]] | None = None,
**kwargs,
): ...
```
Tool calling with stateless Python functions. Automatically converts functions to OpenAI tool format.
**Built-in stop condition:** `no_tools_called` (ends when model responds without tool calls)
**Methods:**
| Method | Description |
| --------------------------- | ------------------------------------ |
| `add_tool(tool)` | Add a tool at runtime |
| `remove_tool(tool)` | Remove a tool at runtime |
| `call_tool(name, args, id)` | Override to customize tool execution |
#### StatefulToolEnv
Tools requiring per-rollout state. Override `setup_state` and `update_tool_args` to inject state.
#### SandboxEnv
```python theme={null}
class SandboxEnv(StatefulToolEnv):
def __init__(
self,
sandbox_name: str = "sandbox-env",
docker_image: str = "python:3.11-slim",
start_command: str = "tail -f /dev/null",
cpu_cores: int = 1,
memory_gb: int = 2,
disk_size_gb: int = 5,
gpu_count: int = 0,
timeout_minutes: int = 60,
timeout_per_command_seconds: int = 30,
environment_vars: dict[str, str] | None = None,
team_id: str | None = None,
advanced_configs: AdvancedConfigs | None = None,
labels: list[str] | None = None,
**kwargs,
): ...
```
Sandboxed container execution using `prime` sandboxes.
**Key parameters:**
| Parameter | Type | Description |
| ----------------------------- | ------------------------ | ----------------------------------------------- |
| `sandbox_name` | `str` | Name prefix for sandbox instances |
| `docker_image` | `str` | Docker image to use for the sandbox |
| `cpu_cores` | `int` | Number of CPU cores |
| `memory_gb` | `int` | Memory allocation in GB |
| `disk_size_gb` | `int` | Disk size in GB |
| `gpu_count` | `int` | Number of GPUs |
| `timeout_minutes` | `int` | Sandbox timeout in minutes |
| `timeout_per_command_seconds` | `int` | Per-command execution timeout |
| `environment_vars` | `dict[str, str] \| None` | Environment variables to set in sandbox |
| `labels` | `list[str] \| None` | Labels for sandbox categorization and filtering |
#### PythonEnv
Persistent Python REPL in sandbox. Extends `SandboxEnv`.
#### OpenEnvEnv
```python theme={null}
class OpenEnvEnv(MultiTurnEnv):
def __init__(
self,
openenv_project: str | Path | None = None,
num_train_examples: int = 100,
num_eval_examples: int = 50,
seed: int = 0,
prompt_renderer: Callable[..., Messages] | None = None,
max_turns: int = -1,
rubric: Rubric | None = None,
**kwargs,
): ...
```
OpenEnv integration that runs OpenEnv projects in Prime Sandboxes using a prebuilt image manifest (`.build.json`), supports both gym and MCP contracts, and requires a `prompt_renderer` to convert observations into chat messages.
#### SandboxDebugEnv
```python theme={null}
class SandboxDebugEnv(SandboxMixin, MultiTurnEnv):
def __init__(
self,
taskset: SandboxTaskSet,
dataset: Any = None,
*,
run_setup: bool = True,
debug_step: Literal["none", "gold_patch", "command", "script"] = "gold_patch",
run_tests: bool = True,
debug_command: str | None = None,
debug_script: str | None = None,
debug_script_path: str | None = None,
debug_timeout: int | None = None,
test_timeout: int = 900,
cpu_cores: int | None = None,
memory_gb: int | None = None,
disk_size_gb: int | None = None,
labels: list[str] | None = None,
timeout_seconds: float = 1800.0,
output_tail_chars: int = 2000,
**sandbox_kwargs,
): ...
```
No-agent debugger for sandbox-backed `SandboxTaskSet` instances. It creates the task sandbox, optionally runs task setup, runs one debug step (`none`, `gold_patch`, `command`, or `script`), and optionally runs tests and scores the result. `SWEDebugEnv` remains as a deprecated wrapper for older callers.
#### EnvGroup
```python theme={null}
env_group = vf.EnvGroup(
envs=[env1, env2, env3],
env_names=["math", "code", "qa"], # optional
)
```
Combines multiple environments for mixed-task training. Combined datasets use `info["env_id"]` as internal routing metadata; it is not a top-level input, state, or output field.
***
### Parser Classes
#### Parser
```python theme={null}
class Parser:
def __init__(self, extract_fn: Callable[[str], str] = lambda x: x): ...
def parse(self, text: str) -> Any: ...
def parse_answer(self, completion: Messages) -> str | None: ...
def get_format_reward_func(self) -> Callable: ...
```
Base parser. Default behavior returns text as-is.
#### XMLParser
```python theme={null}
class XMLParser(Parser):
def __init__(
self,
fields: list[str | tuple[str, ...]],
answer_field: str = "answer",
extract_fn: Callable[[str], str] = lambda x: x,
): ...
```
Extracts structured fields from XML-tagged output.
```python theme={null}
parser = vf.XMLParser(fields=["reasoning", "answer"])
# Parses: ......
# With alternatives:
parser = vf.XMLParser(fields=["reasoning", ("code", "answer")])
# Accepts either or for second field
```
**Methods:**
| Method | Returns | Description |
| -------------------------- | ----------------- | ------------------------------------------- |
| `parse(text)` | `SimpleNamespace` | Parse XML into object with field attributes |
| `parse_answer(completion)` | `str \| None` | Extract answer field from completion |
| `get_format_str()` | `str` | Get format description string |
| `get_fields()` | `list[str]` | Get canonical field names |
| `format(**kwargs)` | `str` | Format kwargs into XML string |
#### ThinkParser
```python theme={null}
class ThinkParser(Parser):
def __init__(self, extract_fn: Callable[[str], str] = lambda x: x): ...
```
Extracts content after `` tag. For models that always include `` tags but don't parse them automatically.
#### MaybeThinkParser
Handles optional `` tags (for models that may or may not think).
***
### Rubric Classes
#### Rubric
```python theme={null}
class Rubric:
def __init__(
self,
funcs: list[RewardFunc] | None = None,
weights: list[float] | None = None,
parser: Parser | None = None,
): ...
```
Combines multiple reward functions with weights. Default weight is `1.0`. Functions with `weight=0.0` are tracked as metrics only.
**Methods:**
| Method | Description |
| ----------------------------------- | ----------------------------------------- |
| `add_reward_func(func, weight=1.0)` | Add a reward function |
| `add_metric(func, weight=0.0)` | Add a metric (no reward contribution) |
| `add_class_object(name, obj)` | Add object accessible in reward functions |
**Reward function signature:**
```python theme={null}
def my_reward(
completion: Messages,
answer: str = "",
prompt: Messages | None = None,
state: State | None = None,
parser: Parser | None = None, # if rubric has parser
info: Info | None = None,
**kwargs,
) -> float: ...
```
**Group reward function signature:**
```python theme={null}
def my_group_reward(
completions: list[Messages],
answers: list[str],
states: list[State],
# ... plural versions of individual args
**kwargs,
) -> list[float]: ...
```
#### JudgeRubric
LLM-as-judge evaluation.
#### MathRubric
Math-specific evaluation using `math-verify`.
#### RubricGroup
Combines rubrics for `EnvGroup`.
***
## Client Classes
### Client
```python theme={null}
class Client(ABC, Generic[ClientT, MessagesT, ResponseT, ToolT]):
def __init__(self, client_or_config: ClientT | ClientConfig) -> None: ...
@property
def client(self) -> ClientT: ...
async def get_response(
self,
prompt: Messages,
model: str,
sampling_args: SamplingArgs,
tools: list[Tool] | None = None,
**kwargs,
) -> Response: ...
async def close(self) -> None: ...
```
Abstract base class for all model clients. Wraps a provider-specific SDK client and translates between provider-agnostic `vf` types (`Messages`, `Tool`, `Response`) and provider-native formats. The `client` property exposes the underlying SDK client (e.g., `AsyncOpenAI`, `AsyncAnthropic`).
`get_response()` is the main public method — it converts the prompt and tools to the native format, calls the provider API, validates the response, and converts it back to a `vf.Response`. Errors are wrapped in `vf.ModelError` unless they are already `vf.Error` or authentication errors.
**Abstract methods (for subclass implementors):**
| Method | Description |
| ----------------------------------------- | -------------------------------------------------------- |
| `setup_client(config)` | Create the native SDK client from `ClientConfig` |
| `to_native_prompt(messages)` | Convert `Messages` → native prompt format + extra kwargs |
| `to_native_tool(tool)` | Convert `Tool` → native tool format |
| `get_native_response(prompt, model, ...)` | Call the provider API |
| `raise_from_native_response(response)` | Raise `ModelError` for invalid responses |
| `from_native_response(response)` | Convert native response → `vf.Response` |
| `close()` | Close the underlying SDK client |
### Built-in Client Implementations
| Class | `client_type` | SDK Client | Description |
| ---------------------------------- | --------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `OpenAIChatCompletionsClient` | `"openai_chat_completions"` | `AsyncOpenAI` | Chat Completions API (default) |
| `OpenAICompletionsClient` | `"openai_completions"` | `AsyncOpenAI` | Legacy Completions API |
| `OpenAIChatCompletionsTokenClient` | `"openai_chat_completions_token"` | `AsyncOpenAI` | Custom vLLM token route (`/v1/chat/completions/tokens`) — server-side templating + token IDs returned alongside content |
| `OpenAIResponsesClient` | `"openai_responses"` | `AsyncOpenAI` | OpenAI Responses API |
| `RendererClient` | `"renderer"` | `AsyncOpenAI` | Renderer-backed token-in generate client (client-side tokenization via the `renderers` package) |
| `AnthropicMessagesClient` | `"anthropic_messages"` | `AsyncAnthropic` | Anthropic Messages API |
| `NeMoRLChatCompletionsClient` | `"nemorl_chat_completions"` | `AsyncOpenAI` | NeMo-RL Chat Completions variant |
All built-in clients are available as `vf.OpenAIChatCompletionsClient`, `vf.AnthropicMessagesClient`, etc.
### Response
```python theme={null}
class Response(BaseModel):
id: str
created: int
model: str
usage: Usage | None
message: ResponseMessage
class ResponseMessage(BaseModel):
content: str | None
reasoning_content: str | None
finish_reason: Literal["stop", "length", "tool_calls"] | None
is_truncated: bool | None
tokens: ResponseTokens | None
tool_calls: list[ToolCall] | None
```
Provider-agnostic model response. All `Client` implementations return `Response` from `get_response()`.
### Tool
```python theme={null}
class Tool(BaseModel):
name: str
description: str
parameters: dict[str, object]
strict: bool | None = None
```
Provider-agnostic tool definition. Environments define tools using this type; each `Client` converts them to its native format via `to_native_tool()`.
***
## Configuration Types
### ClientConfig
```python theme={null}
class ClientConfig(BaseModel):
client_idx: int = 0
client_type: ClientType = "openai_chat_completions"
preserve_all_thinking: bool = False
preserve_thinking_between_tool_calls: bool = False
api_key_var: str = "PRIME_API_KEY"
api_base_url: str = "https://api.pinference.ai/api/v1"
endpoint_configs: list[EndpointClientConfig] = []
timeout: float = 3600.0
connect_timeout: float = 5.0
max_connections: int = 28000
max_keepalive_connections: int = 28000
max_retries: int = 10
extra_headers: dict[str, str] = {}
extra_headers_from_state: dict[str, str] = {}
```
`extra_headers_from_state` maps HTTP header names to state field names. For each inference request, the header value is dynamically read from the rollout state dict. For example, `{"X-Session-ID": "trajectory_id"}` adds a `X-Session-ID` header with the value of `state["trajectory_id"]`, enabling sticky routing at the inference router level.
`client_type` selects which `Client` implementation to instantiate (see [Client Classes](#client-classes)). Use `endpoint_configs` for multi-endpoint round-robin. In grouped scoring mode, groups are distributed round-robin across endpoint configs.
`preserve_all_thinking` and `preserve_thinking_between_tool_calls` are forwarded to the underlying renderer when `client_type == "renderer"`. They control whether past-assistant `reasoning_content` is re-emitted on subsequent renders — `preserve_all_thinking` keeps every past-assistant turn's thinking, and `preserve_thinking_between_tool_calls` keeps thinking only inside the in-flight assistant→tool→…→assistant block after the most recent user turn (when that block contains at least one tool response). Both default to `False` (template default applies).
When `api_key_var` is `"PRIME_API_KEY"` (the default), credentials are loaded with the following precedence:
* **API key**: `PRIME_API_KEY` env var > `~/.prime/config.json` > `"EMPTY"`
* **Team ID**: `PRIME_TEAM_ID` env var > `~/.prime/config.json` > not set
This allows seamless use after running `prime login`.
### EndpointClientConfig
```python theme={null}
class EndpointClientConfig(BaseModel):
client_idx: int = 0
api_key_var: str = "PRIME_API_KEY"
api_base_url: str = "https://api.pinference.ai/api/v1"
timeout: float = 3600.0
max_connections: int = 28000
max_keepalive_connections: int = 28000
max_retries: int = 10
extra_headers: dict[str, str] = {}
```
Leaf endpoint configuration used inside `ClientConfig.endpoint_configs`. Has the same fields as `ClientConfig` except `endpoint_configs` itself, preventing recursive nesting.
### EvalConfig
```python theme={null}
class EvalConfig(BaseModel):
env_id: str
name: str | None = None
env_args: dict
env_dir_path: str
endpoint_id: str | None = None
model: str
client_config: ClientConfig
sampling_args: SamplingArgs
num_examples: int
rollouts_per_example: int
shuffle: bool = False
shuffle_seed: int | None = None
max_concurrent: int
independent_scoring: bool = False
extra_env_kwargs: dict = {}
max_retries: int = 3
verbose: bool = False
state_columns: list[str] | None = None
save_results: bool = False
resume_path: Path | None = None
save_to_hf_hub: bool = False
hf_hub_dataset_name: str | None = None
```
### EndpointConfig
```python theme={null}
class EndpointConfig(BaseModel):
model: str
base_url: str
api_key_var: str
api_client_type: ClientType | None = None
extra_headers: dict[str, str] = {}
Endpoints = dict[str, list[EndpointConfig]]
```
`api_key_var` is a credential reference. Endpoint configs never serialize the materialized API key.
`Endpoints` maps an endpoint id to one or more endpoint variants. A single variant is represented as a one-item list.
***
## Prime CLI Plugin
Verifiers exposes a plugin contract consumed by `prime` for command execution.
### PRIME\_PLUGIN\_API\_VERSION
```python theme={null}
PRIME_PLUGIN_API_VERSION = 1
```
API version for compatibility checks between `prime` and `verifiers`.
### PrimeCLIPlugin
```python theme={null}
@dataclass(frozen=True)
class PrimeCLIPlugin:
api_version: int = PRIME_PLUGIN_API_VERSION
eval_module: str = "verifiers.cli.commands.eval"
gepa_module: str = "verifiers.cli.commands.gepa"
install_module: str = "verifiers.cli.commands.install"
init_module: str = "verifiers.cli.commands.init"
setup_module: str = "verifiers.cli.commands.setup"
build_module: str = "verifiers.cli.commands.build"
def build_module_command(
self, module_name: str, args: Sequence[str] | None = None
) -> list[str]: ...
```
`build_module_command` returns a subprocess command list for `python -m ...`.
### get\_plugin
```python theme={null}
def get_plugin() -> PrimeCLIPlugin: ...
```
Returns the plugin instance consumed by `prime`.
***
## Decorators
### @vf.stop
```python theme={null}
@vf.stop
async def my_condition(self, state: State) -> bool:
"""Return True to end the rollout."""
...
@vf.stop(priority=10) # Higher priority runs first
async def early_check(self, state: State) -> bool: ...
```
Mark a method as a stop condition. All stop conditions are checked by `is_completed()`.
### @vf.cleanup
```python theme={null}
@vf.cleanup
async def my_cleanup(self, state: State) -> None:
"""Called after each rollout completes."""
...
@vf.cleanup(priority=10)
async def early_cleanup(self, state: State) -> None: ...
```
Mark a method as a rollout cleanup handler. Cleanup methods should be **idempotent**—safe to call multiple times—and handle errors gracefully to ensure cleanup completes even when resources are in unexpected states.
### @vf.teardown
```python theme={null}
@vf.teardown
async def my_teardown(self) -> None:
"""Called when environment is destroyed."""
...
@vf.teardown(priority=10)
async def early_teardown(self) -> None: ...
```
Mark a method as an environment teardown handler.
***
## Utility Functions
### Data Utilities
```python theme={null}
vf.load_example_dataset(name: str) -> Dataset
```
Load a built-in example dataset.
```python theme={null}
vf.extract_boxed_answer(text: str, strict: bool = False) -> str
```
Extract answer from LaTeX `\boxed{}` format. When `strict=True`, returns `""` if no `\boxed{}` is found (used by `MathRubric` to avoid scoring unformatted responses). When `strict=False` (default), returns the original text as a passthrough.
```python theme={null}
vf.extract_hash_answer(text: str) -> str | None
```
Extract answer after `####` marker (GSM8K format).
### Environment Utilities
```python theme={null}
vf.load_environment(env_id: str, **kwargs) -> Environment
```
Load an environment by ID (e.g., `"primeintellect/gsm8k"`).
### Configuration Utilities
```python theme={null}
vf.ensure_keys(keys: list[str]) -> None
```
Validate that required environment variables are set. Raises `MissingKeyError` (a `ValueError` subclass) with a clear message listing all missing keys and instructions for setting them.
```python theme={null}
class MissingKeyError(ValueError):
keys: list[str] # list of missing key names
```
Example:
```python theme={null}
def load_environment(api_key_var: str = "OPENAI_API_KEY") -> vf.Environment:
vf.ensure_keys([api_key_var])
# now safe to use os.environ[api_key_var]
...
```
### Logging Utilities
```python theme={null}
vf.print_prompt_completions_sample(outputs: GenerateOutputs, n: int = 3)
```
Pretty-print sample rollouts.
```python theme={null}
vf.setup_logging(level: str = "INFO")
```
Configure verifiers logging. Set `VF_LOG_LEVEL` env var to change default.
```python theme={null}
vf.log_level(level: str | int)
```
Context manager to temporarily set the verifiers logger to a new log level. Useful for temporarily adjusting verbosity during specific operations.
```python theme={null}
with vf.log_level("DEBUG"):
# verifiers logs at DEBUG level here
...
# reverts to previous level
```
```python theme={null}
vf.quiet_verifiers()
```
Context manager to temporarily silence verifiers logging by setting WARNING level. Shorthand for `vf.log_level("WARNING")`.
```python theme={null}
with vf.quiet_verifiers():
# verifiers logging is quieted here
outputs = env.generate(...)
# logging restored
```
# Training
Source: https://docs.primeintellect.ai/verifiers/v0/training
v0 is considered deprecated and will be fully removed in a future release.
This section covers how to use Verifiers environments for RL training with our Hosted Training platform, our open-source `prime-rl` trainer, or other supported libraries.
## Table of Contents
* [Hosted Training](#hosted-training)
* [Configuration](#configuration)
* [Training with `prime-rl`](#training-with-prime-rl)
* [Setup and Configuration](#setup-and-configuration)
* [Prompt Optimization with `prime gepa run`](#prompt-optimization-with-prime-gepa-run)
* [Usage](#usage)
* [Output](#output)
* [RL Rules of Thumb](#rl-rules-of-thumb)
* [Before Training](#before-training)
* [Performance Trade-offs](#performance-trade-offs)
* [Inference Client Types](#inference-client-types)
* [Common Issues](#common-issues)
* [Other Trainers](#other-trainers)
* [Tinker](#tinker)
* [SkyRL](#skyrl)
* [rLLM](#rllm)
## Hosted Training
Hosted Training, available within our Lab platform, enables you to automatically train models via `prime-rl` without needing to manage your own infrastructure. Hosted Training supports LoRA for RL training, and can be used with any environment built with Verifiers.
### Configuration
Use the `prime lab setup` script to download example configuration files for Hosted Training into your workspace:
```bash theme={null}
prime lab setup
```
This will download example TOML configs for Hosted Training into `configs/rl/`, example eval configs into `configs/eval/`, along with `configs/endpoints.toml` and GEPA starter configs in `configs/gepa/`:
```text theme={null}
configs/
├── endpoints.toml
├── eval/
│ ├── qwen-3-5.toml
│ ├── qwen-3-5-moe.toml
│ ├── nemotron-3.toml
│ ├── llama-3.toml
│ └── gpt-oss.toml
├── rl/
│ ├── qwen-3-5.toml
│ ├── qwen-3-5-moe.toml
│ ├── nemotron-3.toml
│ ├── llama-3.toml
│ └── gpt-oss.toml
└── gepa/
├── qwen-3-5.toml
├── qwen-3-5-moe.toml
├── nemotron-3.toml
├── llama-3.toml
└── gpt-oss.toml
```
Example configuration file for the `primeintellect/reverse-text` environment with `Qwen/Qwen3.5-4B`:
```toml theme={null}
# Qwen3.5 dense models. Uncomment exactly one model.
# model = "Qwen/Qwen3.5-0.8B"
# model = "Qwen/Qwen3.5-2B"
model = "Qwen/Qwen3.5-4B"
# model = "Qwen/Qwen3.5-9B"
max_steps = 100
batch_size = 128
rollouts_per_example = 8
[sampling]
max_tokens = 1024
[[env]]
id = "primeintellect/reverse-text"
```
We currently support the following models for Hosted Training:
* `Qwen/Qwen3-30B-A3B-Instruct-2507`
* `Qwen/Qwen3-30B-A3B-Thinking-2507`
* `Qwen/Qwen3-4B-Instruct-2507`
* `Qwen/Qwen3-4B-Thinking-2507`
* `Qwen/Qwen3-VL-4B-Instruct`
* `Qwen/Qwen3.5-0.8B`
* `Qwen/Qwen3.5-2B`
* `Qwen/Qwen3.5-4B`
* `Qwen/Qwen3.5-9B`
* `Qwen/Qwen3.5-35B-A3B`
* `Qwen/Qwen3.5-122B-A10B`
* `Qwen/Qwen3.5-397B-A17B`
* `meta-llama/Llama-3.2-1B-Instruct`
* `meta-llama/Llama-3.2-3B-Instruct`
* `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`
* `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16`
* `openai/gpt-oss-20b`
* `openai/gpt-oss-120b`
* `zai-org/GLM-4.7`
Hosted Training is currently in Private Beta. For access, please fill out [this form](https://form.typeform.com/to/iYn9UliG).
## Training with `prime-rl`
Our [`prime-rl`](https://github.com/PrimeIntellect-ai/prime-rl) trainer is a production-ready async RL training framework that supports large-scale multi-node training, agentic rollouts with Verifiers environments, Mixture-of-Experts (MoE) models, LoRA adapters, and other training algorithms such as SFT and online distillation. We recommend using `prime-rl` for training with Verifiers environments on self-managed GPU infrastructure. The default configuration distills the best practices from our research team's experience and the broader community into a stable, easy-to-use recipe, including advanced features such as online difficulty filtering, continuous batching, in-flight weight updates, importance sampling and logprob clipping for stability, and more.
### Setup and Configuration
To set up your workspace for training with `prime-rl`, run:
```bash theme={null}
prime lab setup --prime-rl
```
This will clone and install the `prime-rl` trainer and its dependencies. For configuration files and launch commands, use the [prime-rl documentation](https://docs.primeintellect.ai/prime-rl).
## Prompt Optimization with `prime gepa run`
`prime gepa run` is the CLI entrypoint for automatic system prompt optimization using [GEPA](https://github.com/gepa-ai/gepa) (Genetic-Pareto prompt optimization). It iteratively refines your environment's system prompt using a teacher LLM to reflect on evaluation results, without requiring gradient-based training. Current support is for system prompt optimization only.
### Usage
Basic usage mirrors `prime eval run`:
```bash theme={null}
prime gepa run wiki-search --model google/gemini-3-flash-preview
```
This will optimize the system prompt for the `wiki-search` environment using the specified model for both evaluation rollouts and reflection. Results are saved to `environments/wiki-search/outputs/gepa/`.
Key options:
* `--model` / `-m`: Model for evaluation rollouts
* `--reflection-model` / `-M`: Teacher model for prompt reflection (defaults to `--model`)
* `--max-calls` / `-B`: Evaluation budget (default: 500)
* `--num-train` / `-n`: Training examples (default: 100)
* `--num-val` / `-N`: Validation examples (default: 50)
* `--minibatch-size`: Number of examples evaluated together per reflection step (default: 3)
* `--perfect-score`: Maximum score for a rollout in your environment (if applicable); minibatches achieving this score are skipped during reflection (useful if your environment has a known max score)
* `--state-columns`: Additional state columns to copy into the reflection dataset. By default, `query`, `completion`, `expected_answer`, `reward`, and `error` are included. Use this to add environment-specific state fields (e.g., `--state-columns tool_calls reasoning_trace`)
In TOML configs, set GEPA parameters such as `max_calls`, `num_train`, `num_val`, `minibatch_size`, and `max_concurrent` under `[gepa]`. Put generation parameters such as `max_tokens` and `temperature` under `[sampling]`; the CLI passes that table through as `sampling_args`. Use `[[env]]` for one or more environments; GEPA samples train and validation examples uniformly by environment. A single `[env]` table is still accepted for older configs.
### Output
After optimization, you'll find:
* `system_prompt.txt` - The optimized system prompt.
* `results.jsonl` - Candidate prompt rows for evaluation upload; GEPA-specific fields live under `info`.
* `pareto_frontier.jsonl` - Best candidate references per validation example
* `metadata.json` - Run configuration and summary
Use `prime eval run` to verify performance before and after optimization.
## RL Rules of Thumb
RL training can be sensitive to implementation details and hyperparameters. Some simple practical guidance:
### Before Training
1. **Evaluate baseline performance**: If your model gets 0% reward after 10+ attempts, the task is too hard
2. **Check task difficulty**: If baseline is already 80%+, consider harder examples
3. **Ensure reward diversity**: You want varied scores within each generation group
### Performance Trade-offs
**For more aggressive training** (higher risk of collapse):
* Increase learning rate (1e-5 to 1e-4 for LoRA, 1e-6 to 1e-5 for full finetuning)
* Decrease `rollouts_per_example` and `batch_size` for faster generation
**For more stable training** (slower progress):
* Increase `rollouts_per_example` (16-32)
* Increase `batch_size` (512-1024)
* Use larger models (14B+)
The best way to improve training is to ensure appropriate task difficulty for your model. When using Hosted Training or `prime-rl`, you can enable online difficulty filtering to ensure that rollout groups used for training always contain a diversity of rewards.
### Inference Client Types
The rollout client's `client_type` controls how prompt assembly and token state flow between the inference server and the trainer. For RL the trainer must see the exact tokens the server sampled — re-tokenization across turns drifts under BPE round-trip and fragments multi-turn rollouts into multiple training samples.
* **`openai_chat_completions`** (MITO, *messages-in*): standard OpenAI-compatible path. Server-side chat templating, returns text. The trainer re-tokenizes — fine for eval and short single-turn training, but can fragment multi-turn rollouts.
* **`openai_chat_completions_token`** (TITO, *token-in*): server-side templating, but returns prompt and completion token IDs alongside text so the trainer doesn't re-tokenize. Use when you trust the server's chat template to be stable across turns.
* **`renderer`**: client-side tokenization via a per-model renderer in the [`renderers` package](https://github.com/PrimeIntellect-ai/verifiers/tree/main/packages/renderers). The trainer renders messages to token IDs locally and sends those to vLLM's `/v1/generate` endpoint. The renderer's `bridge_to_next_turn` extends prior-turn tokens verbatim across multi-turn boundaries (the *extension property*) and synthesizes the canonical turn-close on mid-completion truncation, so multi-turn rollouts merge into one training sample with one clean loss mask.
For production RL training, use `openai_chat_completions_token` — it's the tried-and-tested path with broad model coverage. The `renderer` client is newer and offers stronger token-preservation guarantees in theory, but hand-coded renderers exist only for a subset of models, and corner cases are still being shaken out. See [reference § Built-in Clients](/verifiers/v0/reference#built-in-client-implementations) for the full list.
### Common Issues
**Non-Increasing Chat Templates:** The Qwen3 and DeepSeek-R1 model series both remove `` sections from messages when processing inputs, which violates the increasing context requirement for multi-turn training. We provide versions of many of these models with [modified chat templates](https://huggingface.co/collections/willcb/qwen3-68434f4883925bfdb4570ee5).
**OOM during generation:**
* Reduce `rollouts_per_example` or `micro_batch_size`
* Use LoRA instead of full finetuning
* Check vLLM server has sufficient memory
**Training instability:**
* Decrease learning rate
* Increase `rollouts_per_example`
* Increase `batch_size`
**Slow training:**
* Increase learning rate
* Leverage continuous rewards
* Use online difficulty filtering
* Calibrate difficulty appropriately via smarter models, easier tasks
## Other Trainers
`verifiers` is intended to be largely trainer-agnostic and is straightforward to support for any trainer which can expose an OpenAI-compatible inference client for rollouts.
### Tinker
[Tinker](https://thinkingmachines.ai/tinker/) supports Verifiers environments via the `tinker-cookbook` recipes.
* [Verifiers + Tinker Recipe](https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/verifiers_rl)
### SkyRL
[SkyRL](https://github.com/NovaSky-AI/SkyRL) supports Verifiers environments via its `skyrl-train` integration.
* [Verifiers + SkyRL Integration](https://github.com/NovaSky-AI/SkyRL/tree/main/skyrl-train/integrations/verifiers)
### rLLM
[rLLM](https://github.com/rllm-project/rllm) supports Verifiers environments with both [verl](https://github.com/volcengine/verl) (local GPU) and [Tinker](https://thinkingmachines.ai/tinker/) (remote GPU) backends.
* [Verifiers + rLLM Documentation](https://rllm-project.readthedocs.io/en/latest/examples/verifiers/)
# Architecture
Source: https://docs.primeintellect.ai/verifiers/v1/architecture
verifiers is built out of the following parts:
A server-backed evaluation or prime-rl **orchestrator** creates worker processes and distributes rollout requests among them. The client owns the taskset. It loads the tasks once and ships each task to the workers, which owns the runtime containing the agent(s) to produce a trace out of the data.
The orchestrator and workers are managed by verifiers and prime-rl themselves and thus offer few configurable knobs.
The **rollout** is the executable combination of one loaded task, the harness, and any tools. Each rollout has an independent trace and runtime state. verifiers has three different runtimes which you can use for most tasksets:
* The `subprocess` runtime runs the rollouts in Python subprocesses locally. Thus, it is meant for debugging purposes, as there might be side effects during runtime, such as one subprocess altering the config files of the harness, which then affects the other subprocesses.
* The `docker` runtime runs the rollouts in docker containers on your local machine.
* Sandbox runtimes, such as `prime` or `modal`, are meant for production, especially for training or higher concurrency evaluation. These runtimes run remotely.
The harness runs inside the rollout runtime to interact with the taskset. The harness does *not* call the provider endpoint directly. Instead, model traffic goes through an **interception server** over a local connection or [Prime Tunnel](https://docs.primeintellect.ai/sandboxes/tunnel).
The interception server receives all these requests and then sends them over to the actual API, e.g. the OpenAI responses endpoint. It uses the endpoint that the harness expects, so Codex will use OpenAI Responses, while Claude Code will use the Anthropic Messages API.
The interception server, however, allows several things beyond just replaying the correct API response:
* Traces are built live, thus allowing the collection of the trajectories as they happen
* Setting sampling parameters in harnesses that don't necessarily expose those settings
* Intercepting and rewriting tool responses or server-side web search results to block reward hacks
# Env
Source: https://docs.primeintellect.ai/verifiers/v1/env
An `Env` defines the control flow between `Agents`. In the simplest case, it is just a `SingleAgentEnv` where a single agent solves a task from a taskset.
Its core signature is `Env.run(task: Task, agents: Agents)` -> None — it is passed an initial task and pre-initialized agents and then programs the full multi-agent control flow; every finished agent run automatically joins the resulting `Episode`, which holds all the traces of all the agents.
```python theme={null}
class Env(ABC):
@abstractmethod
async def run(self, task: Task, agents: Agents) -> None:
"""Run a single multi-agent episode."""
...
```
verifiers comes with different pre-built `Env`s to use:
* The `AgenticJudgeEnv` defines the sequential interaction between a solver and judge agent. The judge can re-use the same runtime after the solver (`SharedAgenticJudgeEnv`) or use its own, new runtime `IsolatedAgenticJudgeEnv`.
* The `UserSimEnv` models users as agents, and the episode is a turn-by-turn conversation between the user and assistant agents.
* The `BestOfNEnv` runs n independent attempts at the same task, then marks which attempt achieved the highest reward (best) and whether any attempt crossed a success threshold (pass\_at\_n), which is useful for rejection sampling and pass\@k evaluation.
# Evaluation
Source: https://docs.primeintellect.ai/verifiers/v1/evaluation
To evaluate any taskset, use the `eval` entrypoint:
```bash theme={null}
uv run eval primeintellect/terminal-bench-2
```
You can also use `.toml` files for configuration:
```toml theme={null}
model = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B"
[sampling]
temperature = 1.0
[env.taskset]
id = "primeintellect/terminal-bench-2"
[env.agent.harness]
id = "codex"
version = "0.116.0"
[env.agent.runtime]
type = "docker"
```
Validate the config by using `uv run eval @ config.toml --dry-run`. To run the evaluation, use `uv run eval @ config.toml`.
Use dotted arguments to set values using the CLI, e.g. `--sampling.temperature 0.5`. CLI arguments overwrite toml arguments when both are present.
The output from evaluations are written into `outputs/----//` by default, where `` is the taskset, prefixed by the paired env id when `--env.id` sets one (use `output_dir` to overwrite the folder). The folder contains the used `config.toml`, all the episodes in `traces.jsonl`, as well as logs of the run and workers in `eval.log`.
## Common config values
* `model` — the model id to evaluate, e.g. `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B`
* `sampling` — generation params passed to the model, e.g. `sampling.temperature`
* `env.taskset.id` — pick the taskset (or the positional `eval `)
* `env.agent.harness.id` — pick the agent's harness (`[env.agent.harness]` in TOML)
* `num_tasks` — how many tasks to evaluate. Not setting a value means all tasks; an
infinite taskset (a procedural generator, e.g. `wordle-v1`) requires it
* `num_rollouts` — rollouts per task
* `verbose` — log at debug instead of info
* `shuffle` — samples the task order (fixed seed); an error on an infinite taskset
## Resuming evaluations
`--resume ` re-runs only the rollouts a previous run left missing or errored, appending to that run's own `traces.jsonl`. It reloads the run's saved `config.toml` verbatim, so it takes no other arguments. Good rollouts are kept, while errored ones are dropped and redone.
## Disabling tools
Almost every harness comes with a `disabled_tools` list, which can be used to disable one or multiple tools:
```toml theme={null}
[env.agent.harness]
disabled_tools = ["shell_tool"]
```
The names of these tools are set by the respective harness. Consult the relevant documentation for the given harness for the relevant name(s). Some harnesses do not offer support to disable tools.
## Skills
Harnesses whose program supports SKILL.md skills natively (e.g. Claude Code, Codex) take a `skills` list of local skill folders, each uploaded into the program's skill discovery directory in the agent's runtime as `/`:
```toml theme={null}
[env.agent.harness]
skills = ["path/to/my-skill"]
```
Setting `skills` on a harness without native skill support fails up front.
# Gepa
Source: https://docs.primeintellect.ai/verifiers/v1/gepa
verifiers offers built in support for [GEPA](https://github.com/gepa-ai/gepa), an algorithm that optimizes a system prompt to maximize the downstream reward for a given taskset:
```bash theme={null}
uv run gepa reverse-text-v1
```
`gepa` runs GEPA where a number of rollouts are done before a teacher LLM reflects on the results to propose a better `Task.system_prompt` without any gradient based training. It runs against native v1 tasksets.
GEPA reuses the same `env` (taskset + agent) / `client` / `sampling` config as eval, so the `.toml` config remains very similar:
```toml theme={null}
model = "deepseek/deepseek-v4-flash"
[env.taskset]
id = "reverse-text-v1"
[env.agent.harness]
id = "bash"
[sampling]
temperature = 1.0
```
Validate the config by using `uv run gepa @ config.toml --dry-run`. To run GEPA, use `uv run gepa @ config.toml`. CLI arguments overwrite toml arguments when both are present.
## Common config values
* `model` / `-m` — model for the rollouts under optimization (default: `deepseek/deepseek-v4-flash`, same as eval)
* `reflection_model` / `reflection_client` — model/endpoint that proposes new prompts (default: reuse `model` / `client`)
* `num_train` / `num_val` — train tasks for reflection minibatches and held-out val tasks for the pareto frontier (defaults: 100 / 50)
* `max_total_rollouts` — total rollouts the run may spend (default: 500)
* `max_concurrent` / `-c` — caps how many episodes are in flight at once (default: 128)
## Output
Results go under `outputs/----//`, matching `eval`.
The best system prompt is printed when the run finishes and written to `best_system_prompt.txt` in that folder.
Hand it back to eval or training via the config-layer taskset system prompt:
```bash theme={null}
uv run eval reverse-text-v1 \
--env.taskset.system-prompt outputs//best_system_prompt.txt
```
## Limitations
**Tasksets** — GEPA optimizes `Task.system_prompt`, so the taskset must provide one. Tasksets that bake instructions into the user `prompt` instead (e.g. `gsm8k-v1`) are not supported out of the box.
**Harnesses** — any eval harness works. With `APPENDS_SYSTEM_PROMPT`, the optimized prompt is used as a system message but otherwise is folded into the user prompt.
# Getting Started
Source: https://docs.primeintellect.ai/verifiers/v1/getting_started
verifiers runs locally with `uv`. Install it, clone the repo, and sync dependencies:
```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/PrimeIntellect-ai/verifiers.git
cd verifiers
uv sync
```
You can now run tasksets directly, e.g. `uv run eval `, and scaffold new ones with `uv run init `.
## Skills
To equip your agent with the necessary knowledge, we highly recommend the skills in this repository's [`skills/`](https://github.com/PrimeIntellect-ai/verifiers/tree/main/skills) directory (alongside [`AGENTS.md`](https://github.com/PrimeIntellect-ai/verifiers/blob/main/AGENTS.md)). They are more comprehensive than these docs, which are meant for human consumption.
# Harbor
Source: https://docs.primeintellect.ai/verifiers/v1/harbor
verifiers offers built-in support for Harbor via the `HarborTaskset` class. Creating a Harbor-based taskset is straightforward in most cases:
```python theme={null}
import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset
# Set the dataset to the same name as registered in the Harbor registry
class TerminalBench2Config(HarborConfig):
dataset: str = "terminal-bench/terminal-bench-2"
# The data will get loaded automatically
class TerminalBench2Taskset(
HarborTaskset, vf.Taskset[HarborTask, TerminalBench2Config]
):
pass
```
You can also write custom code for your tasksets. A common customization is to set images for tasks that don’t come with one in their `task.toml`:
```python theme={null}
from pathlib import Path
from typing import Literal
import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset
IMAGE_TEMPLATE = "registry.example.com/openthoughts/{task}:latest"
class OpenThoughtsTBLiteConfig(HarborConfig):
dataset: Literal["openthoughts/openthoughts-tblite"] = (
"openthoughts/openthoughts-tblite"
)
# Tell verifiers to use the pre-built image
ignore_dockerfile: bool = True
class OpenThoughtsTBLiteTaskset(
HarborTaskset, vf.Taskset[HarborTask, OpenThoughtsTBLiteConfig]
):
def load(self) -> list[HarborTask]:
# Use the public image instead to avoid building the image at runtime; the row
# data is frozen, so rebuild each task around an updated copy.
return [
HarborTask(
task.data.model_copy(
update={
"image": IMAGE_TEMPLATE.format(
task=Path(task.data.task_dir).name
)
}
),
task.config,
)
for task in super().load()
]
```
To create and reuse images for your tasksets, build the Dockerfile with Docker and push it to a registry, then set the resulting image reference as the task's `image` field.
On the `prime` runtime any pullable image reference just works: the first sandbox to use an image makes the platform build and cache what it needs from it (for VM sandboxes this build can take \~10 minutes — the eval dashboard marks affected rollouts as `build` and a warning is logged); every later sandbox on the same reference starts in seconds.
## Additional features
By default, each task's declared agent and verifier timeouts are ignored (`ignore_timeouts = true`): Harbor task timeouts are authored against Harbor's own runtime, so enforcing them confounds model capability with the speed of your inference stack. Set `ignore_timeouts = false` (or pass `--no-env.taskset.ignore-timeouts`) to apply them, e.g. for a faithful comparison against the Harbor implementation.
With `ignore_timeouts = false`, every Harbor taskset can also be modified with a `timeout_multiplier`, and any Harbor taskset with a `resource_multiplier`:
```toml theme={null}
[env.taskset]
id = "MY_TASKSET"
ignore_timeouts = false
timeout_multiplier = 2.0
resource_multiplier = 2.0
```
The `timeout_multiplier` multiplies both the agent and verifier timeout, while the `resource_multiplier` multiplies the task's CPU, memory and disk space. You might want to use these multipliers when the tasks set too tight limits and/or the agent is slow.
## Network policies
Harbor's effective agent network policy is applied to Docker or Prime VM harness
runtimes. An `[agent].network_mode` override takes precedence over the `[environment]`
baseline; legacy `[environment].allow_internet` is normalized by Harbor's schema.
| Harbor mode | Task network policy |
| ------------ | ------------------------------------------------------------------------ |
| `public` | Sets the task allowlist to `["*"]`, leaving the evaluator policy intact. |
| `no-network` | Sets the task allowlist to `[]` (framework routes only). |
| `allowlist` | Sets the task allowlist to `allowed_hosts`. |
Trusted task and harness setup remains online. The policy starts immediately before the
agent and stays active through finalization and scoring. Interception and MCP URLs are
added automatically in allowlist and framework-only modes. Concrete task/runtime
allowlists combine, as do blocklists; framework-only access on either side takes
precedence, and concrete allowlists cannot be combined with blocklists. Docker framework
routes take precedence over deny rules, while ordinary Prime deny rules are applied
unchanged and may block a matching route. Restricted Harbor tasks require Docker or a
Prime VM; Prime accepts host-level entries.
## Artifacts and collect hooks
`artifacts = [...]` and `[[verifier.collect]]` are read from `task.toml` ([Harbor Docs](https://www.harborframework.com/docs/run-jobs/results-and-artifacts)). Collect hooks run in the agent's box from the task's `finalize`, which is Harbor's own ordering — after the agent phase, before collection — and declared paths plus the `/logs/artifacts/` convention dir are then carried into the grading box and restored at their original paths ("no translation", as in Harbor).
Two deliberate differences from `harbor run`:
* **A failing collect hook fails the rollout.** Harbor logs it and carries on, because there the output is observability; here it is a grading input, and a silently absent file makes the verifier score a stale state.
* **`destination` has no effect.** It positions a file in Harbor's host trial directory; verifiers has no trial directory (the trace is the record), and Harbor never lets `destination` affect verifier-side placement.
## Separate verifier environments
`[verifier].environment_mode = "separate"` grades in a second box the agent never touched, instead of the one it worked in ([Harbor Docs](https://www.harborframework.com/docs/tasks/verifier)). The harbor env — this taskset's default — grades such tasks in `finalize`: the solver plays the task as usual, its declared artifacts and the `/logs/artifacts/` convention directory are collected while its box is alive, the box is torn down, and the env then provisions a fresh box, restores those artifacts, stages `tests/` fresh, and grades there, recording the verifier's rewards and metrics onto the solver's trace. The grading box derives from the solver's runtime policy unless `--env.verifier-runtime.*` names its own (a network-restricted verifier on Prime needs `vm true`); infrastructure failures around it retry per `--env.verifier-retries` before the episode fails — a grading box that can't be reached never reads as reward 0. The score is read from `/logs/verifier/reward.json` — a finite number, or an object of finite numbers: with a `reward` key that key is the score and the rest are recorded as metrics; without one every key is recorded as a separate reward. Missing or invalid, it falls back to `reward.txt`.
Which image the verifier boots from follows Harbor: a declared `[verifier.environment]` if there is one, otherwise a fresh copy of `[environment]`, which is the task's own image.
A declared `[verifier.environment]` needs a pullable `docker_image`. Without one Harbor would build the verifier image from `tests/Dockerfile`, and verifiers never builds images — so build and push it yourself and name the resulting reference, exactly as for `[environment]`. `ignore_dockerfile` grades in the agent's image instead, which means the verifier runs somewhere the task never declared; it warns when it does.
Under any other env, a separate-verifier task refuses to grade in the agent's box rather than silently losing its isolation. `ignore_separate_verifier = true` forces every task back into shared grading, trading the isolation for one sandbox per task.
## Shortcomings
verifiers does not have parity with Harbor yet, so some features are missing and currently being worked on. The most notable missing features right now are:
* Switching to a different verifier-phase network policy for a *shared* verifier ([Harbor Docs](https://www.harborframework.com/docs/tasks/network-policy)); a separate verifier's own policy is applied
* Building a verifier image from `tests/Dockerfile`, which Harbor does when a declared `[verifier.environment]` names no `docker_image`. A separate verifier image itself is supported — it just has to be pre-built and pullable (see above), because verifiers never builds images
* Sidecar services, and the sidecar artifacts and collect hooks that go with them ([Harbor Docs](https://www.harborframework.com/docs/tasks#sidecar-artifacts-and-collect-hooks))
* Multi-step tasks ([Harbor Docs](https://www.harborframework.com/docs/tasks/multi-step))
# Harnesses
Source: https://docs.primeintellect.ai/verifiers/v1/harnesses
verifiers supports a range of harnesses out of the box, including Claude Code, Codex, the tool-enabled `bash` harness, the CDP-driven `browser_use` harness, and the minimal tool-less `null` harness. However, you may want to build a custom one or extend the selection of third‑party harnesses.
## A minimal harness implementation
```python theme={null}
from verifiers.v1.clients import ModelContext
from verifiers.v1.configs.harness import HarnessConfig
from verifiers.v1.harness import Harness
from verifiers.v1.runtimes import ProgramResult, Runtime
from verifiers.v1.task import TaskData
from verifiers.v1.trace import Trace
class MyHarnessConfig(HarnessConfig):
# These are the values that the users are allowed to set and change.
version: str = "0.0.1"
class MyHarness(Harness[MyHarnessConfig]):
# Set the system prompt of the task as the harness system message; else add it to the first user message
APPENDS_SYSTEM_PROMPT = True
# When the taskset exports a toolset, they are added as MCP. To show that your harness is able to install MCPs, you have to set this flag to true.
SUPPORTS_MCP = True
# Allow user simulation, mostly implemented by supporting the ACP protocol.
SUPPORTS_RESUME = True
async def setup(self, runtime: Runtime) -> None:
# Install the harness in its rollout runtime
await runtime.run(["sh", "-c", "echo installing..."], {})
async def launch(
self,
ctx: ModelContext,
trace: Trace,
runtime: Runtime,
endpoint: str,
secret: str,
mcp_urls: dict[str, str],
data: TaskData,
) -> ProgramResult:
# Run the harness in its respective runtime to completion
# The model (interception) endpoint is in endpoint
# mcp_urls are the URLs of the tools from the toolset (if registered)
# Resolve the task's prompt (and system prompt) for this harness
_, prompt = self.resolve_prompt(data)
# Example: Use the harness, but overwrite the endpoint to use the interception server and the custom model name
env = {
**self.config.env,
"HARNESS_BASE_URL": endpoint,
"HARNESS_API_KEY": secret,
"HARNESS_BASE_MODEL": ctx.model,
}
# Run the harness to completion inside the selected runtime.
return await runtime.run_program(["", str(prompt or "")], env)
```
# Overview
Source: https://docs.primeintellect.ai/verifiers/v1/overview
verifiers is a framework for defining tasks, running agents and harnesses, scoring them on set tasks, and using those for evaluations and reinforcement learning.
The following concepts are important when creating or running tasksets, be it for evals or training:
## Environment Hub
The [Environment Hub](https://app.primeintellect.ai/dashboard/environments?ex_sort=most_stars) is Prime Intellect's collection of user-created tasksets which are installable and ready to use with verifiers.
## Taskset
A taskset is the collection and loader for the work to evaluate or train on. Each task combines a serializable `TaskData` row (prompt, files, references, resource requirements) with its task class's behavior (lifecycle hooks, tools, metrics, and rewards). The taskset's `load()` method constructs those objects and declares their task/config types through `Taskset[TaskT, ConfigT]`.
## Harness
A harness is the program the model is run in, e.g. Claude Code, Codex or mini-swe-agent.
## Agent
An Agent is the combination of harness × model × runtime policy which produces a `Trace`.
## Environment
An environment has one or multiple agents and defines the control flow among them.
## Toolset
A set of tools defined by the taskset that are installed as MCP servers into the harnesses that support them.
## Trace
A trace records the message graph, rewards, metrics, errors, and one per-call record (`ModelCall`) per provider exchange (its model, sampling, finish reason, usage, timing, and any error), etc. When using verifiers for training with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl), it stores additional information such as tokens and logprobs, built incrementally using [renderers](https://github.com/PrimeIntellect-ai/renderers).
## Documentation
* [Getting started](/verifiers/v1/getting_started) - How to install verifiers and the needed skills.
* [Architecture](/verifiers/v1/architecture) — How verifiers works behind-the-scenes
* [Tasksets](/verifiers/v1/tasksets) — How to create tasksets
* [Harbor Tasksets](/verifiers/v1/harbor) — How to create Harbor-based tasksets
* [Evaluation](/verifiers/v1/evaluation) — How to evaluate tasksets
* [Harnesses](/verifiers/v1/harnesses) — How to build custom harnesses
* [Agent](/verifiers/v1/agent) — How to run standalone agents
* [Env](/verifiers/v1/env) — How to build multi-agent environments
For the documentation for legacy environments, go to [the v0 documentation](/verifiers/v0/overview).
# Tasksets
Source: https://docs.primeintellect.ai/verifiers/v1/tasksets
A taskset defines the work to be done, which will be solved by the agent in a *harness* running in a *runtime*.
You can scaffold a new taskset with the following:
```bash theme={null}
uv run init addition-v1
```
The generated package has two important files:
```text theme={null}
environments/addition_v1/addition_v1/
├── __init__.py # exports the taskset entry point
└── taskset.py # defines the data, tasks, and taskset
```
The command also supports:
* `-p`, `--path ` — parent directory, default: `./environments`
* `-T`, `--add-tool` — also scaffold a `vf.Toolset` tool server at `servers/tool.py`
* Use this to create custom tools which are installed into supported harnesses via MCP.
* `-H`, `--add-harness` — also scaffold a custom `vf.Harness` at `harness.py`, selectable via `--env.agent.harness.id `
* Prefer a built-in harness unless the model needs to run inside a custom program.
> For a production-scale catalog of tasksets, see the companion [`research-environments`](https://github.com/PrimeIntellect-ai/research-environments) repository.
## An example taskset
Tasksets are made of the following components:
* The **Taskset** loads the actual **Tasks** from a dataset using the `load()` function. It can be configured with the **TasksetConfig**, to e.g. load a certain split. Configs are exposed to the user and thus should only contain configurable values.
* A **Task** defines the scoring, stop conditions, setup, judging etc. of the task to solve. It also gets the tools or user config. It gets configured by a **TaskConfig**, e.g., to set a specific judge model.
* The **TaskData** is the immutable object that holds the actual data, i.e., the prompts, images, expected outputs etc., as well as other information such as timeouts (if set).
The following taskset generates addition questions and checks whether the model returned the exact answer.
```python theme={null}
import verifiers.v1 as vf
class AdditionData(vf.TaskData):
# One immutable row in the dataset, including its reference answer.
answer: int
class AdditionTask(vf.Task[AdditionData]):
# @vf.reward denotes the scoring function for the task.
# It needs the trace, which contains the whole message graph, including function calls, user messages etc.
# It returns the reward for the single task based on this function.
@vf.reward
async def exact_match(self, trace: vf.Trace) -> float:
return float(trace.last_reply == str(self.data.answer))
class AdditionConfig(vf.TasksetConfig):
# Values users can configure for the whole taskset.
num_tasks: int = 100
# The Taskset itself
class AdditionTaskset(vf.Taskset[AdditionTask, AdditionConfig]):
# The loading function for the actual tasks
def load(self) -> list[AdditionTask]:
return [
AdditionTask(
AdditionData(idx=i, prompt=f"What is {i} + {i}?", answer=2 * i),
self.config.task,
)
for i in range(self.config.num_tasks)
]
```
If a config class is not explicitly created, it means that no configurable, custom values are exposed to the user. In this example, there is no `vf.TaskConfig`, so no task values (like judge models) are configurable.
The scaffold also exports the taskset from `addition_v1/__init__.py`:
```python theme={null}
from addition_v1.taskset import AdditionTaskset
__all__ = ["AdditionTaskset"]
```
The exported `AdditionTaskset` is what verifiers loads and makes discoverable for evaluation.
## Data and configuration
Keep values on the narrowest object that needs them:
* Put load-time values shared across the dataset, such as its split, name, seed, or size, on `TasksetConfig`.
* Put values used by every task during execution or scoring under `TasksetConfig.task`.
```python theme={null}
class AdditionTaskConfig(vf.TaskConfig):
tolerance: float = 0.0
class AdditionTask(vf.Task[AdditionData, vf.State, AdditionTaskConfig]):
@vf.reward
async def exact_match(self, trace: vf.Trace) -> float:
error = abs(float(trace.last_reply) - self.data.answer)
return float(error <= self.config.tolerance)
class AdditionConfig(vf.TasksetConfig):
num_tasks: int = 100
task: AdditionTaskConfig = AdditionTaskConfig()
```
These values can be overridden with `--env.taskset.num-tasks` and `--env.taskset.task.tolerance`, or with the equivalent TOML fields (`[env.taskset]`).
## Lazy and infinite tasksets
`load()` may be a generator instead of returning a list: yield each task as it's built. Consumers iterate the taskset lazily (`Taskset.head` pulls only what a run needs — `eval -n 5` builds 5 tasks, not the whole set) — so a generator pays off whenever building a task is expensive.
A procedural taskset can keep yielding forever. Declare `INFINITE = True` so consumers know the stream never ends — infinity is inherent to the taskset, not a config knob; how many tasks a run takes is the run's choice (`-n`), not the taskset's:
```python theme={null}
import itertools
from collections.abc import Iterator
class AdditionTaskset(vf.Taskset[AdditionTask, vf.TasksetConfig]):
INFINITE = True
def load(self) -> Iterator[AdditionTask]:
for i in itertools.count():
yield AdditionTask(
AdditionData(idx=i, prompt=f"What is {i} + {i}?", answer=2 * i),
self.config.task,
)
```
Two rules follow from infinity: a run over an infinite taskset must be bounded with `num_tasks` (`-n` on the CLI — omitting it is an error), and `shuffle` is an error: there is no whole set to sample from — bound the stream first (`taskset.head(n).shuffle()`). The generator runs once, client-side (the eval entrypoint or the prime-rl orchestrator pulls tasks off it and ships each task's data to the env server), so nothing needs to re-produce the same sequence across processes; keep `load()` deterministic only if you want `--resume` to regenerate the same first `n` tasks (see `alphabet_sort_v1`, `color_codeword_v1`, or the built-in `textarena` taskset).
## Adding Tools
Some tasksets require custom tools, which are bundled as a `vf.Toolset` (similar to how a `vf.Taskset` bundles `vf.Task`). Tools are exposed as MCP servers to the given harness and thus need a harness which exposes MCP support (via `SUPPORTS_MCP`).
You can create them like this (remember the bootstrapping with `uv run init MY_ENV -T`):
```python theme={null}
DATABASE = None
class SearchToolset(vf.Toolset[vf.SharedToolsetConfig]):
TOOL_PREFIX = "search"
@vf.tool
async def query(self, text: str) -> list[str]:
"""Search the task corpus."""
return DATABASE.search(text)
# User-configurable knobs
class SearchConfig(vf.TasksetConfig):
tools: vf.SharedToolsetConfig = vf.SharedToolsetConfig()
class SearchTaskset(vf.Taskset[vf.Task, SearchConfig]):
@classmethod
def toolsets(cls, config: SearchConfig) -> list[vf.Toolset]:
return [SearchToolset(config.tools)]
```
Taskset tools are shared by a worker's rollouts. Tools can also be set per task.
## Using Judges
If your reward is semantic, use an LLM judge.
```python theme={null}
import verifiers.v1 as vf
from functools import cached_property
class Task(vf.Task):
answer: str
class CorrectnessJudge(vf.Judge[bool]):
# The rubric for the judge
prompt = """Question: {question}
Answer: {answer}
Response: {response}
Correct? Reply yes or no."""
# Parse the response from the judge
def parse(self, response: vf.JudgeResponse[bool]) -> bool:
return "yes" in response.text
class JudgedData(vf.TaskData):
answer: str
class JudgedTaskConfig(vf.TaskConfig):
# The judge inherits base_url and api keys from the client config
judge: vf.JudgeConfig = vf.JudgeConfig(model="openai/gpt-5-mini")
class JudgedTask(vf.Task[JudgedData, vf.State, JudgedTaskConfig]):
@vf.reward()
async def correct(self, trace: vf.Trace) -> float:
# Keeping judge configuration on TaskConfig makes it overridable from CLI/TOML.
judge = CorrectnessJudge(self.config.judge)
result = await judge.evaluate(
trace=trace,
question=self.data.prompt_text,
answer=self.data.answer,
# give the last assistant message to the judge
response=trace.last_reply,
)
return float(result.parsed)
class SetConfig(vf.TasksetConfig):
task: JudgedTaskConfig = JudgedTaskConfig()
class JudgeTraceTaskset(vf.Taskset[JudgedTask, SetConfig]):
def load(self) -> list[JudgedTask]:
return [
JudgedTask(
JudgedData(idx=0, prompt="What is 2+2?", answer="4"),
self.config.task,
)
]
```
To override the judge model, set `env.taskset.task.judge.model` in your config (it is a string).
## Beyond one agent
One episode doesn't have to be one agent run: agents, the control flow between agents, and cross-agent rewards are the environment's job — see [The Env](/verifiers/v1/env).