Evaluating Environments - Prime Intellect Docs

The prime eval command provides powerful evaluation capabilities for testing environments against various language models through Prime Inference or other OpenAI-compatible providers. You can run evaluations locally or launch them as hosted evaluations on the platform with --hosted. This guide covers both workflows, model selection, and best practices.

Quick Start: Running Your First Evaluation

Prerequisites

Python 3.10–3.13 — Required for the Prime CLI and verifiers
Install Prime CLI — Follow the installation guide
Set up API keys — Configure your Prime API key via prime login; if you plan to use another provider, also export the provider key you will reference with --api-key-var
Install an environment — Use prime env install owner/environment

Basic Evaluation

# List available environments
prime env list --owner primeintellect

# Install an environment
prime env install primeintellect/gsm8k@latest

# Run a basic evaluation
prime eval gsm8k

Local prime eval runs are automatically uploaded to the platform after each run. Use --skip-upload to disable this.

Hosted Evaluation Quick Start

If your environment is already published to the Environments Hub, you can run it remotely on Prime-managed infrastructure:

prime eval run primeintellect/gsm8k --hosted

Use --follow to stream hosted logs until completion:

prime eval run primeintellect/gsm8k --hosted --follow

See Hosted Evaluations for the full dashboard and CLI workflow.

Using the prime eval Command

Basic Syntax

prime eval ENVIRONMENT [OPTIONS]

This is a shorthand for prime eval run ENVIRONMENT. Both forms work identically.

Available Models

To see all available models for evaluation:

prime inference models

Example models:

Model	Notes
`openai/gpt-4.1-mini`	Fast, cost-effective (default)
`openai/gpt-4.1`	Higher quality
`anthropic/claude-sonnet-4.5`	Strong reasoning
`meta-llama/llama-3.3-70b-instruct`	Open-weight, balanced
`deepseek/deepseek-r1-0528`	Advanced reasoning
`qwen/qwen3-235b-a22b-instruct-2507`	Large MoE model
`google/gemini-2.5-flash`	Fast multimodal

Model availability and pricing may change. Always run prime inference models to get the current list with pricing.

Core Parameters

environment

string

required

Environment to evaluate. Supported forms:

Full slug (e.g., primeintellect/gsm8k) — recommended for hosted runs and Hub environments
Short name (e.g., gsm8k) — local-first resolution for installed environments
TOML config path (e.g., configs/eval/gsm8k.toml) — for config-driven runs

--model, -m

string

Model to use for evaluation. Default: openai/gpt-4.1-miniSee prime inference models for all available models.

--num-examples, -n

integer

Number of examples to evaluate. Default: 5

--rollouts-per-example, -r

integer

Number of rollouts per example for statistical significance. Default: 3

Advanced Options

--max-concurrent, -c

integer

Maximum concurrent requests to the inference API. Default: 32

--max-retries

integer

Maximum number of automatic retries with exponential backoff when rollouts fail due to transient infrastructure errors (e.g., sandbox timeouts, API failures).

--max-tokens, -t

integer

Maximum tokens to generate per request. If unset, uses model default.

--temperature, -T

number

Sampling temperature (0.0–2.0). Higher values = more randomness.

--sampling-args, -S

string

JSON string with additional sampling arguments.Example: '{"enable_thinking": false, "max_tokens": 256}'

--env-args, -a

string

Environment-specific arguments as JSON.Example: '{"difficulty": "hard"}'

Output and Storage Options

--verbose, -v

boolean

Enable verbose output for detailed logging.

--save-results, -s

boolean

Save evaluation results to disk. Default: true

--save-every, -f

integer

Save dataset every N rollouts. Useful for checkpointing. Default: 1

--save-to-hf-hub, -H

boolean

Save results to Hugging Face Hub.

--hf-hub-dataset-name, -D

string

Specify Hugging Face Hub dataset name.

--skip-upload

boolean

Skip uploading results for local evaluations (local runs upload by default).

Hosted Evaluation Options

--hosted

boolean

Run the evaluation on the platform instead of locally. Requires a published environment.

--follow

boolean

Follow hosted evaluation status and stream logs until completion. Only valid with --hosted.

--poll-interval

number

Polling interval in seconds for hosted status and log streaming. Only valid with --hosted.

--timeout-minutes

integer

Optional timeout in minutes for a hosted evaluation. Default: 1440 (24 hours). Min: 120. Max: 1440.

--allow-sandbox-access

boolean

Allow sandbox read/write access for hosted evaluations.

--allow-instances-access

boolean

Allow hosted evaluations to create and manage instances.

--allow-tunnel-access

boolean

Allow hosted evaluations to create and manage tunnels from inside the sandbox. This adds tunnel scopes to the temporary PRIME_API_KEY.

--custom-secrets

string

JSON object of additional secrets to inject into a hosted run.

--eval-name

string

Custom display name for a hosted evaluation.

--api-base-url and --api-key-var from the model configuration section also work with --hosted. When you use a custom endpoint for a hosted run, provide the referenced API key inside the remote sandbox via an environment secret or --custom-secrets.

End-to-End Example

Here’s a complete workflow from installation to viewing results:

# 1. Install an environment
prime env install primeintellect/gsm8k@latest

# 2. Run evaluation with a fast model (5 examples, 1 rollout each)
prime eval gsm8k -m openai/gpt-4.1-mini -n 20 -r 2

# 3. List your evaluation runs
prime eval list

# 4. Get details of a specific evaluation
prime eval get <eval-id>

# 5. View samples from an evaluation
prime eval samples <eval-id>

Example output:

Running evaluation: gsm8k
Model: openai/gpt-4.1-mini
Examples: 5 | Rollouts: 1 | Concurrency: 32

Progress: 100%|████████████████████████████| 5/5 [00:12<00:00, 2.45s/it]

Results:
  Average Score: 0.80
  Total Samples: 5
  Successful: 4
  Failed: 1

Results saved to: outputs/evals/gsm8k--openai-gpt-4.1-mini/...
Uploaded to platform: https://app.primeintellect.ai/...

Using TOML Configs for Multi-Environment Evals

For reproducible evals, you can pass a TOML config file instead of individual CLI flags:

prime eval run configs/eval/my-benchmark.toml

Local TOML configs can define one or more [[eval]] entries, which makes them useful for benchmark suites and multi-environment comparisons:

model = "openai/gpt-4.1-mini"
num_examples = 20

[[eval]]
env_id = "primeintellect/gsm8k"
rollouts_per_example = 2

[[eval]]
env_id = "primeintellect/alphabet-sort"
rollouts_per_example = 2

When you use a TOML config, per-eval settings in [[eval]] override global defaults at the top of the file. For the full config schema, precedence rules, and advanced options like ablations and endpoint registries, see Verifiers Evaluation.

Hosted Evaluations from the CLI

Hosted eval runs are useful when you want the platform to execute the environment remotely and keep logs on the platform.

prime eval run primeintellect/gsm8k \
  --hosted \

Hosted runs can also target a custom OpenAI-compatible endpoint:

prime eval run primeintellect/gsm8k \
  --hosted \
  -m openai/gpt-4.1-mini \
  --api-base-url https://api.openai.com/v1 \
  --api-key-var OPENAI_API_KEY \
  --custom-secrets '{"OPENAI_API_KEY":"..."}'

When --api-base-url is set on a hosted run, Prime still hosts the evaluation sandbox, but model billing comes from your external provider instead of Prime Inference. See Hosted Evaluations for the full hosted billing details. You can also launch a hosted run from a TOML file:

model = "openai/gpt-4.1-mini"
num_examples = 20
rollouts_per_example = 2

[[eval]]
env_id = "primeintellect/gsm8k"
env_args = { split = "test" }

prime eval run configs/eval/gsm8k-hosted.toml --hosted

Hosted run management commands:

prime eval logs <eval-id> -f
prime eval stop <eval-id>

Managing Evaluation Results

List Evaluations

# List all your evaluations
prime eval list

# Filter by environment
prime eval list --env gsm8k

# Output as JSON
prime eval list --output json

# Paginate results
prime eval list --num 20 --page 2

Get Evaluation Details

# Get full details of an evaluation
prime eval get <eval-id>

# Pretty-print output
prime eval get <eval-id> --output pretty

View Samples

# Get samples from an evaluation
prime eval samples <eval-id>

# Paginate samples
prime eval samples <eval-id> --page 2 --num 50

Push Local Results

If you ran evaluations offline or with --skip-upload, you can push results later:

# Auto-discover and push from outputs/evals/
prime eval push

# Push a specific directory
prime eval push outputs/evals/gsm8k--gpt-4/abc123

# Push with environment context
prime eval push --env gsm8k

Model Selection Guide

When choosing models for evaluation, consider:

Task complexity — Harder tasks benefit from larger, reasoning-capable models
Cost — Smaller models are significantly cheaper for large-scale evals
Throughput — Some models handle high concurrency better than others

Self-hosting with vLLM

For most users, we recommend Prime Inference for easier setup. Consider self-hosting only for specialized requirements or very large-scale evaluations.

Self-hosting makes sense when you:

Need specific model variants or custom fine-tuned models
Require maximum cost efficiency for very large evaluations (1M+ examples)
Are testing smaller models not available via API

Configuring for Self-Hosted Models

# Point to your vLLM instance
prime eval gsm8k \
  -m Qwen/Qwen3-4B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  --api-key-var CUSTOM_API_KEY \

Recommended Self-Hosted Models

High Performance (MoE with small active parameters):

prime eval gsm8k \
  -m Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  -n 2000 -c 96

Balanced Performance:

prime eval wordle \
  -m Qwen/Qwen3-4B-Instruct-2507 \
  --api-base-url http://localhost:8000/v1 \
  -n 5000 -c 128

vLLM Configuration Example

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --max-model-len 32768 \
  --max-num-seqs 256 \
  --tensor-parallel-size 2 \
  --quantization fp8

Environment-Specific Dependencies

Some environments may require additional dependencies beyond the base installation. Check the environment’s documentation or use prime env info to see requirements:

prime env info primeintellect/math_python

If an environment needs specific packages (e.g., sympy for math verification), install them before running evaluations.

Troubleshooting

Common Issues

Rate Limiting

Error: “Rate limit exceeded” or 429 responsesSolution: Reduce concurrency

prime eval gsm8k -n 100 -c 8

Insufficient Balance

Error: “Insufficient balance” or payment requiredSolution: Add funds to your Prime Intellect account or use cheaper models

prime eval gsm8k -m openai/gpt-4.1-mini -n 1000

Environment Not Found

Error: Environment not found or installation failuresSolution: Verify environment exists and reinstall

prime env list --owner primeintellect
prime env uninstall gsm8k
prime env install primeintellect/gsm8k@latest

Python Version Issues

Error: Installation failures or import errorsSolution: Ensure you’re using Python 3.10–3.13

python --version  # Should be 3.10, 3.11, 3.12, or 3.13

Performance Tips

Start Small: Begin with -n 5 to test your setup
Monitor Costs: Check token usage before large evaluations
Use Appropriate Models: Match model capability to task complexity
Optimize Concurrency: Balance speed vs. rate limits (default 32 is usually good)
Save Results: Results auto-save and upload by default

Integration with Other APIs

You can use other OpenAI-compatible API providers:

# DeepSeek API
prime eval math \
  -m deepseek-reasoner \
  --api-base-url https://api.deepseek.com/v1 \
  --api-key-var DEEPSEEK_API_KEY

# OpenRouter API
prime eval gsm8k \
  -m meta-llama/llama-3.1-405b-instruct \
  --api-base-url https://openrouter.ai/api/v1 \
  --api-key-var OPENROUTER_API_KEY

Next Steps

Environments Hub

Learn more about creating and managing environments

Inference API Reference

Detailed API documentation for inference endpoints

Prime CLI Reference

Complete CLI command reference

Creating Environments

Build your own evaluation environments

​Quick Start: Running Your First Evaluation

​Prerequisites

​Basic Evaluation

​Hosted Evaluation Quick Start

​Using the prime eval Command

​Basic Syntax

​Available Models

​Core Parameters

​Advanced Options

​Output and Storage Options

​Hosted Evaluation Options

​End-to-End Example

​Using TOML Configs for Multi-Environment Evals

​Hosted Evaluations from the CLI

​Managing Evaluation Results

​List Evaluations

​Get Evaluation Details

​View Samples

​Push Local Results

​Model Selection Guide

​Self-hosting with vLLM

​Configuring for Self-Hosted Models

​Recommended Self-Hosted Models

​vLLM Configuration Example

​Environment-Specific Dependencies

​Troubleshooting

​Common Issues

​Performance Tips

​Integration with Other APIs

​Next Steps

Environments Hub

Inference API Reference

Prime CLI Reference

Creating Environments

Quick Start: Running Your First Evaluation

Prerequisites

Basic Evaluation

Hosted Evaluation Quick Start

Using the prime eval Command

Basic Syntax

Available Models

Core Parameters

Advanced Options

Output and Storage Options

Hosted Evaluation Options

End-to-End Example

Using TOML Configs for Multi-Environment Evals

Hosted Evaluations from the CLI

Managing Evaluation Results

List Evaluations

Get Evaluation Details

View Samples

Push Local Results

Model Selection Guide

Self-hosting with vLLM

Configuring for Self-Hosted Models

Recommended Self-Hosted Models

vLLM Configuration Example

Environment-Specific Dependencies

Troubleshooting

Common Issues

Performance Tips

Integration with Other APIs

Next Steps