End-to-End Training Run

This guide walks you through a complete Hosted Training run — choosing an environment, establishing a baseline, writing a config, dispatching the run, monitoring it, and reviewing results. For the config schema and dispatch options in reference form, see Full Fine-Tuning.

Prerequisites

# Install and authenticate the CLI
uv tool install -U prime
prime login

You also need on-demand training enabled and a funded wallet — see Billing. Start from Getting Started if you haven’t run anything yet. Create a volume to hold the trained weights. Checkpointing is off by default and pods are torn down when a run ends, so without a volume and a [ckpt] block the run leaves you nothing to evaluate:

prime volumes create research --size 100Gi
prime volumes list   # wait for RUNNING before dispatching

Step 1: Choose an Environment

You can use an existing environment from the Environments Hub or create your own. For this walkthrough we’ll use alphabet-sort — a multi-turn game where the model must sort letters into alphabetical order. If you’re new to how environments work, see The Environment Model. Install it:

prime env install primeintellect/alphabet-sort

Step 2: Run a Baseline Evaluation

Before training, evaluate the base model to establish a baseline. This confirms the environment works and shows you where the model starts:

uv run vf-eval primeintellect/alphabet-sort \
  -m Qwen/Qwen3-4B-Instruct-2507 \
  -n 20 -r 1 -s

A good training environment should have a baseline reward between roughly 10–80%. If the model scores 0% after many attempts, the task is too hard. If it’s already at 80%+, consider harder examples or a different environment.

Results are saved under outputs/evals/ (metadata.json, results.jsonl). To view them on the dashboard, upload the run directory:

prime eval push outputs/evals/primeintellect/alphabet-sort--Qwen--Qwen3-4B-Instruct-2507/<run-id>

The baseline vf-eval results above use the legacy sample workflow. They are not a direct Prime Traces import. See supported trace uploads and exporting existing legacy samples.

Step 3: Choose a Model and GPU Type

prime train models

This lists the models you can dispatch, plus an On-Demand Capacity table of purchasable GPU types with their current $/GPU-hour and whether a run starts now or queues. Runs boot from a cluster-side weight cache, so pick a model from that list — see Models. For a first run, pick the smallest model that exercises your environment end to end. Validate cheaply, then scale.

Step 4: Create a Training Config

Training runs are configured with a prime-rl .toml file. The [deployment] block sizes the run and is what routes it onto Hosted Training.

# configs/alphabet-sort.toml
name = "alphabet-sort-rl"
max_steps = 50
seq_len = 2048

[ckpt]                                 # save a checkpoint at the end of training

[model]
name = "Qwen/Qwen3-4B-Instruct-2507"   # cached models in `prime train models` start warm

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[trainer.optim]
lr = 1e-6

[orchestrator]
batch_size = 128

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.source]]
name = "alphabet-sort"

[orchestrator.train.source.env.taskset]
id = "primeintellect/alphabet-sort"

[orchestrator.train.source.env.agent.harness]
id = "default"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

[inference]

The key fields:

[model].name — the model to train. Models listed by prime train models are already cached and start warm; anything else is downloaded first, which adds billable time before the first step
[ckpt] — save checkpoints. Off by default; with no [ckpt] block nothing is written
[deployment] — how many trainer and inference GPUs to reserve; required for dispatch
max_steps — total training steps
[orchestrator].batch_size — rollouts consumed per training batch
[orchestrator].group_size — rollouts generated per dataset example
[orchestrator.train.sampling].max_completion_tokens — max tokens the model generates per turn
[[orchestrator.train.source]] — one block per training environment, carrying a nested env block

Taskset and harness ids can be bare runtime ids ("alphabet-sort-v1", "default") resolved by the installed runtime package, or slash-shaped Hub refs ("primeintellect/alphabet-sort", optionally with @version) resolved through the Environments Hub.

Trace storage and costs

Traces uploaded by platform runs incur Prime Traces ingestion and storage charges, separately from training charges. Review workspace upload controls before launching. See Viewing and deleting traces to manage the resulting data.

Step 5: Launch the Training Run

prime train configs/alphabet-sort.toml --volume research

The CLI confirms the configuration, echoes the GPU type and hourly rate it resolved, and prints a dashboard link:

Dispatched hosted run wn2cjdrzdo6bmfqajoeuu30p

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/wn2cjdrzdo6bmfqajoeuu30p

Pass credentials your environment needs with -e / --env-var or --env-file:

prime train configs/alphabet-sort.toml -e WANDB_API_KEY -e OPENAI_API_KEY

Config errors — bad schema, an uncached model, an oversized deployment — fail the dispatch immediately with a reason, before any GPU is reserved and before anything bills.

Step 6: Monitor the Run

In the terminal — stream logs in real time:

prime train logs <run-id> -f

A run has several components; pick one with -c / --component:

prime train logs <run-id>                  # orchestrator (default)
prime train logs <run-id> -c trainer       # trainer (FSDP / torchrun)
prime train logs <run-id> -c inference     # vLLM inference server
prime train logs <run-id> --env <env-name> # env-server for a specific env

Filter with --search, --regex, --level, and --since. On the dashboard — open the URL printed when the run started, for reward curves, rubric scores, reward distributions, and individual rollouts. Key metrics to watch:

Reward — the overall reward curve should trend upward over time
Rubric — individual rubric component scores
Reward Distribution — should shift from lower to higher values as training progresses

Stop a run early to release its GPUs and stop the meter:

prime train stop <run-id>

Step 7: Review Results

Retrieve the checkpoint

Each run writes to its own runs/<run-id>/ directory on the volume, with the prime-rl run directory nested under outputs/. Browse it first, then pull the whole run directory down — the export step needs the run’s resolved config, not just the checkpoint:

prime volumes ssh research                  # browse what the run wrote
prime volumes get research runs/<run-id> .  # copy it down

Export to HF format

Trainer checkpoints are DCP-sharded, so they need converting before anything else can load them. The converter lives in the prime-rl repo and needs at least one GPU:

# from a prime-rl checkout
uv run python tools/convert_dcp_to_bf16.py <run-dir>/checkpoints/step_<n>

# multi-rank, for faster gathers and models too big for one GPU
uv run torchrun --nproc-per-node 8 tools/convert_dcp_to_bf16.py <run-dir>/checkpoints/step_<n>

This writes sharded safetensors plus config and tokenizer assets to <run-dir>/checkpoints/step_<n>/weights. That directory loads into any HF consumer. See Exporting Checkpoints for FP8 variants.

Evaluate against your baseline

Serve the exported weights, then point the Step 2 evaluation at them:

# serve the exported directory
uv run inference --vllm.model <run-dir>/checkpoints/step_<n>/weights

# evaluate the same environment, same sample size as Step 2
uv run vf-eval primeintellect/alphabet-sort \
  -m <run-dir>/checkpoints/step_<n>/weights \
  -b http://localhost:8000/v1 -k OPENAI_API_KEY \
  -n 20 -r 1 -s

-b points vf-eval at your endpoint and -k names the env var holding its key. Compare the reward against the baseline you recorded in Step 2.

Metrics, samples, and progress are persisted to the dashboard either way, but trainer, inference, and env-server pod logs go away with the pods when the run is torn down — pull anything you need while the run is live.

Run Size Guidelines

Each of these omits the blocks already covered above — add [ckpt] and dispatch with --volume or the run saves nothing.

Small (validation)

Verify your environment and config work before committing to a longer run.

max_steps = 50
seq_len = 2048

[deployment]
num_train_gpus = 1
num_infer_gpus = 1

[orchestrator]
batch_size = 128
group_size = 8

Medium (experimentation)

Good for iterating on environment design and hyperparameters.

max_steps = 200
seq_len = 4096

[deployment]
num_train_gpus = 2
num_infer_gpus = 2

[orchestrator]
batch_size = 256
group_size = 16

[wandb]
project = "my-experiment"

Large (production)

max_steps = 500
seq_len = 32768

[deployment]
type = "multi_node"
num_train_nodes = 2
num_infer_nodes = 2

[trainer.model]
attn = "flash_attention_3"

[orchestrator]
batch_size = 512
group_size = 16

[wandb]
project = "production"

WANDB_API_KEY is required for the [wandb] block to take effect — without it the block is stripped and the run trains without W&B logging.

Full Fine-Tuning

Config reference, dispatch knobs, limits, and run lifecycle.

Billing

How GPU-hours are metered and what gets billed.

Troubleshooting

Solutions for common issues with Hosted Training runs.

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

Prerequisites

Step 1: Choose an Environment

Step 2: Run a Baseline Evaluation

Step 3: Choose a Model and GPU Type

Step 4: Create a Training Config

Trace storage and costs

Step 5: Launch the Training Run

Step 6: Monitor the Run

Step 7: Review Results

Retrieve the checkpoint

Export to HF format

Evaluate against your baseline

Run Size Guidelines

Small (validation)

Medium (experimentation)

Large (production)

Full Fine-Tuning

Billing

Troubleshooting

​Prerequisites

​Step 1: Choose an Environment

​Step 2: Run a Baseline Evaluation

​Step 3: Choose a Model and GPU Type

​Step 4: Create a Training Config

​Trace storage and costs

​Step 5: Launch the Training Run

​Step 6: Monitor the Run

​Step 7: Review Results

​Retrieve the checkpoint

​Export to HF format

​Evaluate against your baseline

​Run Size Guidelines

​Small (validation)

​Medium (experimentation)

​Large (production)

Full Fine-Tuning

Billing

Troubleshooting

Prerequisites

Step 1: Choose an Environment

Step 2: Run a Baseline Evaluation

Step 3: Choose a Model and GPU Type

Step 4: Create a Training Config

Trace storage and costs

Step 5: Launch the Training Run

Step 6: Monitor the Run

Step 7: Review Results

Retrieve the checkpoint

Export to HF format

Evaluate against your baseline

Run Size Guidelines

Small (validation)

Medium (experimentation)

Large (production)