Hosted SFT (Beta)

Hosted SFT is in beta. Contact Prime support if dispatch is refused.

Run prime-rl supervised fine-tuning on Prime’s hosted GPU clusters: create a volume, copy your dataset onto it over SSH, dispatch. No kubectl, no cluster credentials — the trainer reads the dataset from the volume. This walkthrough fine-tunes a small Qwen3 model to reverse text (reverse-text) on one GPU. For SFT on your own infrastructure, see Training.

Prerequisites

uv tool install prime
prime login

Requires prime CLI v0.9.0 or later.

A Prime account with hosted training access — Prime-side enablement; the platform deployment needs the hosted SFT path switched on.
A base model the run can boot from: in the model cache (check with prime train models), or weights already on the volume itself via a /volume/... [model].name path (Volume-hosted models).

prime train models

A Hugging Face dataset you can download with hf download — with prompt/completion or messages columns (Training § Dataset Format).

1. Create a volume

prime volumes create research --size 100Gi

Size the volume for your dataset plus the checkpoints the run keeps on its disk. A checkpoint is about 11–12 bytes per parameter: the 0.6B smoke run writes ~7.2 GB. See Choose a size. Grow it with prime volumes expand <name> --size <size>.

2. Put the dataset on the volume

Open a shell on the volume and download the dataset onto it. The download uses the cluster’s network, not your laptop’s. Shells need a cluster allocation or a cluster your team owns and your primary SSH key:

prime volumes ssh research --read-write

Inside the session:

hf download willcb/R1-reverse-wikipedia-paragraphs-v1-1000 \
  --repo-type dataset \
  --local-dir /volume/datasets/reverse-text
exit

Back in your local terminal, stop the session using the ID printed when it opened:

prime volumes stop research <session-id>

Wait for its final sync before dispatching. Exiting SSH alone leaves the read-write session open and blocks the training run. The trainer sees the volume at /volume, the same path you see in the shell. Without one, download the dataset to your machine and upload it instead: prime volumes put <name> ./my-data datasets/ creates datasets/my-data (paths start at the volume root, and a trailing / means “into this folder”). In the config, data.name accepts either form: a path relative to the volume (datasets/my-data) or the absolute mount path (/volume/datasets/my-data) — the platform resolves both to the same location.

3. Write the config

Save this as sft.toml — data.name points at the dataset on the volume (relative to the volume root; absolute /volume/... paths work too):

max_steps = 10

[ckpt]               # save step 10 to the volume
interval = 10
keep_last = 1

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"  # must be in the cluster model cache

[data]
name = "datasets/reverse-text"  # volume-relative; the platform resolves it to /volume/datasets/reverse-text
type = "sft"
seq_len = 2048        # add splits = ["train"] for multi-split datasets
batch_size = 8

[optim]
lr = 2e-5

[deployment]
type = "single_node"
num_train_gpus = 1
gpus_per_node = 1

[renderer]
name = "prime-qwen3"  # must match the model family

output_dir is injected by the platform — omit it. The walkthrough model is a cached debug sibling already fine-tuned on this dataset — treat the loss curve as a smoke test of the hosted path.This walkthrough is trainer-only. To score checkpoints against an environment while the run trains, see Online evals below.

Instead of a cached model ID, [model].name can point at HF-format weights already on the volume, uploaded with prime volumes put. RL configs support volume model paths too. Run checkpoints are in prime-rl’s distributed format, so export them to HF format before using them as [model].name — or resume the full training state by pointing [resume].dir at the checkpoint’s step directory on the volume (dir = "/volume/runs/<runId>/checkpoints/step_<N>"). See Volume-hosted models.

4. Dispatch

prime train sft.toml --volume research

Creating Hosted Training run...
Dispatched hosted run kqeggj5dl7k85bc0mk1i4y5i

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/kqeggj5dl7k85bc0mk1i4y5i

Use the run ID your dispatch printed (<runId>) below. The run moves PENDING → CREATING → RUNNING → COMPLETED — expect roughly three minutes on one H200 for this walkthrough.

5. Watch it run

prime train get <runId>
prime train metrics <runId>   # the loss series the dashboard renders
prime train logs <runId>      # the live trainer log

Run kqeggj5dl7k85bc0mk1i4y5i

  Status: COMPLETED
  Model: PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT
  Max Steps: 10
  Created: 2026-09-28 20:33
  Started: 2026-09-28 20:34
  Completed: 2026-09-28 20:37

step: [1, 2, 3, 4, 5, 6, 7, 8, 9]
loss/mean:       [1.108, 1.1788, 0.9996, 0.9468, 1.1179, 1.044, 1.0353, 1.0038, 0.9944]
loss/perplexity: [3.028, 3.251, 2.717, 2.577, 3.058, 2.841, 2.816, 2.729, 2.703]

35:38 [INFO] Initializing scheduler with 10 steps (type='constant')
Generating train split: 100%|████████████████████| 1000/1000 [00:00<00:00, 23213.48
examples/s]
35:47 [SUCCESS] Step 1 |    8.5s | Loss 1.1080 | Grad. Norm 12.9495 | LR
00e-05 | Throughput 0 tokens/s | MFU 0.0% | Peak Mem. 10.7/139.8 GiB (7.7%)
35:52 [SUCCESS] Step 3 |    2.4s | Loss 0.9996 | Grad. Norm 7.4464 | LR
00e-05 | Throughput 6704 tokens/s | MFU 2.9% | Peak Mem. 10.7/139.8 GiB (7.7%)

Ten steps on a 0.6B model with a constant LR will not descend cleanly — the point is a visible, NaN-free curve. Occasionally the last metrics row lands during teardown; terminal status and the trainer’s final log lines are the success signal. Teardown is automatic. The step-10 checkpoint and the datasets/ folder stay on the volume. Wait for the run to end and the final volume sync to complete before downloading its checkpoint:

prime volumes get research runs/<runId>/checkpoints/step_10/ ./step_10

(Usage accounting shows 0 tokens / $0.00 for SFT — a reporting quirk, not free usage.)

Online evals

Add an [eval] block to score checkpoints against an environment while the run trains. The platform then runs an inference server alongside the trainer, which broadcasts its weights to it at each eval interval, plus one env server per eval source:

[eval]
interval = 5          # evaluate every 5 trainer steps

[[eval.source]]
name = "reverse-text"
env = { taskset = { id = "primeintellect/reverse-text" } }

[inference]
[inference.vllm]
tensor_parallel_size = 1

Sizing: inference GPUs come out of the same allocation as the trainer, so num_train_gpus + num_infer_gpus must fit gpus_per_node on a single node. On multi_node, set num_infer_nodes to at least 1 and add an [inference.router] block.

[deployment]
type = "single_node"
num_train_gpus = 4
num_infer_gpus = 4    # defaults to 1
gpus_per_node = 8

Eval metrics land on the run’s dashboard next to the loss curve, and each component gets its own log tab (prime train logs <runId> --component inference). The run stays RUNNING until the evals triggered by the final checkpoint have drained, so expect it to outlast the last trainer step.

Online evals need a cluster on prime-rl-fft 0.24.0 or newer; dispatch is rejected with a clear error otherwise. Weight transport defaults to NCCL when [inference] is set, and to filesystem for LoRA runs. Traces are not yet wired for SFT eval rollouts.

Troubleshooting

Online evals rejected at dispatch

The cluster’s chart predates online-eval support. Remove the [eval] block to run trainer-only SFT, or ask Prime support when the cluster will be upgraded.

Error: online evals ([eval]) are not supported by the prime-rl-fft chart this
deployment installs (needs 0.24.0 or newer). Remove the [eval] block to run
trainer-only SFT.

[inference] without [eval]

The inference pool exists only to serve online-eval rollouts, so it needs an [eval] block to go with it.

Error: [inference] is only used for online evals — add an [eval] block or
remove it.

Model not available on this cluster

Hosted SFT boots from the cluster model cache. Check prime train models, or ask Prime support to cache the model. If the weights are already on the volume, point [model].name at their /volume/... path instead — no cache entry is needed.

Error: HTTP 400: Model 'PrimeIntellect/Qwen3-0.6B' is not available on this
cluster. Reach out to Prime support to get it added.

data.name requires --volume

Real data needs a volume. data.type = "fake" is the one no-volume path (synthetic data, connectivity smoke test).

Error: HTTP 400: data.name requires `--volume <name>`. Create a volume with
`prime volumes create <name>`, then rerun `prime train sft.toml --volume <name>`.

Dataset missing on the volume

The trainer reads data.name from the volume mount — a path that is empty or missing fails at startup. Re-run step 2 and check the name matches: the directory you create under /volume/datasets/<name> in the SSH session is exactly the datasets/<name> you point data.name at (relative paths resolve under /volume, so datasets/<name> and /volume/datasets/<name> are the same location).

Volume ran out of space during download

Inside the shell, hf download fails when the session’s disk is full. Exit, end the session, grow the volume, and re-run the download in a new session:

prime volumes stop research <session-id>
prime volumes expand research --size <larger-size>
prime volumes ssh research

Limitations

Requires CLI v0.9.0+ — older versions misroute SFT configs against the RL schema.
SFT-shaped datasets — prompt/completion or messages columns.
Cached or volume-hosted models — a cached HF id (prime train models; support can add others), or weights on the volume via a /volume/... [model].name path.
Online evals need chart 0.24.0+ — on older clusters SFT is trainer-only and [eval] is rejected. No traces for eval rollouts yet.
Checkpoints live on the volume — use prime train checkpoints <runId> to list saved checkpoints, then prime volumes get research runs/<runId>/checkpoints/ to download them after sync.

Scaling up

The same flow carries to multi-node — switch [deployment] to multi_node and add the parallelism knobs. Save as glm53-sft.toml:

max_steps = 4

[env_vars]
TRITON_CACHE_DIR = "/tmp/.cache/triton_cache"

[model]
name = "zai-org/GLM-5.3-BF16"
impl = "custom"
attn = "flash_attention_2"
cp = 8
ep = 8
optim_cpu_offload = true

[model.ac]
freq = 1

[model.ac_offloading]
max_inflight_activations = 5

[optim]
type = "sign_sgd"
lr = 5e-5
weight_decay = 0.01

[scheduler]
type = "linear"
warmup_steps = 50
decay_steps = 0

[renderer]
name = "glm-5.3"

[deployment]
type = "multi_node"
num_train_nodes = 8
gpus_per_node = 8

[model.debug]
force_balanced_routing = true  # debug only

[ckpt]               # save the final checkpoint
interval = 4
keep_last = 1

[data]
type = "sft"
name = "datasets/intellect-3-sft-10k"  # put it on the volume first (step 2); volume-relative
splits = ["math"]
batch_size = 8
micro_batch_size = 1
seq_len = 4096

This config targets 8×8 H200 (64 GPUs) — 4 steps take ~5.5 min on that hardware. If you adapt it: remove force_balanced_routing — it forces round-robin expert routing and is debug-only; with warmup_steps = 50 a 4-step run never leaves warmup, so losses are warmup-dominated rather than convergence; the [ckpt] block saves one checkpoint at step 4, so size the volume for it (about 11–12 bytes per parameter, see Choose a size); with this linear schedule the LR ramps to its peak at step 50 — expect a transient loss spike near the peak before it settles (9.42 at step 55 in a 60-step run). See Scaling and Advanced for the knobs.

Hosted RL

RL-style full-parameter training on a dedicated cluster — same CLI, same dashboard.

prime-rl Training

The underlying SFT trainer: config schema, dataset format, and local runs.

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

Prerequisites

1. Create a volume

2. Put the dataset on the volume

3. Write the config

4. Dispatch

5. Watch it run

Online evals

Troubleshooting

Limitations

Scaling up

Hosted RL

prime-rl Training

​Prerequisites

​1. Create a volume

​2. Put the dataset on the volume

​3. Write the config

​4. Dispatch

​5. Watch it run

​Online evals

​Troubleshooting

​Limitations

​Scaling up

Hosted RL

prime-rl Training

Prerequisites

1. Create a volume

2. Put the dataset on the volume

3. Write the config

4. Dispatch

5. Watch it run

Online evals

Troubleshooting

Limitations

Scaling up