Hosted SFT (Beta)

Hosted SFT is in closed beta — Prime enables it per account. Contact Prime support if dispatch is refused.

Run prime-rl supervised fine-tuning on Prime’s hosted GPU clusters: create a volume, copy your dataset onto it over SSH, dispatch. No kubectl, no cluster credentials — the trainer reads the dataset from the volume. This walkthrough fine-tunes a small Qwen3 model to reverse text (reverse-text) on one GPU. For SFT on your own infrastructure, see Training.

Prerequisites

uv tool install prime
prime login

Requires prime CLI v0.9.0 or later.

A Prime account with hosted training access — Prime-side enablement; the platform deployment needs the hosted SFT path switched on.
A base model the run can boot from: cached on the cluster that owns your volume (check with prime train models --fft-only), or weights already on the volume itself via a /volume/... [model].name path (below). Shared-LoRA models are not valid here.

prime train models --fft-only

A Hugging Face dataset you can download with hf download — with prompt/completion or messages columns (Training § Dataset Format).

1. Create a volume

prime volumes create research --size 100Gi
prime volumes list   # wait for RUNNING (~10 s) before the next step

Size for what you keep: datasets plus weights + optimizer state per checkpoint — the 0.6B smoke run wrote ~7 GB; a 100B-class [ckpt] needs terabytes. Grow with prime volumes resize <name> --size <size>.

2. Put the dataset on the volume

SSH into the volume and download the dataset onto it. The session runs next to the volume in the cluster, so the download uses the cluster’s network — not your laptop:

prime volumes ssh research --read-write

Inside the session:

hf download willcb/R1-reverse-wikipedia-paragraphs-v1-1000 \
  --repo-type dataset \
  --local-dir /volume/datasets/reverse-text
exit

The trainer sees the volume at /volume — the same path you see in the SSH session. Local files work too: prime volumes put <name> ./my-data datasets/my-data — put paths are relative to the volume root (the CLI prepends /volume itself). In the config, data.name accepts either form: a path relative to the volume (datasets/my-data) or the absolute mount path (/volume/datasets/my-data) — the platform resolves both to the same location.

3. Write the config

Save this as sft.toml — data.name points at the dataset on the volume (relative to the volume root; absolute /volume/... paths work too):

max_steps = 10

[ckpt]               # checkpoint at the end of training, saved to the volume

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"  # must be in the cluster model cache

[data]
name = "datasets/reverse-text"  # volume-relative; the platform resolves it to /volume/datasets/reverse-text
type = "sft"
seq_len = 2048        # add splits = ["train"] for multi-split datasets
batch_size = 8

[optim]
lr = 2e-5

[deployment]
type = "single_node"
num_train_gpus = 1
gpus_per_node = 1

[renderer]
name = "prime-qwen3"  # must match the model family

output_dir is injected by the platform — omit it. The walkthrough model is a cached debug sibling already fine-tuned on this dataset — treat the loss curve as a smoke test of the hosted path.This walkthrough is trainer-only. To score checkpoints against an environment while the run trains, see Online evals below.

Instead of a cached model id, [model].name can point at HF-format weights already on the volume, uploaded with prime volumes put. Run checkpoints are in prime-rl’s distributed format and do not work as [model].name — continue those with [resume].dir on the checkpoint’s step directory instead (Volume-hosted models).

4. Dispatch

prime train sft.toml --volume research

Creating Hosted Training run...
Dispatched hosted run kqeggj5dl7k85bc0mk1i4y5i

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/kqeggj5dl7k85bc0mk1i4y5i

Use the run ID your dispatch printed (<runId>) below. The run moves PENDING → CREATING → RUNNING → COMPLETED — expect roughly three minutes on one H200 for this walkthrough.

5. Watch it run

prime train get <runId>
prime train metrics <runId>   # the loss series the dashboard renders
prime train logs <runId>      # the live trainer log

Run kqeggj5dl7k85bc0mk1i4y5i

  Status: COMPLETED
  Model: PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT
  Max Steps: 10
  Created: 2026-09-28 20:33
  Started: 2026-09-28 20:34
  Completed: 2026-09-28 20:37

step: [1, 2, 3, 4, 5, 6, 7, 8, 9]
loss/mean:       [1.108, 1.1788, 0.9996, 0.9468, 1.1179, 1.044, 1.0353, 1.0038, 0.9944]
loss/perplexity: [3.028, 3.251, 2.717, 2.577, 3.058, 2.841, 2.816, 2.729, 2.703]

35:38 [INFO] Initializing scheduler with 10 steps (type='constant')
Generating train split: 100%|████████████████████| 1000/1000 [00:00<00:00, 23213.48
examples/s]
35:47 [SUCCESS] Step 1 |    8.5s | Loss 1.1080 | Grad. Norm 12.9495 | LR
00e-05 | Throughput 0 tokens/s | MFU 0.0% | Peak Mem. 10.7/139.8 GiB (7.7%)
35:52 [SUCCESS] Step 3 |    2.4s | Loss 0.9996 | Grad. Norm 7.4464 | LR
00e-05 | Throughput 6704 tokens/s | MFU 2.9% | Peak Mem. 10.7/139.8 GiB (7.7%)

Ten steps on a 0.6B model with a constant LR will not descend cleanly — the point is a visible, NaN-free curve. Occasionally the last metrics row lands during teardown; terminal status and the trainer’s final log lines are the success signal. Teardown is automatic; outputs and the datasets/ tree stay on the volume. Fetch your outputs any time:

prime volumes get research /runs/<runId>/outputs/fft-<runId>/checkpoints/step_10/trainer .

(Usage accounting shows 0 tokens / $0.00 for SFT — a reporting quirk, not free usage.)

Online evals

Add an [eval] block to score checkpoints against an environment while the run trains. The platform then runs an inference server alongside the trainer, which broadcasts its weights to it at each eval interval, plus one env server per eval source:

[eval]
interval = 5          # evaluate every 5 trainer steps

[[eval.source]]
name = "reverse-text"
env = { taskset = { id = "primeintellect/reverse-text" } }

[inference]
[inference.vllm]
tensor_parallel_size = 1

Sizing: inference GPUs come out of the same allocation as the trainer, so num_train_gpus + num_infer_gpus must fit gpus_per_node on a single node. On multi_node, set num_infer_nodes to at least 1 and add an [inference.router] block.

[deployment]
type = "single_node"
num_train_gpus = 4
num_infer_gpus = 4    # defaults to 1
gpus_per_node = 8

Eval metrics land on the run’s dashboard next to the loss curve, and each component gets its own log tab (prime train logs <runId> --component inference). The run stays RUNNING until the evals triggered by the final checkpoint have drained, so expect it to outlast the last trainer step.

Online evals need a cluster on prime-rl-fft 0.24.0 or newer; dispatch is rejected with a clear error otherwise. Weight transport defaults to NCCL when [inference] is set, and to filesystem for LoRA runs. Traces are not yet wired for SFT eval rollouts.

Troubleshooting

Online evals rejected at dispatch

The cluster’s chart predates online-eval support. Remove the [eval] block to run trainer-only SFT, or ask Prime support when the cluster will be upgraded.

Error: online evals ([eval]) are not supported by the prime-rl-fft chart this
deployment installs (needs 0.24.0 or newer). Remove the [eval] block to run
trainer-only SFT.

[inference] without [eval]

The inference pool exists only to serve online-eval rollouts, so it needs an [eval] block to go with it.

Error: [inference] is only used for online evals — add an [eval] block or
remove it.

Model not available on this cluster

Hosted SFT boots from the cluster model cache. Check prime train models --fft-only, or ask Prime support to cache the model. If the weights are already on the volume, point [model].name at their /volume/... path instead — no cache entry is needed.

Error: HTTP 400: Model 'PrimeIntellect/Qwen3-0.6B' is not available on this
cluster. Reach out to Prime support to get it added.

data.name requires --volume

Real data needs a volume. data.type = "fake" is the one no-volume path (synthetic data, connectivity smoke test).

Error: HTTP 400: data.name requires `--volume <name>`. Create a volume with
`prime volumes create <name>`, then rerun `prime train sft.toml --volume <name>`.

Dataset missing on the volume

The trainer reads data.name from the volume mount — a path that is empty or missing fails at startup. Re-run step 2 and check the name matches: the directory you create under /volume/datasets/<name> in the SSH session is exactly the datasets/<name> you point data.name at (relative paths resolve under /volume, so datasets/<name> and /volume/datasets/<name> are the same location).

Volume ran out of space during download

Inside the SSH session, hf download fails when the volume is full. Exit, grow the volume, and re-run the download:

prime volumes resize research --size <larger-size>
prime volumes ssh research --read-write

Limitations

Requires CLI v0.9.0+ — older versions misroute SFT configs against the RL schema.
SFT-shaped datasets — prompt/completion or messages columns.
Cached or volume-hosted models — a cached HF id (prime train models --fft-only; support can add others), or weights on the volume via a /volume/... [model].name path.
Online evals need chart 0.24.0+ — on older clusters SFT is trainer-only and [eval] is rejected. No traces for eval rollouts yet.
Checkpoints live on the volume — prime train checkpoints doesn’t list SFT ones; use prime volumes get/ssh.

Scaling up

The same flow carries to multi-node — switch [deployment] to multi_node and add the parallelism knobs. Save as glm53-sft.toml:

max_steps = 4

[env_vars]
TRITON_CACHE_DIR = "/tmp/.cache/triton_cache"

[model]
name = "zai-org/GLM-5.3-BF16"
impl = "custom"
attn = "flash_attention_2"
cp = 8
ep = 8
optim_cpu_offload = true

[model.ac]
freq = 1

[model.ac_offloading]
max_inflight_activations = 5

[optim]
type = "sign_sgd"
lr = 5e-5
weight_decay = 0.01

[scheduler]
type = "linear"
warmup_steps = 50
decay_steps = 0

[renderer]
name = "glm-5.3"

[deployment]
type = "multi_node"
num_train_nodes = 8
gpus_per_node = 8

[model.debug]
force_balanced_routing = true  # debug only

[data]
type = "sft"
name = "datasets/intellect-3-sft-10k"  # put it on the volume first (step 2); volume-relative
splits = ["math"]
batch_size = 8
micro_batch_size = 1
seq_len = 4096

This config targets 8×8 H200 (64 GPUs) — 4 steps take ~5.5 min on that hardware. If you adapt it: remove force_balanced_routing — it forces round-robin expert routing and is debug-only; with warmup_steps = 50 a 4-step run never leaves warmup, so losses are warmup-dominated rather than convergence; add a [ckpt] block if you want checkpoints (this one omits it, so none are saved); with this linear schedule the LR ramps to its peak at step 50 — expect a transient loss spike near the peak before it settles (9.42 at step 55 in a 60-step run). See Scaling and Advanced for the knobs.

Full Fine-Tuning

RL-style full-parameter training on a dedicated cluster — same CLI, same dashboard.

prime-rl Training

The underlying SFT trainer: config schema, dataset format, and local runs.

Getting Started

Lab

Inference

Model Factory

Libraries

Compute

Prerequisites

1. Create a volume

2. Put the dataset on the volume

3. Write the config

4. Dispatch

5. Watch it run

Online evals

Troubleshooting

Limitations

Scaling up

Full Fine-Tuning

prime-rl Training

​Prerequisites

​1. Create a volume

​2. Put the dataset on the volume

​3. Write the config

​4. Dispatch

​5. Watch it run

​Online evals

​Troubleshooting

​Limitations

​Scaling up

Full Fine-Tuning

prime-rl Training

Prerequisites

1. Create a volume

2. Put the dataset on the volume

3. Write the config

4. Dispatch

5. Watch it run

Online evals

Troubleshooting

Limitations

Scaling up