Hosted Pretraining (Beta)

Hosted pretraining is in beta. Contact Prime support if dispatch is refused.

Run prime-rl pretraining on Prime’s hosted GPU clusters. The trainer starts from randomly initialized weights and learns next-token prediction on text you tokenize ahead of time. You tokenize your corpus into shards, put them on a volume, and dispatch; the trainer reads the shards from the volume. This walkthrough pretrains a Qwen3-0.6B-sized model on TinyStories on one 8-GPU node. For pretraining on your own infrastructure, see prime-rl’s Pretraining guide.

Prerequisites

uv tool install prime
prime login

Requires prime CLI v0.20.1 or later. Older versions don’t read mode and send a pretraining config as SFT, and it fails validation.

A Prime account with hosted training access.
A model in the cluster model cache (prime train models) whose architecture and tokenizer you want. Pretraining reads only the model’s config and tokenizer, never its weights, so any cached checkpoint of the right architecture works.
datatrove to tokenize your corpus: pip install "datatrove[io,processing]".

1. Tokenize your corpus

prime-rl does not tokenize. It reads the shards datatrove’s DocumentTokenizer writes: *.ds token files, each with a .ds.index and a .ds.metadata. Tokenize with the same tokenizer as the model you will train, and set eos_token to the tokenizer’s end-of-text token so documents stay separated. For Qwen that is <|endoftext|>, not <|im_end|>.

from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import HuggingFaceDatasetReader
from datatrove.pipeline.tokens import DocumentTokenizer

for split, folder in (("train", "train"), ("validation", "val")):
    LocalPipelineExecutor(
        pipeline=[
            HuggingFaceDatasetReader("roneneldan/TinyStories", {"split": split}),
            DocumentTokenizer(
                output_folder=f"tinystories/{folder}",
                tokenizer_name_or_path="Qwen/Qwen3-0.6B",
                eos_token="&lt;|endoftext|&gt;",
            ),
        ],
        tasks=4,
        logging_dir=f"logs/{folder}",
    ).run()

This writes tinystories/train/ and tinystories/val/. Keep validation documents in their own folder.

2. Put the shards on a volume

prime volumes create research --size 100Gi
prime volumes put research ./tinystories datasets/

The second command creates datasets/tinystories/train and datasets/tinystories/val on the volume (a trailing / means “into this folder”). Size the volume for the shards plus the checkpoints the run keeps: a checkpoint is about 11 to 12 bytes per parameter, so about 7 GB for a 0.6B model. See Choose a size. A running run holds its volume’s writer lock: while a run is active on a volume, dispatching a second run against it fails with an “in use by a run” error. Give concurrent runs their own volume (the same dataset can be uploaded to each). For a corpus too large to upload from your machine, open a shell on the volume (prime volumes ssh research --read-write) and tokenize there, or import it from S3.

3. Write the config

The top-level mode = "pretrain" makes this a pretraining run. Save this as pretrain.toml:

mode = "pretrain"
max_steps = 300

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"  # a cached Qwen3-0.6B; only its config and tokenizer are read

[data]
paths = ["datasets/tinystories/train"]  # folders on the volume
seq_len = 1024
batch_size = 64
micro_batch_size = 2

[val]
interval = 50
num_samples = 64

[val.data]
paths = ["datasets/tinystories/val"]
seq_len = 1024  # not inherited from [data]

[ckpt]
interval = 50

[deployment]
type = "single_node"
num_train_gpus = 8

data.paths lists folders on the volume, relative to its root or as absolute /volume/... paths. Every shard in them forms one token pool, read in seq_len windows in a new random order each epoch.
The weights start from the architecture’s own initialization. optim.lr defaults to 1e-3, with a 10-step linear warmup.
output_dir is set by the platform; leave it out.

See prime-rl’s Pretraining guide for every [data] and [val] option.

4. Dispatch

prime train pretrain.toml --volume research

Dispatched hosted run pgdrlpmhh7l7pm62299ppa47

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/pgdrlpmhh7l7pm62299ppa47

The CLI reads mode and validates the config against prime-rl’s pretraining schema before submitting it.

5. Watch it run

prime train get <runId>
prime train metrics <runId>
prime train logs <runId>

The dashboard shows the run as Pretrain, with its training loss, perplexity and validation loss. The walkthrough takes about ten minutes on eight GPUs, and its validation loss falls from about 3.8 at step 150 to about 3.1 at step 250.

Outputs

The run keeps its run directory on the volume at runs/<runId>/, with checkpoints/step_<N>/, logs/ and monitors/. Everything on the volume syncs to cloud storage when the run ends.

prime volumes get research runs/<runId>/checkpoints/step_300/ ./step_300

Troubleshooting

data.paths is not on the volume

Pretraining reads shards only from the run’s volume. Pass --volume and list folders on it, relative to the volume root (datasets/<name>/train) or as /volume/... paths.

Error: data.paths: Hosted pretrain reads tokenized shards from the named volume:
pass --volume and set data.paths to folders on it (e.g. data.paths =
["datasets/<name>/train"]).

Model not available on this cluster

The model’s config and tokenizer come from the cluster model cache. Pick a cached model of the architecture you want (prime train models), or ask Prime support to cache one.

Online evals rejected

Pretraining runs are trainer-only: [eval], [inference] and [weight_broadcast] are rejected. Use [val] to track held-out loss instead.

Loss does not fall, or the model never stops generating

Check that the shards were tokenized with the model’s tokenizer and with eos_token set to its end-of-text token. Without a separator, documents run together. The trainer warns at startup when shards look like they have none.

Limitations

Requires CLI v0.20.1 or later.
Trainer-only. No online evals; validation loss comes from [val].
Shards on a volume. data.paths must be on the run’s volume.
Dense architectures. prime-rl’s from-scratch initialization is tested on dense models.

Hosted SFT

Fine-tune a pretrained model on a Hugging Face dataset, with the same volume flow.

Volumes

Create volumes, upload data, and get checkpoints back.

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

Prerequisites

1. Tokenize your corpus

2. Put the shards on a volume

3. Write the config

4. Dispatch

5. Watch it run

Outputs

Troubleshooting

Limitations

Hosted SFT

Volumes

​Prerequisites

​1. Tokenize your corpus

​2. Put the shards on a volume

​3. Write the config

​4. Dispatch

​5. Watch it run

​Outputs

​Troubleshooting

​Limitations

Hosted SFT

Volumes

Prerequisites

1. Tokenize your corpus

2. Put the shards on a volume

3. Write the config

4. Dispatch

5. Watch it run

Outputs

Troubleshooting

Limitations