> ## Documentation Index
> Fetch the complete documentation index at: https://docs.primeintellect.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hosted Pretraining (Beta)

> Pretrain a model from scratch on Hosted Training: tokenize with datatrove, put the shards on a volume, and dispatch with prime train

> [!NOTE]
>
> Hosted pretraining is in **beta**. Contact Prime support if dispatch is refused.

Run `prime-rl` pretraining on Prime's hosted GPU clusters. The trainer starts from randomly initialized weights and learns next-token prediction on text you tokenize ahead of time. You tokenize your corpus into shards, put them on a volume, and dispatch; the trainer reads the shards from the volume.

This walkthrough pretrains a Qwen3-0.6B-sized model on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) on one 8-GPU node. For pretraining on your own infrastructure, see prime-rl's [Pretraining](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/training.md#pretraining) guide.

## Prerequisites

```bash theme={null}
uv tool install prime
prime login
```

**Requires `prime` CLI v0.20.1 or later.** Older versions don't read `mode` and send a pretraining config as SFT, and it fails validation.

* A Prime account with hosted training access.
* A model in the cluster model cache (`prime train models`) whose architecture and tokenizer you want. Pretraining reads only the model's config and tokenizer, never its weights, so any cached checkpoint of the right architecture works.
* [datatrove](https://github.com/huggingface/datatrove) to tokenize your corpus: `pip install "datatrove[io,processing]"`.

## 1. Tokenize your corpus

prime-rl does not tokenize. It reads the shards datatrove's `DocumentTokenizer` writes: `*.ds` token files, each with a `.ds.index` and a `.ds.metadata`. Tokenize with the same tokenizer as the model you will train, and set `eos_token` to the tokenizer's end-of-text token so documents stay separated. For Qwen that is `&lt;|endoftext|&gt;`, not `<|im_end|>`.

```python theme={null}
from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import HuggingFaceDatasetReader
from datatrove.pipeline.tokens import DocumentTokenizer

for split, folder in (("train", "train"), ("validation", "val")):
    LocalPipelineExecutor(
        pipeline=[
            HuggingFaceDatasetReader("roneneldan/TinyStories", {"split": split}),
            DocumentTokenizer(
                output_folder=f"tinystories/{folder}",
                tokenizer_name_or_path="Qwen/Qwen3-0.6B",
                eos_token="&lt;|endoftext|&gt;",
            ),
        ],
        tasks=4,
        logging_dir=f"logs/{folder}",
    ).run()
```

This writes `tinystories/train/` and `tinystories/val/`. Keep validation documents in their own folder.

## 2. Put the shards on a volume

```bash theme={null}
prime volumes create research --size 100Gi
prime volumes put research ./tinystories datasets/
```

The second command creates `datasets/tinystories/train` and `datasets/tinystories/val` on the volume (a trailing `/` means "into this folder"). Size the volume for the shards plus the checkpoints the run keeps: a checkpoint is about 11 to 12 bytes per parameter, so about 7 GB for a 0.6B model. See [Choose a size](https://docs.primeintellect.ai/hosted-training/volumes.md#choose-a-size).

A running run holds its volume's writer lock: while a run is active on a volume, dispatching a second run against it fails with an "in use by a run" error. Give concurrent runs their own volume (the same dataset can be uploaded to each).

For a corpus too large to upload from your machine, open a shell on the volume (`prime volumes ssh research --read-write`) and tokenize there, or [import it from S3](https://docs.primeintellect.ai/hosted-training/volumes.md).

## 3. Write the config

The top-level `mode = "pretrain"` makes this a pretraining run. Save this as `pretrain.toml`:

```toml theme={null}
mode = "pretrain"
max_steps = 300

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"  # a cached Qwen3-0.6B; only its config and tokenizer are read

[data]
paths = ["datasets/tinystories/train"]  # folders on the volume
seq_len = 1024
batch_size = 64
micro_batch_size = 2

[val]
interval = 50
num_samples = 64

[val.data]
paths = ["datasets/tinystories/val"]
seq_len = 1024  # not inherited from [data]

[ckpt]
interval = 50

[deployment]
type = "single_node"
num_train_gpus = 8
```

* `data.paths` lists folders on the volume, relative to its root or as absolute `/volume/...` paths. Every shard in them forms one token pool, read in `seq_len` windows in a new random order each epoch.
* The weights start from the architecture's own initialization. `optim.lr` defaults to `1e-3`, with a 10-step linear warmup.
* `output_dir` is set by the platform; leave it out.

See prime-rl's [Pretraining](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/training.md#pretraining) guide for every `[data]` and `[val]` option.

## 4. Dispatch

```bash theme={null}
prime train pretrain.toml --volume research
```

```text theme={null}
Dispatched hosted run pgdrlpmhh7l7pm62299ppa47

Monitor run at:
  https://app.primeintellect.ai/dashboard/training/pgdrlpmhh7l7pm62299ppa47
```

The CLI reads `mode` and validates the config against prime-rl's pretraining schema before submitting it.

## 5. Watch it run

```bash theme={null}
prime train get <runId>
prime train metrics <runId>
prime train logs <runId>
```

The dashboard shows the run as **Pretrain**, with its training loss, perplexity and validation loss. The walkthrough takes about ten minutes on eight GPUs, and its validation loss falls from about 3.8 at step 150 to about 3.1 at step 250.

## Outputs

The run keeps its run directory on the volume at `runs/<runId>/`, with `checkpoints/step_<N>/`, `logs/` and `monitors/`. Everything on the volume syncs to cloud storage when the run ends.

```bash theme={null}
prime volumes get research runs/<runId>/checkpoints/step_300/ ./step_300
```

## Troubleshooting

> [accordion] data.paths is not on the volume
>
> Pretraining reads shards only from the run's volume. Pass `--volume` and list folders on it, relative to the volume root (`datasets/<name>/train`) or as `/volume/...` paths.
>
> ```text theme={null}
> Error: data.paths: Hosted pretrain reads tokenized shards from the named volume:
> pass --volume and set data.paths to folders on it (e.g. data.paths =
> ["datasets/<name>/train"]).
> ```

> [accordion] Model not available on this cluster
>
> The model's config and tokenizer come from the cluster model cache. Pick a cached model of the architecture you want (`prime train models`), or ask Prime support to cache one.

> [accordion] Online evals rejected
>
> Pretraining runs are trainer-only: `[eval]`, `[inference]` and `[weight_broadcast]` are rejected. Use `[val]` to track held-out loss instead.

> [accordion] Loss does not fall, or the model never stops generating
>
> Check that the shards were tokenized with the model's tokenizer and with `eos_token` set to its end-of-text token. Without a separator, documents run together. The trainer warns at startup when shards look like they have none.

## Limitations

* **Requires CLI v0.20.1 or later.**
* **Trainer-only.** No online evals; validation loss comes from `[val]`.
* **Shards on a volume.** `data.paths` must be on the run's volume.
* **Dense architectures.** prime-rl's from-scratch initialization is tested on dense models.

> [card] [Hosted SFT](https://docs.primeintellect.ai/hosted-training/hosted-sft.md)
>
> Fine-tune a pretrained model on a Hugging Face dataset, with the same volume flow.

> [card] [Volumes](https://docs.primeintellect.ai/hosted-training/volumes.md)
>
> Create volumes, upload data, and get checkpoints back.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.