> ## Documentation Index
> Fetch the complete documentation index at: https://docs.primeintellect.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pause and resume runs

> Release a run's GPUs at a checkpoint and continue it later, on any cluster

Pausing a run stops it at its next training step, saves a checkpoint of that step to the run's [volume](/hosted-training/volumes), and releases its GPUs. Resuming starts a new run from that checkpoint on whichever cluster has capacity, and training continues from the step after it.

## Quickstart

Install and log in to [Prime CLI](/cli-reference/introduction) 0.16.0 or later. Pausing needs a run launched with `--volume`.

**1. Launch a run on a volume**

```bash theme={null}
prime train config.toml --volume research
```

**2. Pause it**

```bash theme={null}
prime train pause <run-id>
```

The run turns **PAUSING** while it finishes its current step, saves it, and syncs the volume.

**3. Wait for PAUSED**

```bash theme={null}
prime train get <run-id>
```

Once the run is **PAUSED**, its GPUs are released and the meter stops. This usually takes a few minutes.

**4. Resume it**

```bash theme={null}
prime train resume <run-id>
```

This starts a new run with its own ID, which reuses the paused run's config, volume, image, and GPU type and continues at the next step.

## Things to know

* Only a RUNNING run can be paused, and only if it uses prime-rl image `v0.10.1.dev11` or later. The default image qualifies. A run without a volume can only be [stopped](/hosted-training/hosted-rl).
* Pausing doesn't need a `[ckpt]` block. The pause saves its own checkpoint at `runs/<runId>/checkpoints/step_<n>/`.
* A paused run can't be unpaused or restarted in place. Resume it as a new run instead. The new run writes its outputs to its own `runs/<newRunId>/` directory on the same volume.
* Secrets aren't kept after a run starts, so pass them again when you resume: `prime train resume <run-id> -e WANDB_API_KEY`.
* To resume on a different GPU type or image, pass `--gpu-type` or `--image-tag` to `prime train resume`.
* To change the config, for example to raise `max_steps`, resume from your edited TOML with `prime train config.toml --resume-from <run-id>`. Keep the same model, since the checkpoint is loaded as is. If you also pass `--volume`, it must name the paused run's volume.
* The PAUSING phase is [billed](/hosted-training/billing) like a running run, because the run still holds its GPUs while it saves. A PAUSED run isn't billed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.