Skip to main content
Pausing a run stops it at its next training step, saves a checkpoint of that step to the run’s volume, and releases its GPUs. Resuming starts a new run from that checkpoint on whichever cluster has capacity, and training continues from the step after it.

​
Quickstart

Install and log in to Prime CLI 0.16.0 or later. Pausing needs a run launched with --volume.
1

Launch a run on a volume

2

Pause it

The run turns PAUSING while it finishes its current step, saves it, and syncs the volume.
3

Wait for PAUSED

Once the run is PAUSED, its GPUs are released and the meter stops. This usually takes a few minutes.
4

Resume it

This starts a new run with its own ID, which reuses the paused run’s config, volume, image, and GPU type and continues at the next step.

​
Things to know

  • Only a RUNNING run can be paused, and only if it uses prime-rl image v0.10.1.dev11 or later. The default image qualifies. A run without a volume can only be stopped.
  • Pausing doesn’t need a [ckpt] block. The pause saves its own checkpoint at runs/<runId>/checkpoints/step_<n>/.
  • A paused run can’t be unpaused or restarted in place. Resume it as a new run instead. The new run writes its outputs to its own runs/<newRunId>/ directory on the same volume.
  • Secrets aren’t kept after a run starts, so pass them again when you resume: prime train resume <run-id> -e WANDB_API_KEY.
  • To resume on a different GPU type or image, pass --gpu-type or --image-tag to prime train resume.
  • To change the config, for example to raise max_steps, resume from your edited TOML with prime train config.toml --resume-from <run-id>. Keep the same model, since the checkpoint is loaded as is. If you also pass --volume, it must name the paused run’s volume.
  • The PAUSING phase is billed like a running run, because the run still holds its GPUs while it saves. A PAUSED run isn’t billed.