Volume capacity and checkpoint sizing

A volume has a fixed size. Runs that write past it fail. Full fine-tuning checkpoints are large, so size the volume before you launch a run.

What fills a volume

Content	Where	When it is removed
Trainer checkpoints	`runs/<runId>/checkpoints/step_N/`	When the run’s retention settings delete them (see below)
Data you uploaded	for example `datasets/`, `models/`	Only when you delete it
Outputs of earlier runs	`runs/<oldRunId>/`	Only when you delete them; they stay after a run ends
RL rollout traces	under the run’s directory	Never during the run; they grow until the run ends

Every run that uses the volume shares the same space.

How many checkpoints are on disk

These [ckpt] settings control how many checkpoints a run keeps. RL configs use the same fields in the shared [ckpt] block.

Setting	Effect
`interval`	Save a checkpoint every N steps. When unset, the run saves one checkpoint at the end of training.
`keep_last`	Keep only the N most recent checkpoints. When unset, every checkpoint is kept.
`keep_interval`	Also keep every checkpoint whose step is a multiple of N, permanently.

The trainer writes a new checkpoint before it deletes the oldest one. So at peak the run holds keep_last + 1 checkpoints, plus any pinned by keep_interval.

How big one checkpoint is

A trainer checkpoint holds the full training state: model weights in fp32 and the optimizer state. Plan for about 11–12 bytes per parameter. Measured saves: ~341 GB for Qwen3-30B-A3B (30.5B parameters) and ~7.2 GB for a 0.6B model.

Model size	One checkpoint (about 11–12 bytes/param)
0.6B (measured)	~7.2 GB
8B	~90–100 GB
30B (measured: Qwen3-30B-A3B, 30.5B params)	~341 GB
100B	~1.1–1.2 TB

The bf16 weights alone are 2 bytes per parameter (~57 GiB for Qwen3-30B-A3B). That is a lower bound and the size of exported weights. It is not the size of a trainer checkpoint.

ckpt.skip_optimizer = true does not make checkpoints smaller. It only skips loading the optimizer state when a run resumes. The checkpoint is still saved with the full state.

Size the volume

required ≈ (keep_last + 1) × 11–12 bytes × parameters
         + uploaded datasets and models
         + outputs of other runs on the volume
         + about 20% headroom

If keep_last is unset, multiply by the number of checkpoints the run will save instead. Example. Qwen3-30B-A3B (30.5B parameters) with keep_last = 2:

One checkpoint is about 341 GB. Peak is 3 checkpoints: about 1 TB.
With the formula (11–12 bytes per parameter, 20% headroom): about 1.2 TB, plus your datasets. Use a 1.5Ti volume.
On a 200Gi volume (about 215 GB), even the first checkpoint does not fit. The run fails at its first save.

For RL runs, use the same checkpoint math, then add room for rollout traces. Traces are written in chunks of about 5 GiB. That is the file size per chunk, not a limit: traces keep growing for as long as the run trains. Long RL runs need extra headroom, and old runs’ traces should be cleaned up.

The size is a hard quota

The size you set with --size or --volume-size is a hard limit. A write past it fails with one of these errors:

OSError: [Errno 122] Disk quota exceeded
OSError: [Errno 28] No space left on device

Check what uses the space (Prime CLI 0.13.0 or later):

prime volumes du training-outputs -h          # whole volume, and usage against its size
prime volumes du training-outputs /runs -h    # one directory

The volume’s stored contents update when a run or read-write session ends, so du does not show what a run that is still going has written. A volume that already holds more than its size cannot start a run: the run fails before training with volume contents (...) exceed the volume size (...); run prime volumes expand. Expand the volume or delete data, then launch again.

What happens when a volume fills

The checkpoint save fails and the trainer crashes.
The trainer restarts from the last good checkpoint (or from the start, if there is none), trains to the next save, and hits the same limit again. Each retry spends GPU time.
After repeated crashes, the run is marked failed with a disk-quota error.

The run does not recover on its own. Free space or use a larger volume, then launch again.

Fix or prevent a full volume

Keep fewer checkpoints. Set keep_last = 1 or 2 in [ckpt].
Save less often. Raise [ckpt].interval, or leave it unset to save only at the end.
Delete old outputs. Open a read-write session and remove runs you no longer need:
prime volumes ssh training-outputs --read-write rm -rf /volume/runs/<oldRunId>
Do not delete a checkpoint that another run resumes from with [resume].dir.
Grow the volume. Expand it with prime volumes expand <name> --size <size> (see Make a volume bigger).

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

Volume capacity and checkpoint sizing

What fills a volume

How many checkpoints are on disk

How big one checkpoint is

Size the volume

The size is a hard quota

What happens when a volume fills

Fix or prevent a full volume

​What fills a volume

​How many checkpoints are on disk

​How big one checkpoint is

​Size the volume

​The size is a hard quota

​What happens when a volume fills

​Fix or prevent a full volume

What fills a volume

How many checkpoints are on disk

How big one checkpoint is

Size the volume

The size is a hard quota

What happens when a volume fills

Fix or prevent a full volume