> ## Documentation Index > Fetch the complete documentation index at: https://docs.primeintellect.ai/llms.txt > Use this file to discover all available pages before exploring further. # Volume capacity and checkpoint sizing > Size a volume for your checkpoints, and fix runs that fail with Disk quota exceeded A [volume](/hosted-training/volumes) has a fixed size. Runs that write past it fail. Full fine-tuning checkpoints are large, so size the volume before you launch a run. ## What fills a volume | Content | Where | When it is removed | | - | - | - | | Trainer checkpoints | `runs//checkpoints/step_N/` | When the run's retention settings delete them (see below) | | Data you uploaded | for example `datasets/`, `models/` | Only when you delete it | | Outputs of earlier runs | `runs//` | Only when you delete them; they stay after a run ends | | RL rollout traces | under the run's directory | Never during the run; they grow until the run ends | Every run that uses the volume shares the same space. ## How many checkpoints are on disk These `[ckpt]` settings control how many checkpoints a run keeps. RL configs use the same fields in the shared `[ckpt]` block. | Setting | Effect | | - | - | | `interval` | Save a checkpoint every N steps. When unset, the run saves one checkpoint at the end of training. | | `keep_last` | Keep only the N most recent checkpoints. When unset, **every checkpoint is kept**. | | `keep_interval` | Also keep every checkpoint whose step is a multiple of N, permanently. | The trainer writes a new checkpoint **before** it deletes the oldest one. So at peak the run holds `keep_last + 1` checkpoints, plus any pinned by `keep_interval`. ## How big one checkpoint is A trainer checkpoint holds the full training state: model weights in fp32 and the optimizer state. Plan for about **11–12 bytes per parameter**. Measured saves: \~341 GB for Qwen3-30B-A3B (30.5B parameters) and \~7.2 GB for a 0.6B model. | Model size | One checkpoint (about 11–12 bytes/param) | | - | - | | 0.6B (measured) | \~7.2 GB | | 8B | \~90–100 GB | | 30B (measured: Qwen3-30B-A3B, 30.5B params) | \~341 GB | | 100B | \~1.1–1.2 TB | The bf16 weights alone are 2 bytes per parameter (\~57 GiB for Qwen3-30B-A3B). That is a lower bound and the size of exported weights. It is not the size of a trainer checkpoint. `ckpt.skip_optimizer = true` does not make checkpoints smaller. It only skips **loading** the optimizer state when a run resumes. The checkpoint is still saved with the full state. ## Size the volume ```text theme={null} required ≈ (keep_last + 1) × 11–12 bytes × parameters + uploaded datasets and models + outputs of other runs on the volume + about 20% headroom ``` If `keep_last` is unset, multiply by the number of checkpoints the run will save instead. **Example.** Qwen3-30B-A3B (30.5B parameters) with `keep_last = 2`: * One checkpoint is about 341 GB. Peak is 3 checkpoints: about 1 TB. * With the formula (11–12 bytes per parameter, 20% headroom): about 1.2 TB, plus your datasets. Use a `1.5Ti` volume. * On a `200Gi` volume (about 215 GB), even the **first** checkpoint does not fit. The run fails at its first save. For RL runs, use the same checkpoint math, then add room for rollout traces. Traces are written in chunks of about 5 GiB. That is the file size per chunk, not a limit: traces keep growing for as long as the run trains. Long RL runs need extra headroom, and old runs' traces should be cleaned up. ## The size is a hard quota The size you set with `--size` or `--volume-size` is a hard limit. A write past it fails with one of these errors: ```text theme={null} OSError: [Errno 122] Disk quota exceeded OSError: [Errno 28] No space left on device ``` Check what uses the space (Prime CLI 0.13.0 or later): ```bash theme={null} prime volumes du training-outputs -h # whole volume, and usage against its size prime volumes du training-outputs /runs -h # one directory ``` The volume's stored contents update when a run or read-write session ends, so `du` does not show what a run that is still going has written. A volume that already holds more than its size cannot start a run: the run fails before training with `volume contents (...) exceed the volume size (...); run prime volumes expand`. Expand the volume or delete data, then launch again. ## What happens when a volume fills 1. The checkpoint save fails and the trainer crashes. 2. The trainer restarts from the last good checkpoint (or from the start, if there is none), trains to the next save, and hits the same limit again. Each retry spends GPU time. 3. After repeated crashes, the run is marked failed with a disk-quota error. The run does not recover on its own. Free space or use a larger volume, then launch again. ## Fix or prevent a full volume * **Keep fewer checkpoints.** Set `keep_last = 1` or `2` in `[ckpt]`. * **Save less often.** Raise `[ckpt].interval`, or leave it unset to save only at the end. * **Delete old outputs.** Open a read-write session and remove runs you no longer need: ```bash theme={null} prime volumes ssh training-outputs --read-write rm -rf /volume/runs/ ``` Do not delete a checkpoint that another run resumes from with `[resume].dir`. * **Grow the volume.** Expand it with `prime volumes expand --size ` (see [Make a volume bigger](/hosted-training/volumes#make-a-volume-bigger)). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.