Volumes (Beta)

Volumes are in beta for Hosted RL and Hosted SFT.

A volume is named cloud storage for your training data and results. Upload datasets and models to it, use it in training runs, and download the checkpoints each run saves there. A volume belongs to your team (or to you, with no team selected) and isn’t tied to any cluster.

Legacy staging is being retired; volumes work the same on every cluster.

What you can do with and without a cluster allocation

Task	Without a cluster allocation or team-owned cluster	With one
Create, list, grow, and delete volumes	Yes	Yes
Upload and download files (`put` / `get`)	Yes	Yes
Open a shell on a volume (`ssh`)	No	Yes
Use a volume in a training run (`--volume`)	Yes, on on-demand capacity (beta) for GPU types that support it	Yes

On-demand training is in beta and needs beta access for your account or team. On demand, a volume works only on GPU types whose clusters support shared storage. On any other GPU type, the run is refused with a message that names the type. Pick another GPU type with --gpu-type, or run without --volume. Shells need a cluster allocation for your account or team, or a cluster your team owns. On-demand capacity alone doesn’t include SSH. Personal accounts can’t own clusters, so they need an allocation. Ask Prime support if you need one.

Quickstart

Install and log in to Prime CLI 0.11.0 or later. Volumes use the team set in your CLI (prime config set-team-id <team-id>). With no team set, they belong to your personal account.

Create a volume

prime volumes create research --size 500Gi

Upload a dataset

prime volumes put research ./my-data datasets/

This creates datasets/my-data on the volume. Training runs see it at /volume/datasets/my-data.

Optional: open a shell on the volume

Needs a cluster allocation or a cluster your team owns and an SSH key.

prime volumes ssh research

After editing, exit the shell, run prime volumes stop research <session-id>, and wait for the final sync. Exiting alone leaves the session open and blocks training on the volume.

Train with the volume

Add a checkpoint configuration to save weights, then:

prime train config.toml --volume research

Download your checkpoints

Wait for the run to end and its volume sync to finish, then:

prime volumes get research runs/<runId>/checkpoints/ ./checkpoints

Delete the volume when you're done

prime volumes delete research

Things to know

Each run writes checkpoints under runs/<runId>/checkpoints/. Previous runs are visible in the next run or shell at /volume/runs/. Direct put uploads to runs/ are refused.
A volume has one writer at a time: a training run, a read-write shell, or an upload. Wait for that writer to finish syncing before starting another. Read-only shells can coexist with a writer, but see the last saved copy.
Changes made by a run or shell reach cloud storage when it ends. There is no periodic sync. Downloads during a run can return the previous saved contents.
Each run gets a disk the size of your volume. It must fit the whole volume, including earlier runs, plus new checkpoints. See Choose a size.

Create a volume

Names use 1 to 40 lowercase letters, digits, and hyphens, and start and end with a letter or digit. The default size is 5Ti, and the maximum is 20Ti.

prime volumes create research
prime volumes create experiments --size 500Gi

The volume is ready as soon as the command returns. If you have a cluster allocation (or your team owns a cluster) and an SSH key, create also starts a read-only shell session in the background. That session stops after 30 minutes of no use. Pass --no-warm to skip it. List your volumes:

prime volumes list

┏━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Name     ┃ Size  ┃ Status  ┃ Created                          ┃ Created by ┃
┡━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ research │ 500Gi │ CREATED │ 2026-10-12T16:05:11.482000+00:00 │ Jane Doe   │
└──────────┴───────┴─────────┴──────────────────────────────────┴────────────┘

Add -o json for JSON output.

Upload data

put uploads a file or folder. Paths on the volume start at its root:

prime volumes put research ./my-data datasets/        # creates datasets/my-data
prime volumes put research ./my-data/ datasets/my-data # same, copying the folder's contents
prime volumes put research ./model.safetensors models/my-model/

A trailing / on the volume path means “into this folder”.
A trailing / on a local folder copies what’s inside it, not the folder itself.
Volume paths can use letters, digits, and ._-@%+=,:. For other names, use a shell.
Symlinks inside a folder are skipped.
You can’t upload to runs/.

Uploads go straight to cloud storage, so they don’t need a cluster allocation or an SSH key. While an upload runs, nobody can start another writer or delete the volume. A training run or someone else’s read-write shell blocks uploads until it finishes syncing. If you have your own read-write shell open, put can upload through it instead; those changes reach cloud storage when you end the session.

Download data

prime volumes get research datasets/my-data/ ./my-data
prime volumes get research / ./everything

The local destination defaults to the current folder. Direct downloads read the saved copy in cloud storage. Wait for a run or read-write shell to finish syncing to retrieve its latest changes. get can read ordinary files through your own read-write shell, but downloads under runs/ always use cloud storage.

Open a shell on your volume

A shell is handy for downloading data on the cluster’s fast network (for example, with hf download), or for looking around and editing files. It needs a cluster allocation or a cluster your team owns and an SSH key.

prime volumes ssh research              # read-write (the default)
prime volumes ssh research --read-only

When the session starts, the whole volume is copied to the machine, including runs/, and you find it at /volume. A large volume takes a while to copy. The session becomes ready after the download completes.

Read-write: changes are saved back when the session ends. Files you delete in the shell are deleted from cloud storage too. A training run or another read-write shell holds the writer lock until its final sync completes.
Read-only: nothing is saved back. These sessions can coexist with a writer, subject to the session limit. They read the saved contents from when the session started.

Exiting the shell doesn’t end the session. It ends after 30 minutes with no connection, or after 24 hours. To end it now:

prime volumes sessions research
prime volumes stop research <session-id>

Only you can connect to or stop your sessions, even on a team volume. Wait for the final sync after stopping a read-write session before downloading its changes or launching a training run on the same volume.

Set up your SSH key

Shells accept one key: your account’s primary SSH key. The CLI must point at the matching private key.

Get a key pair

Use an existing pair, or create one:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519

Add the public key to your account

Open Dashboard → Tokens → SSH Keys and paste the contents of ~/.ssh/id_ed25519.pub. A new key becomes your primary key. To switch later, click Set Primary on another key.

Point the CLI at the private key

prime config set-ssh-key-path ~/.ssh/id_ed25519

The default is ~/.ssh/id_rsa. prime volumes ssh prints Using SSH key: <path> before it connects.

If you never added a key, you can use the key pair under System generated keys on the same page: download the private key, run chmod 600 on it, and point prime config set-ssh-key-path at it. Once you add your own key, the system-generated key stops working for shells.

Use a volume in a training run

Pass --volume to a Hosted RL or Hosted SFT run:

prime train config.toml --volume research

Or set volume = "research" at the top of the TOML, before any tables. If the volume doesn’t exist, the CLI creates it. Use --volume-size to choose its size. When the run starts, the whole volume is downloaded and mounted read-write at /volume. For example, a Hosted SFT data.name = "datasets/my-data" reads /volume/datasets/my-data. Earlier outputs are available under /volume/runs/. The run’s changes are saved to cloud storage when it ends, after the training pods stop. The final sync runs when the run completes, stops, fails, or pauses, so checkpoints survive prime train stop and crashes. End any read-write shell and wait for its final sync before launching the run. A training run holds the volume’s writer lock until its own sync completes.

Train from weights already on the volume

Both Hosted SFT and RL runs can load Hugging Face-format model weights from an absolute /volume/... path. Set [model].name in a Hosted SFT config, or set the shared [model].name in an RL config with [trainer] and [orchestrator] blocks:

[model]
name = "/volume/models/my-model"

Attach the volume when launching:

prime train config.toml --volume research

The directory must contain the model configuration, weights, and tokenizer assets. It does not need an entry in the cluster model cache. A sharded training checkpoint is a different format; export it to Hugging Face format before using it as a model directory. See Volume-hosted models for config examples and path rules.

Checkpoint settings

Checkpointing is off unless you add [ckpt]. An empty [ckpt] block saves the final checkpoint; set interval to save during training too:

max_steps = 100

[ckpt]
interval = 50       # save a checkpoint every 50 steps
keep_last = 2       # keep the 2 newest checkpoints

Setting	What it does
`interval`	How often the run saves a checkpoint. The final checkpoint is also saved.
`keep_last`	How many recent checkpoints to keep. Pruned checkpoints are not recovered by the final volume sync.
`keep_interval`	Preserve checkpoints at multiples of this step in addition to the recent ones. It does not control upload timing.

If you set only keep_interval, the final checkpoint can be pruned when its step is not a multiple of that interval. Set keep_last to at least 1 to keep the final checkpoint too.

Neither keep_interval nor keep_last is required. All files on /volume are synced when the run ends. RL runs write checkpoints, logs, and other run outputs there. Hosted runs currently do not save the resolved config needed for checkpoint conversion. Hosted SFT writes checkpoints there; its other outputs remain on the run’s scratch disk.

Get your checkpoints back

Each run saves its checkpoints to runs/<runId>/checkpoints/step_<N>/:

prime volumes get research runs/<runId>/checkpoints/ ./checkpoints
prime volumes get research runs/<runId>/checkpoints/step_100/ ./step_100

Wait for the final volume sync before downloading a new run’s outputs. Complete trainer checkpoints contain trainer/.metadata. To load checkpoints outside Prime, convert them first: see Export to HF format.

Choose a size

Each run gets a disk the size of your volume. It must fit:

all existing files, including datasets, models, and earlier runs/ outputs, plus
keep_last + 1 checkpoints (the run writes a new one before it deletes the oldest), plus
about 20% extra.

The keep_last + 1 estimate assumes keep_last is set and keep_interval is unset. If you also preserve checkpoints with keep_interval, include those extra checkpoints. Without retention limits, size for every checkpoint the run saves. A full fine-tuning checkpoint is about 11–12 bytes per parameter (measured: ~341 GB for a 30B save, ~7.2 GB for a 0.6B save):

Model size	One checkpoint
0.6B	~7.2 GB
8B	~100 GB
30B (measured: Qwen3-30B-A3B)	~341 GB
100B	~1.2 TB

For example, Qwen3-30B-A3B with keep_last = 2 holds up to 3 checkpoints, about 1 TB. With 20% extra and your datasets, use 1.5Ti. On 200Gi, even the first checkpoint doesn’t fit. ckpt.skip_optimizer = true doesn’t make checkpoints smaller. For the full sizing formula, the peak checkpoint count, and how to fix a run that fails with Disk quota exceeded, see Volume capacity.

Deleting saved files or checkpoints inside a read-write shell removes them from cloud storage when the session ends. Check what you need to keep before removing earlier runs to free space.

Check disk usage

prime volumes du research

prime volumes du (CLI 0.13.0 or later) shows how much of the volume’s saved contents is in use. It reads cloud storage, so a live run’s or session’s changes appear only after it ends and syncs.

Make a volume bigger

prime volumes expand research --size 2Ti

Volumes can grow up to 20Ti but can’t shrink. Runs and shells already going keep their old size. New ones get the new size.

Delete a volume

Deleting a volume deletes everything on it, including every run’s checkpoints.

prime volumes delete research

Add --yes to skip the confirmation. Only team admins can delete team volumes. You can’t delete a volume while a run uses it, a shell is open on it, or an upload is running. Stop the run (prime train stop <runId>) or the shell (prime volumes stop), then try again. If the status shows DELETE FAILED, run prime volumes delete again.

Troubleshooting

Error	What it means	What to do
`SSH sessions need a cluster allocation or a team-owned cluster with shared storage, and this team has neither (on-demand capacity doesn't include SSH).` (for a personal account: `...a cluster allocation with shared storage, and this account has none...`)	Shells need a cluster allocation or a cluster your team owns. On-demand capacity alone doesn’t include SSH.	Use `put` and `get` instead, or ask Prime support for a cluster allocation. Until the newer wording ships, you may see the older message `no cluster assigned to you has ReadWriteMany storage, which an SSH session needs to stage the volume; ask Prime support for one`, which means the same thing.
`On-demand training is in beta. Contact Prime support to request beta access for this account.`	Your account or team has no cluster allocation, so the run would use on-demand capacity, which is in beta.	Ask Prime support for beta access, or for a cluster allocation.
`No on-demand gpu_type=<type> cluster has ReadWriteMany storage, which a --volume run needs. Try another GPU type, or run without --volume.`	On demand, that GPU type doesn’t support volumes.	Choose another GPU type with `--gpu-type`, or run without `--volume`.
`volume '<name>' has an active read-write session (...); use --read-only, or end that session first`	Someone else has a read-write shell open.	Use `--read-only`, or ask them to end their session.
`volume '<name>' has a live read-write SSH session (...); put through that session or end it first`	You tried to upload while someone else has a read-write shell open.	Upload from inside that shell, or wait for it to end.
`an upload to volume '<name>' is in progress until <time>Z; retry after it finishes`	An upload is running.	Wait for it to finish. A stopped upload clears within 15 minutes.
`volume '<name>' is mounted by N live run(s)` or `has N active volume SSH session(s)`	You tried to delete a volume that’s in use.	Stop the runs or shells, then delete again.
`runs/ holds run outputs and is read-only; put elsewhere.`	You tried to upload to `runs/`.	Upload to another folder, such as `datasets/`.
`volume '<name>' is in use by a run (...) that writes to it` or `is attached read-write by a run (...)`	A training run holds the writer lock.	Wait for the run and its final volume sync to finish.
`volume contents (...) exceed the volume size (...); run prime volumes expand`	The saved contents do not fit in the requested disk.	Expand the volume before starting another run or shell.
`No space left on device` (`Errno 28`)	The volume is too small for your files plus checkpoints.	Stop the run, `prime volumes expand`, and start again. Or lower `keep_last`.
`volumes can only grow`	The new size isn’t bigger than the current one.	Choose a larger size, up to `20Ti`.
`Volume SSH session limit (20) reached`	Your team has 20 shells open.	End some with `prime volumes stop`.
`Permission denied (publickey)`	Your CLI key doesn’t match your primary key.	Repeat Set up your SSH key.
`no SSH public key is configured for your account`	Your account has no SSH key.	Add one in the dashboard.

Getting Started

Inference

Model Factory

Community

Compute

Traces

Libraries

What you can do with and without a cluster allocation

Quickstart

Things to know

Create a volume

Upload data

Download data

Open a shell on your volume

Set up your SSH key

Use a volume in a training run

Train from weights already on the volume

Checkpoint settings

Get your checkpoints back

Choose a size

Check disk usage

Make a volume bigger

Delete a volume

Troubleshooting

​What you can do with and without a cluster allocation

​Quickstart

​Things to know

​Create a volume

​Upload data

​Download data

​Open a shell on your volume

​Set up your SSH key

​Use a volume in a training run

​Train from weights already on the volume

​Checkpoint settings

​Get your checkpoints back

​Choose a size

​Check disk usage

​Make a volume bigger

​Delete a volume

​Troubleshooting

What you can do with and without a cluster allocation

Quickstart

Things to know

Create a volume

Upload data

Download data

Open a shell on your volume

Set up your SSH key

Use a volume in a training run

Train from weights already on the volume

Checkpoint settings

Get your checkpoints back

Choose a size

Check disk usage

Make a volume bigger

Delete a volume

Troubleshooting