Skip to main content
Slurm is an open-source workload manager and job scheduler for high-performance computing clusters. A Slurm deployment runs on worker nodes assigned from your dedicated cluster and mounts shared storage for distributed jobs.

Deploy and connect

  1. Open Dedicated Clusters and select the team that owns your cluster.
  2. As a team admin on a cluster with self-service enabled, choose Deploy > Slurm cluster. Select the dedicated hardware, enter a cluster name, and set Worker nodes, GPUs per node, and Shared /home size. Turn on Enable container jobs (Pyxis/Enroot) if you plan to run container jobs.
  3. Add at least one user with an SSH public key. If your team uses Tailscale, choose a saved credential to join the login node to your tailnet.
  4. Under Slurm Clusters, select the deployment card when it reaches Running. Copy the SSH command from Details and replace username with your Slurm username. If you chose a saved Tailscale credential, the tailnet SSH command appears once the login node joins.
Details dialog for a running Slurm deployment with the SSH command
Team admins can manage SSH access in Users and choose Resize from the deployment’s actions menu. Enter the new total worker count. Growing requires eligible free nodes. Shrinking releases idle workers, so wait for jobs to finish if you need to free more nodes. The Usage tab shows worker allocation history and accounting metrics when available.

Manage Slurm with the Prime CLI

Use Prime CLI 0.21.0 or later and select your team with prime switch. List your team’s deployments.
CLI
Copy the deployment ID from the ID column and use it to inspect the deployment and its members.
CLI
get shows deployment details and SSH information when a login address is configured. Team admins can add or remove members with prime factory slurm add-member and prime factory slurm remove-member. To resize the deployment and wait for the reported worker capacity to update:
CLI
See Slurm cluster permissions for the required API key scopes and team roles.

Essential Slurm Commands

Run these commands from the Slurm login node to manage your jobs and inspect the fleet:

View Cluster Information

GPU Resource Allocation

Prime Intellect clusters use the Generic Resource (GRES) system for GPU allocation. Understanding the correct syntax is crucial for successful job submission.

Interactive GPU Sessions

These examples use --gres=gpu:N to request GPUs per node. Use one GPU request option per job. See the Slurm sbatch reference for supported options.

Batch Job Submission

Batch jobs allow you to queue work that runs without manual intervention. Create a script file (e.g., job.sh) with SBATCH directives:

Basic GPU Job Script

Submit and Manage Jobs

Job Output Location

Job output files are written to the directory where sbatch was executed, which may be on the compute node’s local filesystem if you’re not in shared storage. Always submit jobs from the shared storage directory to ensure output accessibility.

Example: Quick Cluster Test

Verify your Slurm cluster with these simple tests:

Troubleshooting Common Issues

InvalidAccount Error

If you encounter InvalidAccount errors when submitting batch jobs:
  1. Check the account: If your script sets #SBATCH --account=, check that your user can submit jobs under that account in the selected partition.
  2. Check partition availability: Ensure the partition you’re requesting exists with sinfo.

Job Output Not Found

If you can’t find your job output files:
  • Check working directory: Output files are created where sbatch was run
  • Use absolute paths: Specify full paths in --output and --error directives
  • Check other nodes: If submitted from a compute node, outputs may be on that node’s local storage
  • Always submit from shared storage: Change to the shared storage directory before running sbatch

Node Communication Issues

If nodes can’t communicate or jobs hang:
  • Verify all nodes are in idle state with sinfo
  • Check node connectivity: srun --nodelist=<node> hostname
  • Ensure shared storage is mounted on all nodes
  • Consider restarting your cluster if issues persist

Advanced Slurm Commands

Direct Node Access

Access specific compute nodes directly for debugging or monitoring:

Resource Monitoring

Job Arrays for Parameter Sweeps

Run multiple similar jobs with different parameters:

Environment Variables

Useful Slurm environment variables available in jobs:

Shared Storage Integration

Your Slurm cluster comes with shared storage automatically mounted on all nodes. The UI displays the mount path for your shared storage directory. This ensures:
  • Consistent file access across all compute nodes
  • No need to manually copy data between nodes
  • Simplified job submission and management
  • Persistent storage for checkpoints and results

Best Practices

Use the Login Node

Submit jobs and run Slurm commands from the login node

Leverage Shared Storage

Store your code, data, and outputs in the shared storage directory shown in the UI for seamless access across nodes

Monitor Resources

Regularly check cluster utilization with sinfo and squeue to optimize job scheduling

Use Job Arrays

For parameter sweeps or similar tasks, use Slurm job arrays for efficient scheduling