> ## Documentation Index > Fetch the complete documentation index at: https://docs.primeintellect.ai/llms.txt > Use this file to discover all available pages before exploring further. # Slurm Orchestration > Submit and manage distributed jobs on a dedicated cluster with Slurm Slurm is an open-source workload manager and job scheduler for high-performance computing clusters. Dedicated clusters arrive with Slurm configured and shared storage mounted, so you can submit distributed jobs without setting any of it up. # Essential Slurm Commands Run these commands on the cluster's controller node to manage your jobs and inspect the fleet: ## View Cluster Information ```bash theme={null} # Display information about nodes and partitions sinfo # Show detailed node information sinfo -Nel # Display partition summary sinfo -s # Example output: # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST # gpu* up infinite 2 idle node[001-002] ``` ## GPU Resource Allocation Prime Intellect clusters use the Generic Resource (GRES) system for GPU allocation. Understanding the correct syntax is crucial for successful job submission. ### Interactive GPU Sessions ```bash theme={null} # Request an interactive session with GPUs (use --gres for batch scripts) srun --gpus=1 --pty bash # Request multiple GPUs across nodes srun --nodes=2 --gpus-per-node=8 nvidia-smi # Request specific node with GPUs srun --gpus=4 --nodelist=node001 --pty bash # Alternative GRES syntax (required for batch scripts) srun --nodes=2 --gres=gpu:8 --ntasks-per-node=1 nvidia-smi # Once in the session, verify GPU access nvidia-smi # Check available GPUs in your session nvidia-smi -L ``` When writing batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N` to avoid InvalidAccount errors. ## Batch Job Submission Batch jobs allow you to queue work that runs without manual intervention. Create a script file (e.g., `job.sh`) with SBATCH directives: ### Basic GPU Job Script ```bash theme={null} #!/bin/bash #SBATCH --job-name=gpu-test #SBATCH --nodes=2 #SBATCH --gres=gpu:8 # IMPORTANT: Use gres for GPUs in batch scripts #SBATCH --ntasks-per-node=1 #SBATCH --time=01:00:00 #SBATCH --output=%x-%j.out # %x=job-name, %j=job-id #SBATCH --error=%x-%j.err echo "Job started on $(date)" echo "Running on nodes: $SLURM_JOB_NODELIST" # Run nvidia-smi on all allocated nodes srun -l nvidia-smi # Your training or compute commands here # srun python train.py ``` ### Submit and Manage Jobs ```bash theme={null} # Submit a batch job sbatch job.sh # View queued and running jobs squeue # View your jobs only squeue -u $USER # Cancel a job scancel # Cancel all your jobs scancel -u $USER # View detailed job information scontrol show job ``` ### Job Output Location Job output files are written to the directory where `sbatch` was executed, which may be on the compute node's local filesystem if you're not in shared storage. Always submit jobs from the shared storage directory to ensure output accessibility. ## Example: Quick Cluster Test Verify your Slurm cluster with these simple tests: ```bash theme={null} # Test 1: Check all GPUs across nodes (interactive) srun --nodes=2 --gpus-per-node=8 nvidia-smi # Test 2: Verify task distribution across nodes srun --nodes=2 --ntasks-per-node=2 hostname # Test 3: Check task distribution srun --nodes=2 --ntasks-per-node=2 bash -c 'echo "Task $SLURM_PROCID running on node $(hostname)"' ``` # Troubleshooting Common Issues ## InvalidAccount Error If you encounter `InvalidAccount` errors when submitting batch jobs: 1. **Use correct GPU syntax**: In batch scripts, always use `--gres=gpu:N` instead of `--gpus-per-node=N` 2. **No accounting plugin**: Prime Intellect clusters intentionally don't use Slurm's accounting plugin since clusters are single-tenant with dedicated resources. Remove any `#SBATCH --account=` directives from your scripts 3. **Check partition availability**: Ensure the partition you're requesting exists with `sinfo` Prime Intellect clusters run without the Slurm accounting plugin because each cluster is single-tenant with dedicated resources. This simplifies configuration and eliminates account-based resource restrictions, giving you full access to all allocated resources without quota management overhead. ### Incorrect (causes InvalidAccount) ```bash theme={null} #!/bin/bash #SBATCH --gpus-per-node=8 # Wrong for batch scripts ``` ### Correct ```bash theme={null} #!/bin/bash #SBATCH --gres=gpu:8 # Correct for batch scripts ``` ## Job Output Not Found If you can't find your job output files: * **Check working directory**: Output files are created where `sbatch` was run * **Use absolute paths**: Specify full paths in `--output` and `--error` directives * **Check other nodes**: If submitted from a compute node, outputs may be on that node's local storage * **Always submit from shared storage**: Change to the shared storage directory before running `sbatch` ## Node Communication Issues If nodes can't communicate or jobs hang: * Verify all nodes are in `idle` state with `sinfo` * Check node connectivity: `srun --nodelist= hostname` * Ensure shared storage is mounted on all nodes * Consider restarting your cluster if issues persist # Advanced Slurm Commands ## Direct Node Access Access specific compute nodes directly for debugging or monitoring: ```bash theme={null} # SSH into a specific node via srun srun --nodelist=computeinstance-abc123 --pty bash # Run commands on specific nodes srun --nodelist=node001,node002 hostname # Allocate resources without running a command salloc --nodes=2 --gres=gpu:8 --time=01:00:00 # Then use srun within the allocation srun nvidia-smi ``` ## Resource Monitoring ```bash theme={null} # View detailed node status scontrol show node # Check GPU allocation scontrol show node | grep -E "NodeName|Gres" # Monitor job efficiency seff # View job accounting information sacct -j --format=JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode # Check cluster utilization sreport cluster utilization ``` ## Job Arrays for Parameter Sweeps Run multiple similar jobs with different parameters: ```bash theme={null} #!/bin/bash #SBATCH --job-name=param-sweep #SBATCH --array=1-10 #SBATCH --gres=gpu:1 #SBATCH --output=sweep_%A_%a.out # %A=array job ID, %a=array task ID # Use SLURM_ARRAY_TASK_ID for different parameters python train.py --seed=$SLURM_ARRAY_TASK_ID --lr=$(echo "0.001 * $SLURM_ARRAY_TASK_ID" | bc) ``` ## Environment Variables Useful Slurm environment variables available in jobs: ```bash theme={null} echo "Job ID: $SLURM_JOB_ID" echo "Job Name: $SLURM_JOB_NAME" echo "Node List: $SLURM_JOB_NODELIST" echo "Number of Nodes: $SLURM_JOB_NUM_NODES" echo "Tasks per Node: $SLURM_NTASKS_PER_NODE" echo "CPUs per Task: $SLURM_CPUS_PER_TASK" echo "Submit Directory: $SLURM_SUBMIT_DIR" echo "Task ID: $SLURM_PROCID" echo "Node ID: $SLURM_NODEID" ``` # Shared Storage Integration Your Slurm cluster comes with shared storage automatically mounted on all nodes. The UI displays the mount path for your shared storage directory. This ensures: * Consistent file access across all compute nodes * No need to manually copy data between nodes * Simplified job submission and management * Persistent storage for checkpoints and results ## Best Practices Always submit jobs and run Slurm commands from the controller node, not compute nodes Store your code, data, and outputs in the shared storage directory shown in the UI for seamless access across nodes Regularly check cluster utilization with `sinfo` and `squeue` to optimize job scheduling For parameter sweeps or similar tasks, use Slurm job arrays for efficient scheduling This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.