Skip to main content
This page covers common issues you may encounter when using Hosted Training, along with their solutions.

​
Environment Issues

The verifiers library is not installed in your project.
Or if you’re installing a specific environment:
This usually means a module name collision — your environment name conflicts with an existing Python package, or the environment wasn’t installed correctly.Solutions:
  • Rename your environment to avoid conflicts
  • Reinstall: prime env install my-env
  • Check that your environment module exposes a load_environment function at the top level
The environment ID doesn’t match any published environment.
Make sure you’re using the correct owner/name format and that you have access if it’s a private environment.
The environment requires API keys that aren’t set. The error message will list which keys are missing.For local evaluation:
For Hosted Training, the recommended approach is to link secrets to your environment via the Environments Hub. Alternatively, pass them at dispatch:
Or use the CLI to manage secrets:

​
Training Issues

The task is too hard for the model at its current capability level. The model can’t solve any examples, so there’s no reward signal to learn from.Solutions:
  • Try a larger or more capable model
  • Use easier examples (filter your dataset or adjust environment args)
  • Increase [orchestrator.train.sampling].max_completion_tokens — the model may need more space to reason
  • Check your rubric implementation for bugs that might always return 0
  • Run a baseline evaluation first: uv run vf-eval my-env -m <model> -n 20 -r 1
The task is too easy — the model already solves everything.Solutions:
  • Use harder examples or a more challenging dataset split
  • Add more demanding rubric criteria
  • Use a smaller model that has more room to improve
There could be several causes:
  • Low reward diversity: If all rollouts for an example get the same reward, there’s no contrast for the model to learn from. Increase [orchestrator.train].group_size (16–32) to get more variation.
  • Learning rate too low: Try raising [trainer.optim].lr — Hosted Training runs typically sit in the 1e-6 to 1e-5 range, so move in small steps.
  • Batch size too small: Larger batches provide more stable gradient estimates. Try [orchestrator].batch_size = 512.
  • Task mismatch: The task may not be suitable for RL training. Ensure the reward function produces a meaningful gradient of scores, not just binary 0/1.
A field in your TOML config has the wrong type or an invalid value.Common causes:
  • String values not quoted: name = Qwen/Qwen3-4B → name = "Qwen/Qwen3-4B"
  • Integer where float expected or vice versa
  • Missing required sections like [model], [deployment], or [[orchestrator.train.source]]
Double-check your config against the prime-rl config reference.
The run’s volume is full. Usually a checkpoint did not fit: a full fine-tuning checkpoint takes about 11–12 bytes per parameter, and the trainer writes a new one before it deletes the oldest. The trainer crashes and retries into the same limit until the run is marked failed.Solutions:
  • Set keep_last = 1 or 2 in [ckpt], or save less often with a larger [ckpt].interval
  • Delete old runs’ outputs: prime volumes ssh <name> --read-write, then remove /volume/runs/<oldRunId>
  • Grow the volume or use a larger one
See Volume capacity to size the volume.
The model you specified isn’t currently supported for Hosted Training.
Runs boot from a cluster-side weight cache, so a model that isn’t cached can’t be dispatched. See Models.

​
CLI Issues

The CLI isn’t installed or isn’t on your PATH.
If you installed with uv tool install and it’s still not found, make sure ~/.local/bin is in your PATH.
Your CLI session may have expired.
This opens a browser window to re-authenticate.
Check the following:
  • Your config file is valid TOML (no syntax errors)
  • The model is available: prime train models
  • The environment ID is correct and accessible
  • You’re authenticated: prime login
  • Your CLI is up to date: uv tool install -U prime

​
Evaluation Issues

Reduce concurrency when running evaluations:
The -c flag controls maximum concurrent requests. Lower it if you’re hitting rate limits.
Qwen3 and DeepSeek-R1 models have chat templates that automatically remove <think> tags from message history. This conflicts with ThinkParser.Solution: Use MaybeThinkParser or Parser instead of ThinkParser in your environment:
  • Check rollouts_per_example: Low values (1–2) produce noisy results. Use at least 3–5 for reliable metrics.
  • Check num_examples: Very small sample sizes can be misleading.
  • Check sampling temperature: High temperatures produce more variation between runs.
  • Check your rubric: Make sure reward functions handle edge cases (empty responses, malformed outputs, etc.).

​
Environment Development Issues

Common causes:
  • Missing dependencies: Make sure all required packages are listed in your environment’s pyproject.toml
  • Missing secrets: API keys available locally may not be set for hosted runs. Link them to your environment via the Environments Hub, or pass them at dispatch with -e / --env-file.
  • Hardcoded paths: Avoid absolute file paths in your environment code
  • Network access: Some external APIs may not be reachable from the hosted environment
Start with a local evaluation using verbose output:
The -v flag enables verbose logging. You can also test your environment directly in Python:

​
Getting Help

If your issue isn’t covered here:
  • Discord: Join the Prime Intellect Discord for community support and Q&A
  • Research Support: Fill out the research support form for hands-on assistance
  • Feedback: Use the thumbs-up/down on any docs page to let us know what’s helpful or missing

​
Missing traces or upload failures

Check the workspace, upload setting, API permissions, and storage limit. Training completion does not by itself confirm that every trace uploaded. See Traces troubleshooting. For older samples, see legacy storage retirement and exports.