Skip to main content
verifiers offers built-in support for Harbor via the HarborTaskset class. Creating a harbor-based environment is straightforward in most cases: 
import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset

# Set the dataset to the same name as registered in the Harbor registry
class TerminalBench2Config(HarborConfig):
    dataset: str = "terminal-bench/terminal-bench-2"


# The data will get loaded automatically
class TerminalBench2Taskset(HarborTaskset, vf.Taskset[HarborTask, TerminalBench2Config]):
    pass
You can also write custom code for your environments. A common functionality is to set custom images for tasks that don’t come with an image in their task.toml: 
from pathlib import Path
from typing import Literal

import verifiers.v1 as vf
from verifiers.v1.tasksets.harbor import HarborConfig, HarborTask, HarborTaskset

IMAGE_TEMPLATE = "registry.example.com/openthoughts/{task}:latest"


class OpenThoughtsTBLiteConfig(HarborConfig):
    dataset: Literal["openthoughts/openthoughts-tblite"] = "openthoughts/openthoughts-tblite"
    # Tell verifiers to use the pre-built image
    ignore_dockerfile: bool = True


class OpenThoughtsTBLiteTaskset(HarborTaskset, vf.Taskset[HarborTask, OpenThoughtsTBLiteConfig]):
    def load(self) -> list[HarborTask]:
        # Use the public image instead to avoid building the image at runtime; the row
        # data is frozen, so rebuild each task around an updated copy.
        return [
            HarborTask(
                task.data.model_copy(
                    update={"image": IMAGE_TEMPLATE.format(task=Path(task.data.task_dir).name)}
                ),
                task.config,
            )
            for task in super().load()
        ]
To create & re-use images for your environments, build the Dockerfile with Docker and push it to a registry, then set the resulting image reference as the task’s image field.

​
Additional features

Every Harbor taskset can also be modified with a timeout_multiplier and a resource_multiplier: 
[taskset]
id = "MY_TASKSET"
timeout_multiplier = 2.0
resource_multiplier = 2.0
The timeout_multiplier multiplies both the agent and verifier timeout, while the resource_multiplier multiplies the task’s CPU, memory and disk space. You might want to use these multipliers when the tasks set too tight limits and/or the agent is slow.

​
Shortcomings

verifiers does not have parity with Harbor yet, so some features are missing and currently being worked on. The most notable missing features right now are: