​

Multimodal Browser Environments in verifiers

​

Two Modes of Browser Interaction

​

DOM Mode: Natural Language Control

• navigate(url): go to a URL
• observe(instruction): find possible actions matching a natural language description
• act(instruction): execute an action described in natural language (click a button, fill a form)
• extract(instruction, schema_json): extract structured data from the page

​

CUA Mode: Vision-Based Control (Multimodal)

• click(x, y, button): click at screen coordinates
• double_click(x, y): double-click at coordinates
• type_text(text): type text into the focused element
• keypress(keys): press keyboard keys
• scroll(x, y, scroll_x, scroll_y): scroll at a position
• goto(url): navigate to a URL
• back() / forward(): browser history navigation
• wait(time_ms): wait for a specified duration
• screenshot(): capture the current page state

​

Side-by-Side Comparison

​

Getting Started

​

Installation

# Install verifiers with browser support
uv pip install "verifiers[browser]"

​

Environment Variables

# Required for both modes
export BROWSERBASE_API_KEY="your-api-key"
export BROWSERBASE_PROJECT_ID="your-project-id"

# Required for LLM judge model - pick your favorite model/your custom judge model, here we use OpenAI as an example
export OPENAI_API_KEY="your-openai-key"

# Required for DOM mode only (Stagehand's internal LLM) - pick your favorite model provider
export MODEL_API_KEY="your-llm-api-key"

​

Running the Examples

# DOM mode
prime eval run browser-dom-example -m openai/gpt-4.1-mini

# CUA mode (pre-built image, fastest and recommended)
prime eval run browser-cua-example -m openai/gpt-4.1-mini

​

Screenshot Management

• save_screenshots (bool): persist every screenshot to disk as timestamped PNGs in screenshot_dir (defaults to ./screenshots). The CUA example defaults this to False; the BrowserEnv class itself defaults to True.
• keep_recent_screenshots (int | None): how many recent screenshots to retain in the conversation context window sent to the model. Defaults to 2. Set to None to keep all (higher token cost).

​

Example Environments

​

bb_demo: Quick Eval with Click Visualization

from bb_demo import load_environment

env = load_environment(
    save_screenshots=True,
    screenshot_dir="./my_screenshots",
    mark_clicks=True,
    click_marker_radius=15,
)

​

webvoyager_no_anti_bot: Training-Scale Benchmark

# All 600 tasks, DOM mode
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini

# CUA mode, filtered to Amazon tasks
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"mode": "cua", "web_filter": "Amazon"}'

# 10 examples for a quick test
prime eval run webvoyager-no-anti-bot -m openai/gpt-4.1-mini -a '{"num_examples": 10}'

​

Building Your Own Browser Environment

1. Define a dataset: a HuggingFace Dataset with question, answer, and optionally start_url and task_id columns. For task-completion benchmarks where there’s no ground-truth answer (like WebVoyager), set answer to an empty string and rely on a task-completion judge.
2. Define a rubric: typically a JudgeRubric with an LLM judge. For answer-matching tasks, the judge compares agent output to the expected answer. For task-completion tasks, subclass vf.Parser to render the trajectory into a judge-friendly transcript and evaluate whether the task was actually done.
3. Construct a BrowserEnv: pass mode, dataset, rubric, system prompt, and any mode-specific configuration.
4. Package it: add a pyproject.toml with verifiers[browser]>=0.1.8 as a dependency.

import verifiers as vf
from verifiers.envs.integrations.browser_env import BrowserEnv
from datasets import Dataset

def load_environment(max_turns: int = 15, **kwargs) -> vf.Environment:
    dataset = Dataset.from_dict({
        "question": ["What is the title of the first blog post?"],
        "answer": ["Expected answer here"],
        "start_url": ["https://example.com/blog"],
    })

    rubric = vf.JudgeRubric(judge_model="gpt-4o-mini", judge_prompt="...")
    rubric.add_reward_func(your_judge_func, weight=1.0)

    return BrowserEnv(
        mode="cua",  # or "dom"
        dataset=dataset,
        rubric=rubric,
        max_turns=max_turns,
        system_prompt="...",
        **kwargs,
    )

uv pip install -e ./environments/your_env
prime eval run your-env -m openai/gpt-4.1-mini

​

RL Training with Browser Environments

• whether you want mode="dom" or mode="cua"
• whether Stagehand should keep its own model or be routed through the rollout model
• which browser/session settings you want in the [[env]] config

​

Before You Train

1. Validate the environment with evals first. Reward quality matters more than running a large training job quickly.
2. Check reward distribution. If everything is 0.0 or everything is 1.0, the run will not teach you much.
3. Start with a small training run before scaling up browser minutes and GPU time.

​

Training Target in This Repo

• 600 tasks in the filtered WebVoyager dataset
• supports both mode="dom" and mode="cua"
• uses task-completion reward rather than answer matching
• renders assistant text, tool calls, and truncated tool results into the judge transcript
• returns 0.0 reward immediately if a rollout makes no tool calls

​

DOM Training

• model: Qwen/Qwen3-4B-Instruct-2507
• batch_size = 64
• rollouts_per_example = 8
• environment id: browserbase/webvoyager-no-anti-bot

model = "Qwen/Qwen3-4B-Instruct-2507"
env_files = ["secrets.env"]
max_steps = 100
batch_size = 64
rollouts_per_example = 8

[sampling]
max_tokens = 1024

[[env]]
id = "browserbase/webvoyager-no-anti-bot"
args = { mode = "dom", proxy_model_to_stagehand = true }

​

CUA Training

• model: Qwen/Qwen3-VL-4B-Instruct
• batch_size = 32
• rollouts_per_example = 4
• environment id: prime/webvoyager-no-anti-bot
• BrowserEnv args: mode = "cua", max_turns = 10, viewport_width = 800, viewport_height = 600, keep_recent_screenshots = 2, memory_gb = 6

model = "Qwen/Qwen3-VL-4B-Instruct"
env_files = ["secrets.env"]
max_steps = 100
batch_size = 32
rollouts_per_example = 4

[sampling]
max_tokens = 512

[[env]]
id = "prime/webvoyager-no-anti-bot"
args = { mode = "cua", max_turns = 10, viewport_width = 800, viewport_height = 600, keep_recent_screenshots = 2, memory_gb = 6 }

​

Credentials and Workflow Notes

• Always provide BROWSERBASE_API_KEY and BROWSERBASE_PROJECT_ID when BrowserEnv is creating Browserbase sessions.
• DOM mode needs MODEL_API_KEY unless you route Stagehand through the rollout model with proxy_model_to_stagehand=true.
• CUA mode uses the same BrowserEnv backend choices as evaluation. The current bundled CUA server template forwards OPENAI_API_KEY into Stagehand if it is present, but that is a server-template detail rather than a separate BrowserEnv training argument.
• Browserbase credentials belong in environment variables or training secrets, not in the TOML file.
• The same BrowserEnv args used with prime eval run -a '{...}' belong in the training config under [[env]] args = { ... }.
• BrowserEnv does not expose sandbox authentication as a constructor argument.

​

Picking DOM vs CUA for Training

​

Performance Notes

• Browser training rollouts are slower than text-only environments because each rollout step interacts with a real browser session.
• CUA mode is heavier than DOM mode because screenshots must be rendered, carried in context, and consumed by the model.
• If your task does not need visual grounding, DOM mode is usually the faster and cheaper place to start.

​

Quick Reference

​

DOM Mode Arguments

​

CUA Mode Arguments

​

CUA Execution Modes