The Sandbox Mirage: Why Web Agents Fail When Reality Intrudes

Claude Sonnet 4.6 achieves a 75% task completion rate on WebArena and OSWorld. On ClawBench, it scores 33.3%. For GPT-5.4, the drop is even more precipitous: from roughly 67% to 6.5%. These figures, reported in "ClawBench: Can AI Agents Complete Everyday Online Tasks?", represent more than a performance regression. They expose a fundamental disconnect between how we train web agents and the chaotic reality of the live internet. The 40-point gap between sandbox and production performance suggests we have spent years optimizing for pattern matching in sterile environments, while the messy, dynamic, transactional web remains largely unconquered.

The Illusion of Static Benchmarks

Existing web agent benchmarks such as WebArena, VisualWebArena, and OSWorld operate on a seductive premise. By hosting static HTML snapshots of websites in controlled sandboxes, they eliminate the unpredictability that makes real web navigation difficult. These environments feature fixed DOM structures, deterministic element IDs, no authentication flows, and no dynamic JavaScript rendering. An agent navigating these sandboxes can rely on brittle XPath selectors and memorized layout patterns without needing robust error recovery or semantic understanding of page structure.

This controlled setting measures a narrow skill: the ability to map natural language instructions to static interface elements. When confronted with the live web, however, agents face cookie consent popups that block interactions, asynchronously loaded content that shifts element positions, and anti-bot defenses that challenge automated browsers. ClawBench evaluates on 144 live production platforms across 15 life categories including travel booking, job applications, and financial management. These tasks require write-heavy operations; form submissions, purchase completions, and appointment bookings rather than the read-only information retrieval that dominates existing live-web evaluations like WebVoyager or AssistantBench.

The technical constraints of sandbox evaluation create an evaluation mirage. Static benchmarks allow for deterministic verification; the final HTML state can be checked against ground truth with simple string matching. Live web evaluation requires sophisticated instrumentation. ClawBench implements a lightweight Chrome extension paired with a Chrome DevTools Protocol (CDP) based server to monitor network traffic. This system intercepts the final HTTP submission request, the single moment where a task transitions from simulation to real world consequence, allowing safe evaluation without actual transaction completion.

Engineering Realism: The ClawBench Methodology

The authors address the safety and evaluation challenges of live web interaction through a multi-layered capture system that records five distinct data streams: session replay via Xvfb virtual display and FFmpeg encoding, per-step action screenshots, complete HTTP traffic logs, agent reasoning traces and tool calls, and low-level browser actions including clicks, keystrokes, and scrolls. This granularity enables traceable failure analysis rather than binary success metrics.

Human annotators complete each of the 153 tasks to generate reference trajectories. The evaluation then employs an Agentic Evaluator, implemented using a Claude Code sub-agent, which compares agent trajectories against these human references. Unlike simple final-state checking, this evaluator applies predefined rubrics to assess whether the agent correctly populated form fields, navigated required workflow steps, and handled authentication properly. The system produces structured justifications grounded in step-level evidence, revealing not merely that an agent failed, but precisely