Skip to content
Framework Design
Our Approach vs Benchmarks
Approach vs Benchmarks Comparison
Name
Our Approach
WebArena
WebVoyager
OSWorld
Nature of Tasks
Grounded in real-world consumer and enterprise workflows. Tasks span full journeys (e.g., shopping, travel booking, procurement) with system orchestration, ambiguity, personalization, and ROI trade-offs. Tests agents as end-to-end planners, not just tacticians.
Broad set of websites, APIs, and tools across domains. However, sandbox-style setup limits realism; tasks are bounded and optimized for benchmarking repeatability.
Strong focus on realistic consumer tasks (shopping, travel, research) using real sites. Lacks enterprise tasks and higher-order ambiguity.
Simulates OS-level user behavior across desktop apps (email, file, browser). Realistic UI flows but task scope is narrow — small subtasks, not workflows.
Systems Complexity
Explicitly models number of systems involved (1, 1–3, 3+) across workflows. Important for assessing integration needs across apps.
Tasks are all browser-based and mostly single-site.
Mostly browser-based, some tabs or subtasks simulate light multi-system, but no app-switching.
Tasks span multiple desktop apps (file manager, mail, browser). Simulates real OS workflows.
Number & Type of Actions
Categorized into 1–10, 10–20, 20+ actions. Captures task length, UI depth, and whether interactions are simple clicks or complex data inputs.
Ranges from short (fill a form) to long (register + book), but lacks explicit action-type breakdown.
Some multi-step tasks with 10+ steps, especially shopping or research, though many are click-heavy.
Tasks often exceed 20 steps, including app switching and config changes.
Exception Handling
Models task complexity based on system unpredictability: errors, retries, unavailability. Crucial for evaluating robustness and HITL integration.
Ideal environment, no error states or need for retry logic.
No simulation of errors, failures, or recovery.
Tasks are deterministic. No retry logic or exception testing implemented.
Data Types
Differentiates structured (e.g., date), semi-structured (e.g., tables), and unstructured (e.g., PDFs). Essential for real-world consumer + enterprise workflows.
Mostly structured form fields or buttons. No unstructured docs.
⚠️ Includes open-ended queries (e.g., “find something nice”), but still constrained to structured UIs.
✅ Involves some semi-structured data (text docs, settings), though still no OCR or document parsing.
Process Readiness & Orchestration
Distinguishes tasks that are well-bounded vs. those needing sequencing, memory, or coordination. Models orchestration across workflows.
Tasks are short and isolated; no orchestration layer.
Focused on end-to-end completion but lacks true orchestration or branching logic.
Tasks are static. No long-term workflows or process state tracking.
Level of Risk / Consequences
Tasks are classified by risk level (low, medium, high) to model stakes: shopping vs. compliance vs. finance. Evaluates real-world agent trust needs.
No modeling of consequences. All errors are costless.
Same — task failure has no simulated penalty or trade-off.
No outcome penalties; success is binary and isolated.
Task Variability / Ambiguity
Consumer tasks are modeled as highly variable (user intent, preferences), enterprise as repeatable but multi-stakeholder. Requires both adaptability and templating.
Light ambiguity in search tasks (“find a good phone”), but most tasks are templated.
Higher ambiguity in some tasks, e.g., “find a fun trip,” with less constrained expectations.
Some open-ended tasks with flexible completion strategies (e.g., email draft quality).
Personalization & Preference Memory
Evaluates whether agents adapt to user history, preferences, or global cohort memory. Critical for reducing prompting and building trust.
No memory. Tasks are single-shot with no personalization.
No memory of past user behavior or contextual preference setting.
No personalization; each task starts from scratch.
Cross-Task Reasoning & Decomposition
Directly tested via high-complexity workflows involving task breakdown (e.g., travel planning across flights, hotels, cars). Evaluates agent as planner, not just executor.
Tasks are atomic. No support for planning or decomposing larger goals.
Simulates some decomposition (“find X, then check Y”), but sequence is hard-coded.
Some task chaining (e.g., find file → edit → send), but no agent-led decomposition.
Evaluation Grounding
Tasks drawn from real-world consumer and enterprise workflows (e.g., booking, shopping, EHR entries). Grounded in application layer complexity and real ROI trade-offs.
Based on common websites, but focuses on benchmarkable UI manipulation.
Uses real sites and search engines, creating realism, but lacks orchestration or risk context.
Realistic UX environments across apps, but use cases are still simulated.
There are no rows in this table
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.