oqoqo

About oqoqo

oqoqo is a platform for building evals and custom benchmarks that test AI agents on real-world tasks. It launched this week and targets teams building products that agents interact with, as well as teams deploying agents into daily workflows. The tool runs agent experiments in isolated sandboxes, catalogs every step an agent takes, and scores success against user-defined criteria.

Review

Most benchmarks run in curated environments that don't reflect how agents actually behave against live products. oqoqo tries to close that gap by letting teams define tasks as prompts, specify what to test, and set success criteria. The platform then executes those tasks against a range of agents and records tool calls, retries, discovery loops, token consumption, and cost.

Key Features

• Run eval experiments at scale in realistic, isolated sandbox environments
• Define custom task sets to build private benchmarks for your own products
• Measure agent performance against Codex, Claude Code, OpenClaw, Hermes, Pi, Opencode, Cursor, and GitHub Copilot
• Regression test MCP, CLI, skills, SDK, and other agent-facing interfaces
• Compare models and harnesses on the same task set, including different reasoning levels or effort settings
• Generate dynamic insights on product interface friction and token inefficiencies

Pricing and Value

The product page lists "Free Options" and encourages users to try it for free at oqoqo.ai. Specific pricing tiers are not detailed in the reference content. The value is positioned around replacing hand-built eval setups: teams define tasks and success criteria, and oqoqo handles sandbox provisioning, execution, and result collection. The platform also supports sharing custom benchmarks publicly.

Pros

• Tasks are defined as plain prompts, so non-specialists can create evals without deep data science knowledge
• Runs in isolated sandboxes with real dependencies and file contexts, which mirrors production conditions more closely than curated test beds
• Records full agent trajectories including tool calls, retries, and discovery loops, not just pass/fail outcomes
• Supports negative rubric criteria to catch cases where agents fail or leave products in a worse state
• Multiple trials per task help establish statistical significance despite agent nondeterminism

Cons

• Eval rot is a real concern: a test suite that stops failing can look identical to a product that genuinely improved, and the team has not yet shipped tooling to flag outdated eval sets
• Pricing details beyond a free option are not publicly defined, which makes cost planning difficult for teams considering adoption
• Not well suited for teams that need deterministic, unit-test-style verification; the platform embraces agent nondeterminism and requires running multiple trials to get meaningful results

oqoqo fits teams building products that agents will discover and use, especially those shipping MCP servers, CLIs, SDKs, or other agent-facing interfaces. It also works for teams evaluating which model or harness performs best on their specific domain tasks. The tool is new, so expect the feature set to evolve as the team responds to user feedback on evaluation workflows.

Open 'oqoqo' Website