November 3, 2026 | AI | By Mahesh
Momentic, a Y Combinator W24 startup, updated the results for QA Bench on November 2, 2026, the public benchmark it built to show that its AI QA agent Mo finds more bugs than general coding agents [1]. In Momentic's 60-minute test across 1,396 labeled bugs, Mo matched 250.3 bugs for 17.9% recall, against 94.3 and 6.8% for Codex with GPT-6.1 Sol and 39.6 and 2.8% for Claude Code with Opus 5.5 [1]. Momentic built, ran and graded the benchmark itself, and Mo costs more per matched bug than the cheapest rival [1].
- Mo is an AI QA agent from Momentic that tests web, iOS and Android apps, and new sign-ups get $250 in free credits.
- On Momentic's QA Bench, Mo matched 250.3 bugs against 94.3 for Codex with GPT-6.1 Sol, with 17.9% recall against 6.8%.
- Momentic built, ran and graded the benchmark, and Mo costs $2.88 per matched bug against $0.56 for the Codex setup.
What Mo is and what is new
Mo is an AI QA engineer. A team points it at a web, iOS or Android app, describes in plain English what to test, and Mo launches hundreds of parallel agents on hosted browsers and emulators to explore the app [2]. It returns one report with each bug's reproduction steps, logs, screenshots and a video replay, and teams can file the bugs as tickets or pass them to a coding agent to fix [2]. Y Combinator's launch page was created on September 28, 2026, so the product is not new, and the news here is the updated benchmark data [1][2].
On Hacker News, the founders said Mo builds a graph as it explores and forks large product areas into parallel sub-agents, each with its own browser or device [3]. They also shipped pull request testing for Vercel and Cloudflare deploy previews, and a command line tool and skill that let coding agents call Mo [3].
How QA Bench works
QA Bench gives each agent one product area of a running app in a fresh sandbox with the same seeded data, for 30 or 60 minutes [1]. The 1,396 labeled bugs come from 20 product areas in four open-source apps: Gitea with 476, Metabase with 464, Twenty with 374 and Saleor with 82 [1]. An LLM judge, GPT-6.1 Sol, matches each agent report against approved bugs and previously rejected claims, and scores average three runs per area, with one run for Devin [1].
Momentic published the harness and labels on GitHub and the dataset on Hugging Face, so other teams can rerun it [1]. For another recent tool release with a published benchmark, see Claude Haiku 5.5 Launch: $0.10 Input, 1M Context, Benchmarks.
Benchmarks: Momentic's results
| System (60 minutes) | Matched bugs | Recall | Precision | Cost per matched bug |
|---|---|---|---|---|
| Mo | 250.3 | 17.9% | 84.7% | $2.88 |
| Codex, GPT-6.1 Sol, Playwright MCP | 94.3 | 6.8% | 89.0% | $0.56 |
| Claude Code, Opus 5.5, Playwright MCP | 39.6 | 2.8% | 81.5% | $2.79 |
| Devin | 27.0 | 1.9% | Not stated | $3.89 |
At 30 minutes Mo matched 153.4 bugs against 88.0 for Codex with GPT-6.1 Sol and 25.7 for Claude Code with Opus 5.5 [1]. Momentic puts Mo's lead at 2.7 times Codex and 6.3 times Claude Code at 60 minutes [1]. The Codex setup is the most precise at about 89% and the cheapest per matched bug, while Mo's cost per full pass is the highest at about $720 for 60 minutes [1].
The numbers also moved since launch. The Hacker News post showed Mo at 269 bugs, 19.3% recall and $2.67 per bug, and the current page shows 250.3, 17.9% and $2.88 after a rerun [1][3]. Readers should quote the current page and its date.
Pricing, availability and how to access it
Mo is open to sign-ups at momentic.ai, and new accounts include $250 in free credits [2]. Paid plans offer higher infrastructure capacity, included usage and top-ups, but the sources list no plan prices [3]. Founders said bring-your-own-key access may come later, and users objected to paying for proprietary tokens instead of their own subscriptions [3].
Limitations, trade-offs and alternatives
Momentic lists its own caveats. It built Mo and ran and graded the benchmark, some candidate bugs came from bug bashes by Mo, Claude Code and Codex, so the labels may share those agents' blind spots, and an LLM judge can err in either direction [1]. Gitea supplies 11 of the 20 areas, results are averages of three runs with no error bars, and Mo does extra work such as recording reproduction videos [1]. Some Codex and Claude Code runs had web access, and GPT-6.1 Terra refused to run one Gitea area, citing OpenAI's cybersecurity policy [1].
Hacker News commenters asked why not run a frontier model with a custom Playwright harness, and some argued that frontier models will absorb this capability [3]. The founders replied that off-the-shelf models performed poorly in their harness and that tuned models and data give them an edge [3]. Teams weighing Mo should compare it with the cheaper Codex setup on their own app, since the price gap per bug is large. For open-weight options in a different category, see Mistral Large 4 Preview: 1.05T Open-Weight MoE, Specs and Pricing.
Sources
- Momentic. "QA Bench: Evaluating QA Agents on Real Web Applications," updated November 2, 2026 (vendor-run benchmark). momentic.ai
- Y Combinator. "Launch YC: Momentic: Mo, the AI QA engineer that finds bugs before your users do." ycombinator.com
- Hacker News. "Launch HN: Momentic (YC W24)," founder post and comments. news.ycombinator.com
Read More on Depth Grid
- Nvidia in Talks to Deepen Stake in or Buy Reflection AI
- Claude Haiku 5.5 Launch: $0.10 Input, 1M Context, Benchmarks
- Meta Muse Wonder Launch: 1M Context, $4 API Price, Benchmarks
- Mistral Large 4 Preview: 1.05T Open-Weight MoE, Specs and Pricing
Article by Mahesh | Depth Grid
Discussion (0)
Comments are disabled for this post.