SemiAnalysis releases a new benchmark
U.S. firm SemiAnalysis, known for its research on semiconductors and AI, on Nov. 1 (Nov. 2 Japan time) released "GDPx," a new benchmark that measures the capabilities of AI agents (AI that carries out work on its own after receiving instructions). It had the latest models from each company solve a total of 206 tasks across 11 economically valuable domains, including chip design, finance, law and biology. Even the top-ranked model, Anthropic's "Claude Opus 5.5," scored only 12.0%, and SemiAnalysis described GDPx as "a true frontier-level benchmark."
Today we're releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)π§΅
The tasks were created together with 14 companies, including startups that build training environments for AI companies. The time limit for each task is generally 8 hours, and the AI searches for files and uses tools inside a simulated work environment to complete deliverables. Each model ran in its developer's own work tool: Opus 5.5 in Anthropic's "Claude Code," OpenAI's "GPT-6 Astra" in "Codex," and Google's "Gemini 4 Carbon" in "Antigravity."
Even leader Opus 5.5 scores 12.0%
The score is the average accuracy over three attempts at each task, with a maximum of 100%. The 11 domains were averaged with equal weight. Gemini 4 Carbon, which Google announced on Oct. 28, placed second at 10.9%, and OpenAI's GPT-6 Astra was third at 9.6%. Meta's "Muse Wonder Full" was fourth at 9.0%, putting the top four models between 9.0% and 12.0%. Models from China's DeepSeek and Z.ai and from Elon Musk's xAI scored 2.7% to 5.4%.
The leaders differed by domain. According to SemiAnalysis, Opus 5.5 ranked first in quantitative trading, hardware design and chip design, and was the most balanced overall. Carbon ranked first in biology and narrowly first in software development. Astra ranked first in the domain where AI automates AI research and development and in materials science, while Meta's Muse Wonder Full led cybersecurity by a wide margin.
Carbon costs $16 per task, about a third of Opus
Carbon stood out on cost. The average cost to solve one task was $16.29 for Carbon, about a third of Opus 5.5's $51.41 and GPT-6 Astra's $53.99. The line connecting models not beaten by any other on both cost and score (the efficiency frontier) included, from cheapest, OpenAI's "GPT-6 Luna," DeepSeek 4.1 Flash, OpenAI's "GPT 6.1 Sol" and "GPT 6.1 Terra," Carbon and Opus 5.5. Third-place Astra cost about the same as Opus 5.5 but scored 2.4 points lower, leaving it off this line.
Google CEO Sundar Pichai referred to the cost gap on X about two and a half hours after the release.
Great to see high quality benchmarks with headroom and look at that cost line for Carbonπ
Carbon's price rises to Opus's level after an introductory period
Developer API pricing per 1 million tokens is $4 for input and $20 for output for Opus 5.5, and $2 for input and $10 for output for Carbon. However, Carbon's price is an introductory rate, and after the period ends it rises to $4 for input and $20 for output, the same level as Opus 5.5. The length of the introductory period has not been disclosed. SemiAnalysis's release did not make clear which pricing was used to calculate the costs.
Carbon is currently available only to cyber defense organizations vetted by Google, and no timeline has been given for availability to general developers and users. Opus 5.5 is generally available through Anthropic's API and paid plans.
- Claude Opus 5.5
- Gemini 4 Carbon (introductory)
- Gemini 4 Carbon (after period)
Astra leads when safety-feature effects are excluded
The overall ranking changes depending on how safety features are treated. SemiAnalysis explained that Opus 5.5 and GPT-6 Astra refused some work in three domains, cybersecurity, biology and alignment (the safety of AI behavior), because of safety features that block dangerous uses (safety classifiers), pulling down their overall scores. In the ranking of the eight domains excluding those three, Astra took first at 11.8%, followed by Opus 5.5 at 11.7%, and OpenAI's GPT 6.1 Sol rose to third at 10.5%. Carbon fell to fifth at 9.2%.
SemiAnalysis said that even excluding the three domains, OpenAI and Anthropic dominate the top, and Carbon is the only model from another company at the same level. Across the eight domains, the cost per task was $61.34 for Astra, $64.24 for Opus 5.5 and $19.73 for Carbon.
- 11 domains
- 8 domains excluding 3
- GPT-6 AstraOpenAI9.6 β 11.8
- Claude Opus 5.5Anthropic12.0 β 11.7
- GPT 6.1 SolOpenAI8.3 β 10.5
- Claude Sonnet 5.5Anthropic8.2 β 9.7
- Gemini 4 CarbonGoogle10.9 β 9.2
- Claude Fable 5.1Anthropic8.5 β 8.6
- Muse Wonder FullMeta9.0 β 6.2
- Gemini 4 ArgonGoogle7.3 β 5.4
No model could complete finance and law tasks
By domain, there is much work that even the top models cannot complete. In finance and law, nearly every model scored zero under pass-fail grading, so they were ranked by average partial credit. On a law task revising a contract, average partial credit was 44% for Opus 5.5 and 25% for Astra. One finance task involved verifying a company's actual earning power from a data room of about 4,000 files in a private equity fund's acquisition deal; none of 25 attempts fully completed it, and the best run met only 8 of 11 checklist items.
In the quantitative trading tasks, 70 of 111 graded runs underperformed the trading strategy provided as a baseline. SemiAnalysis noted that even the strongest models rarely find an edge that holds up in the production period. In software development, 76% of failed runs failed because they did not meet conditions attached to the main request.
Problems also arose in tasks testing whether models follow rules. In a task assisting with medical administration, only 19 of 100 runs across four models scored above zero, with cases such as reusing an earlier signature on a new consent form and signing medical records in a doctor's name. In a task that imposed cybersecurity rules of conduct, 98% of 200 runs broke at least one rule.
An environment closer to real work than OpenAI's GDPval
SemiAnalysis explained GDPx by comparing it with "GDPval," which OpenAI released on Sept. 25, 2025. According to SemiAnalysis, GDPval is the general-purpose metric most often cited in companies' materials when they announce new models. SemiAnalysis criticized GDPval for putting all needed information into the prompt and having the model produce a single document, a format it said is far from actual work. In finance tasks, GDPval's prompt is about 500 words, while GDPx has the model search a data room of about 4,000 files. SemiAnalysis said agreement on GDPval grading is only 71% even among human experts, and that each GDPx task went through automated quality checks.
GDPx, however, is SemiAnalysis's own measurement and has not yet been reproduced externally. Many of the 14 companies that created the tasks also sell training environments to AI companies. On the overall index of Artificial Analysis, another third-party evaluator, Carbon scored 59, Opus 5.5 scored 58 and Astra scored 57, so rankings vary by metric.
Scores are as of Nov. 1, 2026. The official GDPx site lets users switch between rankings by domain, cost and output tokens.
