Operator says a non-OpenAI model is No. 1 "for the first time in a very long time"
Google's top model, Gemini 4 Carbon, has taken first place on MathArena, a third-party benchmark that measures AI math ability. Jasper Dekoninck, who runs the benchmark, announced it on X on Nov. 13 (12:27:44 a.m. Nov. 14 Japan time). He wrote that "for the first time in a very long time, we have a non-OpenAI model as #1 on MathArena," saying Carbon had taken first place from OpenAI's GPT-6.1 Sol and that Anthropic's "Fable 5.2" placed second.
For the first time in a very long time, we have a non-OpenAI model as #1 on MathArena! Gemini 4 Carbon takes over the first place from GPT-6.1 Sol, with Fable 5.2 taking #2. https://t.co/HOs01YPLf9
Dekoninck is a Ph.D. student at ETH Zurich in Switzerland, and his X profile says he works on mathematical evaluation of AI models, including MathArena. MathArena uses new math competitions and research problems to measure reasoning on questions models have not seen during training. Each model solves each problem four times, and the site shows the average score and the cost per run.
According to updates on the MathArena site, results for Carbon and Claude Fable 5.2 were added on Nov. 13. Google began general availability of Carbon on Nov. 10 (Nov. 11 Japan time), and Anthropic released Fable 5.2 on Nov. 9.
Google CEO Sundar Pichai and Google DeepMind CEO Demis Hassabis reposted Dekoninck's post on X (Hassabis at 2:25:45 a.m. Nov. 14 Japan time and Pichai at 2:55:03 a.m.). Neither added a comment of his own. Google itself has not announced the ranking.
Carbon leads with 85.4% overall expected performance
In the overall table on the MathArena site, Carbon, with its reasoning setting at "high," ranked first with expected performance of 85.4%. Claude Fable 5.2, at the deepest "max" reasoning setting, was second at 82.8%, and GPT-6.1 Sol, also at max, was third at 80.6%. Among models with published weights, China's Moonshot AI's Kimi K3.1 Pro (max) scored highest at 54.3%.
Expected performance is the average of scores on all problems from the competitions currently in use, normalized to 100%. MathArena gives 95% confidence intervals of plus or minus 3.0 points for Carbon and plus or minus 5.3 points for Fable. The 2.6-point gap between Carbon and Fable falls within that range.
Cost per problem is about 15 times Sol's
On cost, Carbon is far above Sol. On the MathArena site, the expected cost of solving one problem once is $15.70 for Carbon, about 15 times Sol's $1.07. Fable's is $40.20, and Kimi K3.1 Pro's is $3.70.
In a follow-up post, Dekoninck wrote that under these reasoning settings Carbon's extra performance comes at "significantly higher costs," which is particularly visible on BrokenArXiv, one of the benchmark's evaluations.
The extra performance does come at significantly higher costs under these reasoning settings, which is particular visible on BrokenArXiv. https://t.co/ZFBVqBF3a1
Because of cost, Carbon has not yet been run on some of the evaluations. According to Dekoninck, Carbon was run only on "ArXivLean," where its performance was "very impressive." He said it will be added to "LeanLean" later. According to MathArena's explanation, scores for problems without results are estimated statistically from the results of other models and problems and included in the average.
Due to costs, we could only run Carbon on ArXivLean. It has a very impressive performance there though. We will add it later to LeanLean as well! https://t.co/89zK0b8s8Q
