A new evaluation of work that involves reading drawings and images
Google DeepMind on Nov. 10 (Nov. 11 Japan time) released "MKA (Multimodal Knowledge Agent)," a new benchmark for AI agents, developed jointly with U.S.-based Artificial Analysis, which compares AI models under the same conditions. It has agents read visually complex files such as architectural drawings, electronic circuit diagrams, X-ray images and maps, produce deliverables such as spreadsheets, slides and reports, and then scores them. On Artificial Analysis' leaderboard, U.S.-based Anthropic's Claude Fable 5.2 ranked first with 57%, Google's Gemini 4 Carbon second with 54% and Anthropic's Claude Opus 5.5 third with 53%.
With @ArtificialAnlys, we built MKA (Multimodal Knowledge Agent): a benchmark for how well agents handle visually complex tasks. Everyday professional work often uses more than just text…
According to Google, many existing benchmarks that measure professional work can be solved just by reading text. It says MKA is new in that it gathers tasks that cannot be answered without looking at images.
200 tasks in 10 industries, graded by an OpenAI model
The tasks cover 10 industries, with 20 tasks each: business, construction, education, engineering, finance, health care, law and government, logistics and mapping, science, and software and data. They were created by experts in 83 occupations, and each task corresponds to work that would take a skilled professional 3 to 8 hours. Each task comes with 2 to 45 input files (7.3 on average), and 88% of tasks require the answer as a file rather than as text.
The grading criteria average about 70 items per task, with 75% of the weight on factual accuracy and 25% on visual presentation. Grading is done by AI. The grader opens the deliverable files, renders the pages as images to check them and even examines spreadsheet formulas. Google describes the grader as a non-Google model, and according to the technical report and Artificial Analysis it uses OpenAI's GPT-6 Luna. In a comparison with expert grading on 50 tasks, the judgments agreed on 96.2% of items.
The score, "Climb WSR," represents the weighted share of difficult grading items that were satisfied. Difficult items are those that at least one of four older-generation models (Gemini 3.8 Flash, Gemini 3.6 Flash, Claude Sonnet 5.5 and GPT-6.1 Sol) could not consistently satisfy. It is not the share of tasks completed perfectly. On "Overall WSR," which measures all items, the four models in the technical report all scored between 69% and 74%.
The models run on "Stirrup," an open-source harness developed by Artificial Analysis, with the same tools and working environment and a limit of up to 250 steps. Artificial Analysis runs the 200 tasks privately to prevent them from leaking into training data. A separate set of 10 tasks was released on Hugging Face.
Google says one-sixth the cost, Artificial Analysis says more than 5 times
The cost of solving a task differed widely between Fable and Carbon. The size of the gap, however, depends on which company did the calculation.
By Artificial Analysis' calculation, the cost per task was $31.97 for Fable 5.2 and $6.29 for Carbon. The company said Fable spent more than 5 times the cost of Carbon for a score about 2 points higher. OpenAI's GPT-6.1 Sol scored 48% at $1.11, about one twenty-ninth the cost of Fable, finishing 9 points lower.
In a blog post, Google claimed Carbon completes tasks at an average of one-sixth the cost of the top-scoring model. In the technical report's calculation behind the claim, based on each company's list prices (including prompt caching), the cost per task was $35.28 for Fable and $5.63 for Carbon, a gap of 6.26 times. Neither company has explained why the two calculations differ.
Fable's costs are higher because it uses more. According to the technical report, Fable used 40.48 million tokens (the units of text an AI processes) per task, 5.74 times Carbon's 7.05 million. It took 128.8 steps versus 46.6, and 83.2 minutes versus 29.2.
API pricing per 1 million tokens for Fable 5.2 is $8 for input and $40 for output. Carbon costs $2 for input and $10 for output during an introductory period through Nov. 30, rising to $4 for input and $20 for output afterward. Google made Carbon generally available on the same day, Nov. 10, through its API for developers and businesses, among other channels. Fable 5.2 had just been released by Anthropic the previous day, Nov. 9.
Top three within margin of error; an evaluation Google co-developed
According to the technical report, the score differences among Fable 5.2, Carbon and Opus 5.5 are not statistically distinguishable. The gap between Fable and Carbon is 2.22 points, with a 95% confidence interval spanning minus 0.94 to plus 5.49 points. Fable and Carbon, however, outperformed GPT-6 Astra by a statistically significant margin.
The report also lists limitations of the evaluation. Each model solved each task only once, and grading was also done only once. Agreement with experts was checked only for heavily weighted items on 50 tasks, and the same agreement rate may not apply to detailed formatting items. Costs were calculated at list prices at the time of writing and will change if prices change. The older-generation models used as the benchmark for selecting difficult items include two Gemini models.
MKA is an evaluation Google co-developed, and its own Carbon placed second. U.S.-based Crypto Briefing pointed out that DeepMind has a direct competitive interest. U.S.-based TechTimes reported that the gap at the top is effectively a tie. The 200 tasks are private, and the results cannot be reproduced externally.
Scores drop when the image-viewing tool is removed
In the technical report, Google also published an experiment showing that MKA measures the ability to read images. When Carbon's image-viewing tool was removed, its Climb-WSR fell 5.94 points, its number of steps rose 1.28 times and its output tokens rose 1.40 times. When the same experiment was run on "GDPval," OpenAI's evaluation of professional work, the score barely changed. Without the tool, Carbon tried to compensate by, for example, converting images into text art, but in cases involving photos and medical images its Climb-WSR fell 34 to 72 points. The experiment was run only on Carbon.
According to Artificial Analysis, by industry Fable ranked first in software and data, engineering, logistics and mapping, business, and finance (tied with Claude Sonnet 5.5 in finance). Opus 5.5 led in education and construction, Gemini 4 Argon in science, Carbon in health care and Sonnet 5.5 in law and government. By deliverable format, Word documents and text files were the hardest, with no model exceeding 52%.
Evaluations that measure AI on long professional tasks have come one after another over the past month. U.S.-based SemiAnalysis released "GDPx" on Nov. 1, on which even the top model, Claude Opus 5.5, had an accuracy of 12.0%. On U.S.-based Mercor's "APEX-Agents," Carbon took first place on Oct. 30.
