23 languages, 16,000 tasks
Researchers at China's Alibaba on Oct. 26 (the same day in Japan time) released the code, data and paper for "BabelArena," a benchmark that compares the abilities of AI agents across 23 languages. @QwenDevs, the developer-focused X account for the company's AI "Qwen," announced it.
The researchers selected 702 tasks from four existing agent benchmarks and translated them into 23 languages, creating 16,146 tasks across 13 domains. The tasks, such as shopping, travel planning and customer support, are carried out while using tools. Agent benchmarks so far have centered on English, and abilities in other languages had not been measured.
Performance falls in languages with fewer speakers
The five models tested were Anthropic's Claude-Opus-4.8, OpenAI's GPT-5.6-Terra, Google's Gemini-3.1-Pro, and Alibaba's Qwen-3.7-Max and Qwen-3.8-Max. No model ranked first on all four benchmarks. Every model performed worse in languages with less training data. For Claude-Opus-4.8, the task success rate fell from 51.3% to 44.8%.
In languages with fewer speakers, errors in tool use and procedures increased, particularly in Thai and Tamil. There were also many cases in which the model switched to English partway through.
| Language | Conversation turns | Tool calls |
|---|---|---|
| English | 9 | 25 |
| Telugu | 64 | 201 |
Repeated searches that come up empty
In the Telugu example, product searches rose from 5 to 158, of which 126 found nothing. This is one case examined by hand, not an overall average. Overall, tool calls in languages with fewer speakers rose to 1.31 to 1.80 times those in English. The amount of text used also grew, which also leads to higher usage fees.
