11 models play 386 games, with GPT-6 Astra on top
Dwarkesh Patel, host of the "Dwarkesh Podcast," on Nov. 5 (Nov. 6 Japan time) released "Clocktower Bench," a benchmark that pits AI models against one another in a game of social maneuvering. Eleven models from OpenAI and Anthropic released in 2026 played 386 games, and OpenAI's "GPT-6 Astra" ranked first.
Patel said on X that he spent about $20,000 on API costs. He expressed the view that this year's models have greatly improved at in-the-moment maneuvering, deception and team reasoning, and wrote that OpenAI's newer models are better than Anthropic's at this game. He noted that this view was his own.
Fwiw, OpenAI’s newer models are much better than Anthropic’s at this game.
A werewolf-style game with a program as moderator
Blood on the Clocktower is a social deception game similar to Werewolf or Mafia. The good team tries to find the evil leader through discussion and voting, while the evil team hides its identity and deceives the good team. Patel says it can measure abilities that are hard to see in other benchmarks.
In each game, one model plays all the good seats and another model plays all the evil seats. Games have 10 to 12 players and use only 1 set of roles, "Trouble Brewing." Games were also played with the teams swapped to cancel out the luck of role assignments. The moderator, normally a human, was replaced with a program that follows probabilities.
Conditions were the same for all models. Reasoning depth was set to each company's "high" setting, with a cap of 8,000 tokens per turn including reasoning and statements limited to 150 words. The rules went through 17 versions, and according to Patel, most of the API costs went toward iterating on this design.
OpenAI holds the top 2 spots, reversing the March rankings
Rankings are shown as Elo-style scores that estimate strength from game results, with the average of the 11 models set at 1,500. GPT-6 Astra ranked first with 1,859, and OpenAI's "GPT-6.1 Sol" was second with 1,734. Anthropic's best was "Claude Fable 5.1" in third with 1,595. The 90% confidence intervals of the top 2 models, 1,779 to 1,983 and 1,684 to 1,815, do not overlap with Fable 5.1's 1,537 to 1,668.
According to Patel, as of March, OpenAI's "GPT-5.4" trailed Anthropic's "Claude Opus 4.6." In the roughly 6 months from then to GPT-6 Astra, OpenAI's models gained about 580 points. By Patel's conversion, GPT-6 Astra beats GPT-5.4 about 97% of the time and Opus 4.6, about 430 points behind, about 92% of the time. Across all models released in 2026, scores are rising at a pace of about 650 points a year (90% confidence interval 553 to 827).
Anthropic's newer models are not always stronger. "Claude Opus 4.7" scored 1,241 and "Claude Opus 4.8" scored 1,198, both below Opus 4.6's 1,432. Patel's article does not appear to explain why.
The site says the order of adjacent models whose confidence intervals overlap is not certain. For example, the intervals of 3rd-place Claude Fable 5.1 (1,537 to 1,668) and 4th-place GPT-5.6 Sol (1,543 to 1,647) overlap.
Most of the gap is in detection; deception "roughly equal"
In addition to the overall score, Clocktower Bench separately shows the ability to detect evil on the good team and the ability to hide one's identity on the evil team. The site says most of OpenAI's advantage lies in detection, and that the 2 companies are roughly equal at deception. Patel also wrote in his article that the gains of OpenAI's newer models come mainly from improvement on the good team.
GPT-6 Astra scored highest in detection at 1,940, while Anthropic's best was Fable 5.1 at 1,582. GPT-6 Astra also ranked first in concealment at 1,806, but Anthropic's best, "Claude Fable 5," scored 1,643, 45 points behind GPT-6.1 Sol's 1,688.
Patel cited a game played under the same conditions as an example of the difference. In a scene where an attack on a villain failed, Claude's good team took that player to be innocent. GPT's good team suspected that the attacking ability might have been blocked, reconstructed what had happened, executed the real leader and won. According to Patel, OpenAI's models narrow down the leader faster and more accurately than Anthropic's, and Anthropic's models are more likely to believe the information reported to them.
Patel's article does not give a standalone figure for gains in deception. In the README table, the concealment score rose from 1,228 for GPT-5.4 in March to 1,806 for GPT-6 Astra in September. The evil team's overall win rate was 33% and falls slightly as both teams get stronger, but Patel says this change cannot be distinguished from chance.
In 132 games with teams made up half of Opus 5.5 and half of GPT-5.6 Sol, the score was slightly below the members' average. The difference was within the range of chance and not statistically significant.
Concealment (evil team) ratings: GPT-6 Astra 1,806, GPT-6.1 Sol 1,688, Claude Fable 5 1,643, Claude Fable 5.1 1,588, Claude Opus 5.5 1,585, GPT-5.5 1,560, GPT-5.6 Sol 1,519, Claude Opus 4.6 1,338, Claude Opus 4.8 1,320, GPT-5.4 1,228, Claude Opus 4.7 1,224.
Only 2 companies tested, and rankings differ from other evaluations
All 11 models tested are OpenAI and Anthropic products, and Patel's article, the site and the README do not mention models from Google or xAI. No explanation for limiting the test to the 2 companies could be found. Patel designed and ran the measurements himself. Agents start each game from a blank slate and do not carry experience over to the next game.
The win rates show only strength at this game. On Artificial Analysis's composite index, Claude Opus 5.5 ranks first with 58, and GPT-6 Astra scored 57. On SemiAnalysis's "GDPx", GPT-6 Astra scored 9.6% against Opus 5.5's 12.0%. On Clocktower Bench, Opus 5.5 scored 1,549 against GPT-6 Astra's 1,859.
Another team also published a benchmark using the same game on LessWrong on Sept. 30. It included Google's "Gemini 3.1 Pro" and China's DeepSeek's "DeepSeek V4 Pro," and "GPT-5.6 Sol" had the best overall result. Because the models and conditions differ, it does not amount to a reproduction of Patel's results.
Code and game logs released; no third-party reproduction confirmed
Patel released the code on GitHub under the MIT License. About 1,100 games' worth of logs are available on Hugging Face, and the site lets visitors view replays of 538 games along with each seat's reasoning. He also set up a site where humans can play against AI.
The site is an unofficial fan project and is not affiliated with The Pandemonium Institute, which owns the game's trademark.
What could not be confirmed
No reports of a third party reproducing these results were found. There were also 0 open issues on GitHub as of Nov. 6. The roughly $20,000 figure comes from Patel, and its breakdown could not be confirmed. Results of games against humans have not been published. ORIENS did not check the contents of the roughly 18-minute video attached to the post. An assessment of this benchmark by OpenAI researcher Noam Brown could not be confirmed in Patel's article and its comments, the site or the README. No comment from OpenAI or Anthropic was found either.
