AI Benchmarks Are Saturated. 3D Games Are the New Test

Most AI benchmarks are saturated. Static test benchmarks where models read a question and write an answer, are now all bunched up at the top, with some retired all together. Vals.ai, a company only founded in 2024, has already archived 17 of them. That’s how fast this field is moving. Static tests no longer capture where the real leaps in development are happening: agents acting on their own in interactive worlds, especially 3D ones.

Below are some examples of saturated benchmarks. What do they have in common? They all only have one question and one answer, simple tests. Many of these were hard in the past for models, but are now no longer a challenge.

• MMLU-Pro (graduate-level questions across 14 subjects, multiple-choice quiz with 10 options): 89.8%, Gemini 3 Pro Preview (Dec 2025). Now labelled Legacy, no new models scored since. Source: Artificial Analysis
• GPQA Diamond (PhD-level science questions you can’t just Google, multiple-choice quiz): 96.3%, GPT-6 Astra (Oct 2026). Source: Artificial Analysis
• ProofBench (machine-checked maths proofs, graduate level, text in, proof out): 100%, four-way tie incl. Claude Opus 5.5 (Oct 2026). Source: Vals.ai
• IOI (olympiad-level competitive programming, problem in, code out): 100%, Gemini 4 Argon and GPT-6 Astra (Oct 2026). Source: Vals.ai
• BrowseComp (finding hard-to-locate facts on the web, question in, short answer out): 91.5%, GPT-6 Astra (self-reported, Oct 2026). Top 5 all within ~1 point. Source: LLM Stats
• SWE-bench Verified (fixing bugs from GitHub projects, bug report in, code fix out): 80.9%, then retired by OpenAI (Feb 2026). Source: OpenAI

However, interactive benchmarks are a different story. There, we’re only just starting to see real differences between models. The progress we saw in the last few months was especially astounding.

Understanding a 3D world has a lot in common with understanding the real one, which is why so many labs and start ups are working on it (also making it highly relevant for furthering robotics progress). It’s the next big hurdle for AI. We find that 3D games depict this progress incredibly well, and therefore, these benchmarks are far more important, for us and other tech startups.

We built and launched Game Gym, now live at evals.nunu.ai, to track that progress. But what are we actually measuring? It depends heavily on the game, which is why we launched with 11 of them.

What Games Measure

Recently, one of our own benchmarks became saturated, we had AI agents play Pokemon Emerald. It’s a 2D, turn-based game, and every model we tested reached the Hall of Fame. Even so, it showed us a lot:

2D Games

• Long-Horizon Memory, Planning & Learning: The model has to remember something from ten hours ago to act now. Like getting an item and using it much later (e.g. the Master Ball in Pokémon Emerald). Over the last years, we noticed Pokémon Emerald performance increase as coding performance did, in part due to improvements in long-horizon planning. This includes learning and not only recalling (Recalling means pulling up something the model already absorbed during training. Learning means working out something new from what's in front of it).
• Spatial Reasoning: Understanding that the map is bigger than the screen (it exists outside their field of view), and keeping track of where you are after every move. A kind of proprioception for models.
• Recovering from Strategic Failures: Losing a gym battle in Pokemon, planning badly, or getting lost navigating the 2D map. Smaller models used to fall into "death spirals", especially when they weren't allowed to stop.
• Efficiency: If a model completes a task, how efficient was it? What methods did it use to complete it and what was the overall cost?
• Behaviour and Personality: The same model plays differently depending on the prompt, and some behaviours suit certain tasks better.

3D games push each of these further, and add more:

• Recovering From 3D Failures: Walking into walls, not understanding where they are in 3D space, or walking off a cliff in Minecraft and losing your items.
• Continuous Movement: Judging speed and distance to know how long to hold a key, and which way.
• Real-Time Pressure: In fast games like shooters and racers, the world doesn't wait while the model thinks.
• New and Changing Interfaces: Every game has its own, and it often gets more complex as features unlock.

Our Own Experience

For a long time, when a new model launched, there were only small incremental changes in our product. Our harness did most of the heavy lifting, and we had to write in-depth prompts just for the AI to understand its world.

But, with the launch of Claude Fable 5, Gemini 4 Argon and, above all, GPT-6 Astra, released in September 2026. Suddenly we needed to know which models were best at games, so we could help our customers find bugs in theirs. Before Game Gym, we tracked this with what you could call "backyard benchmarks".

Minecraft

After each model release, we asked the AI to build a house in Minecraft. Most models couldn't do it without writing code to cheat, and they usually ended up with a pile of logs and no real structure.

With Astra, that changed. It built a house from scratch, furnished it, and even put lanterns outside.

https://youtu.be/RQaz6qVx81o

The Sims 4

We saw the same story in The Sims 4.

• [September/October] 2025: it could keep Sims alive, create a Sim and play a little in live mode, mostly UI tasks. Moving the camera was hard, so we kept it fixed over the house.

• July 2026: it could move around and place essential items on a lot, but still got lost in the world. Here it got stuck caring for Sims on two different lots instead of bringing them home.

• September 2026, with Astra: it can fully play the game. It builds a house without cheats, travels around the world, handles romance and needs, and in further tests kept 8 Sims alive with no problems.

Why The Jump?

We think it’s largely because of the increasing shift towards computer use. OpenAI wrote in their release blog that they focused on computer use with Astra, calling it its strongest model yet for controlling browsers and desktops. The shift had already started with GPT-5.5, and we saw that jump in our Pokémon Emerald benchmark too (harness was kept the same to accurately compare). Independent testing agrees: Vals.ai ranks Astra #1 on its computer-use benchmark, CUA-bench.

Time for Game Gym

So we built Game Gym properly: 11 games, chosen to cover as many different skills as possible. They're also mostly 3D games people know, so you can watch the difference instead of just reading a number.

• Cities: Skylines (city-builder sim, 3D, self-paced): Long-term planning, many systems at once, dense menus
• Dark Souls (action RPG, 3D, real-time): Hard combat, timing, learning from repeated deaths
• DOOM (2016) (first-person shooter, 3D, fast real-time): Aiming, reflexes, moving through 3D space under pressure
• Hogwarts Legacy (open-world action RPG, 3D, real-time): Navigating a huge world, following quests, camera control
• Hollow Knight (metroidvania platformer, 2D, real-time): Precise jumps and timing, remembering the map
• Kerbal Space Program (space-flight and physics sandbox, 3D, self-paced): Physics, engineering, multi-step planning, complex UI
• Palworld (open-world survival and crafting, 3D, real-time): Juggling goals: gathering, crafting, base building
• Poly Bridge (physics puzzle, 2.5D, self-paced): Spatial and engineering reasoning, trial and error
• Portal (first-person puzzle, 3D, mostly self-paced): Spatial reasoning and an unfamiliar mechanic
• Slay the Spire (roguelike deck-builder, 2D, turn-based): Pure strategy and probability, no time pressure
• TrackMania (racing, 3D, very fast real-time): Reaction speed, where slow thinking hurts most

The Results So Far

Even the best model averages 53 out of 100, against 90–100% on the static tests above. Given time to think, models do well. But, when pausing isn’t possible and speed is a factor it gets almost impossible for the agents. We’re hoping to see gains in this direction in the coming years.

• GPT-6 Astra: 52.97 overall score, 325,564 credits used
• GPT-6.1 Sol: 34.91 overall score, 48,303 credits used
• Gemini 4 Argon: 32.30 overall score, 43,252 credits used
• Claude Opus 5.5: 31.68 overall score, 91,716 credits used
• Gemini 3.8 Flash: 10.96 overall score, 23,323 credits used
• Claude Haiku 5.5: 8.93 overall score, 12,522 credits used

Astra leads by 18 points, but it used about 7× the credits of Sol.

Go Explore

Game Gym grew out of years of our own learnings and tens of thousands of runs. Explore every game and model at evals.nunu.ai. We'll keep adding models as they launch.

Which model, or which game, should we test next?

Sources

• Vals.ai benchmarks, ProofBench, IOI
• TechCrunch on Vals (founded 2024)
• BenchLM GPQA leaderboard
• OpenAI: Why we no longer evaluate SWE-bench Verified
• Kili Technology: AI benchmarks in 2026 (MMLU)
• MindStudio: GPT-6 Astra release overview
• Game Gym leaderboard
• GPT-6 Astra Launch Page
• Artificial Analysis: MMLU-Pro
• PricePerToken: MMLU