Close Menu
AI News TodayAI News Today

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Copilot Pro Real World Testing: 47 Days, 312 Prompts, and an Honest Scorecard

    DeepMind’s AGI Benchmarks: What the Scores Actually Measure

    ChatGPT for Students: Which Courses Are Worth Your Time (and Which Are Just Slideshows)

    Facebook X (Twitter) Instagram
    • About Us
    • Contact Us
    Facebook X (Twitter) Instagram Pinterest Vimeo
    AI News TodayAI News Today
    • Home
    • AI News
    • AI Reviews
    • AI Tools
    • AI Tutorials
    • Chatbots
    • AGI
    • Artificial Intelligence
    AI News TodayAI News Today
    Home»Artificial General Intelligence (AGI)»DeepMind’s AGI Benchmarks: What the Scores Actually Measure
    Artificial General Intelligence (AGI)

    DeepMind’s AGI Benchmarks: What the Scores Actually Measure

    By No Comments8 Mins Read
    Share Facebook Twitter Pinterest LinkedIn Tumblr Reddit Telegram Email
    DeepMind's AGI Benchmarks: What the Scores Actually Measure
    Share
    Facebook Twitter LinkedIn Pinterest Email

    Google DeepMind publishes a lot of numbers. In a single quarter you might see a model score on a competition maths set, a percentage on an abstraction puzzle, a win rate in a 3D game, and a table of “critical capability levels” that reads like a risk register. All of it gets flattened into headlines about progress toward AGI.

    The trouble is that these benchmarks measure very different things, and several were never built to answer the question people are asking. Knowing what each one actually tests is the difference between reading a research update and being sold a story. The strategy behind those numbers is covered well in this breakdown of Google DeepMind’s AGI plan; here the focus is narrower and more mechanical: the benchmarks themselves, and what they do and don’t tell you.

    AGI is not a single number, and DeepMind knows it

    In November 2023, a DeepMind team published Levels of AGI, a paper that did something useful and slightly unfashionable. It refused to define AGI as a threshold score. Instead it split the concept along two axes: performance, running from Emerging through Competent, Expert and Virtuoso to Superhuman; and scope, running from narrow to general. Autonomy is tracked separately again, because a system that needs a human to prompt it every thirty seconds is a different object from one that pursues a goal for a week.

    That framing matters because it makes an obvious point easy to see. Within DeepMind’s own taxonomy, AlphaFold sits at Virtuoso or Superhuman on a narrow axis. It is also utterly incapable of anything outside protein structure prediction, and nobody at the lab pretends otherwise, Nobel Prize or not. A benchmark score is always a coordinate, never a destination.

    The benchmarks DeepMind actually puts weight on

    BIG-bench and its harder descendants

    Beyond the Imitation Game Benchmark, released in 2022, pulled together 204 tasks from researchers at 132 institutions. It was deliberately scattershot, covering everything from logical deduction to translating invented languages. The interesting part came after: models improved so fast that BIG-bench Hard was carved out specifically to keep only the tasks that frontier systems were still failing. That recycling pattern has repeated ever since, and it is one of the more honest signals of progress in the field. The underlying research culture that keeps producing these evaluations is worth understanding on its own terms, as this piece on Google AGI research and the papers behind it sets out.

    ARC-AGI: the abstraction puzzle

    François Chollet built the Abstraction and Reasoning Corpus in 2019 while at Google. Each task is a handful of coloured grid puzzles with a couple of worked examples, and the model has to infer the rule. No prior training on the exact format helps much. It has become the most quoted single benchmark in AGI arguments, largely because it is cheap to understand and brutally hard to fake.

    ARC-AGI-2 arrived in 2025 and reset the scores downward by design. For context on how quickly the ground moves, models sat in the low single digits on the new set at launch and Gemini variants have since climbed well past that. If you want the longer argument about whether Google’s flagship model crosses the line, this assessment of whether Gemini is AGI is a reasonable place to start.

    Frontier mathematics and the exam-style tests

    FrontierMath, built by Epoch AI, consists of original research-level problems written to resist both guessing and memorisation. Models scored under 2% when it launched in late 2024. Humanity’s Last Exam, released in January 2025 with roughly 2,500 expert-written questions, produced similarly humbling first results in the low single digits before climbing through the year. These tests are useful because the questions are hard to contaminate, and because a private held-out subset exists precisely because labs have an incentive to train on anything public.

    Agentic suites: SIMA, GATO and game arenas

    DeepMind’s most distinctive benchmarking work isn’t on paper exams. GATO, from 2022, trained a single network on 604 distinct tasks spanning text, images and robotic control. SIMA pushed further into interactive territory, learning to follow natural-language instructions inside commercial 3D game environments and accumulating hundreds of skills across multiple titles. In 2025 the lab ran a public chess exhibition through Kaggle’s Game Arena, pitting models against each other in a setting where the rules are fixed and the outcome is unambiguous. These evaluations matter because they test sequences of decisions rather than single answers.

    AlphaProof: a benchmark of one

    At the 2024 International Mathematical Olympiad, AlphaProof and AlphaGeometry 2 combined for 28 out of 42 points, a silver-medal performance. AlphaGeometry 2 alone cleared around 84% of a curated set of geometry problems drawn from the competition between 2000 and 2024. It’s a narrow result, and DeepMind framed it that way. Still, formal proof is one of the few domains where a machine’s output can be checked completely, which makes it unusually trustworthy as evidence of reasoning rather than pattern matching.

    From task lists to cognitive faculties

    In March 2025, DeepMind published a paper called Measuring Progress Toward AGI: A Cognitive Framework. Rather than another task suite, it proposed ten faculties to evaluate: perception, generation, attention, learning, memory, reasoning, metacognition, executive functions, problem solving and social cognition. The suggested method is comparison against human baselines, including children at different developmental stages.

    The key word is faculties, plural. The paper implicitly argues that you cannot average a strong memory score with a weak metacognition score and call the result a percentage of AGI. Executive function, in particular, is where current systems look thinnest, and it is barely touched by a multiple-choice exam.

    Every benchmark eventually gets eaten

    Saturation is the norm. MMLU, once a respectable proxy for general knowledge, now produces scores clustered so tightly near the ceiling that differences of a point or two are noise. GSM8K, a grade-school maths set, went from a genuine hurdle to a formality in under three years.

    Contamination is the quieter problem. If a benchmark’s questions live on the public web, they are in the pretraining corpus, and a model can score well by recalling rather than reasoning. Labs increasingly rely on private sets, fresh questions and post-cutoff data. Goodhart’s law applies with force here: once a benchmark becomes a marketing target, it stops being a good measure of the thing it was built to measure.

    Safety evaluations are benchmarks too

    DeepMind’s Frontier Safety Framework, first published in 2024 and updated since, defines Critical Capability Levels across domains including cyber offence, biological and chemical risk, machine learning research and development, and autonomous operation. Models are tested against these thresholds before release, and the results appear in model cards alongside the capability scores.

    This is the half of benchmarking that rarely makes headlines. It’s also the half that determines what actually ships. A model that scores brilliantly on ARC-AGI but trips a CCL in an uplift study gets gated, delayed or wrapped in mitigations.

    How to read a DeepMind benchmark claim

    Most confusion comes from comparing numbers that were never comparable. A few checks handle the majority of cases:

    • Version. ARC-AGI and ARC-AGI-2 are different tests. So are MMLU and MMLU-Pro. A score without a version number is close to meaningless.
    • Pass@1 versus best-of-N. A single sample and a thousand samples with a verifier are wildly different claims, and labs don’t always lead with the distinction.
    • Bare model or scaffolding. Agentic wrappers, tool use and search can add tens of points. Ask what was actually run.
    • Public or held out. Benchmarks with a private subset exist for a reason.
    • Human baseline. A number means little without a reference point, whether that’s a domain expert or a member of the public.
    • Date. A benchmark published in 2021 was designed for a world two model generations old.

    The same skepticism applies across labs, not just to Google. Claims that a competitor’s model has crossed some threshold deserve identical scrutiny, as this analysis of whether ChatGPT is AGI makes clear.

    The evaluation fight that’s coming next

    Static question sets are running out of road. The open problem is measuring sustained, goal-directed behaviour over hours or days: whether a system can hold a plan together, notice it has gone wrong, and recover without a human stepping in. DeepMind’s agentic work in simulated environments points in that direction, and the cognitive framework paper names the missing pieces explicitly.

    That is also why the timeline conversation keeps shifting. Predictions move less because of any single benchmark result and more because the evaluations themselves get better at catching failure modes. If you want to track which signals to watch and which to discount, DeepMind’s AGI timeline and the evidence behind it is the more useful thread to follow than any leaderboard. The scores will keep climbing. The question worth asking is always who wrote the test, what it left out, and what the model had to do to get there.

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Previous ArticleChatGPT for Students: Which Courses Are Worth Your Time (and Which Are Just Slideshows)
    Next Article Copilot Pro Real World Testing: 47 Days, 312 Prompts, and an Honest Scorecard

    Related Posts

    Artificial General Intelligence (AGI)

    DeepMind AGI Safety: The Frameworks, Evals and Hard Problems That Matter

    Artificial General Intelligence (AGI)

    DeepMind’s Artificial General Intelligence Bet: What’s Built and What’s Still Missing

    Artificial General Intelligence (AGI)

    Is Gemini AGI? What Google’s Model Can and Can’t Do Yet

    Add A Comment
    Leave A Reply Cancel Reply

    Top Posts

    Copilot Pro Real World Testing: 47 Days, 312 Prompts, and an Honest Scorecard

    0 Views

    DeepMind’s AGI Benchmarks: What the Scores Actually Measure

    0 Views

    ChatGPT for Students: Which Courses Are Worth Your Time (and Which Are Just Slideshows)

    0 Views
    Stay In Touch
    • Facebook
    • YouTube
    • TikTok
    • WhatsApp
    • Twitter
    • Instagram
    Latest Reviews
    AI Tutorials

    Quantization from the ground up

    AI Tools

    David Sacks is done as AI czar — here’s what he’s doing instead

    AI Reviews

    Judge sides with Anthropic to temporarily block the Pentagon’s ban

    Subscribe to Updates

    Get the latest tech news from FooBar about tech, design and biz.

    Most Popular

    Copilot Pro Real World Testing: 47 Days, 312 Prompts, and an Honest Scorecard

    0 Views

    DeepMind’s AGI Benchmarks: What the Scores Actually Measure

    0 Views

    ChatGPT for Students: Which Courses Are Worth Your Time (and Which Are Just Slideshows)

    0 Views
    Our Picks

    Quantization from the ground up

    David Sacks is done as AI czar — here’s what he’s doing instead

    Judge sides with Anthropic to temporarily block the Pentagon’s ban

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    Facebook X (Twitter) Instagram Pinterest
    • About Us
    • Contact Us
    • Terms & Conditions
    • Privacy Policy
    • Disclaimer

    © 2026 ainewstoday.co. All rights reserved. Designed by DD.

    Type above and press Enter to search. Press Esc to cancel.