Ask ten AI researchers to define AGI and you will get roughly eleven answers. Ask them how they would test for it and the argument gets sharper still. That is the awkward position agi intelligence tests find themselves in: everyone agrees we need them, nobody agrees what they should measure, and the benchmarks that exist keep getting flattened faster than the field can build replacements.
The past three years have produced a strange loop. A new benchmark appears, framed as the one that will finally separate general intelligence from clever pattern matching. Models score near zero. Eighteen months later they are at 85 percent and the benchmark quietly retires. Understanding why that keeps happening tells you more about AGI than any single leaderboard.
What an AGI test would actually have to measure
François Chollet’s framing is the most useful starting point. He argues that intelligence is not skill. It is the efficiency with which you acquire new skill. A calculator has skill. A chess engine has skill. Neither can walk into an unfamiliar domain and get good at it quickly.
That distinction matters, because most benchmarks measure skill of a specific kind. They check whether a system can answer graduate-level physics questions or solve competition math. Impressive, genuinely hard, but they score the output rather than the learning machinery behind it. For a deeper walk through that machinery, there is a solid explainer on the mechanisms behind general-purpose AI that covers memory, reasoning loops, and transfer between domains.
The benchmarks we actually have
ARC-AGI and the case for efficiency
The Abstraction and Reasoning Corpus, released by Chollet in 2019, is built on a deliberately awkward premise. Each puzzle shows a handful of coloured grid transformations and asks you to infer the rule from two or three examples. Prior knowledge is nearly useless. You cannot have seen the answer before, because the answer is a rule invented on the spot.
Humans solve most ARC puzzles in under a minute. For years the best AI systems hovered around 20 percent on the original set. OpenAI’s o3 hit roughly 76 percent on the semi-private ARC-AGI-1 evaluation in December 2024, at about $20 per task. Then ARC-AGI-2 arrived and frontier scores fell back into single digits. That oscillation is the design working as intended: the test was built to lose to humans and keep losing.
Humanity’s Last Exam and the saturation problem
In January 2025 the Center for AI Safety and Scale AI released Humanity’s Last Exam, about 3,000 questions written by roughly 1,000 domain experts. Early frontier models scored in the low single digits. Within a year, top systems were clearing 20 percent and climbing. That is fast movement on a benchmark explicitly designed to be the last one.
GPQA went through the same arc. So did FrontierMath and MMMU. The pattern has a name, benchmark saturation, and it has a consequence. Once a test becomes the target, labs optimise against it, and the score stops telling you what it used to.
Goodhart’s law and the contamination problem
Charles Goodhart’s 1975 observation about economic indicators applies here with unusual force: when a measure becomes a target, it ceases to be a good measure. Benchmarks leak into training data. Test questions surface in scraped web content. Labs iterate on evaluation sets, sometimes deliberately, more often through the ordinary pressure of leaderboard incentives.
Contamination is the quiet killer. If a model has seen the answer during pre-training, its score reflects recall rather than reasoning. Benchmark authors now keep sets private, rotate questions, and generate fresh items on the fly. The arms race never really ends.
There is a definitional trap underneath all this. The line between narrow and general keeps sliding as tools get more capable, which is why the argument over where AGI ends and narrow AI begins keeps restarting. Testing for general intelligence needs a stable target, and the target is precisely the thing nobody can pin down.
A different approach: measuring task horizon
METR, a nonprofit evaluation group, took a sidestep that may be more informative than any static exam. Instead of asking what a model knows, they ask how long a task it can complete autonomously. Their 2024 paper measured the length of tasks, in human-expert hours, that models could finish with 50 percent reliability. In 2023 that number sat around a few minutes. By 2025 it ran into hours for reasoning-heavy work. METR’s estimated doubling time is roughly seven months.
A capability curve is harder to game than a pass/fail score. You cannot memorise your way into finishing a forty-hour software project, at least not yet. It also maps onto what deployment actually looks like: an agent that reliably handles a two-hour task is a different product from one that handles eight.
Why a single AGI test probably will not exist
The idea of one clean test, a Turing Test 2.0 that settles the question, is appealing and almost certainly wrong. General intelligence is a bundle: transfer to novel domains, long-horizon planning, calibration about its own uncertainty, learning from sparse feedback, social reasoning, self-correction without a human in the loop. No single number captures that bundle, and any attempt to compress it gets gamed within a release cycle.
There is also genuine disagreement about whether the whole target is even coherent, covered well in the real arguments for and against human-level AI. If AGI means “does everything a human can do,” then no finite test can confirm it, because the space of human tasks is unbounded.
What a practical AGI test suite looks like
The realistic answer is a battery rather than an exam, with each component built to resist memorisation:
- Novel-domain learning. Drop a system into a game, tool, or rule set it has never seen and give it two or three demonstrations. ARC-AGI-2 is the current reference point.
- Long-horizon autonomy. Multi-hour tasks with checkpoints, where partial credit reflects real progress rather than one lucky final answer.
- Cost-adjusted performance. A score means nothing without a token or compute budget attached. Solving a task for $60 is not the same as solving it for $0.06.
- Calibration. Does the system know when it does not know? Overconfident wrong answers are more dangerous than admitted uncertainty.
- Cross-modal transfer. Skills gained in one domain applied in another without retraining, which is exactly where today’s AI capabilities tend to break down under real workloads.
- Contamination controls. Rotating test sets, held-out private splits, and questions generated after the training cutoff.
For a concrete sense of what passing such a suite would look like, it helps to study examples that count as AGI and ones that do not, because a lot of what gets called general turns out to be narrow in a better costume.
What to watch instead of scores
Numbers will keep climbing. They nearly always do. The more useful signals sit underneath: how fast a model picks up a genuinely new skill, whether it can work for hours without drifting, whether it recognises the edge of its own competence, and what all of it costs per task.
A benchmark hitting 90 percent tells you a lab solved the benchmark. A system that learns an unfamiliar tool in twenty minutes, uses it accurately for six hours, flags three points where it needs input, and does the whole thing for under a dollar tells you something real about general intelligence. The field is slowly shifting attention from the first thing to the second, and that shift is the story worth tracking.