Every few months a new number lands and the internet briefly loses its mind. A model clears 87% on a puzzle set designed to resist memorisation. Another posts a score above the human expert average on graduate-level physics questions. The headline writes itself: we’re nearly there.
Then nothing changes about your actual life, and the conversation quietly moves on. That gap between a benchmark chart and a working system is the whole story of OpenAI’s AGI benchmarks, and it’s worth understanding which tests are genuinely informative, which are marketing, and which ones tell you something uncomfortable about how we measure progress at all.
What OpenAI Actually Means by AGI
OpenAI’s charter defines AGI as “a highly autonomous system that outperforms humans at most economically valuable work.” That’s a strange sentence when you sit with it. It’s not a capability claim, it’s an economic one. It doesn’t say the system is conscious, or general, or good at everything. It says the system is cheaper and better than people at most of the jobs people get paid to do.
No single benchmark tests that. So OpenAI does what everyone else does: it assembles a portfolio of proxy tests, each measuring a slice of the thing, and lets the aggregate stand in for the whole. That’s a reasonable engineering compromise and a terrible basis for a public claim, which is why the question of whether ChatGPT counts as AGI depends entirely on which definition you picked up first.
The Benchmarks Everyone Cites
ARC-AGI: The One That Made People Sit Up
François Chollet’s Abstraction and Reasoning Corpus is a set of visual grid puzzles that reward learning a new rule from two or three examples. Humans score around 85%. GPT-4 managed single digits. Then in December 2024, OpenAI’s o3 hit 87.5% on the semi-private evaluation set.
The catch arrived in the footnotes. That performance cost roughly $20 per task and ran through thousands of dollars of compute for the full set. The ARC Prize team was explicit that this was a genuine reasoning result, not a lookup trick, but also that brute-force search at inference time is a different proposition from efficient intelligence. A useful reminder that a score without a cost curve is half a result.
FrontierMath and GPQA: Specialist Knowledge
FrontierMath, built by Epoch AI with working mathematicians, consists of original problems that haven’t leaked into training data. When o3 launched it cleared around 25%, up from roughly 2% for earlier models. That’s dramatic movement on a benchmark where a research mathematician might grind for hours.
GPQA Diamond covers PhD-level biology, physics and chemistry. Top models now sit above the scores of domain experts answering with web access. This is real, and it’s also narrow. A model that can answer a hard physics question is not the same as a model that can run an experiment, notice the equipment is miscalibrated, and decide what to do next.
Humanity’s Last Exam: Designed to Humiliate
A 2,500-question set written by experts specifically to resist saturation. The best models have been crawling through the 20s, with tool use pushing higher. When a test is explicitly built so that no one can score well on it, a low number tells you almost nothing about capability. It’s a saturation-resistance device, not a progress meter.
The Stopwatch: METR’s Time Horizons
The most interesting measurement work isn’t coming from OpenAI directly. METR, a non-profit evaluator, tracks the length of task, measured in human working time, that a model can complete with 50% reliability. That horizon has been doubling roughly every seven months for several years.
Framing capability as a duration rather than a percentage is genuinely clarifying. If a model can reliably handle a task that takes a skilled person 30 minutes, you can imagine deploying it for a slice of work. If the horizon reaches eight hours, you’re talking about replacing whole job functions rather than accelerating them. The gap between thinking longer and thinking bigger shows up here in practical terms: a reasoning model that spends 40 minutes of compute on a five-minute task hasn’t extended its horizon, it’s just spent more.
Benchmarks With Money Attached
The field has shifted toward tests with direct economic meaning, which is the right direction given OpenAI’s own definition.
- SWE-bench Verified asks models to resolve real GitHub issues from open-source Python repositories. Top systems now clear 70%+, up from around 2% in 2023. It’s the closest thing to a standardised software-engineering test anyone has.
- GDPval, released by OpenAI in 2025, scores models on real deliverables from 44 occupations, judged blind by human professionals against human work. Early results put frontier models at or above human quality on a meaningful share of tasks, with the failures clustered where judgement and context matter.
- PaperBench tests whether a model can replicate a published ML paper from scratch, including the code, the experiments and the write-up. Nobody scores well. That’s the point.
Where the Five Levels Fit
OpenAI has its own internal scale, first shared in July 2024 and still loosely used: Chatbots, Reasoners, Agents, Innovators, Organisations. o1 and o3 were placed at Level 2. Agents were flagged as Level 3. Nothing so far has been publicly certified as Level 4.
This is a taxonomy, not a benchmark, and it’s worth being honest about the difference. The five-rung climb toward AGI has never come with a published schedule, and each level is defined by capability adjectives rather than measurable thresholds. When OpenAI says a model has reached Level 3, you’re being told a story, not shown a score.
The agentic benchmarks are where this gets tested hardest. Web-browsing and computer-use evaluations measure whether a model can complete a multi-step task in a live environment without breaking, which is much closer to the economic definition than any question-answering set. Watching a model book a flight, handle a captcha and recover from a failed payment teaches you more than a leaderboard row. There’s a useful breakdown of what browsing agents actually demonstrate about AGI if you want the detail.
Why High Scores Keep Not Adding Up to AGI
There are structural reasons the numbers overshoot, and they’re not conspiracy, just statistics.
- Test contamination. Public benchmarks end up in training data. ARC-AGI sidesteps this by keeping a private set; most tests don’t bother.
- Benchmark-shaped thinking. Models get tuned by teams who can see the eval. That’s not cheating, it’s optimisation, and it means the eval measures the tuning as much as the capability.
- Reliability collapse. A 90% score often means 90% on a curated distribution and considerably less on messy real inputs. Long task chains multiply small failure rates until success odds approach zero.
- Missing the middle. Benchmarks test the hardest problems and the easiest ones. The vast middle of ordinary work, which is where the economy lives, gets skipped.
Those blind spots are exactly the gap between demo performance and dependable output that almost never appears in a launch deck, and it’s why practitioners tend to discount benchmark jumps until they’ve used the thing for a week.
What to Watch Over the Next Year
Three signals are worth tracking more closely than any single leaderboard. First, cost per completed task on ARC-AGI or its successors: a 90% score at $20 a pop and a 90% score at 20 cents describe two different worlds, and only one of them is deployable. Second, METR’s time horizon curve, which has been the most honest predictor of real-world usefulness so far. Third, human-preference evaluation on long, open-ended deliverables, where a model has to hold context for hours and make judgement calls nobody wrote into a rubric.
OpenAI’s own stated AGI definition gives the cleanest test available. Find a measure of economically valuable work being completed end to end, at human or better quality, reliably, at a price that undercuts a salary. That benchmark doesn’t exist yet. When it does, you won’t need a chart to tell you the result changed.