OpenAI has spent a decade telling anyone who’ll listen that AGI is the most important thing it will ever build, and that safety is the reason the company exists at all. Those two claims have never sat comfortably together. One points toward slowing down, verifying, refusing to ship. The other points toward bigger training runs, more compute, and getting there first.
Both things are true at once inside the same organisation. That tension is the actual subject of OpenAI AGI safety, and it’s more useful to understand than any single press release.
What OpenAI Means When It Says AGI
The company’s charter defines AGI as “highly autonomous systems that outperform humans at most economically valuable work.” Fourteen words that quietly do an enormous amount of work, because they set the threshold at which safety obligations, governance rules, and Microsoft’s IP rights all change.
Notice what the definition doesn’t require. It says nothing about consciousness, nothing about general reasoning across arbitrary domains, nothing about a system that can act in the world without supervision. A model that clears human performance on a large basket of paid tasks would qualify. That definition deserves closer reading than it usually gets, because the safety commitments are written against it.
There’s a second wrinkle. Under the original Microsoft agreement, once OpenAI’s board determines AGI has arrived, Microsoft’s access to the technology is cut off. A safety-relevant judgment call is also, in that clause, a commercial one. Who makes it, and on what evidence, matters more than the paperwork suggests.
The Preparedness Framework Is Real, and It Is a Self-Audit
Published in December 2023, the Preparedness Framework is OpenAI’s main pre-deployment safety process. It tracks four risk categories: biological and chemical, cybersecurity, persuasion and manipulation, and model autonomy. Each new frontier model gets scored Low, Medium, High, or Critical in each area.
Medium means deploy with monitoring. High means the Safety Advisory Group has to sign off and the board gets told. Critical means the model doesn’t get developed or deployed at all until mitigations are proven to work. In principle, that’s a real stop sign.
What scorecards catch, and what they don’t
Capability evaluations are a young science. Benchmark contamination is a persistent problem, since training data scraped from the web can include the very test questions being used to measure danger. Evals run in controlled conditions, then deployment happens in a messier world with users who improvise. A model that scores Medium in March might behave differently inside a product with tools, memory, and multi-turn agent loops by June.
The bigger structural issue is that the company grading itself is the company that profits from a passing grade. Third-party access exists, and evaluators like METR and the UK AI Safety Institute have run pre-deployment tests. Their findings mostly stay private.
The Superalignment Bet, and Why It Lost
In July 2023, OpenAI announced a team to solve the hardest technical problem in the field: how do you supervise a system smarter than the people checking it? The plan was to spend 20% of the company’s compute over four years on the problem, with a team of roughly thirty researchers. Ilya Sutskever and Jan Leike led it.
By May 2024 it was gone. Sutskever left to found a new lab. Leike resigned publicly, writing that safety culture had “taken a back seat” to shipping, and that the compute promise had been hard to obtain. OpenAI confirmed the team was dissolved and its members redistributed across other groups.
Worth being precise about what failed here. The technical bet, largely built on ideas like weak-to-strong generalisation and scalable oversight, didn’t produce a decisive breakthrough. That’s a research outcome, not a scandal. The governance question is different: a public, quantified commitment about resources disappeared with no public accounting of where the compute went. If you’re trying to follow what OpenAI’s research roadmap actually contains, that gap between announcement and delivery is the pattern to watch.
Safety Work That Happens at the Product Layer
Plenty of OpenAI’s safety effort is less dramatic than alignment theory and more concrete. The Model Spec, published in 2024, sets out behavioural rules for the assistant. Deliberative alignment trains reasoning models to recall and reason about those rules before answering, rather than relying on patchy fine-tuning. Monitored chain-of-thought gives researchers a window into how a model reached a conclusion. Classifiers and abuse detection catch misuse, and human review teams handle the rest.
This is where most users actually experience safety, and it’s also where the gap between demo and deployment bites. A model answering a question on a livestream is not the same object as a system with tool access, persistent memory, and an API key tied to your credit card. The gap between the chatbot and the AGI it’s supposed to become is where a lot of risk quietly accumulates.
The Compute Race Turns Safety Into a Race Condition
Safety commitments only bind if stopping is affordable. OpenAI has signed compute and infrastructure deals in the tens and now hundreds of billions of dollars, with data centre buildouts designed to train models several orders of magnitude larger than GPT-4. That five-rung, half-a-trillion-dollar staircase creates a structural incentive to keep climbing. A pause on a single training run is no longer just a delay; it’s a write-down against infrastructure that’s already been paid for.
One detail worth remembering. OpenAI’s charter once said the company would stop competing if a value-aligned, safety-conscious project got close to AGI first. In 2024, leadership reaffirmed that. In 2025, reporting suggested the commitment was being softened as the competitive pressure from other labs grew. That’s a useful marker of how stated principles bend under conditions.
External Pressure Is Doing More Than Internal Policy
Real constraints on OpenAI AGI safety come largely from outside the building:
- Regulation. The EU AI Act’s rules for general-purpose AI models, including systemic-risk obligations for the largest systems, apply to OpenAI regardless of where it’s headquartered.
- Government testing. Pre-deployment access agreements with the UK AI Safety Institute and, in the US, the Center for AI Standards and Innovation give regulators an early look, even if results stay confidential.
- Legislation OpenAI opposed. California’s SB 1047 would have required safety testing and an emergency kill switch. OpenAI wrote to the bill’s author in August 2024 arguing it would slow innovation and that federal rules were better. The bill was vetoed.
- Litigation and leaks. Lawsuits, former employee statements, and subpoenas in various proceedings have surfaced internal documents that no one planned to publish.
None of these are a substitute for a company genuinely able to say no to itself. They do raise the cost of saying yes to something it shouldn’t.
How to Read the Next Safety Announcement
When OpenAI publishes its next framework update or safety pledge, apply a few filters before the news cycle moves on:
- Is it quantified? “Significant compute” means nothing. “20% over four years” means something, and can be checked.
- Is it about a named model or an abstract future one? Commitments tied to specific systems get tested. Commitments to “AGI” never come due.
- Who can actually halt a launch? Find the name, the role, and the reporting line. A safety body that advises the CEO is not a safety body that can overrule the CEO.
- Has anyone ever been blocked? In the public record, no OpenAI deployment has been cancelled for reasons disclosed under the Preparedness Framework. Absence of evidence isn’t proof, but it’s a data point.
The honest summary of OpenAI’s position is that it has built real safety infrastructure, staffed some of it with serious researchers, published more than most competitors, and repeatedly restructured or deprioritised that work when it got expensive. Separating the roadmap from the marketing is a job that never ends, because the two are updated on different schedules. The frameworks are worth watching. The financial commitments behind them tell you which one actually runs the company.