A toddler knocks a glass off the table and flinches before it lands. Nobody taught her the trajectory. She has a rough, fast, useful simulation running behind her eyes, one that already knows glasses fall, floors are hard, and the sound arrives after the shatter.
Researchers call that internal simulator a world model. For the past few years a growing share of the AI field has decided that teaching machines to build one is the shortest route to systems that can plan, drive, and pick things up in physical space.
The term started as academic shorthand. Then Fei-Fei Li’s World Labs raised $230 million for it in 2024, Google DeepMind shipped Genie 2, and Nvidia built an entire CES keynote around Cosmos. Suddenly “world models AI” was on every roadmap deck in the industry.
What a world model actually is
Strip away the branding and a world model is a learned representation of how an environment behaves. Give it the current state and a proposed action, and it predicts the next state. Chain those predictions together and you get a rollout: a cheap, imagined future you can plan inside.
The 2018 paper that popularised the name, David Ha and Jürgen Schmidhuber’s “World Models,” trained a small network to play racing games entirely inside its own hallucinations. It never touched the real game while learning. That party trick still explains the appeal. If a model can simulate, you can train, test, and stress-test at a fraction of the cost of doing it in reality.
The three pieces every world model needs
- An encoder that compresses messy reality (pixels, LiDAR, joint angles) into something compact enough to reason over.
- Dynamics, the part that predicts how that compressed state evolves under an action. This is the actual world model, and it’s where most of the difficulty lives.
- A decoder or value head that turns the imagined state back into something useful: a reconstructed image, a predicted reward, or a planning score.
Why not just use a language model?
Large language models are extraordinary at the statistics of text. But text is a lossy description of reality. A language model can write a convincing paragraph about a glass falling; it cannot tell you where the shards scatter, because that information was never in the training data. If you want a deeper walkthrough of where those systems genuinely shine and where they fall over, our breakdown of how large language models work and where they break covers the ground.
World models usually work in a different substrate: video frames, sensor streams, point clouds. Video has become the default because it is the cheapest large-scale recording of physics humanity has ever produced. Hundreds of millions of hours of it are sitting on public platforms, showing objects falling, water pouring, fabric folding, and people walking through doorways.
The machinery underneath is mostly familiar. If the phrases encoder, latent space, and attention feel foggy, this explainer on what’s really happening inside a deep learning black box is a decent place to start.
Three approaches, three philosophies
Video generation as simulation
OpenAI’s Sora, DeepMind’s Genie 2, and Nvidia’s Cosmos all bet that a good enough video generator is a de facto world model. Genie 2 will take a single reference image and generate a playable 3D environment you can move through with keyboard input, holding the scene together for roughly a minute. A minute is not much. Two years ago it was zero.
Latent prediction and the JEPA line
Yann LeCun has argued for years that predicting exact pixels is mostly wasted effort. Most of a frame is unpredictable detail: leaf positions, creases in a shirt, the precise blur of a passing car. His Joint Embedding Predictive Architecture predicts in abstract representation space instead, asking what is going to happen rather than what every pixel will look like. Meta’s V-JEPA 2, a 1.2-billion-parameter model trained on over a million hours of video, shipped in 2025 with zero-shot robot arm demos attached.
Model-based reinforcement learning
The quiet workhorse. The Dreamer family of agents learns a compact world model and then trains its policy entirely inside the imagination, which is why DreamerV3 could learn to collect diamonds in Minecraft from scratch without human demonstrations or hand-tuned rewards. Wayve’s GAIA-2 applies the same instinct to driving, generating controllable multi-camera scenes to test a self-driving stack on situations that are too rare or too dangerous to film.
Where this stops being research
Four places are already feeling it.
- Autonomous driving. Generating rare edge cases, a child chasing a ball from behind a parked van, is far cheaper than waiting to encounter one.
- Robotics. Simulated rollouts let a grasping policy fail ten thousand times before it ever touches a real gripper.
- Industrial operations. Digital-twin style forecasting is creeping into maintenance and logistics planning, which is the logic behind these 90-day enterprise AI rollout plans for operations teams.
- The edge. Compressed, task-specific simulators are small enough to run on a drone or a phone, part of the same push toward efficiency driving the rise of deliberately tiny language models.
The failures nobody has fixed yet
Compounding error is the big one. A model with 2% per-step error looks fine for ten steps and nonsense after a hundred. Small mistakes feed into the next prediction and snowball until the imagined world has drifted somewhere the real one never goes.
There’s also the cheating problem. Models trained on video learn the visual grammar of a bouncing ball without learning the physics. They produce a plausible arc, then fumble contact, object permanence, or anything involving counting. Ask a video model to show a hand with six fingers turning a Rubik’s cube through a full rotation and you’ll see exactly how much is understanding and how much is pattern completion.
Then comes the question of trust. A simulator that drives your test conclusions needs to be inspectable, not just impressive, which is why the field’s debate over what explainable AI actually requires matters more for world models than for chatbots. If you can’t tell whether a generated scenario reflects reality or a training-set artefact, the whole evaluation collapses.
And the uncomfortable questions arrive quickly. A convincing simulator of human behaviour is also a machine for generating synthetic people, synthetic evidence, and synthetic consent. That’s not a hypothetical for the far future. It’s why responsible AI has to be a set of hard questions rather than a signed PDF.
The metric that will actually tell us something
Pretty demo videos are cheap now. The test that matters is transfer: does a policy trained inside a world model perform measurably better in the real world than one trained without it, on a task nobody curated for the demo? Wayve, Google, and a handful of robotics labs are publishing those numbers. Most are modest. A few are real.
Watch for action-conditioned data, not just video. A model that predicts the next frame is a video generator. A model that predicts the next frame given what the agent chose to do is a world model, and it needs paired action labels that are far rarer than raw footage. That dataset gap, not the architecture race, is probably what sets the pace over the next two years.