A chess engine that beats every human grandmaster alive cannot tell you why a joke is funny. A model that writes clean Python will happily invent a case citation that doesn’t exist. Both are extraordinary tools. Neither is general, and the gap between them and something truly general is not a matter of degree.
Ask what makes an AI system general and you tend to get one of two answers: a parameter count, or a vague wave toward “human-level intelligence.” Neither is much use. Generality is a property of behaviour under unfamiliar conditions, and it can be probed, at least roughly. Here is what actually separates a general system from a very good narrow one.
Generality Is a Range, Not a Size
Scale helps. A model trained on more varied material picks up more varied competence, which is part of why GPT-4 cleared the Uniform Bar Exam around the 90th percentile while earlier systems stumbled on it. But size and generality come apart constantly.
Why parameter counts mislead
A 70-billion-parameter model fine-tuned exclusively on insurance claims is narrow the way a scalpel is narrow. Meanwhile a smaller model with broad training data and a decent tool-use harness can book a flight, debug a script and summarise a lease in the same afternoon. What matters is the distribution of inputs the system handles competently, not the number on the spec sheet.
There’s a labelling problem too. Any model that produces fluent language looks general at first glance, which is exactly the trap AGI vs generative AI: the difference that actually matters unpacks. Fluency is one skill. Generality is a claim about the whole set of them.
Five Traits That Actually Signal Generality
1. Transfer to tasks it was never trained on
Real generality shows up on problems nobody prepared the system for. Hand it an unfamiliar format, an unfamiliar domain, a constraint it has never encountered. If performance collapses to random, the earlier competence was pattern-matching against a familiar shape.
2. One set of weights, many domains
This is where multi-task systems get interesting. DeepMind’s Gato ran 604 different tasks, from stacking blocks with a robot arm to captioning photographs, through a single network. A genuine engineering achievement. It was also, by most per-task measures, behind the specialists. Being one model for everything is necessary for generality and nowhere near sufficient for it.
3. Composition
Ask a system to plan a three-day trip for someone with a peanut allergy who refuses to fly, in a city hosting a major conference. You are testing whether it can combine constraint handling, geography, food knowledge, scheduling and a feel for cost. Composition is the trait people underestimate most, because each ingredient is easy on its own and the combination is where narrow systems come apart.
4. Learning from very few examples
A child sees three dogs and knows what a dog is. Most machine learning needs thousands of examples to get there. In-context learning, where a model picks up a new task from two or three examples inside the prompt, closed part of that gap when GPT-3 demonstrated it in 2020. It’s a real capability and a fragile one: rephrase the examples and the results wobble.
5. Pursuing open-ended goals
A general system should be able to take an objective it wasn’t designed around and work toward it across many steps without falling off a cliff at step forty. This is the weakest area for current models. Long-horizon tasks drift, earlier constraints get forgotten, the thread is lost. Session amnesia doesn’t help either. Close the chat window and yesterday’s context is gone unless somebody engineered a memory layer.
Benchmarks Are a Poor Proxy
Scores on MMLU, GSM8K and HumanEval get cited as proof of generality, and they are weaker evidence than they look. Test sets leak into training data. Formats get gamed. A model can hit 90% on a benchmark while failing a lightly reworded version of the same question.
There’s a second problem: the definitions move. Every time a benchmark falls, someone argues it was never a proper test of intelligence in the first place. That isn’t always bad faith, but it does mean a leaderboard position tells you less than the launch post implies. Knowing how to spot AGI hype is a decent filter for exactly this.
Generality Arrives on a Spectrum
You also have to be careful about which kind of generality is being claimed. A system that’s excellent inside one field but broad across tasks within it is a different animal from one that’s shallow everywhere. The classic strong AI vs weak AI distinction is a useful starting point, and a proper look at what AGI actually means covers the definitional mess in more depth. The practical takeaway: treat generality as a dial with many settings, not a certificate a system either has or lacks.
What’s Still Missing
Today’s general-purpose models are genuinely broad in language-heavy work and strikingly brittle almost everywhere else. A short list of the gaps:
- Sample efficiency. Picking up a new concept still takes far more data than a person needs.
- Calibration. Models are poor at knowing what they don’t know, which matters more as they get more autonomy.
- Continual learning. Updating on new information without forgetting the old is unsolved at scale. Retraining is the usual workaround.
- Grounding. Physical causality, spatial reasoning and real-world consequence remain hard-won, task by task.
- Long-horizon reliability. An agent that’s 95% accurate per step is roughly 60% accurate across ten steps, and most real work runs longer than ten steps.
None of that is a reason for cynicism. It’s a reason to evaluate claims concretely, which is where the harder discipline of responsible AI practice begins: documenting what a system can’t do, and testing it against the conditions your users will actually create.
A Practical Checklist for Judging Generality Claims
Next time a vendor deck or a paper calls a system general, run it past these.
- Does it hold up on a task outside its training distribution, in a fresh format?
- Can it reuse a skill from domain A to solve a problem in domain B?
- Does it stay coherent over a hundred steps, or only five?
- Will it tell you when it’s guessing?
- Did it learn the new task from three examples, or did it need a fine-tuning run?
- Does it keep working when the world shifts slightly?
Systems that pass most of that list are rare, and the ones that fail all of it are still often worth buying. Generality isn’t the only thing that matters. It’s just the claim people make most loosely, and the one that most repays being checked.