a bit whimsical on a Saturday and decided to ask ChatGPT a fairly simple question: “Who won the Nobel Prize in Physics in 2025?”
ChatGPT responded immediately: “The 2025 Nobel Prize in Physics was awarded to…” It even provided names, research areas, and an explanation of the specific research that earned them the Nobel Prize!
There was just one problem—a very small one, actually. The Nobel Prize had not yet been announced. Yet, the model did not hesitate; it did not pause; it surely did not say, “I don’t have enough information” or, even better, “the Nobel Prize winner for 2025 has not been announced yet!”
Instead, it confidently walked into the room, sat down, and delivered fiction with the energy of someone defending a PhD dissertation. As a person who once defended a PhD, I wish I had ChatGPT’s confidence when it makes stuff up!
As humans, we tend to do something interesting with confidence; we associate it with correctness, but that’s not always the case. If someone says, “I think the answer might be 42” and another person says, “The answer is absolutely 42,” most of us instinctively trust the second person more, even when both are equally likely to be wrong. For us, confidence sometimes acts as a useful signal of correctness. For AI systems, however, confidence can be a surprisingly unreliable narrator.
In this article, we will explore why.
Confidence Feels Like Probability
Let’s say we asked an LLM to predict what animal it is in a given picture. It says:
Cat: 0.97
Dog: 0.02
Bird: 0.01Most will interpret that as: ”The model is 97% sure this is a cat.”
That is a reasonable assumption. Unfortunately, that is often not what those numbers mean. We need to remember that many AI models use a function called Softmax to generate predictions.
The Softmax function converts raw outputs (called logits) into values that sum to one and resemble probabilities. The important thing to notice here is the exponential term, which can cause small differences to suddenly become very large.
So basically, the model isn’t saying, “I have overwhelming evidence that this is a cat.” It may simply be saying: “Among these options, cat happened to win by a small margin.” Those are very different statements with completely different meanings.
Humans and AI Handle Uncertainty Differently
Though it might be uncomfortable to sit with, humans are surprisingly good at expressing and dealing with uncertainty.
We constantly hear: “I might be wrong…”, “I’m pretty sure…”, “Maybe…”, or “I think…”. Our confidence tends to exist on a spectrum. AI systems, however, often behave like that one person in a group project who confidently explains something they learned three minutes ago (I am sure we all had that classmate…).
So, when chatting with an LLM, both telling it “I think Paris is the capital of France,” and it responding “Paris is the capital of France with 99.8% probability,” gives the same energy as telling it “I think Atlantis is fictional,” and it responding “Atlantis is located approximately 400 miles west of Portugal with 98.7% confidence.”
Although the two cases have very different outcomes, the LLM treats them equally.
The Confident Fool Problem
This creates what I think of as the confident fool problem. Where a system can be spectacularly wrong while sounding spectacularly certain. And unfortunately, confidence often increases exactly when we would prefer more caution.
This becomes especially noticeable when LLMs encounter situations outside their training distribution.
Suppose we train an image classifier to identify cats and dogs. But then we decided to give it a picture of a toaster! Ideally, the model should say, “I have absolutely no idea what this is.” What would be the response of most people when shown something they have never seen before? Instead of saying that, the model might respond:
Dog: 98%
Cat: 2%Now, unless your toaster is poodle-shaped, that answer is clearly false!
Why does this happen? The answer is simpler than most people think. Simply, it happens because the model was never trained to say: “None of the above.” So, when it encounters something unfamiliar, it chooses the highest available score among the options.
It is like forcing someone to answer “What fruit is this?” while pointing at a bicycle. Eventually, they will choose a fruit just to resolve the situation and say, “Banana?”
Let’s simulate a model that is overconfident.
If the model reports “90% confidence”, we would hope it is correct roughly 90% of the time. Instead, many systems look more like “90% confidence, 65% accuracy.” This gap between confidence and accuracy is why the way we choose to train these LLMs matters a lot.
Teaching Models to Be More Honest
Okay, we know why models tend to be so confidently wrong, but how can we overcome that to have better models with higher accuracy, or accuracy that matches their confidence? This is where calibration comes into play.
Calibration does not necessarily improve predictions. Instead, it improves honesty! So, if a model says 90% after calibration, it should mean: “Historically, predictions at this confidence level were correct about 90% of the time.”
Methods such as:
- Platt Scaling
- Temperature Scaling
- Isotonic Regression
attempt to align predicted confidence with observed outcomes.
Let’s see what this looks like:
Why This Matters
It is easy to laugh when an AI thinks a toaster is a dog. Because that is, arguably, very funny. However, many less funny situations exist. Not just less funny, but critical, and maybe even life-threatening. Using LLMs in medical diagnosis systems, autonomous vehicles, fraud detection, and financial forecasting requires high accuracy.
If a model tells a doctor: “Cancer probability: 99%” or “Cancer probability: 62%,” the doctor’s response will vary significantly!
If confidence scores are poorly calibrated, people may trust predictions that do not deserve trust. And humans are particularly vulnerable here because confidence feels persuasive. Even when we know better.
As models continue moving into real-world workflows, we may need to stop asking: “How accurate is the model?” and start asking: “When the model says 90%, does it actually mean 90%?” Because there is a difference between a smart model and a trustworthy model.
Humans are not perfect at uncertainty, either. We become overconfident all the time. We think we can finish a project in two days. We think we can assemble furniture without reading the instructions. We think we only need one trip from the car to bring in groceries. Even when history suggests otherwise.
Maybe AI is simply inheriting some of our bad habits? The difference is that when humans are confidently wrong, usually only a few people suffer. When AI is confidently wrong, the mistake can scale to millions, and confidence at scale is a very different problem.
Final Thoughts
For years, we have measured AI progress by asking increasingly impressive questions:
Can it write code? Can it generate art? Can it pass exams? Can it reason?
Those questions are useful, but they can sometimes distract us from a more important one:
Can we trust it?
A model producing the right answer once is exciting. A model that produces the right answer repeatedly while knowing when it might be wrong is something entirely different. Reliability rarely creates flashy headlines.
Confidence itself is not the problem. The problem begins when confidence becomes a performance rather than a meaningful measure of certainty. As AI systems continue moving into healthcare, education, finance, research, and decision-making pipelines, we may need to stop treating confidence scores as truth meters and start treating them as estimates that require validation.
Because a model sounding certain is easy, where a model knowing when not to be certain may be one of the hardest problems we still have left to solve.