🎧 Listen to this article (5:58)
Spend a little time listening to children in a school playground, and you’ll hear all sorts of confident statements. Some are true, some are exaggerated, and others are completely wrong. Yet the children making those claims often sound absolutely certain.
One child repeats something they’ve heard from an older sibling. Another insists something is true because a friend told them. Before long, an unlikely story can become accepted as fact, particularly if the child telling it is popular, persuasive or simply louder than everyone else. We don’t suddenly grow out of this behaviour as adults. We just get better at sounding convincing.
Who gets challenged?
I’ve experienced conversations where someone confidently makes a statement I know is wrong. What interests me isn’t just the mistake, but how we decide whether to correct it.
If the person is your boss, you might hesitate to challenge them. If it’s a close friend, you might gently point out the error. If you dislike them, you might be surprisingly quick to prove them wrong. Sometimes you remain quiet because correcting someone isn’t worth the disagreement.
The information hasn’t changed. What changes is our willingness to challenge the person delivering it.
Psychology offers two useful explanations for how confidence and credibility can become disconnected from actual knowledge.
Think of those memorable auditions on The X Factor or America’s Got Talent, where someone appears convinced they’re an extraordinary singer and seems genuinely surprised when the judges disagree. The Dunning–Kruger effect describes a related possibility: people with limited ability in a particular area may struggle to recognise their own shortcomings. It doesn’t explain every unsuccessful contestant, but it illustrates how confidence can exist without competence.
The halo effect works from the other direction. When we admire someone for one quality, we may assume they’re equally capable in other areas. For example, we might treat a successful businessperson as an authority on subjects they know little about.
When AI sounds convincing
People make honest mistakes, misremember information and sometimes bluff to protect their reputation. Being sincerely mistaken isn’t the same as deliberately pretending to know.
Artificial intelligence introduces a different problem. It can generate fluent, convincing explanations that include fabricated information, even when no one intends to mislead. These errors, commonly called hallucinations, can be hard to recognise when we don’t know much about the subject.
In 2023, New York lawyer Steven Schwartz used ChatGPT to research legal precedents for a case involving the airline Avianca. The resulting court papers cited at least six judicial decisions that didn’t exist.
Schwartz also produced screenshots showing him asking ChatGPT whether the cases were real. It assured him they were real and could be found in established legal databases. The lawyers continued to stand by the fabricated authorities even after the court questioned their existence. In June 2023, the court imposed a $5,000 penalty jointly on both lawyers and their firm.
The failure wasn’t simply using AI. It was relying on fabricated information without independently verifying it, then continuing to defend it after serious doubts were raised. Asking the same AI whether its own answer is correct isn’t independent verification.
Unlike questioning a boss or correcting a friend, challenging AI carries little social discomfort. Yet we may still treat it as an authority because it sounds knowledgeable. The halo effect can influence us here too, when we mistake polished communication for factual reliability.
There’s a further complication. Some AI systems can become overly agreeable, changing their answers to accommodate our objections even when their original responses were correct.
We shouldn’t accept an answer just because AI sounds confident, but we also shouldn’t accept a revised answer just because it agrees with us.
Learning to question without becoming an expert
You don’t need to be an expert in medicine, finance, science or history before asking AI questions. Part of its value is helping us explore subjects we don’t yet understand. But there’s a difference between using AI to learn something and accepting everything it tells us as gospel.
If an answer contains an important statistic, quotation or factual claim, ask where the information came from. Check the original source when possible and see whether it supports the explanation. Even a convincing citation isn’t necessarily reliable.
Asking another AI can help identify mistakes or alternative interpretations, but agreement between models doesn’t establish the truth. They might be relying on the same incorrect information.
Of course, verifying every answer would make AI exhausting to use. An inaccurate film recommendation might waste an evening, while incorrect medical, legal or financial information could have much more serious consequences. The amount of checking should reflect what’s at stake.
AI developers have responsibilities too. Their systems should communicate uncertainty rather than present guesses as established facts.
How do you know?
I would rather hear someone say, “I’m not sure,” than listen to a convincing explanation built on information they haven’t checked. There’s nothing wrong with making an educated guess, as long as we recognise it as a guess.
Perhaps the most useful question we can ask, whether we’re talking to another person or an AI, is simply, “How do you know?”
It asks for evidence rather than turning disagreement into a contest over who is right. It’s also a question worth asking ourselves when we’re tempted to speak with more confidence than our knowledge justifies.
