At least when Dave from accounting invents a statistic, he looks nervous about it. AI doesn’t sweat.
There’s a particular kind of hell reserved for the moment you cite something an AI told you, confidently, fluently, with impeccable grammar and a fake citation, in front of a room full of people who actually know what they’re talking about.
A New York attorney named Steven Schwartz discovered this the hard way in 2023, when he submitted a brief containing six entirely fictional court cases that ChatGPT had invented wholesale.
The judge was not amused. The internet was delighted.
The thing is, Schwartz’s mistake wasn’t stupidity. It was a failure to account for what researchers now call “confidence miscalibration”, the maddening tendency of large language models to deliver hallucinated nonsense with the same assured tone they use to explain photosynthesis.
There’s no verbal tell. No hesitation. No “I think” or “you might want to double-check this.” Just clean, authoritative prose, walking you off a cliff at a steady conversational pace.
“We’ve essentially built the world’s most eloquent bullshitter,” one AI safety researcher told us, asking not to be named because their employer is, predictably, an AI company. “The eloquence is the problem.
People are evolutionarily wired to trust fluent speakers. It’s why con artists spend years perfecting their delivery.”
This is meaningfully different from the problem of human overconfidence, and not just in degree. When your dumbest colleague states something wrong with conviction, you have years of data on their track record.
You know Dave is unreliable about spreadsheets. The AI has no track record you can intuit. It speaks with identical confidence whether it’s correctly solving a differential equation or inventing a senator who doesn’t exist.
The epistemological floor has been removed.
Some companies are trying to fix this. Anthropic has published research on what they call “honest uncertainty”, training models to flag their own shaky ground. OpenAI has experimented with grounding responses in real-time web search.
The results are mixed at best, and often create a new problem: models that hedge so aggressively on well-established facts that they become useless. “I’m not sure, but water might be wet” is not a helpful answer.
The uncomfortable truth is that the fix isn’t really a technical one. It’s a cultural one. We need to learn to treat AI outputs the way seasoned journalists treat a tip from an anonymous source: as a lead to be verified, not a fact to be published.
The problem is that the whole value proposition of AI tools, the reason people pay for them, the reason corporations are laying off knowledge workers to adopt them, is speed.
Verification takes time. And so the hallucination economy hums along, one confident lie at a time.
The AI Productivity Boom Is Real. So Is the Productivity Theater That Came With It.
Workers aren’t using AI to do more. They’re using it to look busier while doing less, and management has no idea.
This is the dirty secret at the center of the AI productivity narrative. Consulting firms keep publishing breathless reports about how AI will add trillions to global GDP. McKinsey says it. Goldman Sachs says it.
The reports all agree: massive productivity gains are coming. What they don’t explore, because it would undermine the whole sales pitch, is the question of who captures that productivity, and what happens when “more output per hour” becomes “same output, more personal free time, nobody tell the boss.”
Economists call this “shirking,” a word that doesn’t quite capture the ethical ambiguity of the situation.s
Is it wrong to use a tool that makes you faster and then use the extra time for yourself, rather than working more? Workers have done exactly this with every labor-saving technology in history.
The assembly line didn’t make factory workers produce cars faster and then spend the extra time on more cars.
It made them produce cars faster, and then their hours got cut, and then they went home.
The difference now is the opacity. It’s genuinely difficult for a manager to know whether a report that used to take three hours and now takes forty-five minutes reflects a productivity miracle or someone who discovered that Claude can write a first draft in ninety seconds.
About the Author
