AI Advice Cut People's Accuracy to a Third and Doubled Their Confidence. It Also Nearly Deleted 'I Don't Know.'

TL;DR

A new PsyArXiv preprint gave people trivia questions, handed some of them AI advice that was deliberately configured to be mostly wrong, and watched what happened to their answers. The people using AI got worse and surer at the same time. Accuracy fell from 27% to 9%, self-rated confidence rose from 30% to 76%, and the share who admitted I don't know collapsed from 44% to 3%. Paying them real money for correct answers barely helped. The result is not that AI is dumb. It is that bad AI advice quietly removes the human habit of doubt, which was the exact thing supposed to catch the AI when it is wrong.

What they actually measured

The study comes from Valerio Capraro at the University of Milano-Bicocca, Chiara Marcoccia at the École Normale Supérieure, and Walter Quattrociocchi at Sapienza University of Rome. Participants answered questions about small visual details from films, the kind of thing you either noticed or did not, such as the color of the team uniforms in Bend It Like Beckham. The researchers picked questions that current models tend to get wrong.

Then came the twist that makes the design honest. The AI assistant was set up to give mostly incorrect answers on purpose. That matters, because it rules out the boring explanation: if people had simply deferred to a reliable tool and scored higher, that would be rational delegation, not a cognitive failure. Here the tool was bad, so any gain in confidence is confidence in nonsense.

The researchers configured the model to be wrong on purpose, which, depending on how your week of shipping LLM features has gone, may or may not count as a controlled laboratory condition.

The numbers

The headline pair is the crossover. Without AI, people were right 27% of the time and rated their confidence at 30%. With the (wrong) AI, they were right 9% of the time and rated their confidence at 76%. Accuracy dropped by roughly two thirds while confidence more than doubled. As Capraro put it, People became much worse, the accuracy was only one third, but they were twice as confident.

"I don't know" was the real casualty

The most striking number is not accuracy, it is the disappearance of honest uncertainty. Without AI, 44% of people were willing to say they did not know. With the AI, that fell to 3%. Three of the most useful words in engineering, and one session with a confident bot nearly bred them out of the sample.

The mechanism is old, and it has a name: automation bias, the tendency to trust a confident machine over your own judgment. It behaves like a passenger with a very sure sense of direction. You stop reading the map, they turn out to be wrong, and you only notice when the road ends at a lake.

The researchers even tried to buy the doubt back. When they paid people for correct answers, the willingness to say I don't know recovered only from 3% to 8%, and accuracy from 9% to 16%. Both stayed far below the no-AI baseline. Incentives helped a little; they did not undo the effect.

Why this should bother anyone building with AI

You already know your model hallucinates. The working assumption behind most human-in-the-loop designs is that a person will catch it. This study is a direct measurement of how well that assumption holds, and the answer is: worse than you think, because the AI does not just supply the wrong answer, it also suppresses the reviewer's instinct to question it.

That has concrete design consequences. A confident, fluent completion is the most dangerous kind, because fluency is exactly what disarms the checker. If your product presents model output as a crisp verdict with no visible uncertainty, you are optimizing for the failure mode this paper measured. Surfacing calibrated confidence, forcing an explicit "verify this" step, or making the model show its sources are not UX niceties; they are the guardrails standing between your users and a 76% confidence in a 9% answer.

It also lands on how teams use coding agents and research assistants. The value of a skeptical senior reviewer is not just catching bugs, it is being unbothered by a slick explanation. This is a reminder to keep that skepticism a deliberate, structured step rather than a vibe, because the vibe is precisely what erodes first.

The caveats, kept straight

This is a preprint on PsyArXiv, not yet peer reviewed, so treat the exact figures as provisional pending review and replication. The task was one narrow genre, visual film trivia chosen specifically because models fail at it, which is not a stand-in for the coding, drafting, or analysis work most builders actually delegate. The AI was rigged to be wrong, so this measures overreliance on a bad assistant, not the everyday case of a mostly-good one.

One more honest gap: the secondary coverage disagrees on which model was used to generate the bad advice, so we are describing it by its behavior (deliberately unreliable) rather than naming it. The direction of the effect is consistent with a wider literature, including a 2025 Microsoft and Carnegie Mellon survey that linked higher confidence in AI to lower self-reported critical thinking. The size and cleanliness of this effect are what make it worth your attention.

Key Takeaways

• In a new PsyArXiv experiment, wrong AI advice cut accuracy from 27% to 9% while raising confidence from 30% to 76%.
• Willingness to say "I don't know" fell from 44% to 3%, the study's most consequential number.
• Paying people for correct answers recovered "I don't know" only to 8% and accuracy only to 16%, both far below baseline.
• The design deliberately used an unreliable AI, so this isolates a cognitive effect (automation bias) rather than rational delegation.
• For builders: fluent, confident output is the most dangerous kind, because fluency is what disarms the human reviewer you are counting on.
• It is an unreviewed preprint on a narrow trivia task, so weigh the mechanism more than the precise percentages.

Sources: PsyArXiv preprint (Capraro, Marcoccia, Quattrociocchi), The Register, The Next Web, Microsoft/CMU on AI and critical thinking (CHI 2025)