COLM Study of 1,222 People: 10 Minutes of GPT-5 Help Lowered Unaided Solve Rates in 3 Trials

TL;DR

AI Assistance Reduces Persistence and Hurts Independent Performance, now a peer-reviewed COLM 2026 paper, reports three randomized controlled trials with 1,222 US participants. People who got a GPT-5 sidebar for roughly ten minutes of fraction or SAT reading problems solved fewer problems and skipped more of them once the sidebar was taken away, compared with people who never had it. The gap replicated across both domains, and it was largest among the 61% of AI users who admitted they mostly asked the model for the answer. The authors' read: an assistant that never says no trains you to stop trying.

What they actually ran

The team is Grace Liu at Carnegie Mellon, Brian Christian at UC Berkeley's Center for Human-Compatible AI, Tsvetomira Dumbalska at Oxford, Michiel Bakker at MIT, and Rachit Dubey at UCLA. The preprint first appeared in April, but the version presented at COLM in San Francisco this week is the revised, reviewed one, and Berkeley News pushed it out on 9 October. The experiment code is on GitHub.

The design is deliberately boring, which is the point. Participants recruited on Prolific got a run of problems with a skip button and no pay penalty for wrong answers, so a skip is a choice not to engage rather than a cost calculation. Half were randomly given an AI sidebar for the learning block; the other half worked alone. Then, without warning, the sidebar disappeared and everyone did three identical test problems unaided.

The sidebar was GPT-5, pre-prompted with each problem and its solution. Typing "answer?" worked. Which, to be fair, is also how most of us use it at 11pm.

The numbers

Experiment 1 (354 recruited, 307 analyzed, fractions): on the three unaided test problems the AI group's mean solve rate was 0.57 against 0.73 for controls (Cohen's d = -0.42, P < 0.001). Their skip rate was 0.20 against 0.11 (d = 0.25, P = 0.031). During the twelve assisted problems the AI group had been ahead on both measures, so this is not a weaker cohort. It is the same people, minus the crutch.

The authors flag a confound in that first run: exclusions were based on in-experiment performance, which could have kept weaker participants in the AI arm because the AI was answering for them. So Experiment 2 (667 recruited, 585 analyzed) added an unassisted pretest for exclusions and gave the control group its own disappearing sidebar (pretest worked solutions they had already seen), so both arms experienced the same interface shock. Exclusion rates came out nearly identical (11.7% AI, 12.9% control).

The effect shrank but held: solve rate 0.71 versus 0.77 (d = -0.19, P = 0.020). The aggregate skip-rate difference (0.10 versus 0.07) did not reach significance. The interesting part is what happened when they split AI users by how they used it.

Asking for answers is the problem

At the end of Experiment 2, AI-arm participants self-reported their usage, cross-checked against chat logs. 61% said they mostly asked for answers directly, 27% used it for hints or clarification, and 12% barely touched it. The groups were statistically indistinguishable on the pretest. On the unaided test, they were not.

Answer-seekers scored 0.65 against 0.77 for controls (d = -0.36, P < 0.001), skipped more (0.13 versus 0.07, P = 0.033), and, uniquely among the four groups, declined relative to their own pretest (mean change -0.10). Hint-seekers landed at 0.76, right on top of the control group. The authors are careful here: this split is cross-sectional within the randomized arm, so it is suggestive, not causal. People who prefer shortcuts may also be people who quit sooner. But the randomized arm-level result does not depend on it.

It is not a fractions thing

Experiment 3 (201 recruited, 168 analyzed) swapped arithmetic for SAT reading comprehension, five assisted passages then three unaided. Same shape: solve rate 0.76 versus 0.89 (d = -0.42, P = 0.007), skip rate 0.08 versus 0.01 (d = 0.42, P = 0.008). An answer submitted in under five seconds counted as a skip, since nobody reads a passage that fast.

Why it happens, per the authors

The paper proposes two mechanisms. The first is a reference-point shift they compare to hedonic adaptation: once a task routinely takes seconds, the same task done by hand feels unreasonably expensive, so you offload again, which moves the reference point further. Think of an escalator that is switched off. The stairs did not get steeper, but your legs are convinced they did, because the last ten rides set the price of climbing at zero.

The second is lost self-knowledge. Productive struggle is where you find out what you can do, and that calibration is what keeps you going on the next hard thing. Skip the struggle and you never learn you were capable, so quitting feels rational. Both mechanisms predict the effect compounds, which is exactly the part this study cannot measure.

Caveats, straight

• Brief exposure, immediate test. Participants were tested seconds after losing the AI. Whether the dip lasts an hour, a day, or evaporates is unknown, and the authors say so.
• Maximally helpful AI. The sidebar was seeded with the correct solution. A Socratic or hint-only assistant was not tested; the hint-user subgroup only hints that it might behave differently.
• Experiment 1's confound. The headline 57%-vs-73% gap comes from the run with the attrition problem. The cleaner replication found six points, not sixteen, and its skip-rate difference was not significant in aggregate.
• Online sample, school-style tasks. US Prolific workers paid about $15 an hour doing fractions and SAT passages for 13 to 15 minutes. Christian's line that "this is not a story about fractions and SAT problems" is a hypothesis, not a result.
• Effect sizes are small to medium. Cohen's d ranges from 0.19 to 0.42. Real, replicated, and not apocalyptic.

Why builders should care

If you ship anything with an assistant in a sidebar, this is a product finding, not just a psychology one. The one usage pattern that did no measurable harm was hint-seeking, and the default UX of every chat assistant steers users toward the pattern that did. The paper's own recommendation is blunt: "band-aid" interventions like usage caps will not fix an incentive that rewards instant, complete answers; the objective itself has to include what the user can do afterwards.

It also lines up with what the labs are finding in their own backyard. Anthropic's January coding-skills trial had 52 developers learn a Python library with or without AI help; the AI group scored 50% on the follow-up quiz against 67% for the hand-coders, and the developers who asked conceptual questions retained more than those who delegated. Different task, same split between asking to understand and asking to be done. For your own tooling, the practical move is to make "explain the approach" as cheap a click as "give me the code," and to measure what your users can still do when the model is down. There is some irony in a language-modeling conference publishing the best case yet for occasionally closing the chat window. The reviewers accepted it anyway.

Key Takeaways

• Three randomized trials (1,222 recruited) found that about ten minutes with a GPT-5 sidebar lowered unaided solve rates and raised skip rates once the assistant was removed, in both fractions and SAT reading.
• The cleanest replication puts the solve-rate gap at six points (0.71 vs 0.77); the first run's sixteen-point gap includes an attrition confound the authors acknowledge.
• The damage concentrates in the 61% of users who asked for direct answers: they scored 12 points below controls and declined from their own pretest, while hint-seekers matched the no-AI group.
• The paper was presented this week at COLM 2026 in San Francisco as a peer-reviewed paper, after circulating as an April preprint.
• Limits: effects were measured immediately, the AI was maximally helpful, and effect sizes are small to medium (d = 0.19 to 0.42). Durability and accumulation are open questions.
• For product teams, the actionable lever is interaction design: make hints and explanations the path of least resistance, and track what users can do without the model.

Sources: arXiv 2604.04721 (COLM 2026 version), project page, Berkeley News, EurekAlert release, experiment code, COLM 2026, Anthropic: How AI assistance impacts the formation of coding skills