Is learning supposed to be hard?
If AI makes learning easier, does that defeat the point? Two studies point in different directions.
Daniel Gibbon8 January 2025Is learning supposed to be hard? If AI makes it easier, does that defeat the point?
In July 2024, researchers from the University of Pennsylvania published a paper titled "Generative AI Can Harm Learning." They found that students with access to ChatGPT performed 48% better than a control group on practice problems, but 17% worse on exams once ChatGPT was taken away. This seems like an obvious result: ChatGPT gives you the answers, and can often be wrong.
But they also tested a version set up as a tutor, and found a 127% improvement on practice problems, yet no improvement on exams once it was gone. That's a bit more puzzling.
A 2024 study out of Harvard, since published in Scientific Reports, found students learned significantly more in less time with an AI tutor, but it only measured learning immediately before and after each session.
“Assessment still forces students to struggle with retrieval and generation before getting help.”
So what's happening here? In 1994 Robert Bjork coined the term "desirable difficulties": the idea that making the learning process more difficult, in very specific ways, can result in better performance over the long run.
Maybe AI tutors make learning easier, but they do so by sacrificing desirable difficulties? Maybe we can still find a way to build them in?
Either way, I think we can unlock a huge benefit with AI feedback on student assessment, because assessment still forces students to struggle with retrieval and generation before getting help.