Study: Readers Rate AI Stories Higher, Until They Learn a Machine Wrote Them

A new study finds that readers rate AI-generated short stories higher than human-written ones on both quality and immersion, but the moment readers are told a story came from a human, their scores climb regardless of who actually wrote it. The research, titled "Bot or not: Can people tell the difference between stories written by a human or by an AI system?", was conducted by Sydney Sears and Dr. Deena Skolnick Weisberg of Villanova University's Department of Psychological and Brain Sciences and published in the journal Judgment and Decision Making, according to The Decoder and a university press release on EurekAlert.

How the study worked

The researchers used three stories published by human authors in a literary journal or short story collection, paired with ChatGPT-generated versions of the same stories. In the first experiment, 1,682 participants each read one story and were told, correctly or incorrectly, that it had been written by a human or by AI, then rated the story on quality and how immersive they found it. Two follow-up experiments, with 424 and 481 participants, gave readers one human-written and one AI-generated story side by side, without labels, and asked them to guess which was which.

What the numbers show

On a scale from minus 3 to plus 3, AI-written stories scored an average of 1.54 for quality against 0.97 for the human-written stories, and 1.42 for immersion against 1.00. But labeling mattered more than authorship. Across the experiments, a story labeled as human-written was rated more generously than the same story labeled as AI-written, regardless of which one had actually produced it. In the blind comparisons, readers performed no better than chance at correctly identifying which story was AI-generated, though each one-point increase in a participant's self-reported AI expertise raised their odds of guessing correctly by about 14 percent.

Dr. Weisberg said the pattern "reveals a bias towards narratives written by real people," adding that people tend to assume creative writing requires uniquely human qualities such as emotional understanding and lived experience, which leads readers to underestimate what AI-generated text can do on the page.

Why this matters for you as an author

This study lands next to a string of publishing controversies this year in which authors have had book deals paused or canceled over unproven AI suspicions, simply because a story read as unusually polished. The Villanova findings complicate that instinct: readers cannot reliably tell AI prose from human prose by feel alone, and the label attached to a piece of writing shapes its reception more than the writing itself does.

For your own work, the practical takeaway is not that polish is dangerous. It is that voice, not surface fluency, is what still separates a memorable book from a competent one, and voice is exactly what a generic AI draft does not have until you shape it. Our guide on how to edit and humanize AI-generated book content and our deeper look at what readers really prefer between AI and human writers both walk through how to keep that voice intact through revision.

PageWriter Studio is built around that same idea: AI drafts the outline and chapters from your input, and you revise every page with a style profile trained on your own voice, so the finished manuscript reads like you, not like a prompt. If you want to try that workflow, you can start a free trial and see how it fits your next book.