AI May Well KILL Your Writing (whether or not it kills us all)

You probably heard the news recently that the tech bros building the major AI tools — including the ones most used by writers, ChatGPT and Claude — are predicting anywhere from a 10% to 65% likelihood their creations will wipe out humanity in the next 10 years.

Then, this week, the CEO of chipmaker NVIDIA disputed those claims, asserting there’s a 0% chance of the AI apocalypse any time soon, saying such claims are “not grounded in science.”

But amid the p(doom) hype, you may have missed some observations over the last year or so about AI’s impacts on thinking and writing — possibly even more frightening than extinction (to writers serious about their craft).

The AI Dangers to Writers, Grounded in Science

We’ll start with a June 2025 study from the MIT Media Lab, “Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task.”

The MIT Study

(Note: as the study authors did, I’ll say if you hate reading about the study itself, feel free to skip to “Accumulating Cognitive Debt” below.)

A summary of the methodology helps in grasping the results. The researchers divided 54 participants aged 18-39 into three groups and assigned them an essay writing task. One group was allowed to use ChatGPT (LLM Group), one was allowed to use search engines (Search Group), and one was not allowed to use either tool (Brain-Only Group).

The study ran through four sessions, with the participants given a choice of three topics in each session in the form of a question and 20 minutes to produce an essay. In each of the first three sessions a different set of topic questions were given, so a total of nine possible topic choices were available to each participant and they each ended up with three essays.

In Session 4, the participants were individually allowed to choose from the three topics they’d written about in the first three sessions and assigned to write about it again. Their groupings also changed, however, with Brain-Only participants from Sessions 1-3 switching to the LLM Group and vice versa.

The participants brains were monitored with an EEG throughout the sessions.

After each session, participants were asked about their essay, including whether they could quote any sentence from it; whether they could summary the main points or arguments they’d made; and for the LLM/Search groups how much of the essay was taken from ChatGPT/Search and how much was their work. After Session 4, in addition to those questions, they were asked whether they remembered their previously chosen topics and what they’d previously written.

The Results, Sessions 1-3

The results are startling. On the question whether they could quote any sentence from their essays, after Session 1 only 16.7% of the LLM Group could do so. For both the Search and Brain-Only Groups, 88.9% could.

On the ability to summarize their points and arguments, none of the LLM Group could, whereas 83.3% of the Search Group and 88.9% of the Brain-Only Group did so.

When it came to how much of the essay was their own, 50% of the LLM Group claimed full ownership (despite their failures just noted), 33.3% reported partial ownership (ranging from 50%-90% their own), and 16.7% admitted they had no ownership at all of what was written.

Nobody in the Search or Brain-Only Groups reported zero ownership of their work. The Search Group showed fewer claims of full ownership (33.3%), with 66.7% reporting partial ownership. And the Brain-Only Group stood out on this question, with 88.9% full ownership and 11.1% partial.

As the researchers noted, they expected those gaps to narrow after Sessions 2 and 3, “as the participants now knew what types of questions to expect, specifically with respect to our request to provide quotes.” On the ability to quote from their essays, the LLM and Search Groups both scored 88.9% after Session 2 and 72.2% and 100%, respectively, after Session 3. But the Brain-Only Group scored 100% after both Sessions 2 and 3.

Similar narrowing occurred on the other interview questions, though there was a slight drop-off in the LLM Group scores from Session 2 to 3. It’s worth emphasizing that the Brain-Only Group scored no lower than 88.9% on these questions across all three sessions, with several 100% scores.

Results, Session 4

Then came Session 4. At the outset, all those from the original Brain-Only Group were able to recall all of the topics they’d written about in Sessions 1-3. Of the original LLM Group, only one-third of them could recall all three of their previous essays.

After switching roles and writing their fourth essay, this time on a repeat topic, however, only 22.2% of those original Brain-Only participants who’d now been reassigned to the LLM Group could quote from their essay and only 11.1% could summarize their points or arguments. Not great, but these are actually small improvements over the original LLM Group’s Session 1 results.

Of those reassigned to the Brain-Only Group, 88.9% quoted a sentence and 77.8% summarized their points or arguments. Good, but that 77.8% score represents a sharp drop-off from the 88.9% scored by the original Brain-Only Group in Session 1.

On the ownership question, the results were similar to Session 1, although nobody admitted feeling no ownership. Of those reassigned to the LLM Group, 55.6% claimed full ownership of the essay and 44.4% partial. Those reassigned to the LLM Group reported 88.9% full ownership, 11.1% partial.

Accumulating Cognitive Debt

The study authors offer a number of findings from their test results. From Sessions 1-3, they’re summed up this way:

“Taken together, the behavioral data revealed that higher levels of neural connectivity and internal content generation in the Brain-only group correlated with stronger memory, greater semantic accuracy, and firmer ownership of written work. Brain-only group, though under greater cognitive load, demonstrated deeper learning outcomes and stronger identity with their output.”

In contrast,

“The Search Engine group displayed moderate internalization, likely balancing effort with outcome. The LLM group, while benefiting from tool efficiency, showed weaker memory traces, reduced self-monitoring, and fragmented authorship.”

They described these differences in the participants’ memory, accuracy, and ownership of their work as “the same hierarchical pattern: Brain-only group > Search Engine group > LLM group.” Alluding to the impact in educational settings, they wrote:

“This trade-off highlights an important educational concern: AI tools, while valuable for supporting performance, may unintentionally hinder deep cognitive processing, retention, and authentic engagement with written material. If users rely heavily on AI tools, they may achieve superficial fluency but fail to internalize the knowledge or feel a sense of ownership over it.”

The findings from Session 4, after Brain-Only participants from Sessions 1-3 switched to the LLM Group and vice versa, tell a worrisome story. For those who’d used AI through the first three writing sessions, “removing AI support significantly impaired” their ability to quote from or discuss their arguments in their fourth essay. despite it being on the same topic as one of their earlier writings.

Combining the “quoting failure” and the EEG results and comparing them for the LLM-to-Brain with the Brain-to-LLM Groups, the authors found evidence that:

Early AI reliance may result in shallow memory encoding
Withholding AI tools during early stages might improved memory formation
For the Brain-to-LLM Group, “metacognitive engagement” (recalling their previous Brain-Only essay and comparing it with the AI suggestions in Session 4) resulted in “stronger performance and more cohesive neural signatures”

A further finding that the authors labeled preliminary and in need of confirmation with a larger sample size has garnered support in commentary from the lived experience of writer and Yale writing instructor Meghan O’Rourke. The MIT study authors pointed to results from the LLM-to-Brain Group showing that they “repeatedly focused on a narrower set of ideas” and “may not
have engaged deeply with the topics or critically examined the material provided by the LLM.” As a result, “their writing might become biased and superficial.”

The authors coined the term “cognitive debt” for this over-reliance on the AI output:

“This pattern reflects the accumulation of cognitive debt, a condition in which repeated reliance on external systems like LLMs replaces the effortful cognitive processes required for independent thinking.”

I like that description of writing as an “effortful cognitive processes required for independent thinking.”

The Brain-Only results in Sessions 1-3 and Brain-to-LLM in Session 4 also demonstrate that engaging in that effortful cognitive writing process produces better learning and stronger short and longer term memory of the material.

If independent thinking and learning are what you give up by using AI to write material that in the end “might become biased and superficial” the cost is too high.

More Science

After studying the MIT paper, which is still labeled “in review,” I asked Perplexity the following:

“What other research can you find that examines the impacts of AI use in writing on the cognitive abilities of human users, the diversity of their thinking and writing, the writing process as a thinking tool, and the “ownership” (mastery) of the output? What other concerns should be examined when writers rely on AI tools?”

Perplexity responded: “I found about 20 relevant studies. Taken together, they support O’Rourke’s concerns better than the single MIT study does.” We’ll get to Meghan O’Rourke’s concerns in a moment, I promise.

I’m not going to discuss 20+ more studies here. After all, I’m urging you to engage in your own “effortful cognitive process.” So I’m going to list the studies Perplexity found, with links, so you can go check the AI’s work — and my conclusions — for yourself. I think the titles of the papers make my point quite well:

I know it can feel like you’re being more efficient and productive using AI in your writing. But is it worth producing more, if your writing gets shallower, becomes homogenous with everyone else’s, and you feel less ownership of the ideas?

The authors of #3 above summed up what they found comparing written advice on a variety of topics by writers using AI versus standard web searches to gather information this way:

“… compared with gathering and synthesizing information via standard search links, the lower effort involved in gathering information from LLM syntheses can lead people to develop shallower knowledge on a topic in certain contexts. When they subsequently form advice on the topic based on what they learned from their LLM (vs. web) search, people are less personally invested in forming their advice, and write advice that is shorter, contains fewer references to facts, is less original to them, and is ultimately less likely to be adopted by recipients.”

AI’s “Fake News” Problem

The last paper on the list raises another danger: AI’s penchant for making up sources for the answers it gives you. The problem seems to flow from that “sycophancy” issue, where the AI is so desperate to please that it makes up both facts and sources.

I’ve flagged examples before of lawyers being sanctioned by courts for filing briefs with invented case law and quotes of judges’ opinions, but a new one hit the news this month. The New Mexico Supreme Court fined a lawyer $5,000 for filing a brief that “contained false testimony from wholly fabricated witnesses.”

The lawyer claimed he did not know that AI could hallucinate facts, saying he’d fed the trial transcript into ChatGPT and expected to receive “a bulletproof summary” of the testimony. What he got was “fictional statements that the shooter was wearing dark pants and a white shirt.” He pled “honest mistake.”

At his hearing, an incredulous justice asked,

“Counsel, do you watch the news? Do you listen to the radio? Do you read anything about what’s going on in the world? Because the problem with lawyers relying on AI hallucinations is an ‘above-the-fold’ story every single day.”

The justice may be understating the volume, since a web page dedicated to tracking AI hallucination cases lists 2,077 cases from 22 countries as of September 24, 2026, and includes 53 cases so far this month!

And the problem may be getting worse. Amid the AI apocalypse news we started with, we saw “OpenAI reports 6 new instances of ‘concerning model behavior’ since March.” Most relevant here, the bad behavior included “two training examples of models uploading files to the internet so they could cite them as relevant answers to human evaluators.”

If our main way to verify the AI’s research output has been to make sure the source exists and that it says what the AI claims, what do we do when the AI can create a fake document or page and upload it to the internet as part of it’s “thinking” process? Do we now have to contact the human author of the document or website to see if it’s real?

Or do we just avoid using AI when our writing matters?

Meghan O’Rourke and the Writer’s Mind

After her NYT opinion piece (paywalled) on the effects of AI on writing and learning, O’Rourke added some thoughts on her Substack. She opened by describing her experience in writing the book she was working on:

“It’s been an onerous few writing weeks of learning to let go of parts of the book that are not working … there is a this profound sense of heavy lifting and real uncertainty when you are at the point of a book where meaning has not yet taken shape. It’s a place I find really painful to inhabit, and yet, paradoxically, I know I often need to go through it to get to real insight.”

She then raises her concern that using AI in writing will “erase this painful experience of ‘sitting with’ uncertainty.” She explains:

“And when it comes to writing itself, I both love and hate the fact that I need to puzzle the bulk of a draft out on my own. Writing is how I make sense of the world. I don’t outline easily. I don’t always know what I think until the structure begins to emerge from the sentences. Writing is part of how I metabolize experience; often, it is the way I come to understand some of my deepest ideas and feelings about life.”

She goes on to describe the challenges and experiments in her teaching, including more in-class writing “to foster the kind of present slowness that having AI at home might foreclose.”

In December last year, she spoke at Dartmouth “On Creativity, Teaching, Writing, and Generative AI” and once again I urge you to read it for yourself. But I’ll leave you with three of the recommendation she made for teachers of writing that I think we can all apply to ourselves to limit the impact of AI on our own writing:

Emphasize process over product.
Normalize the challenge of the writing/learning process.
Offer spaces that remove the temptation to use Gen AI: Reading rooms, writing labs.

Consider how you can adapt these to your writing. Let writing remain that slow, sometimes painful process for reaching your deepest understanding of your ideas and the world around you.

AI May Well KILL Your Writing (whether or not it kills us all)

AI May Well KILL Your Writing (whether or not it kills us all)