Is AI Changing How We Write? A Reddit Analysis Tutorial

If you've ever read something online and thought, "Was that written by an AI?", you're not alone. Since ChatGPT went mainstream, there's been an explosion of AI enhanced content, with people using it to generate their posts, proofread their drafts, and even beginning to sound like AI themselves after talking to it too much.

Recent research has attempted to gauge the prevalence of AI assisted writing in different media such as biomedical publications [1] and YouTube videos [2]. In this tutorial, we'll use our new Reddit analysis platform to do the same for Reddit.

We'll walk through a real investigation, step-by-step, showing how you can go from a simple question to a nuanced answer in minutes. Our goal isn't to find bots, but to track LLM-influenced language, the characteristic writing style of AI, whether it's used by bots or by humans who've started to adopt it.

Let's get started.

Finding the AI Fingerprint in Reddit's Data

Following the recent research [2], we begin by selecting five words known to be far more common in AI-generated text: "delve", "underscore", "comprehend", "meticulous", and "intricate". As ChatGPT was released in 2022, we'll use 2021 as a baseline, though we expect content from 2022 to be mostly unaffected as ChatGPT was released November 30th and adoption in December was negligible.

Our first step is to see how often these words appeared on Reddit over the years. Using the Time Series Studio, we can quickly create a chart of their raw annual counts.

The chart shows interesting trends. The only word that hasn't grown from 2021 is "underscore", with "delve" and "intricate" seeing an expected jump from 2022 to 2023. But there's a catch: as we can see on Platform Overview, Reddit itself has grown significantly from 2021. More users and more comments mean the raw count of many words will go up even if they haven't become more popular. To account for this, we have to normalize our data.

A better metric would be a word's relative frequency or share, aka its proportion of all words used on the platform in a given year. This would tell us if a word has actually become more popular relative to others.

This normalized view is much clearer. The growth for "meticulous" and "comprehend" vanished, as their rise was an illusion of platform growth. But the spike for "delve" and "intricate" remains, confirming their growth is real and disproportionate.

To make the magnitude of growth more apparent, we can index the data. By setting 2021 as a baseline (value = 100), we can see how much each word grew relative to its own starting point.

The chart makes the platform-wide trend clear. The popularity of "delve" and "intricate" has grown more than 35% and 50% respectively since the end of 2021.

Drilling Down

We've seen a platform-wide trend, but Reddit isn't a monolith, it consists of thousands of subreddits. Where exactly did this growth happen?

To find out, we can use our Term Share Analysis tool to rank the subreddits by growth of our five target words between 2021 and 2024.

Clicking on a few of the subreddits to read public-facing descriptions makes it clear that the top is dominated by subreddits dedicated to tabletop games and roleplaying such as:r/lfgpremium, r/roll20LFG, and r/pbp. It makes a lot of sense, as LLMs are a perfect match for roleplaying and generating worldbuilding texts.

Investigating an Outlier: The Curious Case of r/wow

The top result, however, is a surprise: r/wow, the main community for the game World of Warcraft. Why would this specific gaming sub be the epicenter?

To investigate, we isolated r/wow in our Time Series Studio and plotted the occurrences of our five words just within it.

While four out of five words are flat, "delve" dramatically rose from a few hundred mentions in 2023 to over 15,000 in 2024. A sudden change like this often points to a one-off external event. Let's plot a monthly chart to confirm this.

We see that the big spike actually happened in September 2024 with other months having much fewer occurrences. To find out what was really going on, we can use the Post Finder tool to look at the raw posts like this:

How to create this: set the target year, month and subreddit, then configure a filter. In this case we want to see example posts that contain 'delve' in their main text, so we add rule 'text content' 'with substring' = 'delve' to filter

As we can deduce from posts like that, users weren't "delving into topics"; they were talking about something similar to dungeons called delves. Googling "wow delves 2024 september" explains the sudden flood of such discussions. In September 2024, a major WoW update "The War Within" introduced short-form instanced scenarios that you can run solo or in a group of 2-5 players called "delves".

This is an important lesson in data analysis: context is everything. Without drilling down into the source data and cross-checking our assumptions, we can easily misinterpret the results.

A Different Clue

The r/wow case shows that single words can sometimes be misleading. As an independent check on our findings, we can track the use of the em-dash (—), a well-known tic of many LLMs.

Let's examine a couple of larger roleplaying subreddits from our top results (r/DiscordRP, r/roleplaying) where we suspect genuine LLM influence, compared against our new control, r/wow.

The results support our theory of AI-assisted writing in roleplaying subreddits. They start the year with around 5% of posts containing em-dashes, and end it with a stunning 10%! That's more than 10x our control r/wow at year's end, which has less than 1% of such posts.

Your Turn to Explore

In this tutorial, we went from a broad question about online language to a nuanced, community-specific answer. We found clear evidence that "LLM assisted writing" is on the rise on Reddit, saw where it's most prevalent, and learned how to spot and correct for false positives.

More importantly, we've demonstrated a repeatable research workflow. By combining theTime Series Studio for trends, the Term Share Analysis for hotspots, and the Post Finder for context, you can answer your own research questions with confidence.

We've shown you the method, now it's your turn. Go forth and explore!