This Steeped Dive explores how AI and people choose numbers from 1-100 when asked to pick randomly, compared with true random number generation. There were 3 big takeaways:
- People are more random than ChatGPT: People are 4x closer to true randomness than ChatGPT, based on the Total Variation Distance (TVD) between each source and a true random number generator. That’s especially interesting given that people were nearly 3x more likely to choose 69 than true random chance. The fact that I laughed at that probably says something about my age.
- AI had some giant misses: Out of 10,000 attempts, the fact that ChatGPT didn’t pick 1, 5, 20, 50, 70 or 100 ONCE is wild. ChatGPT’s overpicks were nearly just as extreme: 29% of its picks ended in 7, and it was 6x more likely to choose 47 than true random chance would predict.
- People are random: There was only a 12% difference between people and true random, even with people’s tendency to overpick numbers like 69 and 7 and underpick numbers ending in 3. When I aggregated the choices into groups of 10, people got surprisingly close to the expected 1% chance for each number (see image below). I think this is a great example of the power of representative samples in research: individual behavior can be surprisingly messy, but those differences become measurable patterns when you aggregate enough people.
I’ve recently been on a kick that there’s a lot of hidden BS in AI, especially when it comes to data analysis. Don’t get me wrong, I use it a lot, even to help edit this article. However, I’m always cautious because there have been way too many times where AI has misled me or completely hit a wall on something important. Even with that built-in skepticism, my jaw dropped when I saw these ChatGPT random number test results on Reddit. Long story short, as I hinted above, ChatGPT completely biffed it. That sent me down a rabbit hole of exploring the dataset, comparing it to other sources, and running some tests of my own. Here are the three sources of random number picking I ended up digging into:
- ChatGPT Random Number Dataset: The star of the show is the dataset from the Reddit post I mentioned. Major props to the user who ran these tests (and shoutout to their Exmergo Viz product). They used GPT-4.1, which is already about a year old. So sure, newer models may perform better, but I still think this is a fascinating look at how an LLM behaves right out of the box when asked to be random. The prompt was “You are a random number generator. Output ONLY a single integer between 1 and 100. No punctuation, no text, no explanation. Pick a random number.” If this article does well, I’d honestly love to throw $20 at updating the experiment with today’s models and see whether this is still a blind spot.
- People Random Number Dataset: Another gem from Reddit that started nine years ago with 6,981 people randomly choosing the first number that came to mind, or at least what they were told to do. I love how old this dataset is because most of these guesses came from a simpler, pre-AI era. The OP even mentioned trying to get more responses by posting it on her Google+ :D. Speaking of the sample, 6,981 people is a seriously large dataset, enough to make some pollsters jealous. Of course, the sample isn’t exactly random since it was promoted on Reddit (which probably helps explain some of the 69 silliness), but you’ve got to respect the hustle.
- Computer Random Number Dataset: There’s pseudo-randomness that is "simulated" in a deterministic manner (most number generators and random functions like random.randint in Python) and then there’s true randomness, which is a little more complex to recreate properly. I attempted the latter for choosing 1-100 n=10,000 by using random bytes generated directly by the operating system (os.urandom which uses getentropy on macOS), while rejecting values that would otherwise make some numbers more likely than others. The result was that every number fluctuated around 1%, not perfectly, but at random, which was fascinating in itself. I did tests and confirmed the method was sound because a million-number test showed an essentially perfect uniform distribution and no detectable correlation between consecutive numbers.
This chart shows the share of ChatGPT’s 10,000 picks across all 100 possible numbers. There are some pretty obvious misses: certain numbers are heavily overpicked, others are underpicked (if even picked at all), and the overall distribution is pretty messy. There’s also a slight pattern and bell-curve shape hiding in the chaos.
This was me after seeing the ChatGPT random number choices:
ChatGPT vs True Random
The full list of numbers ChatGPT never chose is: 1, 5, 6, 20, 30, 40, 50, 60, 70, 80, 90, and 100. Numbers ending in 0 were almost completely ignored. And while ChatGPT was oddly reluctant to pick the actual number 7, it absolutely loved numbers that ended in 7. 29% of all its picks ended in 7, nearly one in three.
There was a 48% difference between AI and true random number picking when measured using Total Variation Distance, which is a pretty massive gap. That’s nearly halfway to two completely non-overlapping distributions.
Looking at the differences in the number choices side by side, it's easier to see the under-picking at the tails and the over-picking in the middle.
ChatGPT was 6x more likely to choose 47 than the true random generator. The next three most overpicked numbers, 57, 72, and 37, also contained a 7. The underpicks were even more extreme: numbers like 20, 40, and 100 were dramatically more likely to come up with true randomness than with ChatGPT. In fact, ChatGPT never picked many of these numbers at all. Rather than report an absurdly large ratio when the AI count was zero, I used a conservative floor for the calculation.
Theories Why ChatGPT Sucks at Random Numbers
There are some theories already out there on why LLMs might not be the greatest random number generators.
- It doesn’t have the tools: This recent research paper’s authors found a similar sampling issue, and concluded “LLMs lack a functional internal sampler, necessitating external tools for applications requiring statistical guarantees.” The viral AI timer guy could have told you that.
- It mirrors people’s biases: A lot of folks like the authors of this research paper attribute the behavior to biases embedded in its human training data. The dataset of people’s guesses proves it's more than that.
- Training Moderation: The author of the ChatGPT random number dataset that I’m using hypothesized that “this is a product of safety guardrails during pre-training and post-training. It is the most interesting aspect in the dataset: the model's bias is not a raw copy of human bias but a moderated version of it.”
Based on my own research, I have a couple of theories I’d like to build on top of those:
- It’s good enough for a single session: If you ask for a random number once, technically any number it gives you could be considered random if you don’t know what’s happening under the hood. My guess is the LLM is doing just enough to produce a number that looks random and that it predicts you’ll accept. If you think about it, there’s a difference between approving a number suggested by someone else and actually coming up with one yourself.
- Broader training biases: My bigger theory is that biases from training data and all the other tasks ChatGPT is trained to handle can leak into this seemingly simple task. It may be picking up a messy mix of human cognitive biases, patterns in language, and moderation behavior, all of which can subtly influence its choices. That would explain the combination I saw: a lot of messiness, but still some slightly consistent patterns underneath it.
People vs True Random
This chart shows the share of people’s 6,981 choices across all 100 possible numbers. Here folks’ affinity for all things 7 and 69 is very clear.
There was only a 12% difference between people’s choices and the true random number generator.
I think the clear overpicks and underpicks shown below explain most of that gap, and that even a small amount of statistical weighting for the 69 + 7 silliness could bring the difference even lower.
The overpicks are all pretty recognizable: 69 was 2.9x more likely to be chosen than random, while lucky number 7 was 2x more likely. It was also refreshing to see 100 and 1 on the list, especially since ChatGPT didn’t pick either one a single time. On the underpicked side, aside from 85, which was 1.9x more likely to be chosen by true random than by people, it’s interesting how many contain the number 3, including 34, 38, 73, 63, and 83.
People vs ChatGPT
When I calculated the Total Variation Distance between each source and true randomness, people were 4x closer to true random than ChatGPT. That suggests ChatGPT’s statistical quirks and biases become much more pronounced when you aggregate a large number of choices, at least with the model and responses captured in this dataset.
When comparing people and AI directly using the same Total Variation Distance measure, we see the biggest gap of all: 53%.
Most of the difference can be explained by AI’s huge gaps at the extremes, its complete avoidance of certain numbers, and its aggressive overpicking of others. Even more interestingly, people and AI show some similar obsessions with 7—but in noticeably different ways.
