Log-Likelihood, Rank, and Entropy: The Three Token Statistics Most AI Detectors Rely On

Open up almost any AI detector and you’ll get a clean little percentage: “92% likely AI.” It feels like the tool read your essay and made a judgment. It didn’t. Under the hood, most detectors never understand a word you wrote—they just score how *predictable* each word was, one token at a time. Three numbers do most of that work: log-likelihood, rank, and entropy.

If you’ve ever wondered what those scores actually measure, this is the layer beneath the dashboard. Once you can picture these three statistics, the whole behavior of detectors—including their mistakes—starts to make sense.

Key takeaways

• Detectors don’t read meaning. They compare your text to what a reference language model *expected* at each position.
• Log-likelihood = how probable your exact word was. Rank = where that word sat in the model’s sorted guess list. Entropy = how unsure the model was at that spot.
• Words that are high-probability, low-rank, and low-entropy look machine-written. Surprising choices and uneven predictability look human.
• Perplexity and burstiness are summaries built on top of these per-token numbers—not separate measurements.
• These same statistics explain why plain, formulaic, or non-native English writing sometimes gets flagged even when a human wrote every word.

First: what a “token” is, and why detectors think in them

Language models don’t see whole words the way you do. They chop text into tokens—often whole words, sometimes word fragments like “un-” or “-ing.” When a model writes, it predicts the next token from everything before it, and it assigns a probability to every option in its vocabulary.

A detector flips that process around. It feeds your finished text into a reference model and asks, position by position: *given the words so far, how likely was this next token?* Your writing turns into a long string of these per-token measurements. The three statistics below are just different angles on that one question.

Log-likelihood: how surprised the model was by your exact word

Start with raw probability. At each position, the model assigns your actual word some probability—say, a high one for an expected word, or a tiny one for a strange one. Log-likelihood is just the logarithm of that probability. Taking the log keeps the math stable when you’re multiplying thousands of tiny numbers together.

Here’s the intuition. A high log-likelihood means the model basically saw your word coming. A low one means you surprised it.

Take this sentence: *”The results clearly demonstrate a significant improvement in overall performance.”* To a reference model, almost every word is a safe, expected continuation, so each token scores a high probability and the sentence ends up with a high average log-likelihood. That’s the profile of fluent, smoothed-out machine text.

Now this: *”The numbers wobbled, then snapped into a pattern nobody on the team had bet on.”* “Wobbled,” “snapped,” “bet on”—these aren’t the model’s obvious picks. Lower probabilities, lower log-likelihood, more surprise. More human-looking, statistically speaking.

When a detector averages log-likelihood across your whole document and transforms it a bit, you get something close to perplexity. So when you read about perplexity and burstiness, remember they sit one level *up* from log-likelihood. They’re the summary; this is the raw ingredient.

Rank: where your word sat on the model’s guess list

Probability isn’t the only thing you can read off that prediction. At every position, the model produces a full ranked list of candidates, from most to least likely. Rank asks a simpler question: where did the word you actually used land on that list?

Rank 1 means your word was the model’s single top guess. Rank 12 means it was the twelfth choice. A rank in the thousands means the model considered an awful lot of other words more likely first.

Why bother with rank when you already have probability? Because they can disagree in a useful way. Picture a spot where the model is genuinely uncertain and no single option scores high. Your word might have a low probability and still be rank 1, because everything else was even less likely. Rank catches a pattern raw probability can blur: text where nearly every word is the model’s *first or second* pick reads as eerily on-rails. The well-known GLTR visualization (Gehrmann, Strobelt, and Rush, 2019) leaned on exactly this—it colored each token by its rank bucket so you could literally see how often the writer chose the obvious next word.

Here’s a quick scenario. Two students hand in paragraphs, both grammatically perfect. In the first, almost every token is rank 1 or 2—the writer (or the model) kept taking the expected path. In the second, the ranks scatter: a 1 here, a few hundred there, an 8 after that. A rank-based detector leans toward “AI” on the first and “human” on the second, even though both read smoothly to a teacher’s eye.

Entropy: how confident the model was *before* it saw your word

Log-likelihood and rank both depend on the word you actually wrote. Entropy is different. It measures the model’s state of mind at that position *regardless* of your choice.

Entropy summarizes the whole probability distribution. Low entropy means the model was confident—one or two candidates hogged most of the probability. High entropy means it was torn across many plausible options.

Picture the phrase *”Once upon a ___.”* The model is nearly certain the next token is “time.” That’s a low-entropy position; there are barely any real choices. Now picture *”My favorite thing about weekends is ___.”* Dozens of continuations work: sleeping, hiking, coffee, silence. High entropy.

Why detectors care: machine text tends to live in low-entropy neighborhoods, because models often get tuned to play it safe and pick from a narrow, confident set. Long runs of low-entropy positions—where the writing keeps landing in spots the model found “easy”—make a quiet tell. Human writers wander into high-entropy territory more often, then make an unexpected pick anyway.

One subtlety is worth flagging. Entropy and a *human* sense of difficulty don’t always line up. A spot can be low-entropy—the model is confident—even where a person would feel the sentence is wide open, because the model has strong stylistic habits. That gap is one root of detector errors.

How the three combine into a score

No serious detector leans on a single statistic. It gathers all three per token, then aggregates:

• Average or transformed log-likelihood → roughly perplexity.
• Variance of these numbers across sentences → roughly burstiness.
• Distributions of rank and entropy → extra features fed into a classifier.

Some detectors stop at simple thresholds. Newer ones train a small model on these features, or use tricks like DetectGPT’s “curvature” test, which perturbs your text and watches how the log-likelihood shifts. The common thread holds: it’s all built from per-token probability geometry, not comprehension.

Why this explains the false positives

Once you see the machinery, the failures stop being mysterious.

Plain, careful, predictable writing produces high log-likelihood, low rank, and low entropy—the exact fingerprint of “AI.” That’s why simple academic phrasing, formulaic business writing, and especially non-native English can get flagged. A writer who sticks close to common, textbook constructions is, statistically, doing what a cautious model does. A 2023 Stanford study by Weixin Liang, James Zou, and colleagues found detectors disproportionately flagged essays by non-native English speakers, and here’s the mechanical reason: a narrower vocabulary range pushes text into low-rank, low-entropy patterns.

It also explains the disagreement problem. Every detector uses its *own* reference model. If that model finds your wording predictable, your scores look machine-like; a different reference model might shrug at the same sentence. Two tools, same paragraph, different verdicts. OpenAI itself retired its AI text classifier in July 2023, citing a low rate of accuracy. Even the lab that builds the models couldn’t make these signals reliable enough on their own.

A heatmap is a window into all of this: the “hot” spans are where your words were too predictable to the reference model. Want more of this under-the-hood stuff? You can browse more on detection mechanics or, if you’re weighing tools, check the plans.

Frequently asked questions

What’s the difference between log-likelihood and perplexity? Log-likelihood is a per-token number: how probable each word was according to the model. Perplexity is a document-level summary built from those numbers—roughly the average surprise across the whole passage. So log-likelihood is the raw ingredient, and perplexity is one dish you can cook with it. A detector usually computes log-likelihood for every token first, then averages or transforms it into something like perplexity for scoring.

Why does rank matter if the detector already has the probability? Probability and rank capture slightly different things. A word can have a modest probability but still be the model’s single top guess, if every other option was unlikely too. Rank tells you where the actual word sat in the sorted list of candidates—1st, 5th, 500th. Text where almost every word is the model’s first or second guess looks unusually “on-rails,” and that’s a tell raw probability alone can miss.

Does low entropy always mean text is AI-written? No. Low entropy means the model was confident about what came next, and that happens a lot in fluent, predictable, template-style writing—human or machine. Boilerplate, formal academic phrasing, and simple sentences can all produce low-entropy stretches. Entropy is a useful signal, but on its own it produces false positives, which is part of why scores are probabilities, not proof.

Can I see these statistics for my own text? Usually not in consumer detectors—they hide the raw numbers and show you a single percentage or a colored heatmap. The heatmap is a friendlier view of the same underlying token scores: “hot” spans are where words were too predictable to the reference model. Tools that surface a token- or sentence-level heatmap let you see roughly where these statistics are working against you.

If detectors use a reference model, can different detectors disagree on the same text? Yes, and they often do. Each detector measures your text against its own reference language model. If that model finds your wording predictable, your log-likelihood looks high and your rank looks low; a different reference model might find the same wording less expected. That mismatch is a big reason two detectors can hand the same paragraph very different scores.

The takeaway

AI detectors aren’t reading your argument. They’re measuring, word by word, how closely your choices match what a reference model expected—through log-likelihood, rank, and entropy. That’s powerful for spotting genuinely formulaic text, and genuinely fragile when a real person happens to write in plain, predictable patterns. Knowing what’s being measured is the difference between trusting a number blindly and reading it for what it is: a probability, not a verdict.

If you want to see these signals at work on your own writing—and find out where your draft reads as predictable—run your draft through a detector and look at the token-level heatmap.