How to Read an AI Detection Score Correctly (Probability, Not Proof)

Someone pastes an essay into a detector, sees “98% AI,” and treats it like a confession. That reading is wrong, and the mistake costs people grades, bylines, and trust. The number on the screen is a probability, not a verdict, and once you understand what it’s actually measuring, you’ll stop over-trusting the green checkmarks and the red flags alike.

This is a literacy guide. Not “how to make the number go down,” but how to read the number in the first place, whether you’re a teacher staring at a flagged paper, an editor vetting a freelancer, or a writer trying to figure out why your own work tripped an alarm.

Key takeaways

• An AI detection percentage is a probability that text matches AI-like patterns, not a measurement of how much of your document is AI.
• “98% AI” means the model is highly confident based on patterns it has seen, not that 98% of your words were machine-written.
• Document-level and sentence-level scores answer different questions; a clean document score can still hide a few flagged sentences, and a high one can rest on just a handful of lines.
• Turnitin reports a document-level false positive rate under 1% for documents that are more than 20% AI, but around 4% at the sentence level, and that risk climbs further at low scores and for short or formal text.
• Treat a score as a signal that prompts a closer look, never as standalone proof of how something was written.

What the percentage actually measures

Here’s the core misunderstanding. People read “98% AI” as “98 out of every 100 words came from a machine.” That’s not what it is.

Most detectors give you one number for the whole document: an estimate of how likely it is that the text was machine-generated, given the patterns the model has learned to associate with AI writing. GPTZero says this directly in its own documentation. The percentage refers to the model’s probability that the *entire document* was written by AI, not the share of the text that’s AI. In GPTZero’s own example, a 4% reading means the tool would believe only about 4 times out of 100 that the document was completely AI-written. The number is a judgment about the whole piece, not a slice of it.

So the score is a confidence statement about a classification. It answers “how sure is the model?” — and it deliberately does not answer “how much of this is AI?”

What feeds that confidence is statistical, not semantic. Detectors don’t understand your argument. They measure things like predictability — how unsurprising each next word is — and the evenness of your rhythm. If you want the mechanics under the hood, we cover them in Perplexity and Burstiness Explained. The short version: smooth, predictable, uniform text reads as “AI-like” to these systems, and that’s true whether a machine wrote it or a careful human did.

That last point matters more than people expect. A score measures *resemblance to a pattern*, and plenty of humans resemble that pattern.

Probability is not proof, and the tools admit it

The companies building these detectors are clearer about their limits than most of their users are.

Turnitin tells instructors that detection scores should be used as indicators rather than proof. It doesn’t declare misconduct; it surfaces data so a human can decide. That framing is on purpose. The vendor knows the number is the start of an inquiry, not the end of one.

The track record backs up the caution. OpenAI launched its own AI text classifier in early 2023 and quietly pulled it that July, citing a low rate of accuracy. By OpenAI’s own published figures, the tool correctly caught only about 26% of AI-written text and falsely flagged human writing as AI 9% of the time. The company that builds the models generating this text couldn’t reliably detect it. That should reset anyone’s expectations.

And the errors aren’t random. A Stanford study published in *Patterns* (Liang et al., 2023) ran detectors on essays and found they flagged more than half of TOEFL essays written by non-native English speakers as AI-generated, while getting native-written US 8th-grade essays nearly right. The likely cause is the mechanism above: non-native writing often has lower lexical variety and lower perplexity, which reads as “AI” to a detector. So the score isn’t just probabilistic. It skews in a measurable, documented direction.

Two practical conclusions follow. A number with that kind of error profile can’t function as proof. And the people most likely to be wrongly flagged are often the ones least equipped to push back, which is its own reason to read scores carefully.

Document-level vs sentence-level: one paper, two different questions

Most modern tools show you two layers, and they’re easy to conflate.

The document score is the single overall probability we’ve been discussing. The sentence-level view highlights which passages pushed that score up or down. They’re related but not interchangeable, and the gap between them is where misreadings happen.

Here’s the most concrete, defensible number in this whole conversation. Turnitin has published that it aims for a document-level false positive rate under 1% for documents that are more than 20% AI — but the sentence-level false positive rate is around 4%. That’s a roughly fourfold jump. Shorter spans of text give the model less to work with, so individual sentence flags are noisier than the overall verdict. If you’re going to point at a specific sentence and say “this one’s AI,” you’re standing on the shakiest part of the whole report.

The layers also answer different questions. A document can land at, say, 15% overall while still containing two or three highlighted sentences. That doesn’t mean “15% of this is AI and here are the AI parts.” It means the model’s overall confidence is low, but a few passages looked statistically AI-like on their own. GPTZero notes that some sentences in an otherwise human document may still get highlighted, because the tool reads the document holistically — its overall call rests on patterns across the whole text, not on tallying up flagged lines.

Rule of thumb: use the document score to decide whether something deserves a closer look, and use the sentence highlights to guide *where* you look — not to convict individual lines.

A worked example: reading the same report three ways

Say a teacher runs a 600-word student essay and gets back 47% AI at the document level, with four sentences highlighted in the introduction and conclusion.

The bad reading: “Almost half this essay is AI. The intro and conclusion are the AI parts. That’s enough to file an integrity case.”

A better reading, step by step:

1. *47% is a near-coin-flip on confidence.* The model isn’t sure. That alone is a reason to slow down, not speed up.
2. *The highlights cluster in the intro and conclusion* — the most formulaic parts of almost any essay. Students are taught to write tidy, structured openings and closings. Tidy and structured is exactly what reads as low-perplexity, AI-like text. This pattern is consistent with a human following an essay template.
3. *Sentence flags carry a higher error rate* (Turnitin’s own ~4% at the sentence level). Four highlighted sentences in a short essay is not strong standalone evidence.
4. *Context the detector can’t see:* Is this student a non-native English speaker? If so, the documented bias makes the flag even less trustworthy. Does the teacher have earlier drafts, in-class writing, or version history to compare against?

The defensible move isn’t a verdict. It’s a conversation: ask about the writing process, look at draft history, compare to known samples of the student’s voice. The 47% is a prompt to investigate, and that’s all it can honestly be. (If you’re a student on the receiving end of one of these, here’s how students can defend themselves when a detector gets it wrong.)

How different roles should act on a score

The same number lands differently depending on what you’re responsible for.

If you’re a teacher or grader

Use the score as one input, never the deciding one. Turnitin’s own guidance is to apply professional judgment and knowledge of the student. Pair any flag with process evidence — drafts, outlines, version history, a quick chat. Be especially skeptical of high scores on short assignments and on writing from non-native speakers. And remember the noise at the low end: Turnitin applies an asterisk to results in the 0-20% range and withholds a score entirely in the 1-19% band, precisely because false positives are more common there.

If you’re an editor or content lead

A score is a triage tool. A high reading on a freelancer’s draft is a reason to read more closely and ask about their process, not automatic grounds to kill an invoice. Plenty of competent, fully human writing scores high simply because it’s clean and conventional. If detection numbers are part of your workflow, write your policy around “scores are signals, decisions are human,” and put that in your contributor guidelines so nobody gets blindsided.

If you’re a writer

Don’t panic at a high score, and don’t treat a low one as a clean bill of health. Use the sentence highlights to find passages that read as flat or mechanical — often that’s a genuine craft signal that a section is over-smoothed, regardless of who wrote it. If you want to see where your own draft lands, you can run your draft through a free checker and read the heatmap rather than fixating on the headline number. The goal isn’t a magic percentage. It’s writing that actually sounds like you.

Common mistakes when reading a score

• Treating the percentage as a word count. It’s a confidence level about the whole piece, not a measure of which words are AI.
• Trusting a single run. Re-running text, or testing across two tools, often gives different numbers. Disagreement between detectors is itself information.
• Ignoring length. Short text is unreliable across the board; OpenAI’s classifier explicitly didn’t work below 1,000 characters, and short samples remain a weak spot.
• Forgetting the bias. If the writer is a non-native English speaker, the documented skew should lower your confidence in any flag.
• Skipping context. The detector can’t see drafts, timestamps, or the person. You can. That context usually matters more than the number.

For the deeper reasons these tools can’t be airtight, see why detectors will never be 100% accurate and why Turnitin and GPTZero produce false positives.

Frequently asked questions

Does a 98% AI score mean 98% of my document was written by AI?

No. Most detectors report a single probability for the whole document, so 98% means the model is roughly that confident the text matches AI-like patterns it was trained on. It’s not a word count. A document can be 100% human and still draw a high probability if it’s formal and predictable, and a partly AI-assisted document can score low if the patterns are broken up. Read the number as a confidence level, not a tally of which words came from a machine.

Which is more reliable, the document score or the sentence-level highlights?

They answer different questions, so neither is simply “more reliable.” The document score is a single overall probability. Sentence highlights show which passages pushed that score up or down. Turnitin, for example, reports a higher false positive rate at the sentence level than at the document level, because short fragments give the model less to work with. Use the highlights to investigate, not to accuse a specific line.

What score counts as a false positive risk?

It depends on the tool, but risk is usually highest at the low end. Turnitin, for example, applies an asterisk to results in the 0-20% range to signal more frequent false positives there, and it withholds a score entirely in the 1-19% band. The honest takeaway: low scores are noisy, and even mid-range scores carry a nonzero error rate. No single percentage is a clean line between human and machine.

Can I trust an AI detection score as proof for a grade or a client decision?

Not on its own. Detectors output likelihoods, and research has shown they misfire — a Stanford study published in *Patterns* (Liang et al., 2023) found GPT detectors flagged more than half of non-native English TOEFL essays as AI, and OpenAI retired its own classifier for low accuracy. A score is a reason to look closer, ask questions, or request process evidence like drafts and version history, not a final verdict.

Why did my obviously human writing get flagged?

Detectors lean on statistical patterns like predictability and uniform rhythm. Writing that’s formal, tightly structured, or heavily edited can look statistically “smooth,” which is exactly the pattern many detectors associate with AI. That’s also why non-native English writing gets flagged more often. A high score on human text is a known failure mode, not necessarily a sign you did anything wrong.

The bottom line

A detection score is a probability with a published error rate and a documented bias. It makes a useful signal and a lousy verdict. Read it as a prompt — look closer, ask better questions, check the context the tool can’t see — and you’ll make far fewer expensive mistakes than the people treating “98% AI” as proof.

Want to see how the score breaks down on your own writing instead of just trusting a headline number? Run your draft through PaperBleach’s free checker and read the heatmap, sentence by sentence.