How Reliable Are AI Detectors Today? A Plain-English Look at the Accuracy Numbers

Ask ten people how accurate AI detectors are and you’ll get ten different answers, from “they’re basically psychic” to “they’re useless.” The truth sits in an awkward middle, and most of the confusion comes from one move: reading a vendor’s accuracy claim as if it describes your document. It doesn’t. Here’s what the numbers actually mean.

Key takeaways

• Vendor accuracy claims (often 98-99%) are measured on the vendor’s own test sets, not on the messy, mixed, real-world text you actually submit.
• A high overall accuracy number can still hide a painful false-positive rate, which is the error that gets real people accused.
• Detectors output a probability, not a verdict. A score is a signal worth investigating, not proof of anything.
• Accuracy drops on short text, edited text, non-native English, and output from models the detector hasn’t been trained on.
• The reliable move isn’t chasing a score. It’s writing in your own voice and keeping evidence of how you wrote.

What “accuracy” even means here

When a detector advertises “99% accuracy,” that’s usually one number pulled from a controlled test. The vendor takes a pile of documents they already know the answer to, half human and half AI, runs them through the tool, and counts how many it labeled correctly. Divide the right answers by the total, and you get an accuracy percentage.

Sounds solid. The problem is hiding in the setup.

That test set is curated. It’s often clean, full-length, English-language text, with AI samples generated by the specific models the detector was tuned to catch. Real submissions look nothing like that. They’re short. They’re edited. They mix a human draft with an AI rewrite, or an AI draft with human edits. They come from whatever model launched last month. The neat lab number was never a promise about that.

So the honest version of “how accurate are AI detectors” is this: accurate under conditions that rarely match yours, and a lot shakier outside them.

The number that actually hurts people: false positives

Overall accuracy lumps two very different mistakes into one figure, and they aren’t equal.

• A false negative is AI text the detector misses. Annoying for a teacher, but nobody gets accused.
• A false positive is human text the detector flags as AI. This is the one that wrecks someone’s week.

A tool can post a beautiful overall accuracy score while still producing false positives at a rate that matters. Picture a detector that’s 99% accurate with a 1% false-positive rate. Sounds tiny. Now run an entire university’s term papers through it: tens of thousands of documents. That 1% becomes hundreds of real students whose own writing got flagged. At scale, a “rare” error stops feeling rare to the people on the receiving end.

That’s why a single accuracy headline can be genuinely misleading. It buries the error that does the damage.

A quick scenario

Maya writes her own history essay. English is her second language, so she leans on clear, common phrasing and tidy sentence structure. She runs it through a detector out of curiosity and gets back “85% likely AI.”

She didn’t use AI. The detector isn’t lying, exactly. It’s reporting that her text is statistically smooth and predictable, which is one of the patterns it associates with machine writing. The tool measured a real property of the text. It just drew the wrong conclusion about *who* wrote it.

This isn’t hypothetical. A 2023 study from Stanford researchers (Liang, Yuksekgonul, Mao, Wu, and Zou), published in the journal Patterns, tested several detectors and found they misclassified more than half of the non-native English writing samples as AI-generated, while staying highly accurate on native-speaker text. Predictable word choice reads as “AI-like” to a statistical model, even when a real person typed every word. The senior author, James Zou, went as far as recommending that people be extremely careful about using these detectors for high-stakes decisions at all.

Why accuracy slips in the real world

A few specific conditions reliably knock detector accuracy down.

Short text. Detection leans on statistical patterns across many words. Feed a tool two or three sentences and there isn’t enough signal. Verdicts on short passages are closer to coin flips than measurements.

Edited text. The moment a human rewrites AI output (or AI rewrites human writing), the statistical fingerprint smears. Detectors are most confident on raw, untouched output, which is exactly the kind of text careful people don’t submit.

New models. Detectors are usually trained on output from models that already exist. When a newer model ships with more varied, human-like rhythm, the detector is reacting to yesterday’s patterns. Accuracy on the latest output tends to lag until the tool catches up.

Non-native English and plain writing. As above, simpler and more predictable phrasing trips the same wires as machine text.

Want the mechanics behind all of this? The short version is that detectors mostly measure two things: how predictable your word choices are, and how much your sentence rhythm varies. We break that down in perplexity and burstiness. Once you see those two levers, the accuracy numbers make a lot more sense.

A score is a probability, not a verdict

Here’s the reframe that fixes most of the confusion: a detector doesn’t *know* anything. It estimates a probability based on patterns in the text. “92% AI” means the text resembles the AI examples it was trained on. It does not mean a 92% chance a human is lying.

Two reliable facts back this up. First, OpenAI built its own AI text classifier and then quietly retired it on July 20, 2023, citing a low rate of accuracy. The company that makes the model people worry about couldn’t ship a reliable detector for it. Second, run the same paragraph through two different detectors and you’ll often get two different scores, sometimes wildly different. If detection were a measurement of a fixed fact, that couldn’t happen. Different tools, different training data, different thresholds, different answers.

So the responsible way to read any score is this: it’s a flag worth a closer look, not a confession. A 70% score on a draft you wrote yourself isn’t a contradiction. It’s the tool telling you your prose happens to be smooth and predictable.

So are they reliable enough to trust?

Reliable for what? That’s the real question.

As a rough first-pass signal, a good detector can be useful. If you paste in raw ChatGPT output, many tools will catch it, and that’s a fair sanity check on whether your writing reads as generic and machine-flat.

As proof of wrongdoing, no. The false-positive problem, the model-lag problem, and the documented bias against non-native writers all mean a score should never be the sole basis for an accusation or a grade. Used that way, the unreliability isn’t a footnote. It’s the headline.

The reasonable middle: treat detectors as one input among several. A score, plus your draft history, plus a real conversation, beats a number on its own every time.

What to do with all this

If you’re a writer who keeps getting flagged on your own work, the fix isn’t paranoia, it’s habit. Write in your actual voice. Vary your sentence lengths on purpose. Use concrete, specific details that a generic model wouldn’t produce. Keep your drafts and version history so you can show your process if anyone asks.

If you use AI to help, edit it into something genuinely yours rather than submitting raw output, and understand that a low detector score is a side effect of good writing, not the goal of it. The point is to learn what trips detectors, not to game a number.

There’s plenty more on how detection works and where it breaks in our more articles on the blog.

Frequently asked questions

How accurate are AI detectors in real terms?

It depends entirely on the text. On clean, unedited output from a model the detector was trained on, the best tools can be quite accurate. On short passages, lightly edited drafts, non-native English, or output from a newer model, accuracy can fall sharply. The headline 98-99% figures come from vendor test sets and rarely match the conditions of a real submission, so treat any single accuracy number with caution.

What is a false positive, and why does it matter more than overall accuracy?

A false positive is human writing that a detector flags as AI. It matters because it’s the error that harms real people: a student accused over an original essay, a freelancer who loses a client. Even a 1% false-positive rate means roughly 1 in 100 genuine human documents gets flagged, and at the scale of a whole school or platform that’s a lot of wrongly accused writers.

Can AI detectors be wrong about non-native English writers?

Yes, and there’s research on it. A 2023 study from Stanford researchers (Liang, Yuksekgonul, Mao, Wu, and Zou), published in the journal Patterns, found that several detectors misclassified more than half of the non-native English writing samples it tested as AI-generated, while staying highly accurate on native-speaker text. The likely cause is that non-native writing tends to use more predictable, common word patterns. It’s one of the clearest documented weaknesses in detection accuracy.

Why do two AI detectors give different scores on the same text?

Because they’re different models trained on different data with different decision thresholds. One tool’s “likely AI” might be another’s “likely human” on the exact same paragraph. The disagreement is a good reminder that these scores are probabilistic estimates, not measurements of a fixed fact.

If detectors aren’t fully reliable, what should I actually do?

Write in your own voice, be specific, vary your sentence lengths, and keep your drafts and version history. If you use AI to assist, edit it into genuinely your own work rather than submitting raw output. Understanding why text gets flagged is far more durable than trying to outguess a score.

The bottom line

AI detectors aren’t magic and they aren’t junk. They’re probability estimators that work reasonably well in narrow conditions and stumble badly outside them, especially in the one direction that hurts real people. Read the scores as signals, not sentences. And if you want to see exactly which lines in your writing look machine-made before anyone else does, check a draft with PaperBleach.