Why Two Detectors Disagree on the Same Essay, and Which One a Teacher Should Trust

You run a borderline essay through one AI detector and get 12%. Out of curiosity, you paste the exact same paragraphs into a second tool and it spits back 89%. Same words, two machines, two completely different stories — and now you’re the one who has to decide what to tell the student.

This happens constantly, and it’s not a bug in either tool. It’s what you’d expect once you understand what these detectors actually measure. Here’s why they split, and how to think about whose number, if anyone’s, deserves your trust.

Key takeaways

• Detectors disagree because they were trained on different data, draw their decision lines in different places, and chop up text differently. The split is structural.
• Every score is a probability, not a verdict. An 89% and a 12% are two guesses, not one truth and one error.
• Disagreement clusters on the exact writing where false positives already live: short answers, ESL prose, very formal academic writing, and heavily revised drafts.
• “Which one should I trust” has an uncomfortable answer: neither, on its own.
• Authorship questions are settled by process — drafts, version history, a quick conversation — not by reconciling two percentages.

Why two detectors look at the same essay and see different things

It helps to stop picturing a detector as a scanner that reads truth off the page. It’s closer to a weather forecast. Two forecasters with different models can look at the same sky and give you 30% and 70% chance of rain. Neither is lying. They weighted the inputs differently.

AI detectors work the same way, and there are three concrete reasons their forecasts diverge.

They learned from different examples

Each tool is a model trained on its own pile of “this is human” and “this is AI” text. One vendor might have trained heavily on GPT-3.5 output from 2023. Another might include newer models, or more student essays, or more marketing copy. A detector can only recognize patterns it has seen. Feed it writing that looks unlike its training set — a chemistry lab report, a non-native speaker’s careful prose — and it guesses, sometimes confidently and wrongly.

They draw the line in different places

Under the hood, a detector produces a number on a sliding scale, then has to convert that into a label. Where it puts the cutoff is a business and design choice. One tool might call anything above 0.5 “AI.” Another might wait until 0.8 to avoid embarrassing false accusations. So a single essay could land at 0.6 internally and get labeled “likely AI” by the aggressive tool and “likely human” by the cautious one — from nearly identical underlying math.

They cut the text up differently

Detectors don’t always judge a whole document as one block. Some score sentence by sentence and average. Some score in overlapping chunks. Some weight the opening more than the body. Paste a single paragraph versus the full essay and you can get different answers from the *same* tool, because the underlying statistics that drive these scores — how predictable the text is, how much the sentence rhythm varies — get computed over different amounts of material. Short text is the worst case. Too few words, too little signal, jumpy results.

A mini-scenario: Maya’s history essay

Maya is a sophomore, a fluent but non-native English speaker. She writes clean, formal, slightly repetitive prose because that’s what she was taught is “academic.” Her essay on the Treaty of Versailles is entirely her own work.

Detector A scores it 84% AI. Detector B scores it 9%.

What happened? Detector A leans hard on predictability and sentence-rhythm flatness. Maya’s careful, uniform sentences read as low-variation — the same statistical fingerprint these tools associate with machine output. That’s not a quirk. A 2023 Stanford study led by Weixin Liang found that detectors misclassified more than half of non-native English writing samples as AI, while getting native-English samples almost always right. Detector B, trained on a broader set and tuned to be more forgiving, didn’t see enough to flag her.

If you’d only run Detector A, you might have opened a misconduct case against a student who did nothing wrong. The disagreement is the warning light. It’s telling you this essay sits in the zone where these tools are least reliable.

So which one should a teacher trust?

The honest answer: don’t fully trust either, and be suspicious of the question itself.

Picking “the more accurate detector” feels responsible, but accuracy claims come from each vendor testing on their own data, under their own conditions. Those numbers rarely survive contact with your actual classroom — a mix of essay lengths, subjects, ability levels, and first languages that no test set perfectly matches. Independent comparisons keep finding that the “best” tool changes depending on what you feed it. There is no detector that’s reliably best across the board.

There’s also a hard limit worth saying plainly. These tools output probabilities, not proof. Reputable vendors state this in their own documentation. OpenAI went further and pulled its own AI classifier in July 2023 because it just wasn’t accurate enough to be useful. A 90% is not a confession. It’s a model saying “this resembles things I was told are AI.” Resemblance isn’t authorship.

So the move isn’t to crown a winner. It’s to demote the score from *evidence* to *signal*.

What a signal is good for

A disagreement, or even a single high score, is a fine reason to look closer. It’s a terrible reason to conclude anything. Use it the way you’d use a smoke alarm: worth investigating, not proof the house is on fire.

What to actually do when the numbers clash

Here’s a workflow that holds up under scrutiny, including a parent meeting or an academic-integrity hearing.

1. Don’t reflexively believe the scarier number. Two scores that disagree should lower your confidence, not raise it. Ambiguity is the headline, not the high reading.
2. Ask why this essay might be a hard case. Is the writer a non-native speaker? Is the text short? Is it a formulaic genre like a lab report or a five-paragraph template? Each of those is a known false-positive trap.
3. Leave the detector behind. This is the important step. The real evidence isn’t in the tools — it’s in the writing process. Ask to see the Google Docs version history. Look at the drafts. Read the student’s earlier work and check whether the voice matches.
4. Have a low-stakes conversation. Not an interrogation. Ask the student to walk you through one choice they made — why they structured an argument a certain way, where they found a source. A student who wrote the thing can almost always talk about it. A score can’t tell you that; a five-minute chat can.
5. Document what you did, not just what the tool said. “Two detectors disagreed; I reviewed draft history and discussed the work with the student” is defensible. “The detector said 89%” is not.

This is more work than reading a percentage. It’s also the only version that’s fair to the student and safe for you. If you want more on the responsible-use side, you can browse the full library of detection guides for classroom-specific walkthroughs.

The bigger picture for educators

The disagreement between detectors isn’t a temporary problem that better software will fix next year. It’s baked into how the technology works. As writing models improve, the line between human and machine prose keeps blurring, and a tool can sound more sure of itself while still being wrong. Two tools splitting on the same essay is a preview of where all of this is heading.

That’s not a reason to despair. It’s a reason to lean on the things detectors can’t measure: a student’s drafts, their thinking, their ability to defend a choice out loud. Those signals don’t disagree with each other, and they don’t produce false positives against careful ESL writers.

Used well, detectors can start a conversation. They should never end one.

Frequently asked questions

If you want to understand what these tools actually react to — the predictability and rhythm patterns that make even careful human writing get flagged — the fastest way is to see it on real text. You can watch which sentences a detector lights up, and why, on the PaperBleach humanizer. Once you’ve seen the patterns once, you’ll never read a single percentage the same way again.