How Confidence Intervals Should Appear on an AI Detection Result (and Rarely Do)

You run a paragraph through an AI detector and it hands you a number: 72% AI. Two decimal points of false comfort, no error bars, no “give or take.” Every serious measurement in science, polling, and medicine comes with a margin of uncertainty around it, yet the one number people use to accuse students and reject freelancers shows up naked. That missing band is the most important thing on the screen, and it’s almost never there.

A confidence interval isn’t a technicality. It’s the difference between “this is AI” and “this might be AI, or might not, and here’s how unsure we are.” Once you know what the interval would look like, a lone score stops feeling like a verdict and starts looking like a guess wearing a lab coat.

Key takeaways

• Every AI detection score is a point estimate. The honest version would be a range, like “68% give or take 15 points,” not a single confident number.
• A confidence interval shows how much the score would wobble if you ran the same text under slightly different conditions. Wide bands mean the verdict is shaky.
• Intervals get wider on short text, mixed human-AI writing, and unusual styles, exactly the cases where people most want a clean answer.
• Detectors almost never show the band because a wide interval undercuts the product. A lone number looks more authoritative than it has earned.
• When no interval is shown, assume one exists and is bigger than you’d like. Treat a borderline score as a tie, not a conviction.

What a confidence interval actually is

Picture a political poll. “Candidate at 52%” never appears alone in a careful report. It comes with “±3 points,” which means the true figure is probably somewhere between 49% and 55%. The 52% is the best single guess; the ±3 is an honest admission that the measurement could land a bit either way depending on who happened to get sampled.

An AI detection score works the same way, even though the tool pretends it doesn’t. The detector looked at your text, measured some patterns, and produced its best single guess: 72%. But that guess sits on a foundation of choices that could easily have gone differently. How the text was chopped into chunks. Which model version is running today. Where the passage starts and stops. Nudge any of those and the number moves.

The confidence interval is the range it would move across. A tight interval, say 70% to 74%, tells you the detector is fairly stable on this text. A wide one, say 50% to 88%, tells you the 72% is barely more than a coin flip dressed up with a percent sign. Same headline number, completely different meaning. Without the band, you can’t tell which situation you’re in.

Interval is not the same as calibration

Two different honesty problems hide inside one score, and it’s worth keeping them apart. Calibration asks whether a 72% really corresponds to a 72% chance of being AI in the real world. The confidence interval asks something narrower: how much would this specific number jitter if I measured it again? A detector can be perfectly stable, returning 72% every single time, and still be wrong if it’s biased. Stability isn’t correctness. But an unstable score, one with a wide interval, can’t even be wrong consistently. It’s noise. You want both a narrow band and good calibration; most tools show you neither.

Why the band matters more than the number

A point estimate without its uncertainty invites a specific mistake: treating small differences as real.

Say a teacher checks two essays. One comes back 71%, the other 64%. The temptation is to rank them, to decide the 71% student “more likely” used AI. But if each score carries an interval of ±15 points, the first essay’s true value could be anywhere from 56% to 86%, and the second’s from 49% to 79%. Those ranges overlap almost entirely. The seven-point gap is meaningless. Ranking the students on it is like declaring one runner faster because your phone’s stopwatch read a tenth of a second different on two separate days.

This is exactly where the missing band does damage. The number looks precise, so people act on precision that isn’t there. They compare scores across students, across tools, across drafts, as if three or seven points mean something. With the interval drawn in, most of those comparisons collapse into “too close to call.”

Where the interval gets wide

The width isn’t fixed. It balloons in predictable situations, and they tend to be the situations where a confident answer would matter most.

Short text

Fewer words means fewer signals, and fewer signals mean a wobblier estimate. Under the hood the detector is reading the statistical signals a detector measures, things like how predictable each word is and how much sentence rhythm varies. A 150-word passage gives it almost nothing to average over, so the interval stretches wide. This is why many tools quietly impose a minimum word count, and why a sky-high score on two paragraphs deserves a raised eyebrow rather than a verdict.

Mixed and edited writing

Real work is rarely pure. People draft with AI and rewrite, or write themselves and polish with a tool. Detectors are mostly tuned on clean all-human or all-AI samples, so half-and-half text lands in territory the model never mapped well. The point estimate comes out looking as crisp as ever, but the honest interval around it would be enormous.

Unusual or non-native style

Predictable phrasing reads as machine-like, and that pushes both the score and its uncertainty around. A 2023 study from Stanford researchers (Liang et al., published in the journal *Patterns*) found GPT detectors flagged writing by non-native English speakers as AI far more often than native-speaker writing. In their tests the detectors hit near-perfect accuracy on essays by US eighth-graders, while misclassifying a large share of TOEFL essays by non-native speakers as AI-generated. Here a single number is doubly dangerous: it hides both wide uncertainty and a systematic lean against one group of writers.

A quick scenario

Maya pastes a 180-word discussion post into a detector. It reads 66% AI. She panics. Then she does something simple: she splits the post in half and checks each piece. The first half comes back 41%. The second comes back 83%. She runs the whole thing a second time and gets 59%.

Nothing about the text changed. The scores scattered across more than forty points because the passage is short and the estimate is fragile. Maya just reconstructed her own confidence interval by hand, and it’s wide enough to drive a truck through. The “66%” the tool showed her was the midpoint of a guess, not a finding. A teacher who saw only that 66% and acted on it would be acting on noise.

Why detectors hide the band

If intervals are this important, why does almost no consumer detector show one? A few honest reasons.

• It weakens the pitch. “98% AI” sells certainty and ends arguments. “98%, give or take 20 points” starts them. Marketing prefers the bold number.
• Measuring it is work. Reporting an interval means the vendor has to study how unstable their scores are across messy real-world text, then publish results that won’t flatter the product.
• A naked number looks more authoritative. Strip the uncertainty and the tool seems to *know*. That impression is the whole value proposition for a lot of buyers, even though it’s the part that’s least earned.

There’s precedent for the harder, more honest path. OpenAI pulled its own AI-text classifier in July 2023, citing low accuracy. The company building the models declined to keep selling a tool to detect them. That should temper how much trust anyone places in a glossy percentage from a vendor with more to gain.

How to read a score that has no band

You usually won’t get the interval handed to you, so estimate it yourself and read accordingly.

Run the same passage more than once. Split it and check the halves. Try a second detector. If the numbers scatter widely, the real interval is wide and the verdict is unstable, full stop. If they cluster tightly, you’ve earned a bit more trust, though clustering across tools can also mean they share the same blind spots.

Then handle the score like the uncertain thing it is. A borderline number is a tie, not a conviction. Don’t compare small gaps within a tool or across tools as if they’re exact. Distrust any single figure on short, edited, or non-native writing. There’s more on how detection works if you want the full mechanics, and what each PaperBleach plan covers lays out where a detector fits in a sane workflow. The score points you somewhere; your judgment, and the actual writing, decide what it means.

A detector that says 72% with no band is telling you its math leaned toward “AI.” That’s worth knowing. It is not a measurement you should bet someone’s grade on.

Frequently asked questions

What is a confidence interval on an AI detection result? It’s a range around the score that reflects how uncertain the estimate is. Instead of a flat “72% AI,” an honest result would read “72%, plausibly anywhere from 55% to 85%.” The width tells you how much the number would move if you re-ran the same text under slightly different conditions. A narrow band means fairly sure; a wide one means it’s guessing.

Why don’t AI detectors show a margin of error? Mostly because it would weaken the product. A bold “98% AI” sells confidence and ends arguments; “98%, give or take 20 points” invites them. Showing the band also forces the vendor to measure and disclose how unstable their scores are, which many haven’t done or won’t publish.

Does a confidence interval tell me whether the score is actually correct? Not by itself. An interval describes precision, how much the number would jitter on re-measurement. It doesn’t fix bias, so a tool can produce a tight band around a wrong answer. You want both a narrow interval and evidence the tool is unbiased on writing like yours.

How can I estimate the interval myself if the tool won’t show one? Run the same passage a few times, split it into halves and check each, and try a different detector. If the scores scatter widely, the real interval is wide and the verdict is unstable. If they cluster tightly, you’ve got more reason to trust the reading.

What should a teacher or editor do with a borderline score? Treat it as a tie, not a conviction. A 55% with an obvious wide band isn’t evidence of anything alone. Use it to start a review: read the writing, ask about the drafting process, check version history, and weigh what you already know about the person.

See the band for yourself

The fastest way to believe a score has an interval is to make it move. Take one passage, run a passage through a detector, then split it, re-run it, and watch the number wander. Once you’ve seen a single paragraph swing thirty points without a word changing, you’ll never read a lone percentage the same way again.