How Watermark Detection Tests for Statistical Significance (the Z-Score Behind the Flag)

Most people picture watermark detection as a lookup: scan the text, find the hidden tag, done. It isn’t like that at all. A watermark “flag” is the output of a statistical significance test, the same kind a scientist runs to decide whether a result is real or just noise. The whole verdict hinges on one number, the z-score, and on a question that’s surprisingly old-fashioned: could this have happened by chance?

Key takeaways

• A watermark flag isn’t a yes/no lookup. It’s a significance test that asks whether this many “green” words could plausibly be luck.
• The z-score measures how far the green-word count sits above what random text would produce, in units of standard deviation. Bigger z means less likely to be chance.
• The detector compares the z-score to a fixed threshold (often around 4), which maps to a tiny p-value, then declares “watermarked” or “not.”
• Short passages can’t reach significance no matter what, because the signal drowns in statistical noise.
• A high z-score is strong evidence; a low one proves almost nothing, since unwatermarked text is the default for everything humans write.

First, the one-sentence version of how the watermark gets there

To make sense of the test, you need a quick picture of what’s being tested. In a green-list scheme like the influential 2023 Kirchenbauer method from a University of Maryland team, the AI model secretly splits its vocabulary into a “green” set and a “red” set at every word, then nudges itself toward green. Read the output and it looks like normal prose. But across a whole document, green words pile up more often than pure chance would allow.

That surplus is the entire signal. Detection is just the job of deciding whether the surplus is big enough to mean something. Everything below is about that decision.

The null hypothesis: assume there’s no watermark

Here’s the move that throws people. The detector doesn’t start by assuming text is watermarked and looking for confirmation. It starts from the opposite assumption, called the null hypothesis: *this text has no watermark, and any green words are there by random luck.*

Then it tries to knock that assumption down.

If green tokens really were random, you could predict roughly how many to expect. Say the watermark uses a 50/50 split, so half the vocabulary is green at any position. In a passage of 200 words, pure chance would land about 100 of them on green. Get 105? Yawn, that’s well within normal wobble. Get 170? Now the “it’s just luck” story starts to fall apart.

The z-score is how we measure exactly how hard it falls apart.

The z-score formula, unpacked

The math is genuinely simple. For a passage of *T* tokens with a green-list fraction of *γ* (gamma), the formula is:

> z = (green_count − γT) / √(Tγ(1−γ))

Three pieces, that’s it:

• green_count is what you actually observed.
• γT is the expected count under the null hypothesis, the “boring” baseline.
• √(Tγ(1−γ)) is the standard deviation, the size of normal random wobble.

So the numerator is “how much more green did we see than expected,” and the denominator is “how big is normal randomness here.” Divide one by the other and you get the answer in standardized units. A z-score of 1 means the surplus is one normal wobble’s worth, basically nothing. A z-score of 6 means the surplus is six wobbles past expected, which essentially never happens by accident.

If the counting-against-chance logic feels familiar, it’s the same family of reasoning behind the statistics behind AI detectors, even though watermarking and detection are separate systems with separate goals.

A worked mini-scenario

Let’s run actual numbers. Suppose a verifier checks a 200-token passage with a 50/50 green split (γ = 0.5).

• Expected green under chance: γT = 0.5 × 200 = 100.
• Standard deviation: √(200 × 0.5 × 0.5) = √50 ≈ 7.07.

Now two cases:

Case A — a human wrote it. The verifier counts 108 green words. z = (108 − 100) / 7.07 ≈ 1.13. That’s nothing. Random text bounces around by this much constantly. No flag.

Case B — a watermarked model wrote it. The verifier counts 150 green words. z = (150 − 100) / 7.07 ≈ 7.07. That is enormous. A z of 7 means the surplus is seven standard deviations beyond expected, an outcome random text would basically never produce. Flag.

Same passage length, same test, wildly different verdicts, all driven by one count plugged into one formula.

From z-score to a decision: the threshold and the p-value

A z-score alone doesn’t flag anything. The detector compares it to a threshold chosen in advance. Watermarking research often uses something around z = 4.

Why 4, and not 2? Because the threshold encodes how cautious you want to be. Each z-score converts to a p-value, the probability that unwatermarked text would clear the bar by pure chance:

• z = 2 → p ≈ 0.02 (about 1 in 50). Way too loose. You’d false-flag tons of innocent text.
• z = 4 → p ≈ 0.00003 (about 3 in 100,000). Strict.
• z = 6 → p ≈ 1 in a billion. Brutal.

Setting the bar at 4 is a deliberate bias toward *not* accusing human writing. The designers decided that missing some real watermarks is an acceptable price for almost never flagging clean text. That tradeoff is a policy choice baked into a number, which is worth remembering whenever anyone treats a detection result as objective truth. The threshold is a judgment call about which mistake hurts more.

Why passage length quietly controls everything

Look back at the formula and notice that both the expected count and the standard deviation depend on *T*, the number of tokens. This is the part that trips up real-world detection.

The green *surplus* a watermark creates grows roughly in proportion to *T*. But the noise, the standard deviation, grows in proportion to √T. Because the signal grows faster than the noise, longer text produces bigger z-scores for the same watermark strength. Flip it around and the problem appears: short text can’t get there.

Picture a single sentence, maybe 12 tokens. Even if every reasonable word came out green, the expected count is tiny, the standard deviation is a large fraction of it, and the z-score stays small. There simply isn’t enough data to separate a real watermark from a lucky run. This is the statistical reason a watermark on a tweet-length snippet is almost impossible to verify, and it’s the same reason editing matters: paraphrasing replaces green tokens the verifier expected, dragging the count, and the z-score, back toward chance.

So three things weaken the test, all for the same underlying reason of starving it of clean signal:

• Short passages — not enough tokens to clear the noise floor.
• Heavy editing or paraphrasing — swaps expected green words for unexpected ones.
• Mixing in your own writing — dilutes the green concentration across the document.

What a z-score can and can’t establish

A *high* z-score is strong evidence, often stronger than anything a style-based AI detector can offer, because it’s grounded in a known, deliberate mechanism and a secret key rather than guesswork about tone. When z hits 7, the “it was random” explanation is dead.

But a *low* z-score is nearly meaningless. Every human-written document is unwatermarked, so “no significant green surplus” is the expected result for almost any text on earth. Absence of a signal is not evidence of human authorship. And there’s a practical wall on top of the statistical one: you can only run this test if you hold the secret key and know the exact setup the model used. Schools, employers, and random websites don’t, which is why they fall back on AI detectors that estimate from writing patterns and output a fuzzy probability instead of a real significance test. If you want the bigger map of how all these tools differ, browse more on how detection works.

Frequently asked questions

What is the z-score in watermark detection, in plain terms?

It’s a number that answers one question: how surprising is it that this text contains this many green-list words? The verifier counts green tokens, compares that count to what random text would produce, and expresses the gap in standard-deviation units using z = (green_count − expected) / standard_deviation. A z near zero looks ordinary. A z of 5 or 8 means the green surplus is far beyond what luck explains, which points to a watermarked model.

What z-score counts as “significant”?

There’s no universal law, but watermarking papers commonly set the threshold around z = 4. That’s a strict bar on purpose. A z of 4 maps to a one-sided p-value near 0.00003, meaning unwatermarked text would clear it by chance about 3 times in 100,000. The bar is set high because falsely flagging human writing is treated as worse than missing a watermark.

Why can’t short text get flagged even if it’s watermarked?

Significance depends on sample size. The green surplus grows with the number of tokens, but the noise grows with its square root, so the test only separates signal from luck once you have enough words. With a sentence or two, the green count can sit above average and still produce a small z-score. There’s just too little data to rule out chance.

Is a watermark z-score the same as an AI detector’s percentage?

No. A watermark z-score is a real significance test tied to a known mechanism and a secret key. An AI detector’s percentage is an estimate from writing style and statistics like perplexity and burstiness. The watermark test, when it fires, is far stronger evidence, but almost nobody can run it because it needs the key. The tool that actually flags your work day to day is usually a detector.

Does a low z-score mean my writing is safe?

It means no watermark was found, which is the default for nearly every document, including everything humans write. That tells you nothing about how a style-based detector will score you. If you care about that, check your draft against a detector rather than assuming a clean watermark result protects you.

The bottom line

Strip away the mystique and watermark detection is a textbook hypothesis test: assume no watermark, count the green words, measure how far the count strays from chance with a z-score, and flag only if it clears a deliberately strict threshold. The number isn’t magic, and it isn’t certainty. It’s a probability statement with a sample-size requirement and a built-in caution against false accusations.

Since the test that schools and editors can actually run is a style-based detector rather than a watermark verifier, the practical question is how your writing reads to *that*. You can check how your draft scores and see exactly which sentences a detector reacts to, or compare what each plan checks if you’ll be running checks often.