Tokenization and AI Detection: How the Way Text Is Split Changes the Verdict

Before an AI detector decides anything about your writing, it does something you never see: it chops your text into pieces called tokens. That first cut, not the words themselves, is what every later calculation is built on. Get curious about it and a lot of weird detector behavior suddenly makes sense.

Key takeaways

• Detectors don’t read words the way you do. They split text into tokens first, and that split shapes every number that follows.
• The same sentence can come out to fewer tokens in one tokenizer and more in another, which changes perplexity and can nudge the verdict.
• Rare words, names, code, emoji, and many non-English scripts get split into more, smaller tokens, which can make human writing look strange to the math.
• A detector built on one model’s tokenizer can misjudge text from a different model, because the splits don’t line up.
• You can’t see tokenization directly, but a token-level heatmap shows where the detector is reacting, which beats staring at one score.

What a token actually is

A token isn’t a word. It’s whatever unit the model was trained to read as a single step. Sometimes that’s a whole word, sometimes a fragment, sometimes just a space plus a few letters.

Most modern detectors and the models behind them use a method called byte pair encoding. The short version: the tokenizer learns the most common chunks of text from a huge pile of data, then reuses those chunks. Frequent words become one token. Rare ones get assembled from smaller pieces.

Take the word “tokenization.” To you it’s one word. To a typical tokenizer it might land as three tokens, something like “token,” “iz,” and “ation.” The word “the” is one token. A long, uncommon surname might be five or six. None of this is visible in your document, but it’s the raw material the detector works with.

Here’s the part that matters. A detector’s core signal is how *surprised* a language model is by each token, given the tokens before it. Predictable next token, low surprise. Unexpected one, high surprise. Average that surprise across the whole passage and you get perplexity, the backbone statistic of most detectors. And perplexity is measured per token. Change how the text is tokenized and you’ve changed the units you’re averaging over.

How the split changes the math

Picture two tokenizers reading the exact same sentence.

> “The mitochondria powers cellular respiration.”

Tokenizer A, trained heavily on scientific text, knows “mitochondria” and “respiration” as single tokens. It sees a short, clean sequence of familiar, predictable units. Low average surprise.

Tokenizer B, trained on more general web text, breaks “mitochondria” into something like “mit,” “och,” “ond,” “ria,” and chops “respiration” into “resp” and “iration.” Now the model has to predict each fragment in turn. Some of those fragments are genuinely hard to guess mid-word, so the surprise spikes. Same sentence, same author, noticeably different token statistics.

Neither tokenizer is wrong. They were just built on different data with different goals. But a detector that draws a line at some perplexity threshold could read the sentence one way through tokenizer A and another way through tokenizer B. If you want the deeper version of why these statistics behave the way they do, our piece on perplexity and burstiness walks through the full picture.

This is also why running the same paragraph through three different tools can hand you three different scores. People assume the tools “disagree about AI.” Often they just disagree about how to slice the sentence first.

Where tokenization quietly breaks things

The trouble shows up most at the edges, where text stops looking like the ordinary prose the tokenizer was tuned for.

Names, jargon, and rare words

Proper nouns are tokenization landmines. A common name like “Sarah” is probably one token. An uncommon surname, a foreign place name, or a niche brand gets shattered into fragments. A bibliography full of author names, or a history essay packed with unfamiliar places, can produce jagged token statistics that have nothing to do with whether a person or a machine wrote it.

Technical jargon does the same. A medical or legal document loaded with specialized terms forces the tokenizer to keep assembling rare words from small pieces, which makes the surprise readings jump around in ways the detector wasn’t really designed to interpret.

Code, URLs, and symbols

Drop a code snippet or a long file path into a detector and watch the results go sideways. A variable like getUserAuthToken isn’t a word the tokenizer has ever seen as a unit, so it explodes into a string of fragments. URLs, hashes, and config strings do the same. The token-level surprise becomes either very high or weirdly flat, and the score swings to match. That’s a measurement artifact, plain and simple.

Emoji and unusual punctuation

A single emoji can be one token or several, depending on the tokenizer and how the character is encoded. Sprinkle a few through a casual message and you can shift the token count and the statistics enough to move a borderline verdict. The emoji didn’t make the writing more or less human. It just changed the arithmetic.

Non-English and ESL writing

This is the one with real consequences. Most widely used tokenizers were optimized for English. Feed them many other scripts and the same idea takes far more tokens to express, each one less predictable to a model that saw mostly English during training.

That extra fragmentation is part of why detectors have repeatedly stumbled on non-native English writing. A 2023 Stanford study led by Weixin Liang found that several detectors were biased against non-native English writers, flagging their essays as AI-written much more often than essays by native speakers. Tokenization isn’t the whole story there, but a tokenizer that chops your sentences into more, stranger pieces is quietly stacking the deck before any judgment gets made.

A quick scenario

Picture two students turning in the same lab report idea.

Maya writes in plain, native-sounding English with common vocabulary. Her sentences tokenize cleanly into familiar units, the surprise stays moderate, and the detector reads her as comfortably human.

Daniel, an international student, writes the same content but reaches for more formal phrasing and a couple of translated idioms. His sentences get split into more tokens, several of them low-frequency fragments. The per-token statistics look choppier. The detector, working off those token numbers, leans toward “AI-like,” even though Daniel typed every word himself and so did Maya.

Nothing about authorship separates them. The tokenizer’s treatment of their word choices did most of the work. That’s the uncomfortable truth underneath a lot of false positives, and it’s a big reason detectors output probabilities rather than proof.

Why mismatched tokenizers cause cross-model misses

There’s one more wrinkle worth knowing. A detector is usually anchored to a specific model’s tokenizer and its expectations. When you feed it text from a *different* model trained with a *different* tokenizer, the splits don’t line up. The detector is essentially measuring surprise against the wrong baseline.

This is part of why detection accuracy tends to drop when a new model comes out. The text isn’t necessarily more human. The detector’s tokenizer and reference model just don’t match the thing that produced it. And it’s worth remembering that OpenAI quietly retired its own AI text classifier in July 2023, citing low accuracy. Even the people who build these models will tell you the measurement is fragile.

What you can actually do about it

You can’t open a detector and watch it tokenize. But you can work with the signal instead of against it.

Look at *where* a tool reacts, not just the final number. A single “62% AI” tells you almost nothing. A token-level or sentence-level heatmap tells you which phrases the math finds uneasy, and very often those are exactly the spots full of jargon, rare names, or stiff phrasing. From there you can decide whether a passage actually needs a human touch or is just getting punished by an unlucky split. If you’re weighing how much of this you’ll use, you can compare what the plans include too.

Then write in a way that gives the tokenizer less to choke on. Vary your sentence length. Use concrete, ordinary words alongside the technical ones. Don’t strip out necessary jargon, just surround it with plain language so the whole passage isn’t one long string of rare fragments. None of this is about gaming a tool. It’s about not letting an arithmetic quirk speak for your writing.

And keep some perspective. A score that hinges on how a sentence got sliced is not evidence of anything. It’s one model’s reaction to one particular way of cutting up your text. For more on the moving parts behind these tools, browse our more articles on how detection works.

Frequently asked questions

What is a token in AI detection? A token is a chunk of text, usually a word or a piece of a word, that a model reads as one unit. Before scoring anything, a detector splits your text into tokens. Common words become one token; rare words, names, and technical terms get broken into several. Every statistic a detector reports, perplexity included, is calculated over tokens.

Does tokenization change my AI detection score? Indirectly, yes. Tokenization decides what units predictability gets measured over. A passage split into many small, hard-to-predict tokens looks different than the same text split into fewer clean ones. Two detectors with different tokenizers can read one paragraph and land on different scores.

Why does code or text with names get weird results? Tokenizers are trained mostly on ordinary prose. Variable names, file paths, unusual proper nouns, and emoji fall outside that comfort zone, so they shatter into tiny fragments. That produces jumpy token statistics a detector can misread. It’s a measurement artifact, not evidence about authorship.

Can I see how a detector tokenizes my writing? Usually not the raw tokens, but you can see a heatmap of which parts of your text the detector reacted to. That points you at the exact phrases worth a second look, which is far more useful than a single percentage.

Does writing in another language make this worse? Often, yes. Most popular tokenizers were optimized for English, so other scripts get split into far more tokens per word. That extra fragmentation can make non-English and ESL writing look statistically unusual, one reason detectors have shown bias against non-native English writers.

The takeaway

Tokenization is the invisible first step that colors everything a detector does. The same sentence, split two ways, can earn two different verdicts, and the people writing in rare words, code, or another language pay the steepest price for it. So treat any score as a reaction to a particular slice of your text, not a fact about you. When you want to see where a detector is really reacting and clean up only the spots that need it, run your draft through PaperBleach and let the heatmap do the talking.