AI Detector for Code Comments and Docstrings: What Tools Miss in Source Files

You point an AI detector at a .py file, hit scan, and get back a single confident percentage. The trouble is that a source file isn’t one thing. It’s prose and code braided together, and the tool reading it can only really make sense of half. What it misses in the other half is where most of the bad calls come from.

Key takeaways

• A general AI detector reads the prose in your source files (comments, docstrings) but treats the surrounding code as noise it was never trained on.
• Most tools have no idea where code stops and a comment begins, so they score the whole blob and blur a real signal into a meaningless one.
• Docstrings and comments are usually too short and too conventional to produce a reliable score, so a confident number on them is mostly luck.
• The biggest blind spot isn’t a wrong number. It’s everything the tool can’t see: project context, why a comment exists, and whether it’s actually true.
• If you want a usable read, isolate the prose and judge it as prose, then decide based on substance rather than a percentage.

Two languages in one file

Open any source file and look at what’s really there. There’s code, which follows rigid grammar a machine enforces. And there’s natural language, which lives in the comments, the docstrings, the occasional TODO, the module header explaining what the thing does. Those are two completely different kinds of text sitting on adjacent lines.

A general-purpose AI detector (GPTZero, Turnitin’s writing indicator, Originality.ai) was trained on the second kind. Essays, articles, forum posts. It learned to spot statistical patterns in English prose, mostly by measuring how predictable each word is and how much the rhythm varies from sentence to sentence. If you want the mechanics, we walk through perplexity and burstiness behind the scores in a separate piece.

Here’s the catch. The detector can’t tell the two languages apart. It doesn’t parse your file into an abstract syntax tree, find the comment nodes, and score only those. It swallows everything you paste as one undifferentiated stream of tokens. So the three real sentences of a docstring get mathematically averaged with the loops, brackets, and variable names wrapped around them.

What the tool literally cannot see

When people say a detector “missed” something on a source file, they usually mean it returned a number that felt wrong. But the deeper misses are structural. The tool is blind to things that matter more than the score.

It can’t see where the prose is

This is the big one. A detector doesn’t know that lines 14 through 18 are a docstring and everything else is implementation. It has no concept of comment syntax. To the model, # returns the user's active session and return user.active_session() are just more tokens, and it weighs them together. The signal you actually care about, your English, gets diluted by code the tool can’t read anyway.

It can’t see why a comment exists

A good comment explains intent. *Why* the timeout is 30 seconds, *why* you skipped the obvious library, *what* breaks if someone reorders these two calls. None of that “why” is visible to a perplexity model. It’s measuring smoothness, not meaning. A comment packed with hard-won project knowledge and a comment that’s pure filler can score nearly the same, because the thing that makes one valuable is exactly the thing the detector ignores.

It can’t see whether the comment is true

This matters for docstrings especially. An AI assistant will happily generate a docstring that’s grammatical, fluent, and subtly wrong, describing a parameter the function doesn’t take or a return type it doesn’t return. A detector has no way to flag that. It can’t compare the prose to the code. The most useful judgment about an AI-written docstring (is it accurate?) is completely outside what any AI detector measures.

Why the scores come out noisy

Set the blind spots aside and assume you only care about the number. It’s still shaky on source files, for two reasons that stack.

First, length. Detectors need text to work with, because perplexity and burstiness are statistics, and statistics over six words are noise. A one-line comment like // guard against null before refresh gives a tool almost nothing. Most checkers will warn you the input is too short, or worse, return a confident number anyway that swings wildly if you change a single word. A short docstring sits right in that danger zone.

Second, convention. Comments and docstrings are predictable by design. “Returns the parsed config object.” “Raises ValueError if the input is empty.” “Initialize the client with default options.” That phrasing is conventional because clarity beats novelty in documentation. But conventional reads as low-perplexity, and low-perplexity is exactly what detectors associate with machine output. So a perfectly good human-written docstring can score “likely AI” for the same reason standard prose does. It’s predictable, and the tool can’t tell predictable-because-it’s-a-convention from predictable-because-a-model-wrote-it.

This isn’t a PaperBleach talking point about the limits of detection. It’s the tools describing themselves. OpenAI retired its own AI-text classifier in July 2023, citing its low rate of accuracy. And a 2023 Stanford study, Liang et al., found general detectors disproportionately flagged writing by non-native English speakers, which the authors attribute to those writers’ more constrained, formulaic word choices. Constrained, predictable text trips these tools, whether the constraint is a second language or a documentation convention.

A mini-scenario: scanning one function

Say you have a function with this docstring, written by a teammate:

> “Validates and normalizes the incoming webhook payload, returning a typed event. Handles the legacy v1 format by mapping eventType to type before validation, because v1 senders still exist in production as of the March migration.”

Now you paste the whole function (signature, body, docstring) into a detector to check if the docs were AI-generated.

What happens? The tool scores the blob. The code tokens push the statistics around in ways that have nothing to do with the English. The docstring is only a few sentences, so there’s barely enough prose to read. You get back, say, “62% AI,” and you have no idea whether that reflects the docstring, the code, or the collision of the two.

Meanwhile, the thing a human reviewer would notice instantly is invisible to the tool: that docstring names a real format (v1), a real field mapping, and a real reason tied to a dated migration. That specificity is a strong human signal, and it’s the kind of detail a model wouldn’t invent. The detector can’t reward it, because it isn’t looking for it. You learned more in two seconds of reading than the percentage will ever tell you.

Where any of this actually matters

For most working developers, scanning source files for AI is a non-problem. Inside a private repo, nobody is running your docstrings through a checker, and they shouldn’t. The code working and the comment being accurate are what count.

It gets real when prose leaves the codebase:

• Public documentation and tutorials. A docs editor pastes a guide into a checker before publishing, and clean, conventional how-to writing can flag.
• Dev-tool marketing. Marketing teams increasingly gate copy on a detector score.
• Graded coursework. A CS course where you submit a documented project and an instructor runs the whole thing through Turnitin.
• Paid technical writing. Freelancers get work bounced over a number, sometimes unfairly.

In all of those, the answer isn’t to scan the raw file. It’s to pull the prose out and judge it as prose.

How to get a read that means something

If you genuinely need to know whether a chunk of documentation reads as human, work with the tool’s actual strengths instead of against its blind spots.

1. Isolate the prose. Copy the docstring or comment body out of the file. Drop the syntax. Give the checker English, not a mixed blob, so the score reflects something real.
2. Give it enough text. A single line won’t produce anything trustworthy. Score a full paragraph of documentation, not a one-liner.
3. Read the actual sentences. Does the prose carry detail only someone who worked on the code would know? The reason, the gotcha, the version it broke in? That substance beats any percentage.
4. Check the docstring against the code. This is the judgment no detector makes. Is it accurate? An AI-fluent, factually-wrong docstring is the real risk, and only a human catches it.

If you want to sanity-check a passage of documentation, you can check the prose for free and see which sentences a detector reacts to, then use that to find the genuinely flat parts worth improving.

Frequently asked questions

Can an AI detector tell which lines in a source file are comments? Almost no mainstream writing detector does. They ingest whatever you paste as one undifferentiated block, so your real sentences get averaged with brackets and identifiers the model was never trained to read.

Is there a detector made specifically for code comments and docstrings? Not really, and that’s the gap. AI-code detectors are a separate, messy category, and there’s no mature tool that surgically pulls comments out of a file and scores only those. People end up using a general prose detector on text it can’t cleanly separate.

Why does the same docstring get a different score depending on how I paste it? Because context changes the token stream. Alone, it’s scored as a short passage; inside the function, the surrounding code shifts the math. Neither number is the truth. The tool measures smoothness over whatever you give it.

Should my team run an AI detector on internal code review? Usually it’s wasted effort. In a private repo, what matters is whether the comment is accurate and the code works. Detection matters when prose goes public or gets graded by someone else.

How do I get a meaningful AI read on a docstring or comment? Pull the prose out, give it enough text, and read the sentences. Then ask whether the comment carries information only someone who worked on the code would have. That’s a stronger signal than any score.

The bottom line

An AI detector pointed at a source file is reading half the page and guessing at the rest. It can’t find your comments, can’t see why they’re there, and can’t tell you if they’re true, which is the part that actually matters. Isolate the prose, judge it on substance, and treat the number as a starting point. Curious about the wider picture? Browse more on how detection works, or see what a plan costs.