Why AI Text Detectors Fail and What to Use Instead

Somebody in the organization will eventually propose running documents through an AI text detector. The instinct is reasonable and the method is the one approach the published evidence rules out, for reasons a security or risk function should understand before the policy gets written.

‍

The organization that built the most widely used language model tried this and withdrew its own tool after six months. Its reason was accuracy, and the figures it published before withdrawing explain why.

‍

What Did the Vendor's Own Detector Achieve?

‍

Less than a coin flip in the direction that matters.

‍

The Published Figures

‍

When OpenAI launched its text classifier in early 2023 it published its own accuracy figures. The tool correctly identified twenty-six percent of AI-written text while incorrectly labeling human-written text as AI-written nine percent of the time.

‍

What Those Two Numbers Mean Together

‍

Three quarters of machine-written text passed as human, and roughly one in eleven human writers was flagged. The company also noted that human text would sometimes be confidently mislabeled, and that AI text could be edited to evade the tool.

‍

The Withdrawal

‍

In July 2023 the classifier was discontinued, with OpenAI stating it was no longer available because of its low rate of accuracy. The tool built by the organization with the most direct knowledge of how the text was produced could not do the job.

‍

Who Gets Falsely Accused?

‍

Not randomly, which is what turns an unreliable tool into a discrimination problem.

‍

The Stanford Finding

‍

Researchers at Stanford tested seven widely used detectors against writing samples from native and non-native English writers, publishing the results in the journal Patterns. Essays written by non-native speakers drew an average false positive rate above sixty-one percent, while essays by native-speaking American students were classified close to correctly.

‍

Why It Happens

‍

Detectors key on linguistic predictability, and writing with a narrower range of vocabulary and construction reads as more predictable. The method penalizes constrained expression rather than machine authorship, and those are not the same thing.

‍

The Part That Closes the Argument

‍

The same study found that prompting for greater linguistic variety removed the bias and let generated text pass undetected. A detector therefore flags writers whose English is their second language while anybody deliberately using AI evades it with one extra instruction.

‍

What Is the Organization Asking?

‍

Usually one of three questions, none of which a detector answers.

‍

• Did confidential material leave the organization? A security question, answerable from records of what was sent where.
• Is the work correct? A review question, answerable by checking the claims against their sources.
• Did this person do the thinking? A management question, answerable by asking them about it.

‍

Why Separating Them Matters

‍

Each has a reliable method and none of the three needs authorship attribution. Naming the question being asked usually reveals that the detector was never the right instrument for it.

‍

The Question Nobody Should Need to Ask

‍

Whether a tool was used at all, in an organization where using one is permitted. Where policy allows assistants, the use is not the concern and the handling of material is.

‍

What Does Usage Monitoring Answer Instead?

‍

Observed activity rather than inferred authorship, which is a different kind of evidence entirely.

‍

What the Record Contains

‍

Which identity reached which assistant, under which account, at what time, and what category of content went with it. Usage monitoring produces that from systems operating rather than from a judgment about prose.

‍

Why It Is Fairer

‍

A record of an event either exists or does not. It carries no view about a person's writing style and cannot be biased by their vocabulary, so the finding survives a challenge in a way a probability score does not.

‍

What It Does Not Tell You

‍

Whether a specific document was drafted with assistance. It tells you which tools a person uses and what they send to them, and that is usually the governable fact rather than the authorship of any single paragraph.

‍

Where Does the Real Exposure Sit?

‍

In the material that went out rather than in the text that came back.

‍

The Asymmetry

‍

A report drafted with an assistant and reviewed properly carries little risk. A customer list pasted into an unmanaged account carries a great deal, and the second leaves a record while the first does not. What employees share with assistants covers which categories are detectable and how.

‍

Which Tools Are in Use

‍

An assistant nobody approved is the one carrying material nobody is monitoring. Shadow AI detection establishes the population before any question about a document arises, and unsanctioned use is where the governable exposure concentrates.

‍

What Should Be Done Instead?

‍

Three things, and none involves pasting prose into a tool.

‍

Set a Disclosure Expectation

‍

State plainly whether assistance should be declared and in what form. Most disputes over generated work trace back to nobody having said, which makes the ambiguity a management failure rather than an individual one.

‍

Review the Work on Its Merits

‍

Check the claims, the sources and the reasoning. Generated text that survives that review is acceptable work, and human text that fails it is not, which is the standard that applied before any of this existed.

‍

Never Act on a Detector Score

‍

The Stanford authors cautioned explicitly against using these tools in evaluative settings. An accusation resting on a probability from a method with a documented bias is one the organization cannot defend if challenged.

‍

Watch the Activity, Not the Prose

‍

OpenAI's own classifier caught twenty-six percent of generated text while falsely flagging nine percent of human writing, and was withdrawn in July 2023 for low accuracy. Stanford researchers testing seven detectors found a false positive rate above sixty-one percent on writing by non-native English speakers against near-correct classification of native-speaker essays, because the method penalizes constrained expression rather than machine authorship. The same work showed that one additional prompt both removes the bias and defeats detection, so the tools penalize candor and reward evasion simultaneously. The questions behind the request concern whether confidential material left the organization, whether the work is correct, and whether the person understands it, and each has a reliable method that attribution does not.

‍

To see which assistants your teams use and what they send to them, book a demo mapped to your own estate.