What Twelve AI Detectors Really Measure When They Judge Our Writing
We put one of our articles through twelve AI detectors, seeking a verdict on the best possible version of human authenticity. Our robot tools could barely agree on anything.
Anyone who writes for a living now carries a particular modern worry. You finish a piece, you are reasonably pleased with it, and then a small voice wonders whether some piece of software is about to decide you never wrote it at all. Editors fret about it, clients ask about it, and a small industry has sprouted supposedly to settle the matter, often for a monthly fee.
We’re upfront about why this condundrum matters to us. We’re curating submissions from a bunch of different authors, of varying writing ability. Many use a little help from AI for research or the final polish on their articles. Before anything goes live we usually run it through ZeroGPT as a check to avoid publishing content with high AI scores that can be punished by Google Search and feels creepy for readers.
When our One Italian Summer in Positano feature came back at just 4.4% AI, the number felt like a fake love bomb and oddly low for a piece that had some machine assistance along the way. My 100% human writer colleagues were a little miffed their well-written original articles sometimes scored in the mid teens.
A score that flattering invites scepticism, so we decided to run an experiment pitting robots against each other for other opinions.
Our feature article, One Italian Summer in Positano, is a two-day itinerary built around Rebecca Serle’s novel set on the Amalfi Coast that is soon to be released as a movie. The piece runs to 2,116 words and 12,458 characters. We then pasted it, unchanged, into twelve popular AI detectors.
The reasoning was simple. If these tools are measuring something real, they ought to land in roughly the same place. A text either carries the fingerprints of a language model or it doesn’t, and while you would expect some wobble at the margins, twelve instruments pointed at the same object should broadly agree about what they are looking at. That, as it turned out, was a great deal to ask.
From nought to a hundred
Here is the spread. Grammarly, QuillBot and Scribbr each found 0% AI. ZeroGPT, the checker we normally rely on, announced, with some cheer, that “Your Text is Human written”, putting the machine contribution at 4.4%. Clever Humanizer‘s detector judged the article 98% human, adding that “almost all sentences and the overall writing pattern appear human”, and Turnitindetector.ai was close behind at 92% human, 7% mixed and 1% AI. Just Done offered a noncommittal 0–20%. Then came a cluster in the middle, WriteHuman at 25% AI, Detecting AI at 26% (while still labelling the piece “likely human-written”), and Copyleaks at 29.7%, having sorted 629 words into its AI pile and 1,486 into its human one. At the far end sat Humalingo, which decided the piece was 94% AI, and GPTZero, which went all the way to 100% and declared itself “highly confident” that a machine had written it.
So the same article was, depending on whom you asked, entirely human, entirely artificial, or about a quarter suspicious. A nervous freelancer trying to prove their innocence could simply choose a witness.
For this particular article, though, the middle felt like the most believable place to land. A piece written with some AI assistance and a good deal of human editing is, by definition, a mixture, and Copyleaks and Detecting AI seemed to be the ones describing that mixture rather than rounding it down to “innocent” or up to “guilty”. Copyleaks’ split of 29.7% AI to 70.3% human, and Detecting AI’s 26% sitting alongside its “likely human-written” label, are closer to how the piece was actually made than either a clean zero or a confident hundred. That is a judgement about one article, of course, and no proof that either tool gets it right every time.
Detecting AI Scan Result
Copyleaks Scan Result
Some of the disagreement has a mundane explanation, and it is worth being fair about it. Several of the free tools never read the whole article. Grammarly’s scan stops at 1,400 words, QuillBot’s and Scribbr’s at 1,200 (QuillBot’s counter showed 1,139), GPTZero’s at 10,000 characters against our 12,458, and Turnitindetector.ai’s at 2,000 words. A good number of these verdicts, at both extremes, were verdicts on an excerpt. That explains less than you might hope, though. Grammarly and GPTZero both read the opening of the piece, the same paragraphs about the suitcase, the two plane tickets and the trip that was supposed to be taken with a mother. Grammarly found no AI patterns in them whatsoever. GPTZero highlighted them sentence by sentence.
Evidence showing Scribbr using quillbot as it’s AI detector tool
There is a quieter lesson hidden in the numbers too. Scribbr’s result panel carries QuillBot’s own branding and model number, and the two returned identical readouts: 0% AI, 0% mixed, 100% human. What looks like two independent opinions is, as far as the screenshots show, one opinion delivered twice. Agreement between detectors is only reassuring if they are genuinely different instruments, and it isn’t always easy to tell from the outside.
The sentence no bot could agree on
The more revealing evidence is in which sentences the tools chose to worry about. One line was singled out again and again:
“She arrives on the Amalfi Coast carrying all the things grief tends to bring with it, including memories, unanswered questions and the uncomfortable feeling that life is continuing without the person who made it feel familiar.”
Detecting AI marked it as likely AI. WriteHuman shaded it red. GPTZero, which highlighted nearly everything, highlighted this too.
Detecting AI marking the sentence as “likely AI”
WriteHuman shading mentioned sentence in red
GPTZero, which highlighted nearly everything, highlighted this sentence too.
It is easy to see why. The sentence gathers three items into a tidy list, leans on abstract nouns, and finishes on a gently elegiac cadence, and language models are notoriously fond of all three habits. An editor might even quibble with it on its own terms: “all the things grief tends to bring with it” is a slightly roundabout way into the list. But the rule of three was doing honest work in human prose for a couple of thousand years before anyone built a chatbot, and a sentence about grief is allowed to sound a little like a sentence about grief. A detector can tell you that a line resembles what machines tend to produce. It cannot tell you whether someone chose it.
Other flags were harder to account for. WriteHuman also marked “The two had planned a holiday in Positano for years, partly because Carol had spent a summer there before she met Katy’s father and never stopped talking about the place“, which is plain scene-setting of the kind found in any book summary. Its only apparent offence is being clear and orderly. That raises an uncomfortable possibility for anyone who writes straightforward, informative prose for a living: the qualities that make a piece easy to read may be the same ones that make it look statistically suspicious.
Humalingo‘s written explanation illustrates the problem neatly. It complained that the writing “relies on repeated sentence patterns, limiting natural flow and flexibility”, scored sentence structure 1 out of 5 (“Unnatural”) and readability 3 out of 5 (“Hard to read”). Clever Humanizer, reading the same words, found almost every sentence human. Read the opening yourself and there is a grain of truth in Humalingo’s complaint: the first paragraphs do move in a steady march of facts, one clause after another. A human editor might well vary that rhythm. That is a note about style, though, and a detector has no way to keep the question of style apart from the question of authorship.
Detectors that sell the cure
The most telling pattern in the screenshots has little to do with the article at all. A striking number of these detectors sit right beside a product designed to beat detectors. Grammarly’s menu places its AI Detector a tab away from its AI Humanizer. Clever Humanizer is named after the thing it sells. Detecting AI lists an AI Humanizer in its navigation. WriteHuman delivered its 25% with a large “Humanize This” button underneath.
Just Done went furthest. Having rated the article 0–20% AI, it added a red warning that there was “a 73% chance of being falsely flagged as AI” and invited us to “Fix my weak spots”, on a subscription billed at $19.99 a month. The message, in effect, was that the writing was probably fine, that it was probably in danger anyway, and that help was available for a fee.
This is where the question of humanisers comes into focus. A humaniser is built to move a detector’s number, which makes it only as meaningful as the detector it is trying to satisfy. If one tool calls a piece 0% AI and another calls it 100%, rewriting the text until it passes becomes a matter of choosing which judge to please. Rework the grief sentence to get it past Detecting AI and you might lose the one line in the paragraph with any feeling in it. The result would read as less obviously machine-made, perhaps, without reading as anything a person would actually choose to write.
What the numbers can’t read
We went looking for a verdict on one article and came away with a better sense of what these verdicts are. Every detector we tried was measuring how closely a text resembles the kind of writing a language model tends to produce, and each seems to measure that resemblance differently, over different portions of the text, with different ideas about what counts as suspicious. None of them was measuring whether the writing was any good, whether it said something true, or whether a person had stood behind each sentence and decided it should stay.
Those are editing questions, and they remain stubbornly human ones. Does the opening march along too evenly? Probably, a little. Does the grief sentence earn its three-part list, or is it coasting on a familiar shape? That is worth an argument over coffee. Is the closing line, that you are “visiting the place before the film does”, a pleasing turn? We rather think so. No score settles any of it, and a tool that rewrites a sentence purely to shift a score is solving a different problem from the one a writer actually has.
If the experiment taught us anything, it is that sounding human and sounding less like a machine are separate achievements, and only one of them can be measured with a percentage. The other still requires a reader.
AI writing score: not calculated. The twelve detectors in our test gave the same article scores ranging from 0% to 100%, so there is no single scoring method to reproduce here. A figure for this piece would need to come from running it through whichever of those detectors you choose to trust.
