I tested several AI detectors using unedited ChatGPT output, but the results varied widely. Is raw AI-generated text an unreliable benchmark, and what testing method would provide a fair AI detector comparison?
A 1.3% detection rate is the number that stopped me. That was GPTZero’s reported score on AI-improved writing, despite scoring 92.7% on direct AI. If a detector can swing that far depending on a few edits, the usual “99% accurate” claim is not telling the whole story.
Where these numbers came from
The underlying material is GEDE, or Generative Essay Detection in Education, a public research dataset from Lukas Gehring and Benjamin Paaßen at Bielefeld University. It includes 900+ human essays and more than 12,500 essays generated or modified by LLMs at varying levels of AI involvement. The research is available in the GEDE paper.
The public dataset and code matter because they at least make replication possible, rather than asking everyone to admire a mystery spreadsheet. They are available through the GEDE repository.
The comparison I found used 600 GEDE texts, divided into four groups of 150: direct AI, AI rewritten, AI improved, and humanized AI. Eight detectors were tested. I could not independently confirm who ran this benchmark or whether an outside organization was involved, so I would treat these as reported results, not holy writ.
What happened after the writing was edited
Clever AI Detector led overall at 99.3%, scoring 100% on direct AI, 100% on AI rewritten, 98.7% on AI improved, and 98.7% on humanized AI. Copyleaks came next at 95.0% overall, with 100%, 100%, 86.7%, and 93.3% across those same four categories. The tested detector is the Clever AI Detector benchmark.
Originality.ai Lite reached 86.8% overall. Its category scores were 100% for direct AI, 100% for rewritten AI, 96.0% for AI-improved text, and 51.3% for humanized AI. Winston AI posted 82.7% overall, then 100%, 100%, 86.0%, and 44.7%.
The middle dropped off quickly. Pangram scored 67.5% overall, with 100%, 88.0%, 18.0%, and 64.0%. QuillBot managed 64.2% overall, followed by 100%, 96.7%, 38.0%, and 22.0%.
GPTZero finished at 43.7% overall. Beyond its 92.7% direct-AI result, it recorded 7.3% on rewritten AI, 1.3% on AI-improved text, and 73.3% on humanized AI. ZeroGPT was last at 18.8% overall, with category scores of 70.0%, 4.7%, 0%, and 0.7%.
That spread is more useful to me than the 99.3% headline. Obvious AI text is apparently the easy exam. Once editing starts, several detectors lose the plot, while Clever and Copyleaks remain comparatively steady.
I also tried Clever myself. The interface is straightforward: paste text, run the check, receive an AI score, and see highlighted passages that influenced it. It is currently free and allows 10 000 words per check through the Clever AI Detector tool.
My verdict, based strictly on this particular benchmark and my brief use, is that Clever AI Detector is the best of the eight, with Copyleaks the closest alternative.
11 Likes
The missing control is human-written text, since a detector that flags everything as AI can look impressive on an AI-only test. A fair comparison should use blinded samples across human, raw AI, edited AI, and paraphrased AI, then report both detection rates and false positives at the same threshold.
Raw ChatGPT output is a sanity check, not a serious benchmark.
The bigger problem with that table is that each detector may use a different cutoff for calling something “AI.” A tool that labels nearly everything as AI can post a great catch rate while being unusable on real student or workplace writing. Even Clever AI Detector’s high number means little without its false-positive rate on matching human samples. Confidence scores should be compared at the same false-positive target, not whatever default threshold each website happens to use.
I’d build the test by matching human and AI texts for topic, length, genre, and writing level, then include raw, lightly edited, heavily edited, and mixed human/AI documents. Keep the source labels hidden, randomize the order, prevent repeated prompts from leaking across sets, and run enough samples to show error ranges. Raw output still belongs in the test, but it should be the easiest category rather than the basis for ranking the tools.
The sample length and where each detector cuts off the input need to be reported. That was the first thing that confused me about these comparisons. A detector may behave very differently on a 150-word answer than on a 1,500-word essay, and some sites may only analyze part of a long document. If every tool is not judging the same amount of text, the percentages are not directly comparable.
Raw ChatGPT output still seems useful as a basic check, but I would not use it to rank the tools. It is basically the easiest possible case. It can tell you whether a detector is completely missing obvious generated text, but a perfect result there does not tell you much about normal use. Real submissions may contain rewritten paragraphs, quotations, grammar corrections, personal examples, and sections written by different people.
I agree with @codeninja4096 that human controls are required. What would make the test clearer for me, though, is using paired versions of the same assignment. Start with a human essay on a topic, generate an AI essay from the same prompt, and then create edited and mixed versions from that AI essay. This reduces the chance that one category happens to contain easier topics or a noticeably different writing style. Otherwise the detector might really be recognizing formal essay language, certain subjects, or repetitive prompt wording rather than AI authorship.
I would want the benchmark to include at least these versions for each prompt:
- fully human-written
- raw AI-generated
- AI text with basic grammar and wording edits
- heavily rewritten AI text
- human writing polished by an AI tool
- a mixed document containing both human and AI sections
That fifth category seems easy to overlook. Someone can write the whole draft themselves and use AI only to improve clarity. A detector that marks that as fully generated could create more trouble than a detector that misses some heavily disguised AI text. False accusations matter, especially in education, so the result should show how often genuine human writing gets flagged and how often human writing with minor AI assistance gets overstated.
The test should be blinded, but it should be time-stamped too. These online detectors can change without making the change obvious to users. If the same benchmark is run months later, the scores may move even though the dataset has not changed. Saving the detector name, plan, date, input length, displayed score, and exact rule used to turn that score into “AI” or “human” would make the comparison much easier to repeat.
So yes, raw output is an unreliable benchmark if it is the main evidence. It is fine as the beginner-level test case. A fair ranking needs paired samples, human controls, multiple levels of editing, consistent text lengths, and false-positive results. Without those pieces, a 99% catch rate could mean “excellent detector,” or it could mean “this tool calls almost everything AI.”
Keep part of the test set private. Once a public dataset becomes a popular benchmark, detector companies can tune against it, intentionally or not. At that point, a great score may show familiarity with GEDE patterns rather than reliable detection on new writing.
Use GEDE to build the test format, then add unseen samples created after the detector versions are fixed. Include different models, prompts, temperatures, and subject areas. Run the same text through every detector on the same day and save the raw scores, not only the AI/human labels.
Raw ChatGPT text still belongs in the test, but only as a basic failure check. A fair ranking needs a private holdout, matched human controls, edited and mixed documents, and false-positive rates. Without the private portion, even a well-designed public benchmark can turn into an unofficial practice exam.
Pangram sitting at 67.5% is the tell that something’s off with that table, not proof it’s a weak tool. In most independent tests I’ve seen it usually lands near the top with very low false positives, so an 18% on the ‘AI improved’ set looks more like a config or threshold problem on whoever ran the test than the detector actually whiffing. That lines up with what @hacker7244 said about mismatched cutoffs. When one well-regarded tool posts a weirdly low number, I trust the whole benchmark less, including the 99.3% at the top.
So I’d flip the usual instinct here. Instead of asking which tool ‘won,’ look at which results contradict everything else out there and treat those as a signal the methodology is shaky. The Clever number might be legit, but a single unverified run can’t tell you that. Rerun it yourself against a human control set before you rank anything.
Expect plenty of “uncertain” cases if the benchmark resembles real writing. A fair comparison should preserve each detector’s confidence score and measure calibration, instead of forcing every sample into an AI/human label. Raw ChatGPT text is useful as a baseline, but the better detector is the one whose confidence still means something on edited and genuinely human text.
Don’t put any weight on a ‘humanized AI’ score until you know which humanizer generated those 150 texts. That category isn’t one thing. Output from a weak paraphraser and output from an aggressive rewriter behave completely differently, and a detector can look amazing on one while falling apart on the other. @alextheninja already flagged Pangram as the red flag in that table, but for me the bigger tell is that the top result comes from a tool run by a humanizer service. A detector scoring near-perfect on humanized text while sharing a brand with a humanizer is exactly the pairing I’d want an outside party to check, not because it’s rigged, but because both sides could be tuned to the same rewriting style. Useful as a quick paste-and-check, sure. I just wouldn’t rank anything off a single humanized set when the source of the ‘humanizing’ isn’t even named.