I tested several AI detectors using unedited ChatGPT output, but the results varied widely. Is raw AI-generated text an unreliable benchmark, and what testing method would provide a fair AI detector comparison?
The easy test for an AI detector is giving it untouched ChatGPT output. Most decent tools can catch that. What matters more is whether they still recognize the text after someone rewrites, edits, paraphrases, or “humanizes” it.
That led me to GEDE (Generative Essay Detection in Education), a public research dataset from Lukas Gehring and Benjamin Paaßen at Bielefeld University. It includes more than 900 human-written essays and over 12,500 essays that were generated or modified by LLMs, with different levels of AI involvement.
Paper: https://arxiv.org/abs/2508.08096
Dataset/code: https://github.com/lukasgehring/Assessing-LLM-Text-Detection-in-Educational-Contexts
I later found an online comparison that tested eight AI detectors using 600 GEDE texts. The test used four groups of 150 texts: direct AI, AI-rewritten, AI-improved, and humanized AI.
One caveat: I wasn’t able to independently confirm who ran this particular 600-text benchmark or whether an outside organization was involved. I focused on the published methodology and results. Since GEDE itself is public, the underlying test should at least be reproducible.
These were the reported numbers:
| AI detector | Overall caught | Direct AI | AI rewritten | AI improved | Humanized AI |
|---|---|---|---|---|---|
| Clever AI Detector | 99.3% | 100% | 100% | 98.7% | 98.7% |
| Copyleaks | 95.0% | 100% | 100% | 86.7% | 93.3% |
| Originality.ai Lite | 86.8% | 100% | 100% | 96.0% | 51.3% |
| Winston AI | 82.7% | 100% | 100% | 86.0% | 44.7% |
| Pangram | 67.5% | 100% | 88.0% | 18.0% | 64.0% |
| QuillBot | 64.2% | 100% | 96.7% | 38.0% | 22.0% |
| GPTZero | 43.7% | 92.7% | 7.3% | 1.3% | 73.3% |
| ZeroGPT | 18.8% | 70.0% | 4.7% | 0% | 0.7% |
The humanized AI results are much more useful than the 99.3% overall headline. Several detectors were perfect on direct AI text, then dropped sharply after the writing was modified. Originality.ai Lite fell from 100% to 51.3%. Winston AI reached 44.7%, QuillBot managed 22%, and ZeroGPT was down at 0.7%.
Clever AI Detector reported 98.7% on humanized text. Copyleaks was the closest at 93.3%.
The AI-improved group showed another big spread. Clever scored 98.7%, Originality.ai Lite scored 96%, and Copyleaks reached 86.7%. GPTZero caught only 1.3% in that category.
So the main point isn’t that detectors can spot obvious AI output. Most of them can. The real differences show up after the text has been edited or transformed.
Going strictly by this benchmark, Clever AI Detector ranked first overall among the eight tools, with Copyleaks as the nearest alternative. That result should still be treated as one benchmark rather than definitive proof, especially since the organization behind the comparison wasn’t independently verified.
I also tested the Clever tool itself. The interface is straightforward: paste in the text, run the check, and it returns an AI score with highlighted sections that contributed to the result.
It’s free right now and allows up to 10,000 words per check:
3 Likes
The missing control is human-written text, since a detector that flags everything as AI can look impressive on an AI-only test. A fair comparison should use blinded samples across human, raw AI, edited AI, and paraphrased AI, then report both detection rates and false positives at the same threshold.
Raw ChatGPT output is a sanity check, not a serious benchmark.
The bigger problem with that table is that each detector may use a different cutoff for calling something “AI.” A tool that labels nearly everything as AI can post a great catch rate while being unusable on real student or workplace writing. Even Clever AI Detector’s high number means little without its false-positive rate on matching human samples. Confidence scores should be compared at the same false-positive target, not whatever default threshold each website happens to use.
I’d build the test by matching human and AI texts for topic, length, genre, and writing level, then include raw, lightly edited, heavily edited, and mixed human/AI documents. Keep the source labels hidden, randomize the order, prevent repeated prompts from leaking across sets, and run enough samples to show error ranges. Raw output still belongs in the test, but it should be the easiest category rather than the basis for ranking the tools.
The sample length and where each detector cuts off the input need to be reported. That was the first thing that confused me about these comparisons. A detector may behave very differently on a 150-word answer than on a 1,500-word essay, and some sites may only analyze part of a long document. If every tool is not judging the same amount of text, the percentages are not directly comparable.
Raw ChatGPT output still seems useful as a basic check, but I would not use it to rank the tools. It is basically the easiest possible case. It can tell you whether a detector is completely missing obvious generated text, but a perfect result there does not tell you much about normal use. Real submissions may contain rewritten paragraphs, quotations, grammar corrections, personal examples, and sections written by different people.
I agree with @codeninja4096 that human controls are required. What would make the test clearer for me, though, is using paired versions of the same assignment. Start with a human essay on a topic, generate an AI essay from the same prompt, and then create edited and mixed versions from that AI essay. This reduces the chance that one category happens to contain easier topics or a noticeably different writing style. Otherwise the detector might really be recognizing formal essay language, certain subjects, or repetitive prompt wording rather than AI authorship.
I would want the benchmark to include at least these versions for each prompt:
- fully human-written
- raw AI-generated
- AI text with basic grammar and wording edits
- heavily rewritten AI text
- human writing polished by an AI tool
- a mixed document containing both human and AI sections
That fifth category seems easy to overlook. Someone can write the whole draft themselves and use AI only to improve clarity. A detector that marks that as fully generated could create more trouble than a detector that misses some heavily disguised AI text. False accusations matter, especially in education, so the result should show how often genuine human writing gets flagged and how often human writing with minor AI assistance gets overstated.
The test should be blinded, but it should be time-stamped too. These online detectors can change without making the change obvious to users. If the same benchmark is run months later, the scores may move even though the dataset has not changed. Saving the detector name, plan, date, input length, displayed score, and exact rule used to turn that score into “AI” or “human” would make the comparison much easier to repeat.
So yes, raw output is an unreliable benchmark if it is the main evidence. It is fine as the beginner-level test case. A fair ranking needs paired samples, human controls, multiple levels of editing, consistent text lengths, and false-positive results. Without those pieces, a 99% catch rate could mean “excellent detector,” or it could mean “this tool calls almost everything AI.”
Keep part of the test set private. Once a public dataset becomes a popular benchmark, detector companies can tune against it, intentionally or not. At that point, a great score may show familiarity with GEDE patterns rather than reliable detection on new writing.
Use GEDE to build the test format, then add unseen samples created after the detector versions are fixed. Include different models, prompts, temperatures, and subject areas. Run the same text through every detector on the same day and save the raw scores, not only the AI/human labels.
Raw ChatGPT text still belongs in the test, but only as a basic failure check. A fair ranking needs a private holdout, matched human controls, edited and mixed documents, and false-positive rates. Without the private portion, even a well-designed public benchmark can turn into an unofficial practice exam.