Contents
It depends almost entirely on what you ask a detector to do. On the generator and the conditions a tool was trained on, accuracy is genuinely high, often quoted at 99 percent or better. On an arbitrary file arriving in the wild, from an unknown model and after ordinary processing, the same tool can fall to little better than a coin toss. And the number that decides whether a detector helps or harms, its false-positive rate on real human content, is rarely the one on the marketing page. So the answer is that AI detectors are accurate in a narrow, checkable way, and unreliable in exactly the situations where the stakes are highest.
The high numbers are real, and they are narrow
The best results deserve to be met at full strength. In text, Binoculars detects over 90 percent of ChatGPT output at a 0.01% false-positive rate without ever being trained on ChatGPT (Hans, Schwarzschild, Cherepanova et al., ICML 2024). In images, a convolutional detector trained on one generator family can spot related models so well it earned the label “surprisingly easy to spot” (Wang, Wang, Zhang et al., CVPR 2020). In audio, the AASIST detector reaches about 0.83% equal error rate, the point where false accepts and false rejects balance, on the ASVspoof 2019 benchmark (Jung, Heo, Tak et al., ICASSP 2022).
Every one of those figures holds in distribution: on long unedited text from a known model, on a generator family the detector was tuned against, on a benchmark recording. The word “benchmark” carries the whole qualification, because it is not the file you are usually trying to check.
| Medium | Strong lab number | In the wild |
|---|---|---|
| Text | Binoculars over 90% TPR at 0.01% FPR | cross-model accuracy “rarely beyond 60%“ |
| Image | near-perfect on the trained GAN family | 44 to 49% AUC on diffusion after compression |
| Voice | AASIST 0.83% equal error rate | 35 to 55% equal error rate |
What happens off the training ground
Remove those conditions and the accuracy falls away, in the same shape in every medium. In text, the cheapest attack is paraphrase: the DIPPER paraphraser drops DetectGPT from 70.3% to 4.6% detection at a fixed 1% false-positive rate (Krishna, Song, Karpinska et al., NeurIPS 2023), and RAID, a benchmark of over 6 million generations, finds that detectors advertising “99% or more” accuracy are “easily fooled” and that cross-model accuracy “rarely achieves beyond 60%” (Dugan, Hwang, Trhlik et al., ACL 2024). In images, a spectral detector tuned on GANs falls to between 44 and 49 percent AUC, essentially chance, on diffusion models such as Stable Diffusion once ordinary compression is applied (Corvi, Cozzolino, Zingarini et al., ICASSP 2023). In audio, detectors reading in the low single digits on ASVspoof climb into the 35 to 55 percent range on in-the-wild recordings, with RawNet2 rising from 24.32% to 36.74% (Yi, Wang, Tao et al., 2023). Different media, one pattern: a detector mostly knows the generators it was trained on, and a new one is a fresh problem it was never taught.
The false-positive rate is the number that matters
For any decision that carries a consequence, the harm comes from false positives, real content flagged as AI, and that rate is not low on real people or real photographs. Bellingcat, testing a popular image detector, found it labeled 6 of 20 genuine contest photographs as AI. Across five tools on the same 45 images, NewsGuard measured false-positive rates ranging from 0 percent up to 40 percent, one tool wrongly flagging 40 percent of what it saw while another flagged none. The maker of the largest text model conceded the point in its own words: OpenAI withdrew its AI Text Classifier in July 2023 “due to its low rate of accuracy,” disclosing a 26% true-positive rate and a 9% false-positive rate. And because these rates apply per file, even a “1 percent” false-positive rate becomes many wrong accusations once you run it across a school term, a newsroom, or a moderation queue.
A score is not a verdict
An accuracy figure means little without the threshold and false-positive rate behind it. RAID fixes and discloses its operating point, holding the false-positive rate at 5 percent; most viral “99 percent” figures are quoted at a hidden one. In audio the right metric is equal error rate rather than a headline percentage, and the same displayed confidence means different things on a clean studio clip and a compressed phone call. A number with no named false-positive rate behind it is not evidence of anything, which is why the first question about any detector result is not “how accurate is the tool” but “at what false-positive rate was this score read.”
Why detectors disagree, and why that is information
Because each tool reads a different trace and draws its own line, two detectors routinely split on the same file. NewsGuard found at least one of five tools dissenting on 35 of 45 images. That is not malfunction; it is the ordinary behavior of narrow readers pushed off their training ground, and it is examined in why do AI detectors give different results and, for voice specifically, why do two voice detectors disagree. When independent methods that read different signals agree, the convergence is worth something, because they are unlikely to share the same blind spot. When they split, the split itself is telling you the file sits near the edge of what these tools can resolve, and no single score should be treated as the answer.
The one thing that is reliable: cooperative provenance
There is a genuine exception, and it should not be erased. Where the generator or the platform cooperates, an injected watermark or a signed Content Credential behaves like a real trace. A green-list text watermark is detectable from as few as 25 tokens (Kirchenbauer, Geiping, Wen et al., ICML 2023), and SynthID-Text was A/B-tested across roughly 20 million Gemini responses with no statistically significant quality loss (Dathathri, See et al., Nature 2024). But this covers only content a cooperating provider chose to mark, it is opt-in, and the marks can be stripped by ordinary handling or forged onto innocent files, so a clean check is not exoneration. This is provenance, a scoped signal about origin, not a detector that reads AI-ness out of an arbitrary file. How to read it, and its limits, are covered in SynthID check: what it can and can’t tell you and does “no watermark” mean an image is real.
So, are AI detectors accurate?
As a scoped signal, yes: on a known generator, on a clean original file, read against a disclosed false-positive rate, a detector is useful evidence. As a general truth machine that decides whether a stranger’s essay, photograph, or voice note is real, no. The reliable move is the same in every medium: preserve the original file rather than a re-encode, read the score as one probability at an operating point you are usually not shown, corroborate with a second method that works differently, look for provenance, and weigh the source. For each medium in depth, see are AI text detectors reliable, are AI image detectors reliable, and for voice, is this voice AI-generated.
Sources
- Hans, Schwarzschild, Cherepanova et al. (2024). Spotting LLMs with Binoculars: Zero-Shot Detection of Machine-Generated Text. ICML 2024.
- Wang, Wang, Zhang et al. (2020). CNN-generated images are surprisingly easy to spot… for now. CVPR 2020.
- Jung, Heo, Tak et al. (2022). AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks. ICASSP 2022.
- Krishna, Song, Karpinska et al. (2023). Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. NeurIPS 2023.
- Dugan, Hwang, Trhlik et al. (2024). RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors. ACL 2024.
- Corvi, Cozzolino, Zingarini et al. (2023). On the Detection of Synthetic Images Generated by Diffusion Models. ICASSP 2023.
- Yi, Wang, Tao et al. (2023). Audio Deepfake Detection: A Survey.
- Kirchenbauer, Geiping, Wen et al. (2023). A Watermark for Large Language Models. ICML 2023.
- Dathathri, See et al. (2024). Scalable watermarking for identifying large language model outputs. Nature 634:818-823.
- NewsGuard (2024). Leading AI Image Detection Tools Mislead Online Users, Often Declaring Authentic Content Fake.
- Bellingcat (2023). Testing ‘AI or Not’: How Well Does an AI Image Detector Do Its Job?
- OpenAI (2023). New AI classifier for indicating AI-written text.