Contents
AI image detectors are reliable in a narrow way: they work well on the generators and the conditions they were trained on, and they degrade sharply once you move off that ground. For an arbitrary image arriving in the wild, from an unknown model and after unknown processing, a single detector’s verdict is not something you should trust on its own. That is not a defect in one product. It follows from how these tools are built and what they read.
Reliable on the training ground, weak off it
Detectors learn a specific trace from a specific set of generators. Wang, Wang, Zhang et al. (CVPR 2020) trained a convolutional detector on one generator family and showed it could spot images from related models, which is where the “surprisingly easy to spot” reputation comes from. The reputation holds inside that family. It does not transfer cleanly to an architecture the detector never saw.
The clearest failure is cross-architecture. A detector tuned on GAN images does not automatically read diffusion-model images, which leave weaker and differently-shaped traces. Corvi, Cozzolino, Zingarini et al. (ICASSP 2023) reported that a spectral detector fell to between 44 and 49 percent AUC, essentially chance, on diffusion families such as ADM and Stable Diffusion once ordinary compression was applied. Corvi’s team put the underlying reason plainly: most detectors “rely on traces that are invisible to the human eye.” Those traces are real, but they are narrow, and a new generation of models can move or remove them.
The evidence that no single verdict is safe
There is no published matrix that pairs each detector against each kind of content and tells you the accuracy of a given tool on a given image. That gap is itself a finding: the numbers that vendors quote are almost always headline accuracy on a favorable benchmark, not the accuracy you get on your file.
What the independent record does show is how often tools disagree, a split examined in why do AI detectors give different results. NewsGuard tested five detectors on the same 45 images and found at least one tool dissenting on 35 of them. When several tools that each claim high accuracy split on most of a small test set, no single answer among them can be treated as authoritative.
Where the numbers actually come from
Part of the unreliability is that detectors often learn the wrong thing. Grommelt, Weiss, Pfreundt et al. (2024) showed that popular detection datasets carry biases tied to JPEG compression and image size, and that detectors “indeed learn from these undesired factors.” When they stripped those biases, cross-generator performance rose by more than 11 percentage points for two standard detectors, which means the earlier scores were partly measuring compression and resolution rather than whether an image was generated. A tool leaning on that kind of shortcut will look accurate on a clean benchmark and misfire in the wild.
The research direction meant to fix this is feature-space detection. Ojha, Li, Lee (CVPR 2023) built a detector on top of a large vision-language model to generalize across generators rather than memorize one. It helps, but it does not make any single tool a reliable last word, and it does not undo the compression sensitivity that erases traces before the detector ever sees them.
What “reliable enough” looks like in practice
A detector is a strong signal under the right conditions and a weak one otherwise, and you rarely know which case you are in. So the reliable move is to use more than one:
| Detector | False positives on real images |
|---|---|
| ScamAI | 40% |
| ZeroGPT | 20% |
| AI or Not | 6.67% |
| Hive | 0% |
| Sightengine | 0% |
Those per-tool false-positive rates, from the same NewsGuard test of five tools on 45 images, show the spread is large: one tool wrongly flagged 40 percent of the images it saw, another flagged none, the false-alarm side examined in AI image detector false positives. Independent field testing tells the same story from the real-photo side. Bellingcat, testing a popular detector, found it falsely flagged 6 of 20 genuine contest photographs as AI. A verdict from a tool at the high end of that table, on an image it was not trained for, is a coin toss dressed up as a percentage.
So AI image detectors are reliable enough to be useful and not reliable enough to be trusted alone. Read the score as a probability from one narrow reader, corroborate with a second tool that works differently, and weigh the file’s history before you believe the number. For how far any of these tools can be trusted across media, see are AI detectors accurate.
Sources
- Corvi, Cozzolino, Zingarini et al. (2023). On the Detection of Synthetic Images Generated by Diffusion Models. ICASSP 2023.
- Wang, Wang, Zhang et al. (2020). CNN-generated images are surprisingly easy to spot… for now. CVPR 2020.
- Grommelt, Weiss, Pfreundt et al. (2024). Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets.
- Ojha, Li, Lee (2023). Towards Universal Fake Image Detectors that Generalize Across Generative Models. CVPR 2023.
- NewsGuard (2024). Leading AI Image Detection Tools Mislead Online Users, Often Declaring Authentic Content Fake.
- Bellingcat (2023). Testing ‘AI or Not’: How Well Does an AI Image Detector Do Its Job?