Contents
Two AI detectors can read the same image and return opposite verdicts, and most of the time neither one is broken. They are simply not measuring the same thing. Each detector was trained on a different set of generators, keys on a different signal in the file, and draws its own line between real and fake, so disagreement is the normal outcome rather than the exception.
They are reading different signals
An AI detector is not one method but a family of methods that share an interface. One reads the image in frequency space, where generators leave periodic grid patterns from upsampling (Frank, Eisenhofer, Schönherr et al., ICML 2020). Another hands the raw pixels to a convolutional network (Wang, Wang, Zhang et al., CVPR 2020). A third ignores pixels and asks where the image sits in the feature space of a large vision-language model (Ojha, Li, Lee, CVPR 2023). A fourth compares rich-texture and poor-texture regions (Zhong, Xu, Li et al., 2023). These detectors can look at the same picture and genuinely see different things, because each was built to notice a different trace. How each family works is covered in how AI image detectors actually work.
They were trained on different generators
Even two detectors from the same family can split, because a detector mostly knows the generators in its training set. Ojha, Li, Lee (CVPR 2023) measured this directly: one detector scored 100 mean average precision on the GAN family it trained on and 61.32 on an autoregressive model it had not seen. Corvi, Cozzolino, Zingarini et al. (ICASSP 2023) showed the same wall between technology generations, where detectors built for GAN images degrade sharply on diffusion images. A tool tuned on last year’s models can miss this year’s, so a picture from a brand-new generator may read as fake to one detector and clean to another purely because of what each was fed.
They set the line in different places
Every detector ends in a single decision: is the score high enough to call “AI”? That threshold is a choice, and different vendors choose differently. A tool set to catch as many fakes as possible will flag more real images by mistake, and a tool tuned to avoid false accusations will wave more fakes through. This is the same operating-point problem that governs text scores, set out in what an AI detector score proves. Independent testing shows how wide the spread gets: NewsGuard, running five tools across 45 images, found false-positive rates from 0 percent (Hive, Sightengine) up to 40 percent (ScamAI), with ZeroGPT at 20 percent and AI or Not at 6.67 percent. Same images, same task, very different lines.
Compression and file history throw them off
Much of the disagreement is not about AI at all. It is about what happened to the file after it was made. Grommelt, Weiss, Pfreundt et al. (2024) found that many detection datasets carry “biases related to JPEG compression and image size,” so detectors trained on them partly learn to spot compression rather than generation, and removing those biases shifts cross-generator accuracy by more than 11 percentage points. Corvi, Cozzolino, Zingarini et al. (ICASSP 2023) show the effect on a live detector, where social-media-style compression and resizing drop a frequency detector from 70.5% to 52.7% AUC. A screenshot, a re-save, or a platform re-encode can therefore tip one detector and not another, which is why the same image at two compression levels can get two different answers from the same tool.
Newer detectors still disagree
Combining signals narrows the gap but does not close it. AIDE (Yan, Li, Cai et al., ICLR 2025) uses multiple experts, one for high-level semantic features and one for low-level patch statistics, and reports improvements of +3.5% and +4.6% over prior methods, while stating plainly that detecting AI-generated images “remains far from being solved.” That is a useful ceiling. The reason disagreement persists is structural: an image can look close to real on one axis and close to synthetic on another, so a semantic reader and a frequency reader can each be right about their own signal and still land on different labels.
What happens when you run them side by side
The clearest picture comes from tests that put several tools on the same images. NewsGuard found that at least one tool disagreed with the others on 35 of 45 images, a disagreement rate near four in five. Bellingcat, testing a single popular detector, found it labeled 6 of 20 real contest photographs as AI, and called 7 of 10 compressed Midjourney images “real.” Those are not edge cases. They are the ordinary behavior of detectors pushed outside their training conditions.
How to use disagreement instead of being confused by it
Disagreement is not noise to be averaged away, it is information. When independent detectors that read different signals all agree, that convergence is worth something, because they are unlikely to share the same blind spot. When they split, the split itself is telling you the image sits near the edge of what these tools can resolve, and no single score should be treated as the answer. That is the right way to read a detector, as one witness among several rather than a verdict. For the broader question of how far any of these tools can be trusted, see are AI detectors accurate.
Sources
- Frank, Eisenhofer, Schönherr et al. (2020). Leveraging Frequency Analysis for Deep Fake Image Recognition. ICML 2020.
- Wang, Wang, Zhang et al. (2020). CNN-generated images are surprisingly easy to spot… for now. CVPR 2020.
- Ojha, Li, Lee (2023). Towards Universal Fake Image Detectors that Generalize Across Generative Models. CVPR 2023.
- Zhong, Xu, Li et al. (2023). PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detection.
- Corvi, Cozzolino, Zingarini et al. (2023). On the Detection of Synthetic Images Generated by Diffusion Models. ICASSP 2023.
- Grommelt, Weiss, Pfreundt et al. (2024). Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets.
- Yan, Li, Cai et al. (2025). A Sanity Check for AI-Generated Image Detection. ICLR 2025.
- NewsGuard (2024). Leading AI Image Detection Tools Mislead Online Users, Often Declaring Authentic Content Fake.
- Bellingcat (2023). Testing ‘AI or Not’: How Well Does an AI Image Detector Do Its Job?