Contents
Voice detectors fail in the real world because they are trained on curated benchmarks and deployed on everything else. Move off the training distribution, an unseen generator, a phone codec, background noise, unfamiliar playback hardware, and equal error rates that read near 1 percent in the lab climb into the tens of percent. The gap between the benchmark number and the real one is not noise; it is the actual reliability of the tool.
How big is the lab-to-wild gap?
Large enough to change the answer. When Müller and colleagues tested detectors on their In-the-Wild benchmark, audio collected outside the lab, performance dropped sharply from the benchmark figures (Müller et al., Interspeech 2022). The survey by Yi, Wang, Tao et al. (2023) quantifies it: trained on ASVspoof 2021 and tested in the wild, detector equal error rates jump from 20 to 30 percent up to 35 to 55 percent. End-to-end models move the same way, with RawNet2 rising from 24.32% to 36.74% and AASIST from 19.77% to 34.81%. A tool that reads as near-perfect on its benchmark can be wrong a third of the time in the open world.
Why do unseen generators break detection?
Because a detector learns the fingerprint of the generators in its training set, and a new one has a different fingerprint. Sun, Jia, Hou and Lyu showed the cliff directly: a neural-vocoder detector scoring 0.13% equal error rate on the vocoder it trained on rises to between 4.6 and 45.35 percent on unseen vocoders (Sun, Jia, Hou, Lyu, CVPRW 2023). The survey names the mechanism as the “poor generalization of existing detection methods to unknown fake attacks” (Yi, Wang, Tao et al., 2023). Since voice-generation systems ship faster than detectors retrain, the clip you care about is often from a generator the detector never saw.
Why does phone compression make it worse?
Because compression strips the fine detail a detector reads and adds artifacts of its own. Negroni, Cuccovillo, Bestagini et al. (ICASSP 2026) measured detector equal error rates under common codecs at 21.8 percent for M4A, 29.2 percent for EnCodec, and 40.9 percent for MP3. A real-world clip, a voice note, a call recording, a forwarded message, has almost always been compressed at least once, so the detector is reading a degraded signal before the generalization gap is even counted. It is why the first step in how to tell if audio is AI-generated is to test the original recording, not a re-shared copy.
Can the detector be reading the room instead of the voice?
Often, yes, and it inflates lab scores in a way that does not transfer. Negroni and colleagues found that many models “base their decisions on unvoiced timesteps, frequency bands where no speech occurs, or even background noise, rather than the actual speech.” Salvi and colleagues found the same signal from the other side: a detector trained on the noise component alone beat the full-signal and speech-only versions on every dataset they tested, by 0.71 to 0.53 balanced accuracy on FakeOrReal and 0.69 to 0.53 on the In-the-Wild set (Salvi, Balcha, Bestagini, Tubaro, ICASSPW 2024). A model that learned the recording conditions of its training set will read a new clip’s conditions, not its voice, and there is no reason those conditions match. This is why a detector can be both confident and wrong on a clip that merely sounds, at the file level, unlike its training data.
Why doesn’t a harder benchmark or better hardware fix it?
Because the difficulty is built into realistic synthesis, not the measurement. The people who built ASVspoof 2019 noted their database included spoofed audio that “cannot be differentiated from bona fide utterances even by human subjects” (Wang, Yamagishi, Todisco et al., Computer Speech & Language 2020). If the best fakes are indistinguishable to people, a detector’s edge comes from model-specific traces, and those are exactly what an unseen generator, a codec, or a re-recording remove. The replay side shows the same pattern in hardware: a loudspeaker-distortion detector reaches a 97.78 percent true-positive rate on known equipment and falls to a 79 to 90 percent floor on unseen playback hardware (Ren, Fang, Liu, 2019).
What does this mean for trusting a result?
It means a benchmark number is a ceiling, not a promise. The lab produces the reassuring figures and the wild produces the real ones. Treat a confident verdict on an unfamiliar generator, a compressed clip, or a noisy recording as weak evidence, corroborate with provenance and a second method, and reserve “undetermined” for the cases, degraded, unfamiliar, unverifiable, where an honest tool cannot know. For how far any of these tools can be trusted across media, see are AI detectors accurate.
Sources
- Müller, Czempin, Dieckmann et al. (2022). Does Audio Deepfake Detection Generalize? Interspeech 2022.
- Yi, Wang, Tao et al. (2023). Audio Deepfake Detection: A Survey.
- Sun, Jia, Hou, Lyu (2023). AI-Synthesized Voice Detection Using Neural Vocoder Artifacts. CVPRW 2023.
- Negroni, Cuccovillo, Bestagini et al. (2026). Multi-Task Transformer for Explainable Speech Deepfake Detection via Formant Modeling. ICASSP 2026.
- Salvi, Balcha, Bestagini, Tubaro (2024). Listening Between the Lines: Synthetic Speech Detection Disregarding Verbal Content. ICASSPW 2024.
- Wang, Yamagishi, Todisco et al. (2020). ASVspoof 2019: A Large-Scale Public Database of Synthesized, Converted and Replayed Speech. Computer Speech & Language 2020.
- Ren, Fang, Liu (2019). Replay Attack Detection Based on Distortion by Loudspeaker for Voice Authentication. Multimedia Tools and Applications 2019.