An investigation published by WIRED this Monday (2) revealed a network of more than 400 apps and domains connected that presented themselves as ordinary artificial intelligence tools but hid features for producing abusive sexual images and videos from photographs. Some of these apps circulated through Apple's and Google's official stores.
Among the identified services were animal portrait generators, fitness training apps and language learning tools. Depending on access conditions, however, they could load interfaces different from the ones advertised, offering sexual deepfake features and violent content models.
Researchers from the AI Forensics organization identified 27 apps from the network available on Google Play in September, which had accumulated more than 350,000 downloads. One of them, called DiffAI, had surpassed 100,000 installs. The numbers represent downloads, not the number of people who used the abusive features.
Servers allowed features to be hidden during app analysis
The mechanism was identified by researchers who examined the code of the apps and the associated sites. Once installed, the programs sent information to external servers, including language, access origin and user identifiers.
The server's response determined whether the app would present its conventional interface or load a page with deepfake generation tools. One of the possible factors for activating the hidden version was accessing the app through ads published on Facebook and Instagram.
"This is an organized and commercially run operation," Paul Bouchaud, a researcher at AI Forensics, told WIRED. The technical analysis also included work by the security firm Infoblox, which identified nearly 500 related domains.
After being contacted by the reporting team, Apple removed 19 apps from its store. The company said the programs met the rules during the initial review, but that their developers later changed content loaded from external sites.
Google had already removed most of the identified apps before WIRED's contact and did not respond to a request for comment. Meta said it had taken action against 11,600 advertising accounts and blocked 209 apps from advertising on its platforms.
The investigation also found technical and commercial connections with KeLing Technology Limited, a company registered in Hong Kong. Among the evidence were contact information, email addresses and administrative pages associated with the services. The company did not respond to WIRED's requests for comment.
Despite the removals, similar apps and ads kept appearing on the platforms while the report was being completed.
