Concerns Grow Over Hugging Face’s Use in Non-Consensual Deepfakes
An open-source AI platform, Hugging Face, is facing scrutiny following a study indicating its tools are being widely exploited to create non-consensual deepfake pornography. This comes amid ongoing global efforts to regulate such technologies. In the past, deepfake content featuring women and even children has put Elon Musk’s Grok AI in legal jeopardy across various countries.
A detailed investigation by the nonprofit AI Forensics highlighted significant issues related to the creation of sexually explicit deepfakes on Hugging Face, a prominent open-source AI platform valued in the billions. The findings, released recently, raised serious questions about the platform’s content moderation and safety measures.
Researchers from AI Forensics examined nine popular image editing tools, referred to as Spaces, on Hugging Face. Alarmingly, seven of these tools were able to transform images of clothed women into topless versions using just a simple six-word prompt, underscoring gaps in safety protocols on the platform.
To delve deeper into user interactions with these tools, AI Forensics set up a honeypot-style image editing space that monitored user submissions without actually generating images. Over a week, the researchers collected more than 1,000 prompts and images, revealing a troubling trend in how the technology is being misused.
The results revealed that 73% of prompts were of a sexual nature, with 83% aimed at undressing or sexualizing individuals in the photos. Disturbingly, 95% of these targets were women, and about 6.7% of the sexual requests seemed to be directed at children.
Paul Bouchaud, a lead investigator at AI Forensics, stated, “Most spaces are tested. It can be used to generate non-consensual intimate images, and users are actually using it for these purposes.” He emphasized that this is not just a theoretical concern; people are indeed leveraging Hugging Face for such ends.
These findings emerge as regulatory scrutiny on deepfake technology intensifies. Recently, US authorities have acted against deepfake hosting sites, while both the European Union and the UK are planning to ban so-called nudity apps within the year. Nevertheless, AI Forensics’ research indicates that major tech platforms still enable access to tools capable of generating non-consensual images.
The models assessed by AI Forensics did not identify features related to the creation of non-consensual content, presenting themselves merely as general-purpose image editing tools. The researchers pointed out that they did not attempt to bypass security measures, utilizing only straightforward prompts.
“No safety measures are implemented at the platform level. Only developers can choose to implement some if they wish, but most do not,” Bouchaud noted. “Hugging Face makes it easy to filter what comes in and out of the system.”
Leonie Ohmig, from the Institute for Strategic Dialogue, highlighted that many image generation models are trained using sexual content available online. In the absence of effective safety mechanisms, these models can produce explicit content with ease, sometimes openly advertising such functionalities.
Additionally, earlier reports indicated that Apple and Google continue to host numerous “nudify” apps in their app stores. According to findings, these companies actively guide users to these apps through search results and advertisements, as identified by the Tech Transparency Project.
The problem is widespread, with market research firm AppMagic estimating that the identified apps have been downloaded roughly 483 million times, generating about $122 million in revenue. It’s worth mentioning that some apps have been removed or modified in response to advocacy efforts.
Earlier this year, Elon Musk faced backlash after Grok AI released deepfake porn content on the X platform. Following this, xAI, the company behind Grok, restricted its image generation features to paid premium subscribers just a week prior to a significant change in how Grok interacts with users’ image requests.
Moreover, researchers at AI Forensics reported noticing disparities in Grok’s handling of pornographic content production, contrasting public interactions on X with private conversations on the Grok.com site.
While AI companies promote the promising potential of their technologies, the lurking risk is substantial: individuals misusing AI to produce exploitative content. There’s a complex landscape at stake as various factions seek to control the narrative surrounding AI’s advancements.
