Malicious actors are abusing generative AI music tools to create homophobic, racist, and propagandic songs — and publishing guides instructing others easy methods to accomplish that.
Based on ActiveFence, a service for managing trust and safety operations on online platforms, there’s been a spike in chatter inside “hate speech-related” communities since March about ways to misuse AI music creation tools to put in writing offensive songs targeting minority groups. The AI-generated songs being shared in these forums and discussion boards aim to incite hatred toward ethnic, gender, racial, and spiritual cohorts, say ActiveFence researchers in a report, while celebrating acts of martyrdom, self-harm, and terrorism.
Hateful and harmful songs are hardly a brand new phenomenon. However the fear is that, with the appearance of easy-to-use free music-generating tools, they’ll be made at scale by individuals who previously didn’t have the means or know-how — just as image, voice, video and text generators have hastened the spread of misinformation, disinformation, and hate speech.
“These are trends which can be intensifying as more users are learning easy methods to generate these songs and share them with others,” Noam Schwartz, co-founder and CEO of ActiveFence, told TechCrunch in an interview. “Threat actors are quickly identifying specific vulnerabilities to abuse these platforms in alternative ways and generate malicious content.”
Creating “hate” songs
Generative AI music tools like Udio and Suno let users add custom lyrics to generated songs. Safeguards on the platforms filter out common slurs and pejoratives, but users have found out workarounds, in accordance with ActiveFence.
In a single example cited within the report, users in white supremacist forums shared phonetic spellings of minorities and offensive terms, reminiscent of “jooz” as an alternative of “Jews” and “say tan” as an alternative of “Devil,” that they used to bypass content filters. Some users suggested altering spacings and spellings when referring to acts of violence, like replacing “my rape” with “mire ape.”
TechCrunch tested several of those workarounds on Udio and Suno, two of the more popular tools for creating and sharing AI-generated music. Suno let all of them through, while Udio blocked some — but not all — of the offensive homophones.
Reached via email, a Udio spokesperson told TechCrunch that the corporate prohibits the usage of its platform for hate speech. Suno didn’t reply to our request for comment.
Within the communities it canvassed, ActiveFence found links to AI-generated songs parroting conspiracy theories about Jewish people and advocating for his or her mass murder; songs containing slogans related to the terrorist groups ISIS and Al-Qaeda; and songs glorifying sexual violence against women.
Impact of song
Schwartz makes the case that songs — versus, say, text — carry emotional heft that make them a potent force for hate groups and political warfare. He points to Rock Against Communism, the series of white power rock live shows within the U.K. within the late ’70s and early ’80s that spawned whole subgenres of antisemitic and racist “hatecore” music.
“AI makes harmful content more appealing — think of somebody preaching a harmful narrative a couple of certain population after which imagine someone making a rhyming song that makes it easy for everybody to sing and remember,” he said. “They reinforce group solidarity, indoctrinate peripheral group members and are also used to shock and offend unaffiliated web users.”
Schwartz calls on music generation platforms to implement prevention tools and conduct more extensive safety evaluations. “Red teaming might potentially surface a few of these vulnerabilities and might be done by simulating the behavior of threat actors,” Schwartz said. “Higher moderation of the input and output may also be useful on this case, as it’s going to allow the platforms to dam content before it’s being shared with the user.”
But fixes could prove fleeting as users uncover recent moderation-defeating methods. A few of the AI-generated terrorist propaganda songs ActiveFence identified, for instance, were created using Arabic-language euphemisms and transliterations — euphemisms the music generators didn’t detect, presumably because their filters aren’t strong in Arabic.
AI-generated hateful music is poised to spread far and wide if it follows within the footsteps of other AI-generated media. Wired documented earlier this 12 months how an AI-manipulated clip of Adolf Hitler racked up greater than 15 million views on X after being shared by a far-right conspiracy influencer.
Amongst other experts, a UN advisory body has expressed concerns that racist, antisemitic, Islamophobic and xenophobic content might be supercharged by generative AI.
“Generative AI services enable users who lack resources or creative and technical skills to construct engaging content and spread ideas that may compete for attention in the worldwide market of ideas,” Schwartz said. “And threat actors, having discovered the creative potential offered by these recent services, are working to bypass moderation and avoid being detected — and so they have been successful.”
