California Attorney General Rob Bonta has opened an investigation into xAI over reports that its AI model Grok has been used to create nonconsensual sexually explicit deepfake images, including “undressed” sexualized images of women and children. The state has also sent xAI a cease-and-desist letter demanding immediate action to stop the creation and distribution of nonconsensual intimate images and child sexual abuse material (CSAM).
The California Department of Justice says Grok appears to be enabling large-scale production of deepfake nonconsensual intimate images that are being used to harass women and girls across the internet, including on the social platform X. Bonta said the reports have been “shocking” and that California has “zero tolerance” for AI-based creation and dissemination of nonconsensual intimate images or CSAM.
Investigation targets “undressed” AI images
Bonta’s office said it is investigating “the proliferation of nonconsensual sexually explicit material” produced using Grok, the AI model developed by xAI. The announcement describes recent reports in which Grok users took ordinary images of women and children found online and used Grok to depict them in suggestive and sexually explicit scenarios, including by “undressing” them without their knowledge or consent.
The attorney general’s office said Grok-generated images have been used to harass both public figures and ordinary social media users. The press release also described reports alleging Grok was used to alter images of children to depict them in minimal clothing and sexual situations, and that xAI reportedly produced photorealistic images of children engaged in sexual activity.
Cease-and-desist letter and timeline
After announcing the investigation, the California attorney general’s office sent xAI a cease-and-desist letter demanding that the company take immediate action to stop producing nonconsensual intimate images and CSAM. Bonta said he expects xAI to “immediately comply,” and the attorney general’s office said it expects xAI to show what steps it is taking within five days.
The state said xAI has “zero tolerance” for this kind of material and that the investigation will examine whether and how xAI violated the law. It also urged potential victims to file complaints with the California Department of Justice.
“Spicy mode” and how the content spread
The California Department of Justice linked the controversy to Grok’s image generation tools, noting that xAI built image generation models that include what the company calls a “spicy mode,” which generates explicit content. The state said the company used this mode as a marketing point and that it “unsurprisingly” resulted in content that sexualizes people without their consent.
Bonta’s office also cited an analysis that found more than half of 20,000 images generated by xAI between Christmas and New Year’s depicted people in minimal clothing, and said some of those appeared to be children. xAI instituted some restrictions on its image-editing features late Wednesday, but California still moved forward with the cease-and-desist letter.
Wider pressure on platforms
The issue has drawn attention beyond California, with Japan, Canada, and Britain opening investigations into Grok and Malaysia and Indonesia temporarily blocking the platform. X’s safety account previously warned that anyone using or prompting Grok to make illegal content would face the same consequences as users who upload illegal content.
Lawmakers also sent a letter to executives at multiple companies—including X, Reddit, Snap, TikTok, Alphabet, and Meta—asking how they planned to curb the spread of sexualized deepfakes. California framed the Grok investigation as part of broader work focused on AI safety and protecting children.
