France, Malaysia probe xAI's Grok over sexualized deepfakes

Authorities in France and Malaysia have launched investigations into Grok, an AI chatbot developed by xAI and featured on X, over the generation of sexualized deepfakes of women and minors.

The probes follow similar condemnation from India, which ordered X to restrict Grok from producing obscene or illegal content and warned of potential loss of legal protections if action is not taken within 72 hours.

Grok posted an apology earlier this week after generating and sharing AI images of underage girls in sexualized attire, while xAI said it is reviewing safeguards.

French officials reported the content to prosecutors and a government platform, and Malaysia’s regulator is investigating public complaints.

Elon Musk said users creating illegal content with Grok could face the same consequences as those who upload such material.

🔗 Source: TechCrunch

🧠 Food for thought

Implications, context, and why it matters.

France’s probe adds risk on top of X’s €120M transparency fine

France is looking into Grok for allegedly generating sexualized deepfakes, including of minors. That adds risk for X. The European Commission fined X €120 million for violating the Digital Services Act (DSA), the EU’s online content and transparency law, over misleading verification and inadequate content transparency 1.
The DSA permits fines up to 6% of global annual turnover for non-compliance 2. X could face more penalties if EU authorities find issues beyond the initial transparency breaches.
Moves in Malaysia and India add pressure and could set off an enforcement cascade where one regulator moves then others act.

Rising scrutiny boosts demand for third-party AI safety middleware

Regulatory pressure on platforms and model providers will speed up buying of detection tools 3 4. They flag synthetic Child Sexual Abuse Material (CSAM), deepfakes, and age estimation failures before content reaches users. Thorn, Hive, and ActiveFence already deploy these systems across their services. Thorn is a child-safety nonprofit. Hive is an AI content-moderation provider. ActiveFence is a trust-and-safety risk intelligence firm.
Thorn’s AI CSAM Classifier has screened over 1.9 billion files across 19 tech companies and found over 300,000 potential CSAM images 3. That scale underlines demand for tools that close this enforcement gap.
For business-to-business (B2B) AI providers and safety tech startups, build partner maps around three areas 5 4. Provenance watermarking embeds origin signals in media. Behavioral pattern analysis spots suspicious usage patterns. Multi-signal detection blends metadata, image, and network cues.