Shanghai authorities have summoned the operator of Zhumengdao (筑梦岛), an AI chatbot application, over concerns about inappropriate content generated by virtual characters on the platform.

According to a June 19 statement from the Cyberspace Administration of Shanghai, recent media reports revealed that the app had produced vulgar content, posing risks to minors. Regulators confirmed these issues after investigation and demanded immediate corrective action.

Officials ordered the platform to strengthen its content review system for AI-generated outputs, improve technical safeguards, and clean up harmful content involving minors. The company has pledged to comply and carry out a comprehensive overhaul.

🔗 Source: Huxiu

🧠 Food for thought

1️⃣ AI chatbots repeatedly stumble with inappropriate content despite years of warnings

The Dream Island incident follows a well-documented pattern dating back to Microsoft’s Tay chatbot in 2016, showing the industry has struggled with the same issues for nearly a decade.

Microsoft’s Tay was designed to engage with millennials on Twitter but within just 16 hours began posting racist, sexist, and antisemitic content, forcing the company to shut it down after only 95,000 tweets1.

Despite this high-profile failure, AI chatbots continue to face similar challenges with content moderation, suggesting that fundamental technical solutions remain elusive despite advances in AI capabilities.

Microsoft’s experience demonstrated how vulnerable learning systems are to manipulation, as users deliberately fed Tay offensive content that it then incorporated into its responses2.

Dream Island’s struggles highlight how the core challenge of preventing AI systems from generating harmful content persists across geographic regions and technological generations.

2️⃣ Content moderation evolves toward hybrid human-AI approaches as market grows

The Shanghai authorities’ demands for better review mechanisms reflect an industry-wide recognition that pure AI-based content moderation is insufficient for protecting users.

The market for content moderation solutions is projected to reach $5 billion by 2025, driven by growing regulatory pressure and the need to create safer online environments3.

Industry best practices now favor hybrid approaches that combine automated AI filtering with human oversight, balancing speed with contextual understanding and judgment4.

Social media platforms have increasingly experimented with community-based moderation systems, with companies like Meta and X (formerly Twitter) implementing user-driven content flagging mechanisms to supplement their AI systems5.