Tencent: AI models still struggle in real-world settings

Tencent researchers say current AI models remain “brittle” in real-world settings due to challenges in learning from context, according to a technical paper published on February 4, 2026.

The study, co-authored by Tencent’s chief AI scientist Vinces Yao Shunyu, stresses that AI models must prioritize “context learning” to perform better outside labs.

Tencent created CL-bench, a benchmark testing 19 models on nearly 1,900 tasks that require learning from background information like humans.

The top models averaged only 17.2%, with GPT 5.1 and Anthropic’s Claude Opus 4.5 leading. Tencent’s Hunyuan 2.0 ranked sixth with 17.2%.

Yao suggested that using chat histories from Tencent’s WeChat could improve context understanding, though practical use is uncertain.

🔗 Source: South China Morning Post

🧠 Food for thought

Implications, context, and why it matters.

Tencent’s public AI critique follows an internal reckoning

The paper’s lead author, Yao Shunyu, joined from OpenAI to improve Tencent’s Hunyuan large language model 1.
Soon after arriving, Yao told colleagues that Hunyuan’s evaluation had “big problems,” since leaderboard chasing pushed benchmark data into the training set, which caused data contamination 2.
Data contamination can boost test scores while weakening real-world behavior, which fits the paper’s warning that current models can be “brittle” in real-world settings 2.
CL-bench tests models on nearly 1,900 context-dependent tasks, with the goal of shifting evaluation away from narrow benchmarks and toward real-world performance.

WeChat data could help context learning but raises privacy and governance risks

Yao proposed using WeChat chat histories to improve context learning, drawing on a large platform where Citizen Lab (a University of Toronto research group that studies digital rights and surveillance) reports WeChat sends significant data to Tencent servers through messaging and Mini Programs (WeChat’s in-app lightweight apps) 3.
The idea touches a sensitive area because Tencent has denied storing or analyzing WeChat communications, while reporting describes China’s regulatory environment that requires firms to keep some logs and data to assist authorities 4.
Researchers say WeChat does not use end-to-end encryption (a security method where only the sender and recipient can read messages) for chats, which gives Tencent access to messages sent on the platform 3.
Training AI on this data in the open could heighten surveillance worries and privacy concerns, creating a tough trade-off between model gains and public trust.