ByteDance has reported a 1.71-fold increase in the efficiency of its large language model (LLM) training.

This improvement comes from Comet, an optimized Mixture-of-Experts (MoE) system, as detailed in a paper by its Doubao development team on arXiv.

MoE divides tasks into smaller segments, enabling multiple expert networks to process them concurrently. It is widely used to scale LLMs with trillions of parameters while keeping computing costs stable.

The new system has been deployed across clusters using over 10,000 GPUs, resulting in millions of GPU hours saved.

This efficiency boost could reduce the demand for high-end graphics processing units, such as those manufactured by Nvidia.

MoE is also used in other AI systems like Grok and DeepSeek.

Recent ByteDance developments