Alibaba Cloud cuts Nvidia GPU use by 82% with Aegaeon system

Alibaba Cloud reduced its use of Nvidia GPUs by 82% after introducing a new computing pooling system called Aegaeon.

The company, which provides cloud and AI services for Hangzhou-based Alibaba, presented the findings at the 31st Symposium on Operating Systems Principles in Seoul.

Aegaeon was tested for over three months in Alibaba Cloud’s model marketplace, cutting the number of Nvidia H20 GPUs needed for dozens of large language models, some with up to 72 billion parameters from 1,192 to 213.

The research, co-authored by Alibaba Cloud’s chief technology officer Zhou Jingren and researchers from Peking University, said Aegaeon addresses high costs in running concurrent large language model workloads.

The study found that 17.7% of GPUs previously served only 1.35% of requests, highlighting inefficiencies.

Cloud providers, including Alibaba Cloud and ByteDance’s Volcano Engine, are exploring similar pooling approaches to increase hardware utilization.

🔗 Source: South China Morning Post

🧠 Food for thought

Implications, context, and why it matters.

Aegaeon’s 82% GPU cut depends on workload patterns

The 82% figure comes from Alibaba Cloud’s Model Studio (its model marketplace), where over 90% of models are rarely used and 17.7% of GPUs served 1.35% of requests 1.
Production beta used Nvidia H20 GPUs 1 built for China under U.S. export controls 2, while the paper’s testbed ran on Nvidia H800 80GB GPUs 3. The paper omits the network fabric (the high-speed interconnect between servers/GPUs), and results may hinge on Alibaba’s eRDMA (elastic Remote Direct Memory Access) network 1.
The paper omits end-to-end SLO targets but reports goodput gains and a 97% cut in model-switching latency, while it skips the quality-of-service impact of packing up to seven models per GPU versus the usual two or three 2.

AI platforms can try GPU pooling to lower spend

Teams that host model marketplaces or serve multiple large language models (LLMs) often waste capacity since most receive sporadic requests while holding dedicated GPUs. Aegaeon uses token-level scheduling (allocating compute at the token decoding step) to deliver 1.5x to 9x goodput gains 1.
LMDeploy (an inference serving toolkit) supports multi-model serving with up to 1.8x higher throughput than vLLM (an LLM serving engine) using persistent batching (keeping request batches across steps) and a blocked key-value (KV) cache (a memory layout optimization for the attention cache) 4.
Split image encoding (turning images into vector representations) from language model processing, as in multimodal (text + image) serving research, to enable independent scaling and modality-aware scheduling (routing by data type) 5.