🧠 Food for thought

Implications, context, and why it matters.

SOCAMM2 gives data centers a standard memory module

• SOCAMM2 brings low-power LPDDR memory into a removable module. Data centers usually solder this memory to the server board 1.
• The removable design makes repairs and upgrades easier. That matters for large AI systems such as Nvidia’s Vera Rubin, the company’s next-generation AI computing architecture 2.
• SK hynix faces pressure from Samsung. Samsung said it has mass-produced a 192GB SOCAMM2 module and fixed “warpage” issues, when a module bends during manufacturing, before mass production 3.
• Micron is sampling 256GB modules built on a monolithic 32Gb LPDDR5X die, a single-chip design instead of stacked parts. That allows up to 2TB of memory per eight-channel server CPU 4.

A new memory layer for AI systems

• SOCAMM2 sits between High Bandwidth Memory (HBM) on graphics processing units (GPUs) and DDR5 RDIMMs, or registered dual in-line memory modules used in servers. It is built for long-context and agentic AI workloads 4.
• These workloads may need terabytes of memory on the central processing unit (CPU) for Key-Value (KV) caches. KV caches store prior context while a model generates responses, and they can outgrow costly HBM with limited capacity 5.
• More memory can lift performance in uneven jumps. One test found that a 33% capacity increase improved Time-To-First-Token, a measure of how fast a model starts replying, by 2.3 times for large models 4.
• Using more than 70% less power than RDIMMs, these modules cut total cost of ownership (TCO) through lower energy use and less cooling in AI data centers 6.