Why stacking memory chips vertically is the secret engine powering the modern AI revolution.

Why stacking memory chips vertically is the secret engine powering the modern AI revolution.
Image: Intellect (generated)

In short: To feed the insatiable hunger of AI processors, engineers stopped spreading memory chips across a board and started stacking them like skyscrapers. High Bandwidth Memory uses vertical connections to move data at blistering speeds, but this efficiency comes at a cost: it is now crowding out the supply of standard computer memory.

High Bandwidth Memory (HBM) represents a fundamental shift in how computers handle data. By stacking up to 32 DRAM dies vertically and connecting them with through-silicon vias, manufacturers create a dense, high-speed memory interface. This architecture allows for a massive bus width—1024 bits in early generations—which dwarfs the 32-bit width of traditional GDDR memory. Because these stacks sit physically close to the processor on a silicon interposer, they achieve superior bandwidth while consuming less power.

The technology has evolved rapidly since SK Hynix produced the first HBM chip in 2013. While HBM1 offered 128 GB/s per package, the industry has pushed through successive generations like HBM2, HBM3, and the recently standardized HBM4. By 2025, HBM4 reached bandwidths of up to 2 TB/s. This performance is critical for AI-focused hardware, such as Nvidia's H100 and Blackwell GPUs, which require massive data throughput to function effectively. Manufacturers like Samsung, SK Hynix, and Micron are now the primary drivers of this innovation.

However, this specialized memory is causing ripples across the broader semiconductor market. Because HBM production is complex and requires significant wafer capacity—Micron estimates that every HBM unit produced consumes three times the wafer space of standard DDR5 memory—the surge in AI demand has led to supply constraints for commodity DRAM. As a result, prices for general-purpose memory have seen sharp increases since 2025. To mitigate these costs and integration challenges, newer standards like SPHBM4 are being developed to allow HBM-class performance on standard substrates, potentially removing the need for expensive silicon interposers.

Beyond raw speed, the industry is exploring ways to make memory smarter. Samsung has pioneered processing-in-memory (PIM) technology, which embeds AI engines directly into memory banks. By performing computations within the memory itself, this approach aims to double system performance and slash energy usage by over 70%. As the industry moves toward HBM5 and custom HBM configurations, the line between storage and processing continues to blur.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/technology-high-bandwidth-memory.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…