This second article discusses announcements by Samsung, Trendforce, IDC and Neo Semiconductor as well as a combined talk by Western Digital, Meta and Fadu, at the 2024 FMS, gave further insights into the latest developments in NAND flash and DRAM as well as memory controllers.
Jim Elliott, EVP of US Memory Business, Hwaseok Oh, EVP of Solution Prouction Engineering Team and Taeksang Song, CVP of DRAM Solution Development spoke about how the AI revolution is fueling new demands for memory and storage. AI will require memory and storage which is faster, vaster and require less power. This slide was shown to emphasize the growing use of some AI technologies. Using AI users can create large file with images or video more easily than ever before—helping to drive demand for storage capacity.
According to Samsung, each GenAI text-to-image requires the same energy as charging your phone 577 times! This power demand is leading to new energy solutions, such as the use of small modular reactors in data centers (in the US).
Samsung talked about their 9th generation 3D NAND TLC (three bits per cell) product which increases bit density and IO speed by 50% while reducing read by 40% over the prior generation product. The company’s QLC (four bits per cell) product increases bit density by 86% and IO speed by 60% with a 30% read power reduction. The company believes both TLC and QLC products will be needed for AI applications. The image below shows Samsung projections for storage capacities with various form factor server SSDs out to 2025.
Samsung’s 32GB TLC PM9D3a U.2 SSD will be available in the 4th quarter of 2024. Likewise the TLC PM1753 32TB U.2 and E3.S SSD will also be available then. QLC SSDs BM1743 (64/128TB in U.2 and E1.L form factors) are now available. QLC BM1753 32/64/128TB SSDs in U.2, E3.S and E1.L form factors will be available in Q4 2024. Samsung’s 128TB QLC BM1743 has 128% higher sequential writing performance than the prior generation.
Samsung’s PM9D3a 8-channel SSD running on PCIe 5.0 has a 1.9X higher throughput, 40% lower latency, 1.5X higher power efficiency and 2.7X higher performance than the prior generation product in a U.2/E1.L or E3.S FDP/OCP form factor and permits immersion cooling. Samsung also introduced a 16-channel SSD for generative AI server application, the PM1753.
This product saturates the sequential read capability of PCIe Gen 5 and increases sequential writing by 1.6X over the PM1743. Random read performance increases 1.3X and random write performance by 1.7X. Power efficiency is increased up to 1.7X. This product also features post-quantum cryptography and supports SPDM v1.2 and PCIE IDE. Samsung said that idle power for this product is reduced to 5W, with plans for further reductions in coming product generations as shown below.
Reducing the thermal contributions from storage is another important element in data centers and Samsung is addressing this with immersive cooling solutions and materials optimization.
In addition to the impact of AI on storage, Samsung also made announcements on their DRAM to support AI, in particular expanding the memory hierarchy to include near memory, memory expanders and tiered memory to add to main memory, cache and core.
The figure below shows Samsungs projections for HBM and DDR memory advances through 2027. The company’s HBM4 and future solutions will be custom products with a standard core die and customized buffer die. DDR5 RDIMMs will increase from 128GB without TSV (through silicon vias) to 256GB with TSV. Overcoating will be added to reduce damage to die. DDR5 MRDIMM solutions with 8.8Gbps bandwidth and 512GB capacity (with TSVs) are also available.
Samsung also talked about scaling memory capacity and bandwidth using CXL memory pooling. CMM-D memory expansion modules with CXL 2.0 Type 3 interfaces are available with up to 1TB capacity and 36GB/s bandwidth. The companies CMM-H products create new persistent tiered memory options. The companies BMM-B module box provides a pooled memory option with up to 24TB (using CMM-D modules) capacity and hundreds of TB (with CMM-H). The company said that it is seeking a total cost of ownership (TCO) optimized memory expansion solution to enable AI platforms with scalable memory.
Avril Wu from Trendforce said that tight DRAM supplies will persist into 2025. With profit improvements, all the major DRAM companies are increasing their Capex spending by 305% or more. Because of AI he said that 2025 new fab spending is prioritizing HBM supply, with TSV solutions are dominant. AI server bit growth is significantly higher than other applications. Jeff Janucowicz, from IDC, showed that revenues from memory and storage technologies (including HDDs) will undergo significant growth, due to the impact of AI.
Neo Semiconductor CEO Andy Hsu talked about new 3D chip technology to accelerate neural network AI. He saw the company’s 3D memory architecture (3D X-DRAM) in an HBM configuration (3D-XAI) combined with GPUs to provide enhanced AI computational capabilities. 300 memory layers could provide weighting training repositories (synapses) bonded with a neuron chip could provide a faster and more efficient processing architecture. He projected that 120TB/s processing throughput with a 12 die 3D X-DRAM HBM would be possible while consuming much less power than with traditional HBM AI processing
A presentation by Eric Spanneut, VP Marketing, from Western Digital, Ross Stenfort. HW System Storage Engineer at Meta and Anu Murthy, VP of Marketing at Fadu talked about flash memory and AI applications. They pointed out that demand for SSD capacity has increased significantly since 2022, partly due to AI data demands. SSD demand for enterprise applications is driving two different approaches, one involves various type of compute in storage and the other is large object storage as shown in the figure below. AI uses these approaches for disaggregated training and inference and faster access to large curated data repositories. They estimate that over 150EBs of additional storage will be needed by 2028 to support AI (from 2024 demand).
AI with SSDs will lead to increasing use of E1.S form factors (which scales at least to 128TB). Flexible data placement in NVMe will also be important and this will improve endurance, performance and quality of service. Software is being updated to support this feature. The OCP Datacenter NVME SSD V2.6 specification will be released soon with an increased number of hyperscale datacenter and enterprise partners, which enables data management at scale and with open-source tooling support. Open-source SSD storage qualification test cases are available from Meta.
Fadu talked about SSD controller challenges to provide AI performance needs with reduced power and replacing HDDs with high-capacity SSDs. The company says that their SSDs are 36-49% more power efficient that leading competitors, leading to lower total cost of ownership (TCO). Fadu talked about their GEN5 controller, shipping since 2023 14GB/s sequential read and 10GB/s sequential write and 3.3M IOPS random read and 500k IOPS random write using less than 6W of power and built to OCP standards. Fadu also introduced its PCIe Gen6 Sierra controller available in late 2025, which will enable greater than 28GB/s read and write and 6.8M IOPS and supporting up to 256TB SSD capacities.
Samsung, NEO Semicondutor, Meta, Fadu and Western Digital announced major developments and grown of NAND flash, SSDs and DRAM to support AI workloads at the 2024 FMS. High-capacity SSDs with over 500TB capacities could be available before the end of the decade.
