A glimpse of SK hynix’s booth at the AI Hardware & Edge AI Summit 2024

SK hynix unveiled an enhanced Accelerator-in-Memory based Accelerator (AiMX) card at the AI Hardware & Edge AI Summit 2024 held September 9–12 in San Jose, California. Organized annually by Kisaco Research, the summit brings together representatives from the AI and machine learning ecosystem to share industry breakthroughs and developments. This year’s event focused on exploring cost and energy efficiency across the entire technology stack.

Marking its fourth appearance at the summit, SK hynix highlighted how its AiM¹ products can boost AI performance across data centers and edge devices².

¹Accelerator in Memory (AiM): SK hynix’s PIM semiconductor product name, which includes GDDR6-AiM.
²Edge device: Hardware that controls the flow of data at the boundary between two networks. While they fulfill numerous roles, edge devices essentially serve as the entry or exit point to a network.

Attendees gather to learn more about the upgraded AimX card

Booth Highlights: Meet the Upgraded AiMX

In the AI era, high-performance memory products are vital for the smooth operation of LLMs³. However, as these LLMs are trained on increasingly larger datasets and continue to expand, there is a growing need for more efficient solutions. SK hynix addresses this demand with its PIM⁴ product AiMX, an AI accelerator card that combines multiple GDDR6-AiMs to provide high bandwidth and outstanding energy efficiency.

³Large language model (LLM): Advanced AI systems that require extensive datasets to train models to understand and generate human-like language. It enables applications like natural language processing and translation.
⁴Processing-In-Memory (PIM): A next-generation technology that embeds processing capabilities within memory, minimizing data transfer between the processor and memory. This boosts efficiency and speed, especially for data-intensive tasks like LLMs, where quick data access and processing are essential.

The 32 GB AiMX prototype card was shown publicly for the first time at the event

At the AI Hardware & Edge AI Summit 2024, SK hynix presented its updated 32 GB AiMX prototype which offers double the capacity of the original card featured at last year’s event. To highlight the new AiMX’s advanced processing capabilities in a multi-batch⁵ environment, SK hynix held a demonstration of the prototype card with the Llama 3⁶ 70B model, an open source LLM. In particular, the demonstration underlined AiMX’s ability to serve as a highly effective attention⁷ accelerator in data centers.

⁵Multi-batch: A computer processing method in which the system groups together multiple tasks (batches) and processes them at once.
⁶Llama 3: An open source LLM developed by Meta, featuring pretrained and instruction-fine-tuned language models.
⁷Attention: Mechanisms which give LLMs context about text, lessening the model’s chance of misunderstandings and allowing it to generate more accurate and contextually relevant outputs.

The upgraded AiMX was demonstrated with the Llama 3 70B model LLM to highlight its processing capabilities

AiMX addresses the cost, performance, and power consumption challenges associated with LLMs in not only data centers, but also in edge devices and on-device AI applications. For example, when applied to mobile on-device AI applications, AiMX improves LLM speed three-fold compared to mobile DRAM while maintaining the same power consumption.

Featured Presentation: Accelerating LLM Services from Data Centers to Edge Devices​

Euicheol Lim presenting on how the AiMX system accelerates LLM services

On the final day of the summit, SK hynix gave a presentation detailing how AiMX is an optimal solution for accelerating LLM services in data centers and edge devices. Euicheol Lim, research fellow and head of the Solution Advanced Technology team, shared the company’s plans to develop AiM products for on-device AI based on mobile DRAM and revealed the future vision for AiM. In closing, Lim emphasized the importance of close collaboration with companies involved in developing and managing data centers and edge systems to further advance AiMX products.

Looking Ahead: SK hynix’s Vision for AiMX in the AI Era

The AI Hardware & Edge AI Summit 2024 provided a platform for SK hynix to demonstrate AiMX’s applications in LLMs across data centers and edge devices. As a low-power, high-speed memory solution able to handle large amounts of data, AiMX is set to play a key role in the advancement of LLMs and AI applications.

The post SK hynix Presents Upgraded AiMX Solution at AI Hardware & Edge AI Summit 2024 first appeared on SK hynix Newsroom.

SK hynix showcased its next-generation memory semiconductor technologies and solutions at the OCP Global Summit 2023 held in San Jose, California from October 17–19.

The OCP Global Summit is an annual event hosted by the world’s largest data center technology community, the Open Compute Project (OCP), where industry experts gather to share various technologies and visions. This year, SK hynix and its subsidiary Solidigm showcased advanced semiconductor memory products that will lead the AI era under the slogan “United Through Technology”.

▲ Figure 1. SK hynix’s exhibition booth at the OCP Global Summit 2023

At the Booth: Leading Global AI Memory Technologies in the Spotlight

▲ Figure 2. HBM(HBM3/HBM3E), MCR DIMM, DDR5 RDIMM, and LPDDR CAMM products on display at SK hynix’s booth

SK hynix presented a broad range of its solutions at the summit, including its leading HBM¹(HBM3/3E), CXL², and AiM³ products for generative AI. The company also unveiled some of the latest additions to its product portfolio including its DDR5 RDIMM, MCR DIMM, enterprise SSD (eSSD), and LPDDR CAMM⁴ devices.

¹High Bandwidth Memory (HBM): A high-value, high-performance product that possesses much higher data processing speeds compared to existing DRAMs by vertically connecting multiple DRAMs with through-silicon via (TSV).
²Compute Express Link (CXL): A next-generation memory solution that increases the memory bandwidth of server systems with traditional DRAM products to improve performance and easily expand memory capacity.
³Accelerator in Memory (AiM): SK hynix’s PIM semiconductor product name, which includes GDDR6-AiM.
⁴Low Power Double Data Rate Compression Attached Memory Module (LPDDR CAMM): A solution developed based on the LPDDR package in line with the next-generation memory standard (CAMM) for laptops and mobile devices. Compared to conventional So-DIMM modules, LPDDR CAMM has a single-sided configuration which is half as thick and offers improved capacity and power efficiency.

Visitors to the HBM exhibit could see HBM3, which is utilized in NVIDIA’s H100, a high-performance GPU for AI, and also check out the next-generation HBM3E. Due to their low-power consumption and ultra-high-performance, these HBM solutions are more eco-friendly and are particularly suitable for power-hungry AI server systems.

▲ Figure 3. CXL-based CMS 2.0, pooled memory, and memory expander solutions being demonstrated at the event

Three of SK hynix’s key CXL products were demonstrated at the event, including its CXL-based computational memory solution (CMS)⁵ 2.0. A next-generation memory solution that integrates computational functions into CXL memory, CMS 2.0 uses NMP⁶ to minimize data movement between the CPU and memory. CMS 2.0 was applied to SK Telecom’s spatial data analysis and visualization solution based on Lightning DB which analyzes foot traffic in real time, illustrating CMS 2.0’s data processing power equivalent to a server CPU. Moreover, the demonstration highlighted that CMS 2.0 can improve data processing performance and energy efficiency.

⁵Computational Memory Solution (CMS):  A memory solution that offers the functions of machine learning and data filtering, which are frequently performed by big data analytics applications. Just like CXL, CMS’s memory capacity is highly scalable.
⁶Near-Memory Processing (NMP): A memory architecture which moves the compute capability next to the main memory to address CPU memory bottlenecks and improve processing performance.

The company also demonstrated its CXL-based pooled memory solution which can significantly reduce idle memory usage and shorten the overhead time of data movement in distributed processing environments such as AI and big data. The demonstration, which featured a technological collaboration with software provider MemVerge, showcased how CXL-based pooled memory solutions can improve system performance in AI and big data distributed processing systems.

Additionally, SK hynix demonstrated its CXL-based memory expander applied to Meta’s software caching engine, CacheLib, which showed how the CXL-based memory solution optimizes software for improved performance and cost savings.

▲ Figure 4. PIM and AiMX on display at the summit

AiM, a game-changing technology that redefines memory’s role in AI inference, was also presented at the booth. A subset of SK hynix’s PIM⁷ portfolio, AiM promises to revolutionize machine learning, high-performance computing, and big data applications while reducing operating costs. The company demonstrated its GDDR6-AiM, a solution which incorporates computational capabilities into the memory and improves the performance and efficiency of data-intensive generative AI inference systems. Additionally, a demonstration was held for the prototype of AiMX⁸, an innovative generative AI accelerator card based on GDDR6-AiM.

⁷Processing-In-Memory (PIM): An advanced technology that combines computational capabilities and memory within just one die to perform the best computing capability.
⁸Accelerator-in-Memory based Accelerator (AiMX): SK hynix’s accelerator card featuring a GDDR6-AiM chip which is specialized for large language models.

▲ Figure 5. PS1010 E3.S, a PCIe Gen5 eSSD product, showcased at the booth

SK hynix also showcased the PS1010 E3.S, a PCIe (Peripheral Component Interconnect Express) Gen5-based eSSD. Designed for data-intensive applications, cloud computing, and AI workloads, the PS1010 E3.S promises improved performance and reliability with superior speed and lower carbon emissions compared to previous generations.

At the Sessions: Sharing Industry Insights & Key Technologies

▲ Figure 6. SK hynix’s Hoshik Kim, fellow of the System Architecture Group in Memory Forest x&D, holds a session on CXL technology

Through thought-provoking talks and sessions at the summit, SK hynix also suggested the future development of next-generation memory solutions.

In a panel session titled “Data Central Computing, Present and Future”, Hoshik Kim, fellow of the System Architecture Group in Memory Forest x&D, joined other industry experts to omputing and programming models. Kim also held an executive session in which he revealed SK hynix’s progress in developing CXL technology and shared the company’s vision under the theme of “CXL: A Prelude to a Memory-Centric Computing”.

▲ Figure 7. SK hynix’s Youngpyo Joo, Dongwuk Moon and Jungmin Choi of Memory Forest x&D giving talks on CXL-based solutions

The potential of CXL was also the focal point of three other sessions held by the company’s employees from the Memory x&D department. Youngpyo Joo, head of the Software Solutions Group, explored how CXL-based computational memory solution architecture can offer increased usability and flexibility.

Meanwhile, Donguk Moon, technical leader of the Platform Software team, introduced how emerging CXL devices can enhance memory capacity and bandwidth at the caching layer for technologies such as AI and web services. In addition, Jungmin Choi, technical leader of the Composable Memory team, covered how the latest CXL technology will further support the disaggregation of memory. Choi also emphasized that this technology can not only solve problems currently faced by data centers such as idle memory, but also improve system performance.

▲ Figure 8. Euicheol Lim, head of SK hynix’s Memory Solution Product Design Group, discusses PIM-based AiM and PIM technologies

In another engaging session, Euicheol Lim, head of the Memory Solution Product Design Group, explored the potential for PIM technology to meet the high memory and computational demands of AI. Lim highlighted key technologies based on PIM, including SK hynix’s PIM-based AiM accelerator.

Advanced Solutions to Unlock Next-Generation Technologies

As the OCP Global Summit 2023 drew to a close, SK hynix reinforced its commitment to developing groundbreaking solutions which can help realize advanced technologies such as AI. Going forward, the company will continue to tackle industry challenges and make technological breakthroughs for the AI era as the global no.1 AI memory solution provider.

The post SK hynix Displays Next-Gen Solutions Set to Unlock AI & More at OCP Global Summit 2023 first appeared on SK hynix Newsroom.

¹Accelerator-in-Memory based Accelerator (AiMX): SK hynix’s accelerator card product that specializes in large language models (AI that learns with large amounts of text data such as ChatGPT) using GDDR6-AiM chips.

²Accelerator: A special-purpose hardware device that uses a chip designed specifically for processing and computing information.

Figure 1. SK hynix’s exhibition booth at the AI Hardware & Edge AI Summit 2023

Hosted annually by the UK marketing firm Kisaco Research, the AI Hardware & Edge AI Summit brings together global IT companies and high-profile startups to share their developments in artificial intelligence and machine learning. This is SK hynix’s third time participating in the summit.

At the event, the company showcased the prototype of AiMX, an accelerator card that combines multiple GDDR6-AiMs to further enhance performance, along with the GDDR6-AIM itself under the slogan of “Boost Your AI: Discover the Power of PIM³ with SK hynix’s AiM⁴.”

³Processing-In-Memory (PIM): A next-generation technology that adds computational capabilities to semiconductor memories to solve the problem of data movement congestion in AI and big data processing.

⁴Accelerator in Memory (AiM): SK hynix’s PIM semiconductor product name, which includes GDDR6-AiM.

Figure 2. The prototype AiMX card utilizes multiple GDDR6-AiM chips for enhanced performance

As a low-power, high-speed memory solution capable of handling large amounts of data, AiMX is set to play a key role in the advancement of data-intensive generative AI⁵ systems. The performance of generative AI improves as it is trained on more data, highlighting the need for high-performance products which can be applied to an array of generative AI systems.

⁵Generative AI: AI that learns from large amounts of data to actively generate results based on a user’s specific needs.

Figure 3. Demonstrating a large AI language model with AiMX that utilizes GDDR6-AiM

SK hynix also demonstrated Meta’s generative AI Open Pretrained Transformer (OPT) 13B model on a server system equipped with the AiMX prototype. The AiMX system featuring GDDR6-AiM chips reduces data processing time by more than 10 times compared to systems with GPUs, while consuming one-fifth the power. The company’s demonstration piqued the interest of global companies providing AI services by showing that it can deliver higher performance⁶ than the most recent accelerators.

⁶Performance is based on the condition that the AiM Control Hub inside the AiMX card is developed as an application-specific integrated circuit (ASIC).

Figure 4. Eui-cheol Lim, vice president of SK hynix’s Solution Development division, delivers a presentation on AiMX

In addition, the company held a session outlining the benefits of AiMX. In a presentation titled “Cost-Effective Generative AI Inference Acceleration using AiM,” Eui-cheol Lim, vice president of the Solution Development division, compared the performance of GPUs and AiMX and discussed the future of next-generation intelligent semiconductor memories.

“SK hynix’s AiMX is a solution that delivers higher performance while consuming less power, and costing less than conventional GPUs,” Lim explained. “We will continue to develop memory technologies that will lead the way in the era of artificial intelligence.”

The post SK hynix Debuts Prototype of First GDDR6-AiM Accelerator Card ‘AiMX’ for Generative AI first appeared on SK hynix Newsroom.

The post How SK hynix is Set to Power the Generative AI Revolution first appeared on SK hynix Newsroom.

SK Group’s Vision of a Carbon-Free Future

Figure 1. SK hynix and seven other affiliates of SK group presented their ‘Net Zero’ aspiration under the slogan “Together in Action”.

SK companies have teamed up with 10 US-based partners to present their booth under the slogan “Together in Action,” the group’s message to turn their Net Zero aspiration into action. At CES 2022, the group committed to reduce carbon emissions by 200 million tons by 2030, or 1% of the global carbon reduction targets while maintaining its goal of achieving Net Zero by 2050.

This year, eight of SK’s affiliate companies are showcasing a total of 40 carbon reducing technologies as part of the group’s vision of a carbon-free future. SK hynix is expected to attract major tech customers and experts to its booth with its “Green Digital Solution,” the theme of its product lineup at this year’s event.

A Green Digital Solution

Figure 2. SK hynix is exhibiting a range of new and core products which offer significant performance improvements while lessening their environmental impact.

SK hynix is exhibiting a range of new and core products at this year’s CES which will offer significant improvements in performance and energy efficiency while lessening the impact on the environment. These products meet the growing needs of global tech companies which are requiring more powerful memory solutions for evolving advanced technologies such as AI, Big Data, autonomous driving, and the Metaverse.

The focal point of SK hynix’s exhibition is the PS1010 E3.S, the company’s latest enterprise solid-state drive (eSSD) which is being launched at CES 2023. The product features multiple 176-layer 4D NAND flash technology that supports PCIe 5.0 (Peripheral Component Interconnect Express) interface. The PS1010 E3.S, which includes a self-developed controller and firmware, offers increased reading and writing speed of 130% and 49%, respectively, compared to the previous generation of products. Moreover, as its performance per watt is 75% more efficient than its predecessors, it will help to reduce server operating costs and cut carbon emissions. The product is set to strengthen SK hynix’s competitiveness in the NAND sector.

Visitors to SK hynix’s booth can also see its HBM3, the world’s best-performing DRAM developed and mass produced by the company in an industry-first. As a Gen 4 HBM (High Bandwidth Memory) product, it offers increased data processing speeds compared to existing DRAMs, while also improving power efficiency by 23% compared to HBM2. Displayed alongside the HBM3 is another leading innovation, the GDDR6-AiM. This AI accelerator offers its own computational capabilities that significantly increase certain computation speeds and reduces energy consumption by 80% compared to existing products.

Last summer, SK hynix reinforced its superiority in Compute Express Link (CXL) solutions by developing its first DDR5 DRAM-based CXL memory samples which have flexible expansions of memory capacity and performance. The company also collaborated with SK Telecom to produce its Computational Memory Solution (CMS), the industry’s first CXL memory that’s integrated with computational functions. SK hynix’s CXL portfolio, which is planned to be mass-produced later this year, can also be seen at CES.

Sustainability Initiatives

Figure 3. Visitors experienced SK hynix’s innovative technology presented under the theme of a “Green Digital Solution”.

It is clear from the SK hynix product lineup at CES 2023 that the company is focused on simultaneously pushing the limits of technology while promoting sustainability in order to solve market and environmental challenges. These products align with SK hynix’s overall ESG strategy which drive the company’s future plans.

One of the key aspects of SK hynix’s ESG initiatives is undoubtedly the PRISM framework which was unveiled last year to enhance its ESG management. Detailing SK hynix’s ESG goals, PRISM relays the message that the company will transparently communicate with stakeholders while also reflecting its goal of spreading a positive influence around the world.

As a leader in the semiconductor industry’s fight against climate change, SK hynix joined the Semiconductor Climate Consortium (SCC) as a founding member last October. The SCC is the first global consultative body formed to reduce greenhouse gas emissions throughout the semiconductor value chain. SK hynix’s efforts to tackle greenhouse gases are further exemplified by its carbon footprint certification for its eSSD and cSSD products. Issued by the U.K. climate change organization Carbon Trust last summer, the certification assesses the impact of carbon emissions throughout the entire lifecycle of a product.

Building a Greener Tomorrow

Figure 4. SK hynix is working with other SK affiliates to build a greener tomorrow.

At a time when people are concerned about the environmental impact of their products, SK hynix is responding to their needs with its range of energy-efficient, high-performance products displayed at CES 2023. SK hynix is using this opportunity to discuss with fellow industry insiders and tech consumers about key industry and environmental issues to build a greener tomorrow.

The post SK hynix Presents Its Green Digital Solution at CES 2023 first appeared on SK hynix Newsroom.

• Core and brand-new products introduced under the theme of “Green Digital Solution”
• Introduction of eSSD with ultrahigh-performance to solidify SK hynix’s leadership in server memory market
• Solution to solve customers’ pain point proposed

Seoul, December 27, 2022

SK hynix Inc. (or “the company”, www.skhynix.com) announced today that it will showcase a number of its core and brand-new products at the CES 2023, the most influential tech event in the world taking place in Las Vegas from Jan 5^th through Jan 8^th.

The products, introduced under the theme of the “Green Digital Solution,” as part of the SK Group’s “Carbon-Free Future” campaign, are expected to attract Big Tech customers and experts given the significant improvement in performance and energy efficiency compared with the previous generation as well as the effect of lessening the impact on the environment.

Attention on energy-efficient memory chips has been on the rise as global tech companies pursue products that process data faster, while consuming less energy. SK hynix is confident that its products to be displayed at the CES2023 will meet customers’ such needs with outstanding performance per watt* and performance.

* Performance per watt: an indicator of how much computation is performed per watt of power consumed.

The core product put forward at the show is PS1010 E3.S, an eSSD product composed of multiple 176-layer 4D NAND that supports the fifth generation of the PCIe* interface.

*PCIe (Peripheral Component Interconnect Express): a high-speed input/output series interface used in the mainboard of digital devices. PCIe’s data-tranfer speed doubles in accordance with a generation shift.

SK hynix said that the introduction of the PS1010, a combination of the company’s industry-leading technologies, was made timely as the server chip market continues to grow despite the current industry downturn.

The PS1010 product shows improvement both in reading and writing speed by 130% and 49%, respectively, compared with the previous generation. Its performance-per-watt is also improved by more than 75%, helping customers reduce costs to run servers and carbon emission.

“We’re proud to launch PS1010, an ultrahigh-performance product with self-developed controller and firmware, at the CES 2023, the world’s largest technology show,” Yun Jae Yeun, Head of NAND Product Planning, said. “This product will solve pain point of our server-chip customers, while paving the way for a stronger competitiveness in NAND business for us.”

Other products to be introduced at the show are HBM3*, a memory product with the world’s best specification for high performance computing, GDDR6-AiM that adopts the PIM* technology and CXL* memory capable of flexible expansion of memory capacity and performance.

* HBM (High Bandwidth Memory): High-value, high-performance memory that vertically interconnects multiple DRAM chips and dramatically increases data processing speed in comparison to traditional DRAM products.

* PIM (Processing In Memory): A next-generation technology that provides a solution for data congestion issues for AI and big data by adding computational functions to semiconductor memory

* CXL (Compute Express Link): A PCIe-based next-generation interconnect protocol on which high-performance computing systems are based

SK hynix will also present the immersion cooling* technology of SK enmove, which specializes in energy efficiency. The technology, designed to help cool down the heat of the servers generated during the operation, marks a successful case where SK hynix cooperated with other SK companies or external business partners to create new values in the semiconductor business.

* Immersion Cooling: A next-generation thermal-management technology that cools down temperature by submerging data servers into cooling oil. This way, the total electricity consumption can be reduced by 30% compared with the existing technology that uses air to cool down temperature.

About SK hynix Inc.

SK hynix Inc., headquartered in Korea, is the world’s top tier semiconductor supplier offering Dynamic Random Access Memory chips (“DRAM”), flash memory chips (“NAND flash”) and CMOS Image Sensors (“CIS”) for a wide range of distinguished customers globally. The Company’s shares are traded on the Korea Exchange, and the Global Depository shares are listed on the Luxembourg Stock Exchange. Further information about SK hynix is available at www.skhynix.com, news.skhynix.com.

Media Contact

SK hynix Inc.
Global Public Relations

Technical Leader
Kanga Kong
E-Mail: global_newsroom@skhynix.com

Technical Leader
Jaehwan Kevin Kim
E-Mail: global_newsroom@skhynix.com

The post SK hynix to Showcase Energy-Efficient, High-Performance Memory Products at CES 2023 first appeared on SK hynix Newsroom.

The Open Compute Project (OCP) is an initiative that proposes a new direction in which both software and hardware disciplines are tightly connected in order to efficiently address the increasing requirements of compute infrastructure. It gathers major decision-makers, technologists, engineer developers, and suppliers to exchange insights, discuss challenges, and provide solutions in and around the data center.

The OCP is also characterized by open source and open collaboration, with a global committee covering everywhere from the telecommunications industry to edge infrastructure, and everything in between.

The OCP Global Summit, held in the Fall each year, provides a unique opportunity to see and explore the various ways in which the data center industry is innovating and evolving. SK hynix attended the 2022 OCP Global Summit in San Jose, California, from October 18 to 20, to demonstrate its latest products, technologies, and ideas.

SK hynix, a Proud Supporter of OCP

Image 1. SK hynix’s booth at OCP Global Summit 2022 drew enthusiasm from participants and visitors

As a Ruby Sponsor, SK hynix set up a booth at OCP with the theme “Driving New Level of Performance for your Ultimate Data Experience,” showcasing latest research developments in next-generation memory technologies. Having attended every year since 2016, this year’s event was particularly memorable as it reverted to a face-to-face format for the first time since the Covid pandemic began.

At the event, the company unveiled the PCIe Gen5 PS1010 for the first time as well as highlighted offerings such as the DDR5 DRAM-based CXL (Compute Express Link) memory solution and next-generation intelligent semiconductor GDDR6-AiM, demonstrating how the company has truly become a total solution provider.

Demonstrating a Complete Solution Portfolio

Image 2. SK hynix demonstrated a portfolio of products on-site at the OCP Global Summit 2022 including a first look at PS1010

Four products were demonstrated by SK hynix at the OCP Global Summit 2022, starting with the PS1010. The PS1010 is a PCle (Peripheral Component Interconnect Express) Gen5 total solution product that handles all SoC/DRAM/NAND and operations in-house, with data transfer speeds more than doubling over the previous generation due to the usage of the PCIe Gen5 interface. It was the first time SK hynix unveiled the product, emphasizing its use of industry-leading V7 NAND to ensure cost and performance competitiveness.

Second to be featured was the company’s DDR5 DRAM-based CXL memory solution that integrates computational functions into CXL memory for the first time in the industry. The solution is expected to be installed on next-generation server platforms in order to improve system performance and energy efficiency.

Third, a sample of the EDSFF (Enterprise & Data Center Standard Form Factor) E3.S was demonstrated following the announcement of the development of the latest DDR5 24Gb DRAM-based 96GB CXL memory sample. The combination of DDR5 and CXL memory was shown to reduce memory execution time by 20% and increase bandwidth by 50%, resulting in a 60% memory capacity expansion impact.

Finally, a sample of GDDR6-AiM (Accelerator in Memory) demonstrated the behavior of a huge AI model aimed at memory-centric computing. GDDR6-AiM combines processing function onto a typical DRAM, and can reduce energy consumption by up to 80% while providing up to 16 times the performance and lower operating voltage.

Thought Leadership in Memory Solutions

Image 3. Thomas Won Ha Choi sharing SK hynix’s perspective on the value of its new CXL memory

Besides the demonstrations that attracted a great deal of attention, SK hynix also participated in three session discussions at the OCP Global Summit. Thomas Won Ha Choi, Director of DRAM Product Planning at SK hynix, shared the company’s perspective on the new value of CXL memory, as well as its vision for planning and enabling the future development of robust CXL memory. He presented SK hynix’s various memory solutions to provide examples of how CXL devices can enhance memory system values.

Image 4. Donguk Moon introduces SK hynix’s CXL-based CMS at his session discussion

Donguk Moon, SK hynix’s Senior Technical Program Manager for Memory System Research, also introduced SK hynix’s CXL-based CMS, as well as collaboration with SK Telecom, one of Korea’s major telecommunications companies, to improve their Lightning DB real-time data management system by incorporating CMS into it.

A Future-Oriented Approach to Data Center Research

While SK hynix has concluded its successful participation in the OCP Global Summit this year, the company’s research continues, beyond the contents presented in the last few days. Next year, SK hynix looks forward to continuing to present important development achievements and new products for the future.

Observing the enthusiasm and passion shown by participants and visitors was inspiring, and further motivates SK hynix’s determination to serve as the ultimate total solution provider for the data center industry.

The post SK hynix Successfully Establishes Itself as a Total Solution Provider at OCP Global Summit 2022 first appeared on SK hynix Newsroom.

When IBM-developed computer Watson beat out its human competitors on the quiz show Jeopardy in 2011, it was thought to be the beginning of the end of the superior reign of human intelligence. Watson brought discussions of AI to the mainstream. Its ability to apply machine learning to gather and analyze massive amounts of data in a flash was something most thought exclusive to sci-fi.

Quintillions of bytes of data are now being generated each day, with the amount of data generated by 2025 predicted to be 181 zettabytes. While this volume of data exceeds far beyond the realm of human consumption, cloud computing, faster processing, faster networks, and faster chips mean it can be processed and applied efficiently. AI isn’t a pipe dream – it’s a reality.

From Synapses to Circuits

Semiconductors supporting AI functions must capitalize on space and provide means for parallel processing for complex tasks. Enter, Processing in Memory chips. The so-called PIM chip integrates a processor with Random Access Memory (RAM) on a single memory module. This structure removes the boundary between memory and system semiconductors, allowing data storage and data processing to happen in the same place.

By eliminating the need for data to traverse modules, response times are greatly improved, allowing for real-time data processing. More traditional computer architectures, which manage processing and storage in separate modules, often fall prey to latency issues, commonly referred to as the von Neumann bottleneck. Adding processing functions to memory semiconductors presents a unique solution to overcome this long-standing problem.

SK hynix unveiled its next-generation PIM in February 2022 at ISSCC in San Francisco. The GDDR6-AiM (Accelerator in Memory) adds computational functions to GDDR6 memory chips, allowing for data to be processed at speeds of up to 16 Gbps.

GDDR6-AiM is also more energy efficient, reducing power consumption by 80% by removing data movement to the CPU and GPU. Advancing technology in a manner that supports a greener and more equitable world is an integral part of SK hynix future vision. GDDR6-AiM can help reduce carbon emissions and shrink the carbon footprint of any technology it’s applied to, advancing SK hynix’s ESG-related goals and expanding their positive impact across their clients’ industries.

While particularly effective in managing the needs of AI-based systems, PIM can be applied to a broad spectrum of technologies. Databases, query engines, data grids, and more all require some version of data storage and processing coupled with custom applications leveraging a variety of inputs.

The next generation of smart memory

Machine Learning vs. Deep Learning

Unbeknownst to many, artificial intelligence is a broad term that describes the science of creating machines that think like humans. The term machine learning marks functionalities that enable computers to perform tasks without explicit programming and includes deep learning, a subset that relies on artificial neural networks.

Deep learning can be seen as the most independent AI system as it manages both feature input and classification. These systems also require vast amounts of data and rely on parallel processes as their algorithms are primarily self-directed once trained.

AI machines, including deep learning models, are already a part of our lives. There are countless real-world AI applications, which only stand to increase. Everything from mobile devices to autonomous vehicles utilize AI models for tasks like location-based recommendation, auto-braking, camera-based object classification, and navigation through complex environments.

The art of computationally mimicking human intelligence takes many forms

Overcoming the Challenges

The road to PIM development was not without detours, roadblocks, and congestion. As the technology continues to advance, there are still obstacles to surmount across design, manufacturing, cost, and more.

Designing PIM requires the application of novel approaches to chip structures. Traditional semiconductors do not need to accommodate near-memory queues or perform parallel functions in the same way PIM chips do. Once onto the manufacturing stage, space and distance considerations become paramount. It is crucial to reduce how far signals must travel without increased cost or risk of thermal issues.

Furthermore, integrated chips such as PIM have an increased dependency on memory – a unique feature that is both a blessing and a curse. Any damage to the memory components could result in compromised data.

With the AI market expected to reach $190 billion by 2025, investment in AI is ripe. According to a Boston Consulting Group and MIT Sloan Management Review study, 83% of businesses say AI is a strategic priority. SK hynix will continue to advance its expertise in the area and lead this growing sector in the years to come.

The post Let PIM Do the Learning: The Brainpower Behind the AI Memory Chip first appeared on SK hynix Newsroom.

(Check here to discover more insights from SNU professor Deog-Kyoon Jeong about AI semiconductor!)

Cloud Computing vs. Edge Computing

Figure 1. Cloud Computing vs. Edge Computing

Image Download

One AI application, which is an antipode to cloud services, is edge computing¹. Applications that require processing massive amounts of input data such as videos or image data must process data using edge computing or transfer the data to a cloud service through wired or wireless communication preferably by reducing the amount of data. Accelerators specifically designed for edge computing for this purpose take up a huge part of AI chip design. AI chips used in autonomous driving are a good example. These chips perform image classification and object detection by processing images that contain massive amounts of data using CNN² and a series of neural operations.

AI and the Issue of Privacy

Another area of AI application is conversational services like Amazon’s Alexa or SK Telecom’s NUGU. However, such services cannot be used widely if privacy is not protected. Conversational AI services, where conversations at home are continuously eavesdropped by a microphone, cannot be developed beyond a simple recreational service by nature, and therefore, many efforts are being made to resolve these privacy issues.

The latest research trend in solving the privacy issue is homomorphic encryption³ . Homomorphic encryption does not transmit users’ voice or other sensitive information such as medical data as is. It is a form of encryption that allows computations of multiplication and addition on encrypted data in the form of ciphertext, which only the user can decrypt, on a cloud service without first decrypting it. The outcome or results are sent to the user again in an encrypted form and only the user can decrypt to see the results. Therefore, no one including the server can see the original data other than the individual user. Homomorphic service requires an immense amount of computation up to several thousand or tens of thousand times more compared to the general plaintext DNN⁴service. The key area for research in the future will be around reducing the service time by dramatically enhancing computation performance through specially designed homomorphic accelerators⁵.

AI Chip and Memory

In a large-scale DNN, the number of weights is too high to contain all of them in a processor. As a result, it has to make a read access whenever it requires a weight stored in an external large capacity DRAM and bring it to the processor. If a weight is used only once and cannot be reused after accessing it, the data that was pulled with considerable amount of energy and time consumption will be wasted. This is an extremely inefficient method as it consumes additional time and energy compared to storing and utilizing all weights in the processor. Therefore, processing an intense amount of data using enormous number of weights in large-scale DNN requires a parallel connection and/or a batch operation that uses the same weights over several times. In other words, there is a need to perform computations by connecting several processors with DRAMs in parallel to disperse and store weight or intermediate data in several DRAMs to reuse them. High speed connection among processors is essential in this structure, which is more efficient compared to having all processors access through one route. And only this structure can deliver the maximum performance.

Interconnection of AI Chips

Figure 3. Interconnection Network of AI Chips

Image Download

The performance bottleneck that occurs when connecting numerous processors depends on the provided bandwidth, latency as well as the form of interconnection. These elements define the size and performance of the DNN. In other words, if one were to deliver ‘N-times’ higher performance by connecting ‘N’ number of accelerators in parallel, bottleneck occurs in the latency and bandwidth provided by the interconnections and will not be able to deliver the performance as one desires.

Therefore, the interconnection structure between a processor and another is crucial in efficiently providing the scalability of performance. In the case of NVIDIA A100 GPU, NVLink 3.0 plays that role. There are 12 NVLink channels in this GPU and each provides 50 GBps in bandwidth. Connecting 4 GPUs together can be done by direct connections using 4 channels each in the form of a clique. But to connect 16 GPUs, an NVSwitch, which is an external chip dedicated just for interconnection, is required. In the case of Google TPU v2, it is designed to enable a connection of a 2D torus structure using Inter-Core Interconnect (ICI) with an aggregate bandwidth of 496 GBps.

Figure 4. NVIDIA’s GPU Accelerator A100 using 6 HBMs
(Source: The Verge)

Image Download

The way in which processors are interconnected has a huge impact on the whole system. For example, if they are interconnected in a mesh or torus structure, the structure is easy to compose as the physical connection between chips is simple. But latency increases proportionally to the distance as it requires hopping over several processors to interconnect between nodes that are far away. The most extreme method would be in the form of a clique that interconnects all processors one to one, but this would lead to a significant increase in the number of chip pins by N!, causing PCB congestion beyond allowable so that in actual design, connecting up to only four processors would be the limit.

Most generally, using a crossbar switch like a NVSwitch is another attractive option, but this method also converges all connections on the switch. Therefore, the more the number of processors you want to interconnect, the more difficult the PCB layout becomes as transmission lines concentrate around the switch. The best method is structuring the whole network in a binary tree, connecting processors at the bottom end, and allocating the most bandwidth to the top of the binary tree. Therefore, creating a binary fat tree will be the most ideal and will be able to deliver maximum performance with scalability.

Neuromorphic AI Chip

Figure 5. Cloud Server Processor vs. Neuromorphic AI Processor

Image Download

Data representation and processing method of processors for cloud servers that serve as DNN accelerators take the form of digital, since the computational structure is fundamentally simulation of NN through software on top of hardware. Recently, there is an increase in research on neuromorphic AI chip which, unlike the previous simulation method, directly mimics the neural network of a living organism and its signals and maps to an analog electronic circuit and performs in the same manner. This method takes the form of being analog in the representation of original data in actual applications. This means that one signal is represented in one node, and the interconnection is by hardwire and not defined by the software, while the weights are stored in an analog form.

Figure 6. Previous semiconductor vs. Neuromorphic semiconductor

Image Download

The advantage of such structure is that it has maximum parallelism to perform with minimum energy. And neuromorphic chips can secure great advantage in certain applications. Because the structure is fixed, it lacks programmability, but it can offer a great advantage in certain edge computing applications of a small scale. In fact, neuromorphic processor has significance in applications such as processing AI signals of sensors used in IoT by delivering high energy efficiency or image classification that requires processing large amounts of video data using CNN of a fixed weight. However, because the weight is fixed, it will be difficult to use in areas of applications that require continued learning. Also, it is difficult to leverage parallelism that interconnects several chips in parallel due to a structural limitation when it comes to large-scale computations, making its actual area of application restricted to edge computing. It is also possible to realize the neuromorphic structure in a digital form, and IBM’s TrueNorth is an example. It is known, however, that the scalability is limited, making it difficult to find wide practical applications.

Current Status of AI Chip Development

To create a smart digital assistant that can converse with humans, Meta (formerly known as Facebook), which needs to process massive amounts of user data, is designing an AI chip specialized to have basic knowledge about the world. The company is also internally developing AI chips that will perform moderation to decide whether to post real-time videos that are uploaded to Facebook.

Amazon, a technology company that mainly focuses on e-commerce and cloud computing, has already developed its own AI accelerator called AWS Inferentiato power its digital assistant Alexa and uses it to recognize audio signals. Cloud service provider AWS has developed an infrastructure that uses the Inferentia chip and provides services for cloud service users that can accelerate deep learning workloads like Google’s TPU.

Microsoft, on the other hand, uses field programmable gate array (FPGA) in its data centers and has introduced a method of securing the best performance by reconfiguring precision and DNN structure according to application algorithms in order to create AI chips optimized not only in current applications, but also in future applications. This method, however, creates a lot of overhead to refigure the structure and logic circuit even if it has identified an optimal structure. As a result, it is unclear that it will have actual benefit because it is inevitably disadvantaged in terms of energy and performance compared to ASIC chips specifically designed for certain purposes.

A number of fabless startups are competing against NVIDIA by developing general-purpose programmable accelerators that are not specialized to certain areas of application. Many companies, including Cerebras Systems, Graphcore, and Groq, are joining the fierce competition. In Korea, SK Telecom, in collaboration with SK hynix, has developed SAPEON and will soon be used as the AI chip in data centers. And Furiosa AI is preparing to commercialize its silicon chip, Warboy, as well.

Figure 7. SAPEON X220
(Source: SK Telecom Press Release)

Image Download

The Importance of the Compiler

The performance of such AI hardware depends greatly on how optimized its software is. Operating thousands or tens of thousands of computational circuits at the same time through systolic array and gathering the outcome efficiently require highly advanced coordination. Setting up the order of the input data to feed numerous computational circuits in the AI chip and make them to work continuously in a lockstep and then transmitting the output to the next stage can only be done through a specialized library. This means that developing an efficient library and the compiler to use them is as important as designing the hardware.

NVIDIA GPU started as a graphics engine. But NVIDIA provided a development environment, CUDA, , to enable users to write programs easily and enabled them to run efficiently on the GPU, which made it popularly and commonly used across the AI community. Google also provides its own development environment, TensorFlow, to help develop software using TPUs. As a result, it supports users to utilize TPU easily. More and more diverse development environments must be provided in the future, which will increase the applicability of AI chips.

AI Chip and its Energy Consumption

TThe direction of AI services in the future must absolutely focus on enhancing the quality of service and reducing the required energy consumption. Therefore, it is expected that efforts will focus around reducing power consumption of AI chips and accelerating the development of energy-saving DNN structure. In fact, it is known that it takes 10^19 floating-point arithmetic in the training of ImageNet to reduce error rate to less than 5%. This is the equivalent to the amount of energy consumed by New York City citizens for a month. In the example of AlphaGo that was used in the game of Go against 9-Dan professional player Lee Sedol in 2016, a total of 1,202 CPUs and 176 GPUs were used in the inference to play Go and estimated 1 MW in power consumption, which is tremendous compared with the human brain using only 20 W.

AlphaGo Zero, which was developed later, became a system of a performance that exceeds AlphaGo merely after 72 hours of training using self-play reinforcement learning with only 4 TPUs. This case proves that there is potential in reducing energy consumption using a new neural network structure and a learning method. And we must continue to pursue research and development on energy-saving DNN structures.

The Future of the AI Semiconductor Market

Figure 8. AI Chip Market Outlook
(Source: Statista)

Image Download

The successful accomplishments made in the field of AI will expand the scope of application, triggering stunning market growth as well. For example, SK hynix recently developed a next-generation intelligence semiconductor memory, or processing-in-memory (PIM)⁶, to resolve the bottleneck issue in data access in AI and big data processing. SK hynix unveiled the ‘GDDR6-AiM (Accelerator in Memory)’ sample as the first product to apply the PIM, and announced the achievement of its PIM development at the International Solid-State Circuits Conference, ISSCC 2022⁷, an international conference of the highest authority in the field of semiconductor, held in San Francisco in the end of February this year.

Figure 9. GDDR6-AiM developed by SK hynix

Image Download

Application systems will further drive a wider AI market and continuously create new areas, enabling differentiated service quality backed by the quality of inference based on a structure of neural network. AI semiconductors, which are the backbone of the AI system, will be differentiated based on how fast and accurately they can conduct inference and training tasks using low energy. Latest research findings show that energy efficiency per se is extremely poor. Therefore, there is an increasing need for research on new neural network structures with a focus not only on function, but also on energy efficiency. And in terms of hardware, the core element that defines energy efficiency lies around improving memory access methods. As such, Processing-In-Memory (PIM), which processes within a memory and not by accessing memory separately, and neuromorphic computing that mimics the neural network by storing synapse weights in analog memory will become important fields of research.

¹Edge computing is a distributed computing paradigm that brings computation and data storage closer to the sources of data.
²Convolutional neural network (CNN) is a type of artificial neural network used in image recognition and processing that is specifically designed to process pixel data.
³Homomorphic encryption is a form of encryption that permits users to perform computations on its encrypted data without first decrypting it.
⁴A deep neural network (DNN) is an artificial neural network with multiple layers between the input and output layers.
⁵An accelerator is a special-purpose hardware made using processing and computation chips.
⁶Processing in memory (PIM, sometimes called processor in memory) is the next-generation technology that provides a solution for data congestion issues for AI and big data by adding computational functions to semiconductor memory. The product based on such technology is sometimes known as a PIM chip.
⁷The International Solid-State Circuits Conference was held virtually from Feb. 20 to Feb. 28 this year with a theme of “Intelligent Silicon for a Sustainable World.

The post The Present and Future of AI Semiconductor (2): Various Applications of AI Technology first appeared on SK hynix Newsroom.