How Can the Semiconductor Industry Meet AI’s Needs?  

As AI continues to evolve, from early chatbot enhancements to sophisticated, publicly available agentic personal assistants, its full potential remains largely unexplored. While its theoretical limits may be bound only by human imagination, the practical challenge lies in developing scalable, modular architectures capable of supporting the next generation of computing demands. Addressing this need is essential to unveiling AI’s true capabilities and ensuring its seamless integration into increasingly complex applications.

Many of the largest players in the AI and semiconductor industries have come to rely on brute-force solutions: increasingly complex combinations of CPUs, GPUs, neural network accelerators and other processors. These combinations can meet the compute power needs of AI models, as can massive chips like Cerebras’ Wafer Scale Engine 3, the world’s largest chip, which is destined for use in a supercomputer.

Communication bandwidth between compute, memory, and input/output (IO) functions is a bottleneck to the advancement of AI and other data-intensive compute functions, and existing system architectures can’t effectively solve the problem. While there are a number of technologies aiming to solve the scalability issue, such as NVIDIA’S proprietary NVLink and the open-source UALink, neither of them has been widely adopted as yet; indeed, UALink only recently released its first specs.

These issues will only be exacerbated as AI continues to develop. McKinsey analysis suggests that generative AI’s compute demand alone could reach 25 x 10³⁰ FLOPS by 2030. To give an idea of scale, 1 exaFLOP is 10¹⁸ FLOPS, and the supercomputer using the Wafer Scale Engine 3 will clock eight exaFLOPS. Normal PCs or laptops have the power of several hundred gigaFLOPS.

We can theoretically continue to build brute-force architectures to meet AI’s needs for power, but realistically, these complicated architectures will take up too much space, draw too much power, and cost too much to build or re-engineer, leaving AI development in the hands of the very few companies with the money to buy enough chips to make it work. Semiconductors, as an industry, are therefore facing a critical question: to stay with SoCs or not to stay with SoCs?

Confusingly, the answer is more like “yes, and.” The semiconductor industry instead must look to another SoC: System of Chips (plural) or System off Chip. Composed of multiple chiplets, this new SoC architecture prioritizes modularity and flexibility, which by extension reduces costs, as less expensive chiplets can be mixed and matched with higher-end chiplets as needed.

By focusing on chiplets, the industry can reduce the need for monolithic, single-die systems, with all their attendant costs, inefficiencies, and energy demands. For those worried that chiplets trade throughput for flexibility, that’s not the case: Chip-to-chip interconnects retain the throughput benefits conferred by using a NoC architecture.

In the words of Apple (itself a key player in both of these industries), it’s time to “think different.” To keep pressing AI forward, we must broaden our vision beyond compute power demands. They remain important, but they are fundamentally addressable.

Instead, the semiconductor industry must consider multiple dimensions – like cost, energy usage, data transport efficiency, manufacturing capability, and yes, even size. The solutions are already here with NoC and System of Chips architectures. The semiconductor industry can quickly drive this radical rethink by expanding on these technologies and making them more broadly available. Working together with our AI counterparts, we can power the next great AI revolutions.