I bet you woke up this morning saying something like, “I wonder what the latest and greatest interconnect technologies are for System-on-Chip (SoC) devices and multi-die system (MDS) solutions.” Well, this is your lucky day because I’m about to reveal all.

You’ve doubtless heard the popular saying, “Those who fail to learn the lessons of history are doomed to repeat them.” This old chestnut has been passed around and occasionally “tweaked” by many people over the years, including politicians and journalists who sometimes accidentally credit it to Winston Churchill. In fact, this famous phrase originated with the Spanish-born philosopher and writer George Santayana. In George’s 1905 book The Life of Reason, he wrote: “Those who cannot remember the past are condemned to repeat it.”

Talking of tweaks, for the purposes of what is to come, I’d like to offer my own version of George’s aphorism: “Those who fail to learn from history’s mistakes are doomed to repeat them”

The reason for this tweak is that I recently realized there’s another side to the story. In fact, I’m going to coin my own maxim (did you see what I just did there) as follows: “Those who succeed in learning from history’s successes are destined to repeat them.” Remember, you heard this here first.

I can almost hear you muttering under your breath, “What on Earth does any of this have to do with AI and interconnect technologies?” You must learn to control your impatience, Grasshopper, because the short answer is “More than you might think.” 

What’s that, you say? You’d really prefer a longer answer. Well, if you insist… (takes a deep breath)… Over the past sixty-odd years, the computing industry has repeatedly encountered the same fundamental problem: how do you allow an ever-increasing number of things to communicate efficiently with each other? The participants have changed—from functional blocks inside a computer, to entire computers, to intellectual property (IP) blocks inside today’s complex SoCs—but the underlying challenges have remained remarkably constant.

Even more intriguing is that engineers have repeatedly arrived at essentially the same solutions. Different decades. Different names. Different technologies. But the same underlying architectural ideas keep reappearing because they solve the same fundamental problems. In this case, those “successes from history” really are destined to be repeated.

Perhaps the easiest way to wrap our brains around this is to consider three phases of communication in the evolution of computing.

First come simple direct connections. Then comes switching, allowing multiple conversations to occur simultaneously. Finally, as the number of participants grows beyond what centralized switching can economically support, engineers abandon dedicated paths altogether and begin moving packets through distributed networks. This pattern has played out at least three times: first inside computers, then between computers, and now inside today’s SoCs.

History’s successful communication architectures keep repeating themselves (Source: Max Maxfield)

As an aside, an electromechanical form of the crossbar switch first appeared as a practical switching technology in telephone exchanges. It later found its way into computer architecture and eventually became a common interconnect within SoCs, providing a perfect example of a successful idea from history being repeated in different guises.

Good engineering ideas rarely disappear; they simply reappear in new guises (Source: ChatGPT and me)

Since I’m feeling loquacious and my creative juices are flowing (don’t worry, I’ll clean everything up later), let’s take a moment to consider the contents of the above table in a little more detail.

Communication Inside a Computer

The earliest electronic computer systems were relatively modest affairs. A central processing unit (CPU), some random-access memory (RAM), some read-only memory (ROM), a direct memory access (DMA) controller, timers, interrupt controllers, and a handful of peripheral interfaces, all needing to exchange information. The obvious solution was to connect everything to a common bus—a shared collection of wires over which any device could communicate with any other.

The shared bus worked remarkably well for many years. It was simple, inexpensive, and easy to understand (one of the reasons I liked it so much). The catch was that only one conversation could take place at any given moment. If the CPU wanted to access memory while the DMA controller was transferring data to a peripheral, somebody had to wait. As systems became larger and processors became faster, the shared bus gradually transformed from an elegant solution into a performance bottleneck.

One obvious improvement was to replace the shared bus with a crossbar switch, allowing multiple independent transfers to occur simultaneously. For example, the CPU could be fetching instructions from memory while the DMA controller transferred data between a disk controller and RAM, dramatically increasing overall system throughput. Unfortunately, crossbars become increasingly difficult to implement as they grow because each new device may require connections to every other device. As the number of connected devices grows, so do the wiring complexity, routing congestion, silicon area, and cost.

Eventually, designers borrowed ideas from the networking world. Rather than trying to provide a dedicated path between every possible pair of devices, modern interconnects increasingly package information into packets, which are routed through the system by switches. Technologies such as PCI Express may look like point-to-point serial links, but architecturally they are packet-switched interconnects.

Communication Between Computers

The story of computer networking followed a remarkably similar path. In the beginning, computers communicated using dedicated point-to-point connections. If Computer A needed to exchange data with Computer B, you simply ran a cable between them. This approach worked perfectly well—provided you only had a handful of machines. As the number of computers to be connected grew, however, the number of required connections quickly became impractical.

The next step was switching. As we previously discussed, early telephone networks had already demonstrated that a central switching system could establish temporary communication paths between many different users. Computer networks adopted similar ideas, allowing multiple conversations to share the same physical infrastructure without requiring every machine to be permanently connected to every other machine.

As the scale of networking continued to increase, however, even circuit switching gave way to packet switching. Instead of reserving an entire communication path, information was divided into packets that could independently find their way through the network. Today’s Internet remains one of the greatest demonstrations of this principle.

Communication Inside SoCs

Fast-forward to the modern era, and—sure enough—history repeated itself again. Early SoCs typically employed shared on-chip buses such as ARM’s AMBA family or similar proprietary interconnects. At the time, these devices contained relatively few IP blocks, making a shared bus a perfectly reasonable solution.

As SoCs grew to include multiple processor clusters, graphics engines, DSPs, memory controllers, and an ever-expanding collection of specialized IP blocks, shared buses once again became bottlenecks. Crossbar interconnects offered a welcome improvement, allowing multiple transactions to proceed simultaneously, but they too eventually ran into scalability limits.

The natural successor was the Network-on-Chip (NoC), which borrowed many of the concepts pioneered decades earlier in computer networking. In this case, rather than relying on a single enormous centralized switch, packets are routed through a distributed network of on-chip routers, enabling hundreds of IP blocks to communicate efficiently while keeping wiring complexity under control.

But Wait, There’s More!

Just when you thought we’d reached the end of our little history lesson, AI leaped onto the center of the stage with a fanfare of flugelhorns (I’m too young for all this excitement).

Until relatively recently, the communication challenges inside most SoCs revolved around traditional collections of IP blocks: CPU clusters, graphics engines, DSPs, memory controllers, peripherals, and the like. Today’s AI-oriented devices, however, add one or more neural processing units (NPUs)—or, more generally, AI accelerators—into the mix. These accelerators consume and generate enormous amounts of data, fundamentally changing both the volume and the nature of on-chip traffic.

A few days ago, I had a chat with Ashley Stevens and Andy Nightingale from Arteris. This dynamic duo explained that AI isn’t simply demanding “bigger” NoCs. Instead, it’s forcing designers to think about communication in a way that involves a sophisticated combination of coherent and non-coherent interconnect.

One of the most interesting observations Ashley made is that not all traffic deserves to be treated equally. Suppose an NPU is gnawing its way through gigabytes of image data or the intermediate tensors associated with a large language model (LLM). Much of that data may never need to be examined by the CPU. It simply streams from memory, through the accelerator, and back to memory again. Forcing all of this traffic through a coherent interconnect would add unnecessary overhead without providing any tangible benefit.

The situation is very different when the CPU needs to coordinate the work being performed by one or more accelerators. In this case, the information being exchanged is typically much smaller: descriptors, pointers, semaphores, status flags, and other control structures. Although the amount of data is relatively modest, it is essential that every participant sees the most up-to-date version. This is where cache coherency earns its keep.

The result is something of an architectural split. Rather than building one enormous NoC to carry every conceivable type of traffic, many AI SoCs now employ a single coherent NoC that acts as the system’s coherency hub, alongside multiple non-coherent NoCs optimized for moving large volumes of streaming data. Think of the coherent NoC as the conductor of an orchestra, ensuring that every performer plays from the same score, while the non-coherent NoCs quietly move the instruments, scenery, and props backstage as efficiently as possible.

Modern AI SoCs typically combine a single coherent NoC with multiple non-coherent NoCs (Source: Arteris)

This is precisely where Arteris has focused its efforts. Its Ncore IP provides the cache-coherent backbone that allows CPUs and other coherent agents to share data consistently across an SoC. Complementing this is the company’s FlexGen smart NoC IP, which uses AI-assisted automation to generate and optimize non-coherent interconnects for moving large volumes of streaming data efficiently while minimizing wire length, latency, power consumption, and design effort. Rather than treating coherent and non-coherent communication as competing approaches, Arteris treats them as complementary pieces of the same architectural puzzle.

Another interesting point Ashley made is that the coherent NoC doesn’t always look the same. In smaller devices, it may take the form of a centralized, coherent hub that connects processors, memory controllers, and other shared resources. As designs grow larger, however, even the coherency fabric begins to repeat history. Rather than relying on a single central hub, it can evolve into a distributed mesh, enabling coherent communication across the entire device.

And the story doesn’t end at the edge of a single piece of silicon. Increasingly, AI systems are being partitioned across multiple chiplets or dies to create multi-die system (MDS) solutions. From the software’s point of view, many of those chiplets need to communicate as though they are part of a single coherent system. The same architectural principles that transformed communication inside computers, between computers, and inside SoCs are now extending naturally across multiple dies.

All of which returns us to the beginning of our tale. Good engineering ideas rarely disappear; they simply wait for the next generation of engineers to rediscover them. Shared buses gave way to crossbars, crossbars gave way to packet-switched fabrics, and today’s AI systems are teaching us that even Networks-on-Chip must continue to evolve.

George Santayana warned us about repeating history’s mistakes. Fortunately for all concerned, engineers have become remarkably adept at repeating history’s successes. Long may that continue. It certainly makes me proud to be an engineer!