At the beginning of the modern AI era, researchers discovered a relatively simple pattern: as you increase compute, training data, and model size, neural networks tend to become significantly more capable. One of the most important milestones came in 2017, when Google introduced the Transformer architecture in the paper Attention Is All You Need. Transformers were initially demonstrated on tasks such as machine translation, but it quickly became clear that the architecture could be used for much more.
The use of GPUs for neural network training, however, started even earlier. One of the key moments came in 2012 with AlexNet, which demonstrated just how effective GPUs could be for machine learning.
Why?
A CPU is extremely good at executing complex operations and handling a wide variety of general-purpose tasks, but it has a relatively small number of powerful cores. A GPU is designed differently. It was built to process huge numbers of relatively simple operations at the same time. Instead of having dozens of powerful CPU cores, a GPU can contain thousands of computational units capable of performing mathematical operations in parallel.
Training a neural network happens to require an enormous number of repetitive operations on matrices and tensors. That made GPUs almost perfectly suited to the job, and this is where the real boom in AI accelerators began.
NVIDIA, a company that for decades had been associated primarily with gaming graphics cards, suddenly found itself serving an entirely different market. Gamers were no longer the main source of its growth. The largest customers were now companies building enormous AI data centres. The more computing power they could deploy, the larger the models they could train and the more users they could serve.
A global race for GPUs began. NVIDIA became one of the most valuable companies in the world, while demand for AI accelerators grew so rapidly that AI infrastructure effectively became an industry of its own.
But then another problem appeared.
The painters and the paint bucket
Imagine that we need to paint an apartment. We hire one painter, and the work progresses slowly. So we hire ten painters, and suddenly the job can be completed much faster. There is only one problem: one person is responsible for bringing them paint, and he can carry only one bucket at a time.
What happens?
The ten painters use the paint very quickly and then stand around waiting for the next bucket. We could hire another ten painters. We could hire a hundred. But if the speed at which paint is delivered does not change, overall productivity will barely improve.
Something similar started happening with AI.
A GPU can perform an extraordinary number of operations every second, but it constantly needs data. If that data cannot arrive quickly enough, thousands of computational units simply sit idle waiting for it.
A new bottleneck emerges: not the ability to compute data, but the ability to move it.
And this is where memory becomes critical.
Enter HBM
But there is an important distinction. We are not talking only about the ordinary DDR5 RAM found in desktop PCs. One of the most important resources for modern AI accelerators is HBM — High Bandwidth Memory. It is extremely fast memory located as close as possible to the GPU.
A highly simplified version of the data path looks like this:
SSD → System RAM → HBM → GPU Cache → GPU Compute Units
The closer the memory is to the GPU, the faster the accelerator can access its data. This is exactly why HBM has become so important.
Modern AI models can contain hundreds of billions of parameters. In addition to the model weights themselves, systems need memory for intermediate calculations, activations, KV cache, and user context. The problem becomes particularly important during inference, when an already trained model is serving users.
The larger the context window, the more simultaneous users the system needs to serve, and the more complex AI agents become, the more memory the infrastructure requires.
That is why we are now seeing a memory boom. Major DRAM and HBM manufacturers such as SK hynix, Samsung, and Micron have suddenly found themselves at the centre of the AI infrastructure market.
The bottleneck keeps moving
But the story does not end there, because solving one bottleneck almost inevitably creates another.
At first, the bottleneck was compute, so we adopted GPUs. Then GPUs became so fast that memory could no longer feed them quickly enough, so the industry moved toward HBM3, HBM3E, HBM4, and increasingly larger and faster memory systems.
Now another question appears: what happens when one GPU is no longer enough?
Modern AI models may run not on one GPU, or even eight GPUs, but across clusters containing thousands or tens of thousands of accelerators. All of them need to exchange enormous amounts of data continuously.
A GPU might have incredibly fast HBM, but what happens when another GPU needs that information? The data now has to move from one accelerator to another, or even from one server to another.
And suddenly we have the same person carrying buckets of paint again.
The difference is that the bottleneck is no longer between memory and the GPU. It is now between the GPUs themselves.
This is why interconnect technology — the systems that connect accelerators — is becoming another major area of development. Technologies such as NVLink, InfiniBand, high-speed Ethernet, and other interconnect systems allow hundreds or thousands of GPUs to operate more like a single computing system.
But there is a physical limit here as well. Electrical signals have to carry increasingly enormous amounts of data, while the distance between servers and racks continues to grow. This leads to another important technology: silicon photonics — using light to move data.
Instead of transmitting ever-growing amounts of information entirely through electrical signals and copper connections, parts of the infrastructure are gradually moving toward optical communication.
That gives us an interesting progression:
CPU → GPU → HBM → Interconnect → Optical Networking
And then, electricity
And then we encounter an even more fundamental problem: electricity.
We can build a million GPUs. We can give them extremely fast HBM. We can connect them through optical networks. But all of this infrastructure still needs power.
Modern AI data centres can consume tens or even hundreds of megawatts of electricity. That means the next bottlenecks increasingly become power and cooling.
The more GPUs we place into a rack, the more electricity we need to deliver to it and the more heat we need to remove. Traditional air cooling becomes increasingly difficult at these power densities.
This is driving demand for liquid cooling, advanced power delivery systems, transformers, electricity generation, and data-centre infrastructure.
A moving target, not a single boom
The progression is therefore becoming quite interesting. First, we did not have enough computing power, so GPUs became the solution. Then GPUs became so fast that memory struggled to provide data quickly enough, creating demand for HBM. After that, a single accelerator was no longer enough, so AI systems began scaling to thousands of GPUs, increasing the importance of NVLink, InfiniBand, Ethernet, and other interconnect technologies. As those clusters became larger, moving data between machines became another limitation, creating demand for optical networking and silicon photonics. And now electricity and cooling are becoming increasingly important constraints of their own.
This is why I think it is a mistake to look at the AI boom simply as a GPU boom.
Something much more interesting is happening: AI is continuously moving the bottleneck.
Today, one technology limits the system. The industry pours billions of dollars into solving that problem. Once the problem is reduced, the bottleneck simply moves somewhere else.
And from an investment perspective, this creates perhaps the most interesting question of all:
Not what is in highest demand today, but what will become the system’s next major constraint tomorrow?
HBM is already an obvious problem, which is why memory manufacturers are experiencing enormous demand. But the next major battlegrounds are likely to include GPU-to-GPU communication, optical networking, power infrastructure, and cooling.
And perhaps somewhere in those industries is a company that today looks like just another infrastructure supplier, but several years from now could find itself in a position similar to where NVIDIA stood at the beginning of the AI boom.
Because ultimately, AI is no longer simply a race for more computing power. It is a race to determine how quickly we can store data, move data, compute data, transmit the results, and supply enough energy to keep the entire system running.
And every new generation of AI will demand more from every link in that chain.