Generative AI · AI Infrastructure · Private AI · Architecture Strategy

In June 2017 eight researchers published a 15-page paper about translating English into German. It quietly removed the one design choice that stopped AI from using more hardware. Everything that followed, from ChatGPT to 16,000-GPU training clusters to the GPU shortage, comes from that change. This post reads the paper as an infrastructure architect would: what it changed, why it moved AI budgets from research to compute, and what that means for anyone designing a data centre today, and how far the effects reach beyond it.

Most articles about the Transformer explain how attention works. I have done that already in Attention, the Idea That Made Modern AI Work. This one asks a different question. Why did a change to how a model reads a sentence end up deciding what servers companies buy, how much power a rack draws, and why NVIDIA became one of the most valuable companies in the world?

Short answer: the Transformer made AI parallel. Once AI could be parallel, more hardware reliably meant a better model. From that point on, AI strategy became infrastructure strategy.

Before 2017AI that read one word at a time

Before the Transformer, the best language models were recurrent neural networks (RNNs) and their improved version, LSTMs. They read a sentence the way you read a news ticker: word one, then word two, then word three. That design had two problems, and both mattered.

“This inherently sequential nature precludes parallelization within training examples.”

Vaswani et al., Attention Is All You Need (2017), Section 1

2017What the paper changed, in plain words

In a Transformer, every word in the input looks at every other word at the same time and decides which ones matter to it. This is called self-attention. Nothing waits for anything else. A whole sentence, or a whole document, is processed in one parallel pass.

Table 1 of the paper puts this in one line. A recurrent layer needs a number of sequential steps that grows with sentence length. A self-attention layer needs one. That single row is the reason this post exists.

Step by step: what happens inside

You do not need the maths to follow the idea. Here is what a Transformer does with one short sentence, in six steps:

Reading the paperWhat it actually claimed, and how modest it was

It is worth reading the paper itself (link at the end). Its claims are small and specific. Nowhere does it say “this will change the world”. It says: we built a translation model that is better and cheaper to train.

Look at the first number. One server, eight GPUs, under four days. A competent university lab could reproduce it. Seven years later, models built on exactly this architecture were trained on clusters of sixteen thousand GPUs. Nothing in the architecture had to change to make that jump. That is what “it scales” means.

Why it changed strategyMore hardware finally meant a better model

Parallelism alone would have been a nice speed-up. What turned it into a strategy was a second discovery three years later.

In January 2020 a team at OpenAI published Scaling Laws for Neural Language Models. They trained many Transformers of different sizes and found that model quality improved as a smooth, predictable power law in three things: model size, amount of data and amount of compute. The trend held across more than seven orders of magnitude.

Read that as a CFO would. If you can predict how much better a model gets for each extra unit of compute, then AI progress stops being a research gamble and becomes a capital investment with a forecastable return. Spend ten times more on GPUs, get a known improvement. Boards understand that sentence. Few of them understood “we have a promising new architecture”.

Seven years, measured in hardware

Here is the original paper next to Meta’s Llama 3 405B (July 2024), which uses the same basic architecture and published its training details openly:

Sources: the Transformer paper (Tables 2 and 3, Section 5) and Meta’s “The Llama 3 Herd of Models” (arXiv 2407.21783).

A 1.6-million-fold increase in compute in seven years did not come from cleverer algorithms. It came from buildings, power contracts, GPUs and networks. Here AI strategy and infrastructure strategy became the same conversation.

TimelineFrom a translation paper to the AI industry

1. Jun 2017
   ResearchTransformer paperTranslation model, trained on 8 GPUs.
2. 2018
   ResearchGPT-1 and BERTGPT-1 (June) uses the decoder half to generate text; BERT (October) uses the encoder half to understand it. One pre-trained model, many tasks.
3. Jan 2020
   ResearchScaling lawsQuality is predictable from size, data and compute.
4. May 2020
   ModelGPT-3, 175 billion parametersScale alone produces abilities nobody explicitly trained for.
5. Oct 2020
   ResearchVision TransformerImages cut into patches and treated like words. Same architecture, new data type.
6. Mar 2022
   HardwareNVIDIA H100 with a Transformer EngineA chip designed around one model architecture.
7. Nov 2022
   ProductChatGPT launchesAI becomes a board-level topic overnight.
8. Jul 2024
   ModelLlama 3 405BOpen weights, trained on up to 16,384 H100s, with its infrastructure problems documented in public.
9. Today
   IndustryTransformers everywhereText, code, images, speech and video. Enterprises now plan GPU platforms, not AI experiments.

Wider impactNot only infrastructure

Infrastructure is where this post focuses, but it is only one part of what changed. Once one architecture could do almost everything, and do it better with more compute, the effects spread well beyond the data centre.

Each of these deserves its own post. What connects them is the same chain this post follows: a parallel architecture, predictable returns from scale, and therefore a race for compute. Infrastructure is not the whole story, but it is the part every other card depends on.

For infrastructureSix things the Transformer changed in the data centre

Each change below follows from the same root: one parallel architecture that gets better with scale. For every one, here is how things worked before, how they work now, and the evidence.

Failure figures: Meta, “The Llama 3 Herd of Models” (arXiv 2407.21783): 419 unexpected interruptions, about 78% hardware, 58.7% GPU-related.

For the enterpriseWhat this means for your AI strategy

Very few enterprises will train a frontier model. What they will do is run them, for chat over internal documents, coding assistants, agents and search. That is inference, and it is where most enterprise AI infrastructure decisions now sit.

For most organisations, the first real decision is where models should run. Two questions settle most cases:

LimitsWhere the Transformer struggles, and what may come next

Eight years on, Transformers still dominate, but their weak spots are well known. Each one already has a fix in progress, and each fix changes what the hardware needs to look like.

Read it yourselfHow to read the original paper in 30 minutes

You do not need to follow the maths to get value from it. Six stops, about 30 minutes, in this order:

1. 1
   Abstract and Section 1Introduction

   One page. Problem and claim.

   6 min
2. 2
   Figure 1Architecture diagram

   Famous diagram. Encoder on the left, decoder on the right.

   4 min
3. 3
   Section 4 and Table 1Why Self-Attention

   Parallelism argument and the cost of the square.

   8 min
4. 4
   Section 5.2Hardware and Schedule

   8 P100 GPUs, 12 hours, 3.5 days. Every infrastructure person should know this paragraph.

   3 min
5. 5
   Table 2Quality vs training cost

   Compared with older models. Here the economic case is made.

   5 min
6. 6
   Section 7Conclusion

   Last paragraph predicts images, audio and video.

   4 min