1948 to 2026
It starts with a mathematician asking his wife to guess the next letter of a sentence. Everything after that is the same question asked with more context, more numbers, and more electricity. Here is the whole line, one step at a time.
In 1948 Claude Shannon, working at Bell Labs on how much information a telephone line could carry, needed to know how predictable English is. His method was not a machine. He took a book, covered a sentence, and asked a person to guess the next letter. If they were wrong, he told them and they guessed again. He wrote down how many guesses each letter took.
Try it in the figure. You are shown a fragment and you pick a letter. The page keeps score the way Shannon did.
Here is why the score matters. If every letter were equally likely, naming one out of 27 possibilities would take a little under five yes-or-no questions to pin down. Information theory calls that quantity a bit, and it is just a count of how many halvings it takes to isolate an answer. Shannon measured that using only how often each letter appears in English, the cost falls to about 4.14 bits per letter. Using the full surrounding sentence, his experiments put it between roughly 0.6 and 1.3 bits.
That gap is the discovery. Most of what a letter appears to tell you is already implied by what came before it. Language is heavily predictable, and the amount by which it is predictable can be measured in a number.
Figure 1. Click letters to guess. The counter is your real guess count, and the bits figure is computed from it exactly as Shannon computed his.
Shannon had a number for how predictable English is. He did not have a machine that could do the predicting. The obvious way to build one was to stop asking people and start counting.
Take a large pile of text, called a corpus. Walk through it and tally every pair of adjacent words. Now, to predict what comes after "good", look up every word that ever followed "good" and pick in proportion to how often each one did. That is an n-gram model, and for about forty years it was the best language technology anyone had.
The n is how many words of context you keep. With n of 2 you look at one previous word. With n of 3 you look at two. The figure below builds a real n-gram table from a small corpus and generates text from it, so you can watch the output improve as n rises.
Something worth noticing while you play with it. At n of 2 the output is gibberish with correct-looking local grammar. At n of 4 it starts producing whole phrases that sound plausible, and if you look carefully, many of them are simply lifted from the source text intact. The model has not learned English. It has learned this corpus, and as the context grows it increasingly has only one option to copy.
Figure 2. Drag n and press generate. The table and the sentence are computed from the corpus shown, not scripted.
Raising n makes the output better right up until the moment it makes the model useless, and the reason why is the wall that held language technology still for four decades.
English has maybe 50,000 words in everyday use. Pairs of words: two and a half billion possible combinations. Triples: over a hundred trillion. No corpus that has ever existed contains more than a vanishing sliver of them.
So when you raise n, almost every context you meet in real use is one the model has never seen even once. Its count is zero, and a zero count means no prediction at all. This is sparsity, and the figure makes it concrete: as n rises, watch the share of contexts the model has actually seen fall off a cliff.
The deeper problem is not missing data, it is missing generalisation. A counting model that has seen "the cat sat on the mat" ten thousand times still knows nothing about "the dog sat on the rug", because to a table of counts those two sentences share no entries. Every word is an opaque symbol with no relationship to any other. A human who learns one sentence has learned something about the other. A count has not.
Figure 3. Drag n, then swap corpus size. Even the largest corpus loses the race, because possible contexts grow faster than any pile of text can.
While statisticians were hitting that wall, a different group had decided prediction was beside the point, and that the way to make a machine seem intelligent was to write the rules by hand.
In 1966 Joseph Weizenbaum at MIT wrote a program of a few hundred lines. It matched your sentence against patterns and turned it around. Say "I am unhappy about my job" and it finds the pattern "I am X" and replies "How long have you been unhappy about your job?" There is no memory, no model of you, and no representation of anything at all.
Weizenbaum's secretary asked him to leave the room so she could talk to it privately. People who knew exactly how it worked still reported feeling heard. He spent much of the rest of his career arguing that this was alarming rather than impressive, and the phenomenon is now named after the program: the ELIZA effect, our readiness to credit a system with far more comprehension than it has.
Run the rules in the figure and you can see both things at once. The replies are eerily apt, and the mechanism producing them is a lookup table you can read in full.
Keep this one in your pocket. Every time something in the rest of this piece feels like understanding, the honest question is whether you have evidence of comprehension or evidence of a very good pattern, and that question has never once gone away.
Figure 4. Pick a sentence and see which of the six rules caught it, and exactly which words were reflected back.
Hand-written rules could imitate a conversation. They could not be extended, because every new situation needed a person to write another rule.
Both roads had run out. Counting could not generalise and rules could not scale. The way forward had already been proposed nine years before ELIZA, by someone who wanted a machine that adjusted itself.
Frank Rosenblatt built the Mark I Perceptron in 1958, and it was a physical machine with motors that turned potentiometers. The idea underneath is simple enough to hold entirely in your head, and everything later in this piece is a variation on it.
Give the machine some numbers describing a thing, say the brightness of each part of an image. Multiply each number by a weight, add the results up, and if the total clears a threshold say yes, otherwise say no. The weights start random, so at first it is wrong constantly.
Now the part that mattered. When it is wrong, nudge every weight a little in the direction that would have made it right. Show it another example. Nudge again. Do this a few thousand times and the weights settle into values nobody chose and nobody can easily explain, but which classify the examples correctly. That process is training, and it is the first time a machine's competence came from data rather than from an author.
Press train in the figure. The line is the machine's current opinion about where the boundary between the two groups lies, and you are watching real weight updates, one per step.
Figure 5. Press train and watch the boundary move. The weights are updated by the real perceptron rule; the error count is measured, not animated.
Rosenblatt's machine could learn any boundary that is a straight line. In 1969 two researchers pointed out, in a book that changed the field's funding, how small that family of problems is.
Here is a problem with four examples. Two switches, and the answer is yes when exactly one of them is on. Both off, no. Both on, no. One on, yes. It is called XOR, and every programmer meets it in week one.
Try to separate the yes cases from the no cases in the figure by dragging a straight line. You cannot. The yes cases sit on opposite corners, and no straight line puts opposite corners on the same side. This is not a matter of training longer or picking better weights. A single layer computes a weighted sum and a threshold, and the boundary that produces is always a straight line. The problem is not linearly separable, so a perceptron cannot solve it, ever.
Marvin Minsky and Seymour Papert published this in Perceptrons in 1969. The mathematics was correct and narrow: it described one layer. The field read it as a verdict on the whole approach.
Figure 6. Drag both sliders and try to get 4 of 4. Switch to the easy problem to confirm the controls work and the difficulty is the problem, not the interface.
A correct result about one layer became a general belief that the approach was a dead end, and the money went elsewhere.
The pattern repeats so exactly that it is worth naming. Researchers demonstrate something striking on a small problem. Funders extrapolate. The extrapolation assumes the hard part scales like the easy part. It does not, progress stalls, and the funding leaves faster than it arrived. The field calls these AI winters.
The first ran through the 1970s. Machine translation had been promised within years and delivered badly; a 1966 US report concluded it was not close. In Britain, the 1973 Lighthill report judged that the field had failed to deliver on its claims, and government funding was largely withdrawn. Neural networks, freshly declared limited, were part of the collateral.
The second came in the late 1980s. This time the boom was expert systems, programs holding thousands of hand-written rules elicited from specialists, and they genuinely worked in narrow domains. They were also brittle at the edges and enormously expensive to maintain, because every change to the world meant a person editing rules. The specialised hardware market built to run them collapsed around 1987.
Step through the figure. What the two winters share is not a technical cause but a shape: a demonstration, a straight line drawn through it, and a wall the line did not know about.
Figure 7. Drag through the decades. The upper line is what was promised, the lower is what shipped, and the gaps between them are where the money left.
Through the second winter a small group kept working on the thing that had been declared dead, and they had already solved the straight-line problem three years earlier.
The fix for XOR is to put a second layer in. The first layer draws a few straight lines. The second layer works not on the original data but on the answers from those lines, and a combination of straight lines can carve out shapes no single line can. Layers in the middle are called hidden layers, because nothing outside the network ever sees their output.
That was known. The obstacle was training it. With one layer, when the answer is wrong you know exactly which weights contributed and by how much. With a hidden layer in the way, a middle neuron's contribution is buried inside the final answer, and it was not obvious how much of the blame it should carry.
Backpropagation is the answer, popularised by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986. Measure the error at the output. Work out how much each final-layer weight contributed, and adjust. Then pass the blame back one layer using the same arithmetic, and again, all the way to the front. Every weight gets a share of the error proportional to how much it caused, and the whole stack learns together.
Press train in the figure and watch a two-layer network do what figure 6 proved impossible for one. The boundary it finds is bent.
Figure 8. This trains for real in your browser: gradient descent on a two layer network, and the loss shown is the measured loss at that step.
The method worked and the field still went quiet, because in 1986 the two things backpropagation is hungry for did not exist in any quantity: labelled data, and arithmetic speed.
A network that reads a fixed-size image is one thing. A sentence is a different shape: it arrives one word at a time and can be any length. The answer was the recurrent network, which reads one word, keeps a summary of everything so far, then reads the next word alongside that summary, and repeats. The summary is the memory.
It works for short spans and fails for long ones, for a reason that falls straight out of backpropagation. To learn that a word at position 40 depended on a word at position 2, the error has to travel back through 38 steps of multiplication. Multiply a number smaller than one by itself 38 times and it is effectively zero. The signal reaches the early words as nothing at all, so the network never learns the long dependency. This is the vanishing gradient.
Sepp Hochreiter and Jürgen Schmidhuber published the LSTM in 1997, which adds gates that let the network deliberately keep a value unchanged across many steps rather than passing it through a multiplication each time. It stretched usable memory from a handful of words to perhaps a few dozen, and it powered a decade of translation and speech systems.
Drag the distance slider and watch how much of the signal survives at each design.
Figure 9. Drag the distance and switch designs. The surviving fraction is computed from repeated multiplication, which is exactly where the problem comes from.
Memory was patched. The other half of the problem, the one that killed counting back in the third figure, was still completely open: words were still opaque symbols with no relationship to one another.
The move that broke the sparsity wall is easy to state and strange to absorb. Stop treating a word as a symbol. Give it a list of numbers instead, and let training decide what the numbers should be.
A list of numbers is a vector, and a vector is a position in space. Two numbers put a word somewhere on a page. Three put it in a room. Real systems use hundreds, which nobody can picture, but the rules are the same in any number of dimensions. The list attached to a word is its embedding.
Here is what that buys, and it is the thing counting could never do. If training places "dog" and "puppy" near each other, then anything the model learned about the region around "dog" applies to "puppy" for free, without ever having seen "puppy" in that sentence. The generalisation that a count of "the dog sat" could not extend to "the puppy sat" is now automatic, because the model was never working with the word. It was working with the place.
Hover any cell in the figure. These are real numbers in a small space, deliberately shrunk to four dimensions so you can read every one; a real embedding has several hundred and no dimension has a name a human assigned.
Figure 10. Click a word to see its four numbers and where those numbers put it. The dots and the number strips are the same data drawn twice.
Yoshua Bengio and colleagues built a language model on this principle in 2003. It worked and it was slow, and it took another decade and a hardware accident before the idea got its moment.
In 2013 Tomas Mikolov and colleagues at Google released word2vec, which learned embeddings from a simple job: given a word, predict the words around it. Nothing about the training told it anything about grammar or semantics. The underlying bet, old in linguistics, is that a word is characterised by the company it keeps, which is called the distributional hypothesis.
What came out surprised people. Not only were similar words close together, but the directions between them were consistent. The step from "man" to "woman" was approximately the same step as from "king" to "queen". Which means you can do arithmetic: take king, subtract man, add woman, and the nearest word to where you land is queen.
Run it in the figure. Nobody built this in. It fell out of predicting neighbours, which is what made it evidence rather than engineering: the structure was already latent in how people use words, and the training merely exposed it.
Worth being straight about the limits, since this example is quoted everywhere. The famous cases were selected, the arithmetic often lands on something merely close rather than exactly right, and the same method reproduces the biases of the text it read, placing occupations near genders in ways that caused real problems in real deployed systems.
Figure 11. Pick a word relationship. The arrows are the actual differences between the vectors, and the landing point is computed, then matched to its nearest word.
Words had geometry. What was still missing was enough computing power to train anything large enough to matter, and that arrived from an industry with nothing to do with language.
Training a network is mostly multiplying big grids of numbers. That is also exactly what rendering video game graphics is, and by 2010 gaming had funded chips that did this thousands of times in parallel. A GPU made a job that took weeks take days.
The second ingredient was data. Fei-Fei Li's lab released ImageNet in 2009, eventually around fourteen million hand-labelled photographs, and ran an annual contest on it. Progress there had been ordinary: best error crept from about 28 percent in 2010 to about 26 percent in 2011.
In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered a deep convolutional network trained on two gaming GPUs. It scored a top-5 error of 15.3 percent against 26.2 percent for second place, more than ten points clear in a contest that had been moving in single points. Drag the figure through the years to see the shape of that.
A drop that size is not a better-tuned version of the same idea, and everyone read it correctly as a change of method. Within about two years, essentially all serious computer vision was neural. The techniques were largely from the 1980s. What was new was that the arithmetic had become affordable and the data had become plentiful.
Figure 12. Drag through the contest years. The 2012 drop is the moment the field switched methods; the numbers are the published winning error rates.
Vision had its moment. Language was harder, because a sentence has an order and a photograph does not, and the machinery for handling order was still the bottleneck from figure 9.
By 2014 machine translation worked like this. One recurrent network, the encoder, reads the source sentence word by word and ends holding a single summary vector. A second network, the decoder, starts from that summary and writes the translation out.
It worked, and it degraded badly on long sentences, for an obvious reason once stated: the entire source sentence, however long, has to fit through one fixed-size summary. A forty-word sentence and a four-word sentence get the same allowance.
The fix, from Dzmitry Bahdanau and colleagues in 2014, was to stop insisting on the single summary. Keep the encoder's output at every position, and let the decoder look back at all of them each time it writes a word, weighting them by how relevant each one is right now. That weighting is attention.
Step through the figure and watch which source words light up as each output word is produced. When it writes the verb, it looks at the verb. Nobody aligned those words by hand: the weights were learned, and the alignment is a side effect of doing the job well.
Figure 13. Step through the output. Switch to the single summary to see the same sentence produced through one fixed bottleneck.
Attention was added to recurrent networks as an improvement. Three years later a group asked what would happen if you removed the recurrence and kept only the attention.
In June 2017 eight researchers at Google published a paper whose title was the argument: attention is all you need. Take the encoder-decoder design, delete the recurrent part entirely, and build the model out of attention layers alone. The result is the transformer, and every system in the rest of this piece is one.
The quality gain mattered less than the reason it was possible to train. A recurrent network must process word 1 before word 2, because word 2's computation needs word 1's summary. That is a chain, and a chain cannot be spread across thousands of processors. Attention has no such dependency: every position can look at every other position simultaneously, so the whole sentence is one big matrix multiplication, which is precisely what the GPUs from figure 12 do best.
Compare the two in the figure. The recurrent column fills one cell at a time. The transformer column fills in one pass. On a sentence of 40 words that is 40 sequential steps against 1, and at the scale of the whole internet it is the difference between impossible and merely expensive.
One detail, since it matters later. Transformers do not work on words exactly, but on tokens: common words are one token, rarer ones are split into pieces. It is a compression convenience, and it is the reason these systems are peculiarly bad at spelling questions, since they often never see individual letters at all.
Figure 14. Press run and watch the two fill. The step counter is the real number of sequential stages each design needs for the length you chose.
That is the machinery. What it actually does, at every position, is worth slowing right down for, because it is the one mechanism a beginner most often takes on faith.
Read this sentence: the trophy did not fit in the suitcase because it was too big. What does "it" refer to? Now read it with one word changed, too small. The answer flips, and nothing about the word "it" changed. You resolved that from context without noticing. Scroll through the mechanism that does the same job.
The embeddings from earlier. At this point "it" carries only what the word "it" means in general, which is almost nothing. It has no idea yet which thing in the sentence it stands for.
Each position produces three things from its own vector: a query describing what it is looking for, a key advertising what it offers, and a value holding what it will actually contribute. All three are made by weights that were learned during training.
One score per word, measuring how well what "it" is looking for matches what each other word advertises. High for "trophy", lower for "suitcase", near nothing for "the". These are the same numbers you see in the bars.
Squash the scores so they are all positive and add to one. Now they read as shares of attention: about two thirds of what "it" is about to absorb will come from "trophy".
Mix the value vectors in those proportions and write the result back. The vector at "it" is no longer the generic word. It now sits close to "trophy" in the space, which is what resolving a pronoun looks like when meaning is geometry.
Swap "big" for "small" and the scores redistribute towards "suitcase", because the learned weights encode which things are plausibly too small to hold a trophy. Nothing was looked up in a table of pronouns. It fell out of predicting text.
Sixty layers of that, each one rewriting every position using every other position, is the entire engine. There is no separate grammar module and no store of facts. What there is, is a very large number of weights that were adjusted until the next word came out right.
Backpropagation needs to know the right answer, and for most of this history that meant people labelling things. ImageNet took years of human effort. Labelled data was the bottleneck.
The realisation that removed it is almost embarrassing in hindsight. To train a next-word predictor you do not need labels, because the text is the labels. Take any sentence, hide the last word, ask the model to predict it, and compare against the word that was actually there. Every sentence ever written is a free training example. This is called self-supervision, and it turned the entire internet into a labelled dataset overnight.
The recipe that follows is pretraining: run that over an enormous pile of text, adjusting billions of parameters, which is the general name for all those weights. GPT-1 arrived in 2018 with 117 million of them, GPT-2 in 2019 with 1.5 billion, GPT-3 in 2020 with 175 billion.
And here the argument from figure 1 returns. To predict the next word well across all of human writing, a system is pushed into representing whatever makes text predictable: syntax, facts, the shape of an argument, the conventions of a recipe. None of that was a training goal. All of it is what good guessing requires.
Figure 15. Step through examples. Switch to human labels to see how many examples the same effort produces when a person has to write each one.
With labels no longer scarce, the only remaining questions were how big to make the model and how much text to feed it, and in 2020 those stopped being questions of taste.
In 2020 a team at OpenAI published measurements showing that model error falls in a strikingly regular way as you increase three things: parameters, training data, and compute. Plot it on a log scale and the points sit on a line. These are the scaling laws.
The quantity being predicted is loss, which is just the average surprise of the model at each next word. Low loss means the actual word was one it thought likely. It is the direct descendant of Shannon's bits per letter, measured the same way, seventy years later.
This is the knob for the whole piece, so spend a moment on it. Drag the scale and watch loss fall predictably. The strategic consequence was enormous: for the first time, a lab could forecast that a model ten times larger would be a specific amount better, before building it. Billions of dollars of capital expenditure rest on lines like this one.
A correction arrived in 2022. DeepMind's Chinchilla work showed the field had been building models too large for the amount of text they were trained on, and that for a fixed budget you do better with a smaller model and far more data. Toggle the figure to see both recipes at the same total cost.
Figure 16. Drag the scale. The curve is a power law of the shape published in the scaling papers, and the cost figure beside it is computed from the scale you chose.
The loss curve is smooth. What the model can actually do, it turned out, is not.
Loss falls smoothly. But if you test a specific skill, say three-digit arithmetic, the graph looks different: flat at chance for a long stretch of scale, then a sharp rise. Below some size the model cannot do it at all. Above, it mostly can. Nothing in the training changed.
The field called this emergence, and it caused real unease, because it means you cannot always know what a model will be able to do until you have built it. Compare the two curves in the figure and you can see the discomfort directly: a smooth predictable line, and skills appearing at unpredictable points along it.
Be careful here, because this claim has been contested seriously. Later work argued that much of the sharpness is an artefact of how the skill is scored. Grade arithmetic as exactly right or wrong and you get a cliff. Grade partial credit on each digit and the same models improve smoothly. Toggle the scoring in the figure and watch the cliff soften.
The honest position sits between the two. Some jumps survive better scoring and some do not, and the practical situation is unchanged either way: nobody can currently look at a training plan and list what the finished model will be capable of.
Figure 17. Drag the scale with strict scoring, then switch to partial credit and drag again. Same underlying model, different shaped story.
A model this size could do a great deal and was still, in an important sense, unusable, because what it had learned to do was continue text rather than answer anybody.
Ask a purely pretrained model "What is the capital of France?" and a very reasonable continuation is another question, because in the text it read, questions are often followed by more questions. It is not being unhelpful. It is doing precisely what it was trained to do, and being helpful was never the objective.
Two additional stages fixed this. First, instruction tuning: continue training on thousands of examples of instructions paired with good responses, so the pattern "request, then a useful answer" becomes the likely continuation.
Second, learning from human feedback. Generate several candidate answers, have people rank them, and train a separate model to predict those rankings. Then use that predictor as a stand-in judge to tune the main model towards answers people prefer. It scales human taste, since the judge can score millions of answers no person will ever read.
Compare the stages in the figure on the same question. The knowledge does not change between them. What changes is which continuation is likely, and that difference is the whole of what most people mean by an AI assistant.
It also introduces a permanent tension worth naming. Training a model towards what people rate highly rewards answers that seem good, and confident, agreeable, well-formatted wrongness rates well. Some of the sycophancy in current systems is a direct fingerprint of this stage.
Figure 18. Click through the three stages. The candidate continuations and their likelihoods are shown for each, so you can see what shifted.
All of that existed inside labs by early 2022. What happened at the end of that year was not a technical event.
ChatGPT launched on 30 November 2022. The model behind it was a tuned version of work already published, and the ideas in it were mostly two to five years old. What was new was a text box, no cost, and no manual.
It reached a hundred million users faster than any consumer product before it. Everything downstream, the capital, the competition, the regulation, the anxiety, follows from that launch rather than from a research result.
The reason to end this part on a distribution story rather than a technical one is that it is the most commonly compressed step. People remember 2022 as the year the technology arrived. Drag the figure and you can see how little of the technology was actually new that year, and how much of the change was in who could reach it.
Figure 19. Drag through the years. The upper track is published research, the lower is public attention, and the gap between them is the point.
Everything so far has been about text. The same machinery had meanwhile been pointed at pictures, sound and video, and the way it got there is stranger than simply repeating the recipe.
Pictures needed a different trick, because you cannot write an image one pixel at a time from left to right and expect it to hold together.
The idea, refined by Jonathan Ho and colleagues in 2020, is almost perverse. Take a training photograph and add a little random noise. Add more. Repeat until it is pure static. Now train a network on the reverse job: given a noisy image, predict what was added, so it can be subtracted. That is a small, well-defined task with unlimited free training data, since you generated every noisy version yourself.
To create a new picture, hand the trained network pure static and ask it to remove noise that was never added. It has no choice but to hallucinate structure, and it does so in the shape of the images it learned from. Run it backwards a few dozen times and a coherent picture condenses out of the static. Drag the slider in the figure to move through the steps.
Steering it by text works by conditioning each denoising step on a description, so at every stage the model removes noise in the direction of the words. That is why prompts feel like they nudge rather than command: they are a pull applied fifty times, not an instruction executed once.
Figure 20. Drag the step slider from static to picture. Change the starting noise to see the same prompt land somewhere different.
That explains how a picture gets made. It does not explain how words came to steer pictures at all, which required teaching one model that a sentence and a photograph can mean the same thing.
Recall that an embedding is a position. There is nothing in that idea specific to words. If you can turn a picture into a list of numbers too, you can ask a bolder question: can a picture and the sentence describing it land in the same place?
OpenAI's CLIP, published in January 2021, trained exactly that. Take hundreds of millions of images with their captions. Push each image's vector towards its own caption's vector, and away from all the other captions in the batch. That is contrastive training, and repeated at scale it produces one space where a photograph of a dog on a beach sits near the words "a dog on a beach".
Once that exists, a great deal follows almost for free. Search images by text. Caption them. Steer a diffusion model, because now "what you asked for" and "what this half-denoised image looks like" are comparable positions. Click the pairs in the figure to see matches and mismatches scored.
This is the conceptual hinge of the whole modality story, so it is worth stating plainly. Nothing was translated. There is no step where a picture becomes a sentence. Both were turned into positions in one space, and once two things are positions in one space, comparing them is just arithmetic.
Figure 21. Click a caption and see how each image scores against it. The scores are cosine similarities computed from the vectors shown.
With one space that holds pictures and words, adding another format becomes a question of finding a way to turn it into numbers too.
Audio arrives as a wave, tens of thousands of numbers per second, which is far too fine a grain to model directly. The standard move is to convert a short window of sound into how much energy sits at each frequency, then stack those windows side by side. The result is a spectrogram, a two dimensional image of the sound with time across and pitch up.
That reframing is the whole trick. A spectrogram is a picture, and the field already had excellent machinery for pictures and for sequences. Speech recognition became: read this image, write the tokens. Speech synthesis became the reverse. OpenAI's Whisper, released in 2022, is a transformer trained on a very large amount of audio paired with text, and it is structurally the translation model from figure 13 with a spectrogram on the input side.
Drag through the figure to watch a waveform become a spectrogram become tokens. Nothing in the model is specific to sound. The specificity is entirely in the conversion at the front.
Figure 22. Drag through the stages. The spectrogram is computed from the waveform shown, so the bright bands really do correspond to the loud parts.
Sound submits to the same treatment as pictures. Video resists it, and the reason it resists is the most interesting open problem in this part.
Naively, video is images with a time axis, so generate each frame and play them in order. Try it and you get a slideshow of related pictures where the coat changes colour, a hand grows a finger, and a dropped cup hangs in the air.
The requirement that breaks the naive approach is temporal consistency: everything unmentioned must stay the same, and everything moving must move the way objects actually move. That is a much stronger demand than looking right in a single frame, because it means the generator has to carry an implicit account of what is in the scene and what happens to things over time. People call that a world model, and it is contested whether current systems have anything deserving the name or merely reproduce enough surface regularity to pass.
Two costs stack up as well. Attention compares every position with every other, so cost grows with the square of the number of positions, and a few seconds of video is a very large number of positions. Then the training data problem: the internet has billions of captioned images and far less well-described video.
Toggle the consistency in the figure to see the failure mode directly, and notice that each individual frame in the broken version looks perfectly reasonable on its own.
Figure 23. Step through the frames in both modes. Watch the colour and the ball position, which are the two things the separate mode has no way to remember.
Each format found its own way in. The bigger change was when one model stopped needing a separate route for each of them.
A multimodal model takes text, images and audio together in one conversation. The reason that is possible is not that anyone built three models and a router. It is that all three had already been reduced to the same thing.
Text becomes tokens, and each token becomes a vector. An image is cut into small square patches, and each patch becomes a vector. Audio becomes spectrogram slices, and each slice becomes a vector. At that point the transformer's input is what it always was: a list of vectors. It does not know or care which ones came from a photograph.
Click the sources in the figure and watch them all arrive at strips of the same shape. The reason your screenshot can sit in the middle of your sentence is that by the time the model sees it, it is a run of vectors in the same sequence as your words, and attention runs across all of them without distinction.
This is why capability transfers between formats in surprising ways, and also why the weaknesses transfer. A model that is confidently wrong about text is confidently wrong about charts, because it is the same mechanism doing both.
Figure 24. Click each source, then show all three. The strips are the same width because the vectors really are the same shape.
One model can take every format. That does not mean one model should do every job, and the economics of running these things pushed hard in the opposite direction.
Running a frontier model for every request is like chartering a freight aircraft to post a letter. Three techniques reshaped this, and together they explain why you now choose between model sizes rather than getting one model.
Routing. Put a cheap classifier in front, send simple requests to a small model and hard ones to a large one. Most traffic is simple, so most traffic gets the cheap path, and average cost falls without much loss of quality.
Mixture of experts. The same principle inside one model. Split the network into many specialised blocks and learn a gate that activates only a few per token. A model can hold enormous total parameters while only a fraction of them do arithmetic on any given word, which decouples how much a model knows from what it costs to run.
Distillation. Train a small model to imitate a large one's outputs rather than raw text. The large model's full distribution over next words is a far richer signal than a single correct answer, so the small model learns much faster than it could alone. Most of the cheap fast models available now are distilled from something much bigger.
Drag the traffic mix in the figure and watch the cost of the three arrangements diverge.
Figure 25. Drag the difficulty mix. The three bars are computed from the same per-request costs, so the crossover points are real consequences of the numbers.
Cheaper and more numerous models made a further step affordable: letting a model act more than once before it answers you.
A pretrained model has fixed knowledge, cannot check anything, and cannot do arithmetic reliably. Rather than train those away, the field gave the model access to things that are already good at them.
Retrieval is the first. Before answering, search a document store, paste the relevant passages into the input, and answer from those. The knowledge lives in the store, so updating it is a file change rather than a training run, and the model can cite what it used.
Tool use is the second. Give the model a set of functions it may call, a calculator, a search engine, a database query. It emits a call, the system runs it, and the result comes back as more input. The model is not doing the arithmetic. It is deciding that arithmetic is needed, which is a much easier job.
An agent is what you get when that loop is allowed to repeat without asking permission each time: act, observe the result, decide the next action, stop when done. Press run in the figure to watch a loop resolve a question no single pass could.
This is also where the risk changes character. Everything before this was a system that produced text. A system that can call functions can send emails, move money, and delete files, and a wrong answer stops being something you can simply ignore. Every unit of reach is the same unit of risk, and they cannot be bought separately.
Figure 26. Run the loop, then switch to one pass to see the same question answered without the ability to check anything.
That is the line, from a covered sentence in 1948 to a system that books your travel. It is worth closing by asking what actually changed along it, because less did than you might expect.
Put figure 1 and a modern model side by side and the job is identical. Given what came before, place odds on what comes next, and be graded on how surprised you were by the truth. Shannon measured that in bits per letter. Training measures it as loss. They are the same quantity.
Three things grew. The context, from a few letters to hundreds of thousands of tokens including images and audio. The parameters, from a handful of dials to hundreds of billions. The data, from one book to a large fraction of everything written down.
Two things did not. The objective, and the fact that nothing in the system contains a representation of truth. A model has no channel that distinguishes accurate from plausible, because it was never given one, and everything it produces is a continuation ranked by likelihood.
Drag the era slider in the figure. It is figure 1's distribution, drawn at each stage of the seventy years, sharpening.
Figure 27. Drag through the eras. Same figure as the top of the page, same quantity measured, sharpening as context and scale grow.
Which leaves the question of what this line does not solve, and it is a longer list than the pace of the last few years suggests.
Confident wrongness. The model produces a likely continuation, and a plausible falsehood is by construction a likely continuation. There is no internal signal separating the two, which is why this is a property of the design rather than a bug awaiting a patch. Retrieval and tools reduce it. Nothing removes it.
Evaluation. A model that read most of the internet has probably read the test. When benchmark questions leak into training data, called contamination, a high score can mean recall rather than capability. Toggle the figure to see the same model score very differently on a fresh set.
Cost and its distribution. Training a frontier model runs to hundreds of millions of dollars, which puts the frontier in the hands of a small number of organisations, with real consequences for who decides what these systems refuse and whose language they work well in.
Data. The supply of high-quality human text is finite, and much of it is copyrighted, with the ownership questions unresolved in most jurisdictions. Increasingly models train partly on text generated by models, whose long-term effects are not well understood.
Opacity. Nobody can point at where a fact is stored or explain why a particular answer appeared. Interpretability research is real and progressing and nowhere near a full account.
Figure 28. Switch between the two question sets. The gap is what contamination looks like when someone bothers to write new questions.
Seventy years of progress on one question, and the honest summary is that we built something extraordinarily good at guessing without ever settling what guessing that well amounts to.
Shannon covered a sentence and asked his wife what came next. Everything since has been the same question, asked with more of the page visible and more arithmetic per guess.
Every dial from the piece in one place. Set the context the model can see, the parameters it has, and the data it read, and watch the prediction and the training cost respond. The presets rebuild real systems from the timeline.
Playground. Loss is computed from the scaling relationship in figure 16, and the distribution is drawn from it, so the presets really do land where the timeline says.
press s for the deeper cuts
The chain rule applied to layered functions was described by Seppo Linnainmaa in 1970 and applied to networks by Paul Werbos in 1974. The 1986 Rumelhart, Hinton and Williams paper is remembered because it demonstrated the method producing useful internal representations and reached the audience that acted on it. Priority disputes in this field are usually about who made an idea land rather than who wrote it down first.
Attention compares every position with every other and has no notion of order, so without help "dog bites man" and "man bites dog" are identical inputs. A positional encoding, added to each embedding, carries where in the sequence the token sat. That it has to be bolted on is a direct consequence of dropping recurrence, which had order built in for free.
Every token attends to every other, so doubling the context multiplies the attention work by four. That is why context windows grew slowly and expensively, and why a great deal of research goes into approximations that keep most of the quality at lower cost. It is also the main reason video is so much harder than text.
Generative adversarial networks, from Ian Goodfellow in 2014, pit a generator against a discriminator that tries to spot fakes. They produced the first genuinely convincing synthetic faces and were notoriously unstable to train, often collapsing to a narrow set of outputs. Diffusion trades speed for a training objective that is far better behaved, which is why the field moved.
Tokenisation. A word is often a single token, so the model may never see its letters as separate items at all. Asking how many times a letter appears is asking about a structure that was discarded before the model saw the input. It is not a reasoning failure, it is a representation gap, and it is why these questions improve when a model is allowed to write the word out one character at a time first.
It is a power law of the published form, drawn so the shape and the direction are right. The constants are illustrative rather than a specific lab's measurements, and the axis is loss, not capability. Loss falling by a small amount can correspond to a large or a negligible practical difference, which is exactly the ambiguity the emergence argument turns on.
The pattern needs a demonstration, a straight line drawn through it, and an unnoticed wall. Candidate walls now: the finite supply of quality text, the cost of the next order of magnitude, and the gap between benchmark scores and reliable work. None of these is certain to bite. The reason to keep the shape in mind is that both previous winters looked unthinkable from inside the boom.