Setting the Stage

I'd been implementing neural networks for years before the Transformer came along. I remember the excitement when LSTMs first clicked for me—finally, a way to model sequences that didn't immediately forget everything! But I also remember the frustration: training was slow, sequences longer than a few hundred tokens were problematic, and there was always this nagging sense that we were fighting the architecture rather than working with it.

Then in June 2017, a paper with possibly the most provocative title in machine learning history dropped: "Attention Is All You Need." The claim was audacious—throw out recurrence entirely and rely solely on attention mechanisms. I was skeptical. How could you model sequences without... sequence processing?

To understand why Transformers were such a breakthrough, we need to understand what they replaced and why that replacement was necessary.

The Sequential Processing Dilemma

Language has order. "Dog bites man" means something very different from "Man bites dog." Any architecture that processes language needs to respect this ordering. The natural approach, which dominated for years, was to process sequences one element at a time.

Recurrent Neural Networks (RNNs) embodied this intuition perfectly. At each timestep t, the network computes a hidden state h_t as a function of the previous hidden state h_{t-1} and the current input x_t:

This is elegant. The hidden state acts as a "memory" that carries context forward through the sequence. Process "The cat sat on the" token by token, and by the time you reach "the," your hidden state hopefully encodes something useful about cats and sitting.

But there's a fundamental problem: sequential computation can't be parallelized. You can't compute h_{10} until you've computed h_9, which requires h_8, and so on. Your expensive GPU with thousands of cores? Most of them sit idle, waiting for the sequential chain to complete.

The Vanishing Gradient Problem

Even worse than the parallelization issue is the vanishing gradient problem. When we train neural networks with backpropagation, gradients flow backward through the network. For RNNs, this means gradients must flow backward through time—through every single timestep.

Here's the mathematical reality: when you backpropagate through a recurrent connection, you multiply by the Jacobian matrix of the hidden state transition. If the eigenvalues of this matrix are less than 1, gradients shrink exponentially. If greater than 1, they explode exponentially.

In practice, this meant RNNs could only effectively learn dependencies spanning maybe 10-20 timesteps. Need to connect a pronoun to its antecedent 50 words back? Good luck.

LSTMs and GRUs: A Partial Solution

Long Short-Term Memory networks were designed specifically to address vanishing gradients. The key innovation was the "cell state"—a highway that allows information to flow through time with minimal transformation. Instead of a single hidden state being multiplied at each step, LSTMs use gating mechanisms:

• Forget gate: decides what to discard from the cell state
• Input gate: decides what new information to store
• Output gate: decides what to output based on the cell state

The cell state updates through addition rather than multiplication, which helps gradients flow more easily. LSTMs could handle dependencies of maybe 100-200 tokens—much better than vanilla RNNs, but still fundamentally limited.

And they were still sequential. Still couldn't parallelize. Still left most of your GPU idle.

The Attention Insight

The breakthrough came from an unexpected direction: machine translation. In 2014, Bahdanau and colleagues introduced attention for sequence-to-sequence models. The idea was simple but profound: instead of forcing all information through a fixed-size hidden state bottleneck, let the decoder "look back" at all encoder states and focus on the relevant ones.

This was attention as an add-on to RNNs. The Transformer paper asked: what if attention was the only mechanism? What if we threw out recurrence entirely?

The visualization above captures the key insight. In an RNN, if position 1 needs to influence position 20, the signal must pass through 19 hidden states—19 chances for information to be corrupted or lost. With attention, position 1 can directly attend to position 20. The path length is constant, regardless of sequence length.

This isn't just theoretically elegant—it's practically transformative. Shorter paths mean better gradient flow. Direct connections mean the model can learn arbitrary dependencies without fighting the architecture.

And crucially: all attention computations for all positions can happen in parallel. Your GPU is finally earning its keep.

Here's what continues to amaze me about modern LLMs: the training objective is almost laughably simple. Next-token prediction. Given the tokens you've seen so far, predict the probability distribution over what comes next.

That's it. No explicit teaching of grammar, no labeled examples of reasoning, no curriculum of skills. Just: predict the next token, billions of times, on trillions of tokens of text.

Why This Works: A Deep Connection

The mathematical justification is elegant. If you model P(x_t | x_1, ..., x_{t-1}) well for all positions in all texts, you're implicitly modeling the entire distribution of human text. Grammar rules? They're just patterns that make certain next tokens more likely. Factual knowledge? It's encoded in which continuations are probable. Reasoning? It's the patterns that connect premises to conclusions.

The loss function is cross-entropy:

Perplexity, the standard metric, is just exp(loss). A perplexity of 10 means the model is, on average, as uncertain as if choosing uniformly among 10 options. Modern LLMs achieve perplexities of 8-15 on standard benchmarks.

Emergent Capabilities: Scale Changes Everything

Emergent capabilities are perhaps the most surprising aspect of LLM scaling. These are abilities that appear suddenly at certain scales but are absent in smaller models.

GPT-2 (1.5B parameters) could generate coherent paragraphs but struggled with few-shot learning. GPT-3 (175B parameters) could solve new tasks from just a few examples in the prompt—a capability that wasn't explicitly trained and that researchers didn't fully anticipate.

Chain-of-thought reasoning emerged similarly. Tell a large enough model to "think step by step," and it produces intermediate reasoning that improves final answers. Smaller models just produce nonsense when prompted the same way.

These aren't linear improvements—they're phase transitions. The capability is essentially zero, then suddenly it's there. We still don't fully understand why.