From Transformer to Modern LLMs

The original Transformer was an encoder-decoder architecture designed for translation. The encoder processed the source sentence with bidirectional attention (each position attends to all others). The decoder generated the target sentence autoregressively, with causal attention (each position only attends to previous positions) plus cross-attention to the encoder outputs.

Modern LLMs almost universally use decoder-only architectures. Why did encoder-decoder lose?

The Case for Decoder-Only

Unified Training: With decoder-only, everything is next-token prediction. You don't need paired data (source-target). Any text works.

Task Flexibility: Encoder-decoder assumes a clear input/output split. Decoder-only treats everything as a sequence to continue. Want translation? Prompt with "Translate to French: [text]". Want summarization? "Summarize: [text]". The task is implicit in the prompt.

Compute Efficiency: No separate encoder phase. During inference, you're running one forward pass per generated token, not encoder + decoder.

The GPT Progression

GPT-1 (2018, 117M parameters) demonstrated that pre-training on next-token prediction followed by fine-tuning could achieve strong results. The insight: unsupervised pre-training provides a good initialization.

GPT-2 (2019, 1.5B parameters) showed zero-shot capabilities. Without any task-specific training, it could perform basic question answering, summarization, and translation. OpenAI initially declined to release the full model due to concerns about misuse.

GPT-3 (2020, 175B parameters) was the breakthrough that launched the current era. Few-shot learning worked: give the model a few examples in the prompt, and it could perform tasks it was never explicitly trained for. This was genuinely surprising.

GPT-4 (2023) introduced multimodality (images) and likely uses a Mixture of Experts architecture (though OpenAI hasn't confirmed details). Estimated at 1.7T parameters.

Sinusoidal positional encodings worked, but they had limitations. The biggest: extrapolation. Train on sequences up to 2048 tokens, and performance degrades on longer sequences. The model hasn't seen those position values before.

RoPE: Rotary Position Embeddings

RoPE, used by LLaMA and many others, encodes position through rotation. Instead of adding positional information to embeddings, RoPE applies rotation matrices to query and key vectors.

The key insight: when you take the dot product of two rotated vectors, the result depends only on their angle difference—which is proportional to the position difference. Position becomes relative, encoded in the geometry of the operation itself.

ALiBi: Attention with Linear Biases

ALiBi takes a radically different approach: no learned positional encoding at all. Instead, it adds a linear penalty to attention scores based on distance: positions farther apart get lower attention scores.

Different heads use different slopes, allowing some to focus locally and others to attend more globally. ALiBi extrapolates perfectly to any length (with the assumption that recency matters), though it can't learn arbitrary position-dependent patterns.

Here's the scaling dilemma: more parameters generally mean better performance, but more parameters also mean more compute per token. Mixture of Experts (MoE) breaks this link.

Replace the dense FFN with N expert FFNs. A router network selects the top-k experts for each token. Most parameters are inactive for any given token—you get the capacity of a large model with the compute cost of a smaller one.

Mixtral: A Concrete Example

Mixtral 8x7B has 8 experts with top-2 routing. Total parameters: 46.7B. Active parameters per token: ~12.9B. It matches or exceeds LLaMA 2 70B on most benchmarks while using similar compute to a 13B dense model.

The Load Balancing Challenge

Without careful design, the router might learn to always use the same few experts— "expert collapse." An auxiliary load balancing loss encourages even distribution:

The infrastructure implications are significant. All expert parameters must fit in memory (or be distributed with expert parallelism), but compute scales only with active parameters. This creates an unusual memory-compute trade-off.

Training a 70B+ parameter model requires careful orchestration of parallelism strategies. No single GPU can hold the model, and naive approaches waste compute or run out of memory.

The Parallelism Strategies

Data Parallelism: Each GPU holds a full model copy and processes different batches. Gradients are synchronized via all-reduce. Scales compute but not memory.

Tensor Parallelism: Split individual layers horizontally across GPUs. The attention and FFN weight matrices are sharded. Requires communication every layer—needs fast NVLink (400-900 GB/s).

Pipeline Parallelism: Split layers vertically—different GPUs own different layers. Communication only at stage boundaries. InfiniBand (200-400 Gb/s) is sufficient. Uses micro-batching to hide pipeline bubbles.

ZeRO: Memory Efficiency for Data Parallelism

ZeRO (Zero Redundancy Optimizer) eliminates memory redundancy in data parallelism. Standard DDP stores optimizer states, gradients, and parameters on every GPU. ZeRO shards them:

• Stage 1: Shard optimizer states (4× memory reduction for Adam)
• Stage 2: Also shard gradients
• Stage 3: Also shard parameters (requires gather before forward/backward)

Scaling Laws: The Chinchilla Revolution

In 2022, DeepMind's Chinchilla paper upended conventional wisdom. GPT-3 (175B params, 300B tokens) was undertrained. The compute-optimal ratio is roughly 20 tokens per parameter.

Chinchilla (70B params, 1.4T tokens) matched GPT-3's performance with 4× fewer parameters— smaller, faster to run, trained with the same compute budget but allocated differently.

Standard attention is O(n²) in both memory and compute. At 128K tokens, that's 16 billion attention elements per layer. The quadratic wall is the fundamental bottleneck for long-context processing.

Sparse Attention Patterns

Sparse attention reduces complexity by only computing a subset of attention pairs:

• Longformer: Local sliding window + global tokens
• BigBird: Random + window + global patterns
• Theoretical result: sparse patterns can be universal approximators

State Space Models and Alternatives

State Space Models like Mamba offer a different paradigm: O(n) training and inference with O(1) memory during inference. They use structured recurrence that can be computed efficiently via convolution during training.

Hybrid approaches like Jamba combine SSM layers with Transformer layers, getting benefits of both architectures.

RAG: Sidestepping the Problem

Retrieval-Augmented Generation avoids the long-context problem entirely. Instead of fitting everything into context, retrieve relevant chunks from an external knowledge base and include only those.

Trade-offs: latency (retrieval adds time), retrieval quality (wrong chunks = wrong answers), and complexity (another system to maintain).