Most modern language models are built from the same architectural family: the Transformer. The original Transformer was introduced in the 2017 paper "Attention Is All You Need" for machine translation. Since then, decoder-only variants have become the dominant architecture for text generation, chat models, coding assistants, and tool-using LLM systems.

The core idea is straightforward: represent text as a sequence of token vectors, then repeatedly let each token update its representation by looking at relevant tokens around it.

The mechanism that makes this work is self-attention.

This chapter explains transformers at the level an AI engineer needs for building systems: enough detail to reason about context windows, latency, model behavior, and architecture choices, without turning the chapter into a linear algebra lecture.

Before transformers, many language systems used recurrent neural networks (RNNs), including LSTMs and GRUs. These models processed text from left to right, passing a hidden state from one step to the next.

This design had two practical limitations.

Problem 1: Long-range information was hard to preserve

If useful information appeared near the beginning of a long sequence, it had to pass through many intermediate updates before influencing later tokens. LSTMs helped, but long-range dependencies were still difficult.

Problem 2: Training was hard to parallelize

Because each step depended on the previous step, the model could not process all token positions independently during training. That limited how efficiently it could use GPUs.

Transformers changed that tradeoff. Instead of passing information through a single hidden state, self-attention lets each token read from other token positions in the sequence. During training, this makes token positions much easier to process in parallel. During generation, decoder-only models still produce one token at a time, but the prompt processing step is highly parallel.

Attention is a learned routing mechanism. For each token position, the model estimates which other positions should influence its representation, and by how much.

Consider:

To interpret "it", the model needs context. In this sentence, "wide" points more naturally to "street" than to "animal". Attention gives the model a way to give more weight to the relevant token positions.

Instead of applying a hand-written grammar rule, the model compares learned vectors.

Queries, Keys, and Values

For every token representation, the model creates three vectors:

Every token position produces all three. Attention compares queries to keys, turns those comparisons into weights, and uses the weights to mix values.

In simplified form:

Input token vectors -> Q, K, V -> compare Q with K -> softmax weights -> weighted sum of V

For a decoder-only LLM, attention is usually causal. A token can attend to earlier tokens, but not future tokens. This masking is essential for next-token prediction: when the model predicts token 20, it must not use token 21.

That distinction matters. Encoder models such as BERT use bidirectional attention over the full input. Generative LLMs use causal attention so they can generate text from left to right.

Real models use large matrices and thousands of dimensions. The mechanics are easier to see with a small example.

Step 1: Create Q, K, and V

Each token starts as a vector. The model multiplies that vector by learned matrices to produce a query, key, and value.

The same matrices are applied across token positions. The model learns those matrices during training.

Step 2: Compute attention scores

The model compares a query with keys using a dot product.

A larger dot product means the query and key point in more similar directions.

Step 3: Scale, mask, and normalize

Attention scores are divided by the square root of the key dimension. This keeps the softmax from becoming too sharp as vector dimensions grow.

In decoder-only models, a causal mask is also applied so a token cannot attend to future positions.

Softmax converts scores into weights that sum to 1.

Step 4: Mix the values

The final output for a token is the weighted sum of value vectors.

This gives the token position a new context-aware representation.

The standard attention equation is:

For decoder-only models, add one more idea: a mask is applied before softmax so future positions receive zero attention weight.

A single attention operation creates one pattern of information flow. Language usually needs many patterns at once: syntactic dependencies, quotation boundaries, variable references in code, pronouns, list structure, and positional relationships.

Multi-head attention runs several attention operations in parallel. Each head has its own learned projections for Q, K, and V. The outputs are concatenated and projected back to the model dimension.

It is tempting to say "this head handles syntax" and "that head handles coreference." Sometimes researchers do find interpretable heads, but the roles are not hard-coded and not always clean. Treat head specialization as a useful intuition, not a guarantee.

The engineering point is more direct: multi-head attention gives the model several learned ways to route information between token positions in the same layer.

A transformer layer contains more than attention. A typical decoder-only block contains:

Modern LLMs often use pre-norm blocks, where normalization happens before attention and before the feed-forward network. Many diagrams show the older post-norm layout because it is easier to draw. The exact ordering varies by architecture, but the main parts are the same.

The feed-forward network is not a side detail. In many transformer models, it accounts for a large share of the parameters and computation. Attention moves information between positions. The feed-forward network transforms each position's representation after that information has been gathered.

Stacking many blocks gives the model depth. The model does not build knowledge in neat stages like "layer 1 is syntax, layer 20 is facts, layer 40 is reasoning." Some patterns are more common in earlier or later layers, but real features are distributed across layers, heads, and feed-forward channels.

Self-attention compares tokens, but by itself it does not know where tokens appear. Without position information, these two sequences would contain the same token set:

The order changes the meaning. Transformers need a way to represent position.

How position is represented

The model combines token identity with position information before or during attention. Different architectures do this in different ways:

Position handling connects directly to context length. A model trained and tuned for a certain context window may behave poorly when pushed far beyond that range. Long-context extensions, RoPE scaling, retrieval, summarization, and attention optimizations all exist because long sequences remain difficult in practice.

Transformers come in three common forms. They share the same building blocks but use different attention patterns.

Encoder-only models

Encoder-only models read the full input with bidirectional attention. Each token can attend to tokens on both the left and the right.

This is useful when the goal is to understand or represent an input:

Examples include BERT, RoBERTa, and DeBERTa.

Decoder-only models

Decoder-only models use causal attention and generate text one token at a time. Each token can attend only to previous tokens.

This is the architecture family most associated with modern LLM APIs:

Examples include GPT-style models, Llama, Mistral, Qwen, and many other open-weight chat models. Some proprietary systems do not publish full architecture details, but the dominant production pattern is decoder-only or decoder-style autoregressive generation.

Encoder-decoder models

Encoder-decoder models use one transformer stack to read the input and another to generate the output. The decoder attends to its own previous outputs and to the encoder's representation of the input.

This architecture is useful for sequence-to-sequence tasks:

Examples include T5, BART, and mBART. The original Transformer paper used an encoder-decoder architecture.

For most of this course, decoder-only models are the default because they power the LLM APIs and open-weight chat models used in AI applications. Encoder models still matter for embeddings, reranking, and classification.

Standard self-attention compares token positions with other token positions. For a sequence of n tokens, full attention computes roughly n x n attention scores.

This O(n^2) scaling is one of the central constraints of the standard transformer.

The practical consequences show up in three places.

1. Prefill latency

Before the model generates the first output token, it has to process the prompt. Long prompts increase time-to-first-token.

2. Memory

Attention and the KV cache consume memory. During generation, the serving system stores keys and values for previous tokens so it does not recompute them from scratch. Long contexts increase KV cache size.

3. Cost

Longer prompts require more computation and more memory bandwidth. Even when an API prices by token count, the serving system is paying a real compute cost underneath.

How modern systems handle long contexts

Engineers use several techniques to make long contexts practical:

Long context is useful, but it is not a substitute for good context engineering. A 200K-token prompt full of irrelevant material can be slower, more expensive, and less reliable than a 10K-token prompt with the right evidence.

Here is what happens when a decoder-only LLM receives a prompt such as:

Step 1: Tokenization

The tokenizer maps text to token IDs. The exact token boundaries depend on the model's tokenizer.

Step 2: Embeddings and position information

Each token ID is mapped to a vector. The model also incorporates position information so it can distinguish order-sensitive sequences.

Step 3: Transformer blocks

The token vectors pass through many transformer blocks. In each block, causal self-attention moves information across positions, and the feed-forward network transforms each position independently.

Step 4: Vocabulary projection

After the final layer, the model uses the hidden state at the last position to produce scores over the vocabulary. These scores are called logits.

Step 5: Decoding

The logits are converted into a next-token choice using a decoding strategy such as greedy decoding, temperature sampling, top-p sampling, or provider-specific constrained decoding.

Step 6: Repeat

The selected token is appended to the context. The model repeats the process until it emits a stop token, reaches a length limit, or the application stops generation.

This is why streaming works: the server can send tokens as they are generated. It is also why long outputs take time: generation is sequential, even though prompt processing is parallelized.

References