Transformer Layers Explained: How LLMs Work

What happens to a token after it enters a Transformer? It passes through neural network operations that help the model combine contextual information, transform its internal representations, and prepare for the next-token prediction process.
The key components are Self-Attention, Multi-Head Attention, Residual Connections, Layer Normalization, and the Feed-Forward Network (FFN). Together, these components form the core of many modern Large Language Models (LLMs).
Let's step inside a Transformer layer and see how each part works. 🧠

---

🏭 1. From Tokens to Transformer Layers​

Consider this sentence:
"Artificial intelligence is changing how businesses operate."

Before reaching a Transformer layer, the text is split into tokens. Each token is converted into a numerical representation called an embedding, and the model incorporates positional information according to its architecture.
The model now has a sequence of vectors representing the input.

For a simplified illustration:

• Token 1 → Vector
• Token 2 → Vector
• Token 3 → Vector
• Token 4 → Vector

These vectors contain numerical information that the neural network can process. However, the model still needs to transform them to capture useful relationships within the context.
That's where the Transformer layers come in.
Each layer applies learned transformations to the representations, allowing the model to build increasingly contextual information as the data moves through the network.

---

👀 2. Self-Attention: How Tokens Use Context​

The first major component is Self-Attention.
Self-Attention allows each token to gather information from other relevant tokens in the sequence. It uses three vectors:

• Query (Q): Represents what a token is looking for.
• Key (K): Helps determine how relevant another token is.
• Value (V): Carries the information that can be combined with other tokens' information.

For example, consider this sentence:
"The student put the book on the table because it was heavy."

To interpret the sentence, a model needs to represent relationships between different words. The word "it" may be associated with "the book" based on the surrounding context.
Self-Attention helps the model combine information from relevant parts of the input when updating its token representations.
The exact relationships learned depend on the sentence, the model's training, and its architecture.

---

🔴 3. Multi-Head Attention: Multiple Attention Paths​

Most classic Transformer designs use Multi-Head Attention, which runs several attention operations in parallel.
Imagine asking eight people to analyze the same sentence. One might focus on grammatical relationships, another might notice relationships between distant words, and another might identify a different pattern.
This is a useful analogy, but there's an important distinction: attention heads do not have fixed, human-assigned roles. Their behavior emerges from the parameters learned during training.
Each head performs its own attention computation. The results are then combined and projected to produce the output of the attention sublayer.

Why use multiple heads?

• They allow the model to learn different attention patterns.
• They provide multiple ways to combine contextual information.
• They increase the flexibility of the attention mechanism.

In short, Multi-Head Attention gives the model several parallel paths for processing relationships within the context.

---

🔄 4. Residual Connections: Preserving the Original Information Path​

After an attention operation, many Transformer architectures use a Residual Connection.
Instead of relying only on the transformed output, the layer combines it with the original input representation.
Conceptually, this looks like:
Output = Input + Transformation(Input)
This is a simplified illustration of a residual connection, not a complete implementation of every Transformer architecture.
Residual connections are useful because they provide a direct path through which information and training gradients can flow across deep neural networks.
Without suitable mechanisms for training deep networks, optimization can become more difficult as the number of layers increases.
Residual connections help address this challenge and are a fundamental part of many modern neural network architectures.

---

🔵 5. Layer Normalization: Supporting Stable Computation​

Another important component is Layer Normalization, commonly called LayerNorm.
LayerNorm normalizes values within a representation using statistics computed across its features. It also typically includes learned scale and shift parameters.
Its purpose is to help make the optimization and computation of deep neural networks more manageable.

LayerNorm is different from Batch Normalization:

• LayerNorm normalizes across features within an individual example or token representation, depending on the implementation.
• Batch Normalization uses statistics calculated across a batch of examples, typically for each feature.

LayerNorm is widely used in Transformer architectures, although its exact position within a layer depends on the model's design.

---

⚙️ 6. Feed-Forward Network (FFN): Transforming Each Token Representation​

Attention allows tokens to exchange contextual information. But the Transformer layer also needs a mechanism to process the resulting representations through additional learned transformations.
This is the job of the Feed-Forward Network (FFN), also commonly described as an MLP, or Multi-Layer Perceptron.
A traditional Transformer FFN typically follows this pattern:
Input → Linear Transformation → Activation → Linear Transformation → Output
The first linear transformation often expands the representation into a larger intermediate dimension. An activation function then introduces nonlinearity, and another linear transformation projects the result back to the model's hidden dimension.
The exact dimensions and activation functions vary by architecture.

Why do we need an FFN if we already have Attention?​

The two components perform different roles.
👀 Attention: Combines information from different tokens in the context.
🧠 FFN/MLP: Applies learned nonlinear transformations to each token's representation.
In many standard Transformer designs, the same FFN is applied independently to every token position, using the same learned parameters.
Think of Attention as a way to exchange contextual information and the FFN as a way to process the information in each representation.
This distinction is simplified, but it provides a useful mental model.

---

🔁 7. How the Components Work Together​

A typical Transformer layer combines attention, residual connections, normalization, and a feed-forward network.
One common design, called a Pre-LN Transformer, follows this conceptual structure:

1. Start with the input hidden states.
2. Apply Layer Normalization.
3. Compute Self-Attention.
4. Add the attention output to the residual path.
5. Apply Layer Normalization again.
6. Process the representations through the FFN.
7. Add the FFN output to the residual path.

The process can be represented as:

Input Hidden States
↓
Layer Normalization
↓
Multi-Head Self-Attention
↓
Residual Connection
↓
Layer Normalization
↓
Feed-Forward Network (FFN)
↓
Residual Connection
↓
Output Hidden States

The diagram is conceptual. It omits implementation details such as dropout and does not represent every Transformer architecture.
Some models use Post-LN designs, while others use different normalization arrangements, attention mechanisms, or FFN variants.
The important point is that a Transformer layer does more than run Attention once. It combines several operations that work together to update the model's internal representations.

---

🤯 8. What Changes as Tokens Pass Through More Layers?​

Now imagine the sentence again:
"Artificial intelligence is changing how businesses operate."
The initial token representations are processed by one Transformer layer, then another, and another.
As they pass through the network, their hidden states are repeatedly transformed.

• Earlier layers contribute to building contextual representations.
• Intermediate layers apply additional learned transformations.
• Later layers produce representations used by the model's output process.

A model may contain dozens or even hundreds of layers, depending on its architecture.
However, don't assume that every layer has one simple, fixed job.

It would be misleading to say:

• Layer 1 understands individual words.
• Layer 2 understands sentences.
• Layer 3 understands emotions.

Real models are more complicated. The patterns represented by each layer depend on the architecture, training data, learned parameters, and task.
A more accurate explanation is that each layer applies learned transformations that can build increasingly complex representations across the network.

---

🧩 9. The Complete Journey Through a Transformer​

Here's a simplified overview of how information flows through a typical Transformer-based language model:

Input Text
↓
Tokenization
↓
Token Embeddings
↓
Positional Information
↓
Transformer Layer 1
↓
Transformer Layer 2
↓
Transformer Layer 3
↓
...
↓
Final Hidden States
↓
Output Projection
↓
Logits
↓
Next-Token Selection

Each Transformer layer typically contains an attention sublayer and a feed-forward sublayer, along with residual connections and normalization.
The exact structure varies between models. For example, some architectures use alternative attention mechanisms or different positional encoding methods.
Notice something important: the model has not necessarily generated any text yet.
It has processed the input and built internal representations that can be used to predict the next token.

---

🎯 10. How Does the Model Turn Hidden States Into a Word?​

Suppose you enter this prompt: "Explain artificial intelligence."
After processing the input through its Transformer layers, a typical autoregressive language model uses its final hidden state for the last relevant token position to calculate scores for possible next tokens.
These scores are called logits.
The model can convert the logits into a probability distribution using Softmax. Depending on the generation settings, a decoding strategy then selects or samples the next token.

For example, the model might assign different probabilities to tokens such as:

• "AI"
• "Artificial"
• "It"
• "The"

These are illustrative candidates, not actual predictions from a specific model.
The selected token is then added to the sequence, and the model repeats the prediction process to generate more text.
One important detail: a token is not always a complete word. It can represent a word, part of a word, punctuation, or another text unit, depending on the tokenizer.
This brings us to the next major topic: Logits, Probability, and Next-Token Prediction.
We'll explore how the model converts its final hidden representations into scores and probabilities, and how techniques such as Softmax, Temperature, Top-k, Top-p, and Sampling affect text generation.

---

✅ 5 Key Takeaways​

1. Self-Attention allows token representations to incorporate information from other tokens.
2. Multi-Head Attention provides multiple parallel attention operations.
3. FFN/MLP applies learned nonlinear transformations to token representations.
4. Residual Connections and Layer Normalization support the design and training of deep Transformer networks.
5. Stacking Transformer layers allows the model to build increasingly complex internal representations.

Most importantly, an LLM is not simply one giant Attention operation. It is a deep neural network made up of multiple layers and learned transformations that work together.
The next step is to understand how those internal representations become the tokens we actually see in a generated response.
Coming next: Logits & Probability - How Does an LLM Predict the Next Token? 🎯