Transformers, ELI5 · Part 4 / 10

Inside the feed-forward network

Widen, drop the negatives, shrink back. A bank of pattern-detectors that stores most of the model's knowledge.

The FFN is the “think alone” half of a block. Three steps, one token at a time: go wide, filter, come back.

Wide

Multiply the token (width 4) by W_up to make it wider (8 here; ~16,000 in a real model). Each new slot is a detector asking “is this feature here?”

Filter — ReLU

ReLU: negative → 0, positive → keep. Only the detectors that fired stay lit.

This bend is the model’s key non-linearity, and it matters more than it looks: without it, stacking a hundred layers would be no smarter than one — they’d fold into a single multiply. (Real models soften the bend — older ones used a smooth cousin, GeLU; today’s gate it instead, see the panel below.)

Back

Multiply by W_down to fold back to width 4. Each lit detector writes its stored knowledge in.

Try it: on × W_up, click a cell to see the multiply-and-add. On ReLU, watch the blue negatives clip to 0. On × W_down, the zeroed detectors add nothing.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FeedForward(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        # blow the vector up to 4x its size, so the block has room to "think"
        self.up = nn.Linear(d_model, 4 * d_model)    # d_model -> 4*d_model
        # squeeze it back down to the original size so it fits the next layer
        self.down = nn.Linear(4 * d_model, d_model)  # 4*d_model -> d_model

    def forward(self, x):
        # x: (batch, seq, d_model) -- runs on EVERY token independently, no token sees another here
        x = self.up(x)        # (batch, seq, 4*d_model): expand
        x = F.gelu(x)         # (batch, seq, 4*d_model): smooth on/off switch, keeps the useful signal
        x = self.down(x)      # (batch, seq, d_model): shrink back to normal size
        return x

Attention moves information between words. The FFN is where a lot of it is stored — most of the model’s weights live here.

Next: stack the block.