Attention Mechanism Notes

Attention lets a model decide which tokens matter for each representation. The core operation compares queries and keys, then uses the resulting weights to mix values.

Inline math works for symbols such as $Q$, $K$, and $V$.

$$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Minimal NumPy sketch

import numpy as np

def softmax(x):
    shifted = x - np.max(x, axis=-1, keepdims=True)
    exp = np.exp(shifted)
    return exp / np.sum(exp, axis=-1, keepdims=True)

def attention(q, k, v):
    scale = q.shape[-1] ** 0.5
    weights = softmax(q @ k.T / scale)
    return weights @ v

Bash experiment note

python train.py --model tiny-transformer --dataset toy-sequence --seed 42

The important habit is to keep the article focused on explanation and link complete experiments elsewhere.