LLM Concept Lab

Scaled dot-product attention and the √dₖ

Lets every token build a new vector from the tokens most relevant to it.

The idea in plain words

A library search. Your query is what you are looking for; each book has a key (its label) and a value (its contents). Compare your query with every key, turn the scores into percentages, and read a blend of the values, mostly from the best matches.

Step by step

  1. Each token makes three vectors: a query Q, a key K and a value V.
  2. Score = Q · K for every pair; high means relevant.
  3. Divide by √dₖ. A dot product of d random numbers has spread about √d, so without the division, scores for big d get huge, softmax saturates to nearly one-hot, and gradients vanish.
  4. Softmax the scores into weights, and output the weighted sum of the V vectors.

Play it on paper first

A room of 8 people each shouts a number; you add them up. With 64 people the total is much louder even if everyone is equally loud. Dividing by the square root of the head-count keeps the volume steady, which is what √dₖ does for the dot product.

Watch it explained

Other labs