LLM Concept Lab

Evaluating language models: perplexity and friends

Gives one number to compare two language models on held-out text.

The idea in plain words

A multiple-choice quiz. If a model is as unsure as someone guessing between 6 equally likely options each time, its perplexity is 6, like rolling a fair die. A perplexity of 1 would be a model that is never surprised; higher means more confused.

Step by step

  1. Hold back some text the model never trained on.
  2. Measure how surprised the model is by each real next word (low probability = high surprise).
  3. Average the surprise and convert it into "effective number of choices": that is perplexity. Lower is better.

Play it on paper first

Shannon game: a friend hides a sentence and you guess it letter by letter. Count your guesses. Text you predict quickly has low perplexity for you; text full of surprises has high perplexity.

Watch it explained

  • Entropy (for data science) Clearly Explained!!! (StatQuest with Josh Starmer): Uses boxes of orange and blue chickens to explain "surprise": a rare pick surprises you more than a common one. Perplexity is just this average surprise turned back into a count of choices, so this is the best warm-up.
  • Language Model Evaluation and Perplexity (DeepLearning.AI NLP course): Splits a corpus into training and test sets, then computes perplexity on a small sentence by hand. Shows why a lower score means a better model.

Other labs