LLM Concept Lab

The Markov assumption and the bigram model

Turns raw counts into next-word probabilities using only the previous word.

The idea in plain words

Phone keyboard autocomplete. When you type "good", it suggests "morning" or "luck" by looking only at that last word, not at your whole message. That shortcut is the Markov assumption: the recent past is a good enough stand-in for the whole past.

Step by step

  1. Count how often each word follows each other word in a pile of text.
  2. Divide: P(next | previous) = times "previous next" appeared ÷ times "previous" appeared.
  3. To write a sentence, keep picking a likely next word and moving forward one step.

Play it on paper first

Take any book page. Pick a word such as "the", and tally every word that follows it. The tallies are your bigram table. Now try the same with the two previous words and notice how few matches you find: that is why we cut the history short.

Watch it explained

  • Markov Chains Clearly Explained! Part 1 (Normalized Nerd): A restaurant that decides tomorrow's dish (burger, pizza or hot dog) only from today's dish. That is exactly the Markov assumption, drawn as a diagram and a table you can read without any maths.
  • The spelled-out intro to language modeling: building makemore (Andrej Karpathy): Takes 32,000 real names, counts which letter follows which, turns the counts into probabilities and generates new names. The first part is a bigram model built live, line by line. Watch the first 30-40 minutes; the rest goes deeper.

Other labs