CBOW: predict the missing word from its neighbours

Averages the context word vectors and predicts the centre word with a softmax.

The idea in plain words

Fill in the blank. Show a neural net "the ___ sat on the mat" and make it guess "cat". Its guess is built from the words around the blank, mixed together like paint colours into one average.

Step by step

1. Take the context words around a hidden centre word.
2. Average their vectors into one summary vector.
3. Use a softmax over the vocabulary to guess the centre word; nudge the vectors so the right word scores higher.

Play it on paper first

Cover one word in a printed sentence and let a friend guess it from the rest. CBOW is exactly this game, played millions of times.

Watch it explained

• Understanding Word Embeddings 2: CBOW and Skip Gram (Rasa (Algorithm Whiteboard)): Whiteboard drawing of a sentence with a sliding window: hide the middle word, use its neighbours to guess it. That is CBOW; the same video then flips it around for skip-gram.
• Word Embedding and Word2Vec, Clearly Explained!!! (StatQuest with Josh Starmer): Already watched it for Lab 5? Jump to the Word2Vec part near the end, where Josh shows the "use the surrounding words to predict the middle word" network.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ