masked language modeling

Masked Language Modeling (MLM) is the pre-training objective introduced by BERT where a percentage of input tokens are hidden (masked), and the model must predict them using bidirectional context — typically masking 15% of tokens and minimizing the cross-entropy loss of the prediction.

The "Cloze" Task

• Input: "The quick [MASK] fox jumps over the [MASK] dog."
• Target: "brown", "lazy".
• Refinement: 80% [MASK], 10% random token, 10% original token (to prevent mismatch between pre-training and fine-tuning).
• Efficiency: Only 15% of tokens provide a learning signal per pass (unlike CLM where 100% do).

Why It Matters

• Revolution: Started the Transformer revolution in NLP (BERT) — smashed records on benchmarks (GLUE, SQuAD).
• Representation: Creates deep, context-aware vector representations of words.
• Pre-training Standard: Remains the standard for encoder-only models (BERT, RoBERTa, DeBERTa).

MLM is fill-in-the-blanks — the bidirectional pre-training task that teaches models deep understanding of language structure and relationships.