span masking

Span Masking is a pre-training strategy for masked language models where contiguous sequences (spans) of tokens are masked instead of individual random tokens — popularized by SpanBERT and T5, this approach forces the model to predict entire phrases using only the surrounding context, encouraging the learning of longer-range dependencies and phrasal semantics.

Span Masking Details

• Contiguous Spans: Instead of masking single tokens, mask a sequence like "New York City" as [MASK] [MASK] [MASK] or a single [MASK] token.
• Geometric Distribution: Span lengths are often sampled from a geometric distribution (e.g., mean length 3) — favoring short phrases but allowing longer ones.
• Objective: Usually combined with a span boundary objective (SBO) or standard MLM stability.
• T5 Approach: T5 replaces the entire span with a single sentinel token (e.g., <extra_id_0>) and trains the model to generate the missing span.

Why It Matters

• Harder Task: Predicting a span is harder than predicting a single token — reduces the reliance on shallow local cues.
• Downstream Performance: Significantly improves performance on span-selection tasks like Question Answering and Coreference Resolution.
• Efficiency: Can be more sample-efficient than single-token masking for learning structural relationships.

Span Masking is hiding chunks of text — forcing the model to reconstruct entire phrases from context, fostering deeper semantic understanding.