salient span masking

Salient Span Masking is a domain-specific masking strategy (used in REALM, RetriBERT) where spans that are "salient" (named entities, dates, key technical terms) are masked preferentially — specifically designed to force the model to look up external knowledge or learn facts, rather than just guessing common words.

Mechanism

• Identification: Use a tagger (NER) or frequency analysis (TF-IDF) to find "salient" terms.
• Masking: Mask these terms.
• Purpose: "The capital of France is [MASK]." -> Model MUST know/retrieve "Paris". Random masking "The [MASK] of France is Paris" is trivial grammar.

Why It Matters

• RAG (Retrieval-Augmented Generation): Crucial for training retrievers — the retriever must find a document containing "Paris" to solve the mask.
• Question Answering: Improves performance on Open-Domain QA.
• Fact Learning: Shifts focus from syntax ("The cat sat on [MASK]") to semantics/facts.

Salient Span Masking is fact-checking tests — specifically hiding the answers to factual questions to force the model to learn or retrieve knowledge.