stemming

Stemming is a heuristic-based text normalization process that reduces words to their root form (stem) by chopping off ends of words — a crude but fast approximation of morphological analysis used to reduce vocabulary size and group related word forms.

Algorithms

• Porter Stemmer: The most famous algorithm (1980) — a cascade of regex rules ("-ing" → "", "-ies" → "i").
• Snowball: An improvement over Porter, supporting multiple languages.
• Lancaster: A more aggressive stemmer (often over-stems).

Issues

• Over-stemming: "Universe" and "University" → "Univers". (Bad, they mean different things).
• Under-stemming: "Alumnus" and "Alumni" → "Alumnus", "Alumni". (Failed to merge).
• Not Words: Stems are often not real words ("Apples" → "Appl").

Why It Matters

• Search: Essential for Information Retrieval (query "run" matches doc "running").
• Efficiency: Reduces vector space dimensionality in Bag-of-Words models.
• Modern NLP: Largely replaced by Lemmatization or Subword Tokenization (BPE) in Deep Learning.

Stemming is chopping off the suffix — a rule-based approach to word normalization that favors speed over linguistic correctness.