target encoding
Target Encoding (Mean Encoding) is a supervised categorical encoding technique that replaces each category with the mean of the target variable for that category — transforming "New York" into 0.82 (82% of New York customers bought) and "Paris" into 0.23 (23% of Paris customers bought), providing a single numeric column that captures predictive signal for high-cardinality features (1000+ categories) where one-hot encoding would create an impractically wide matrix, but requiring careful regularization to prevent severe overfitting.
What Is Target Encoding?
- Definition: For each category, calculate the mean of the target variable across all training examples with that category, then replace the category string with that mean value — converting a categorical feature into a continuous numeric feature that directly encodes predictive information.
- Why It's Powerful: It converts high-cardinality categories (10,000 zip codes, 50,000 product IDs) into a single informative numeric column — impossible with one-hot encoding (10,000 new columns) and uninformative with label encoding (arbitrary integers).
- Why It's Dangerous: Without regularization, target encoding is the most common source of data leakage in machine learning pipelines — a category with 1 observation and a positive target gets encoded as 1.0, giving the model perfect (but fake) signal.
How Target Encoding Works
| City | Target (Bought?) | Count | Mean Target | Encoded Value |
|---|---|---|---|---|
| New York | 1,1,1,0,1 | 5 | 0.80 | 0.80 |
| Paris | 0,0,1,0,0 | 5 | 0.20 | 0.20 |
| Tokyo | 1,1,0,1,1 | 5 | 0.80 | 0.80 |
| Berlin | 1 | 1 | 1.00 | ⚠️ Overfitting! |
The Overfitting Problem and Solutions
| Problem | Example | Solution |
|---|---|---|
| Small category | Berlin has 1 sample → mean = 1.0 (perfect but fake) | Smoothing with global mean |
| Target leakage | Encoding uses the same data the model trains on | Compute encoding inside cross-validation folds |
| Rare categories | A new city in test data has no encoding | Fall back to global mean |
Smoothing Formula
$\text{Encoded} = \frac{n \times \text{category\_mean} + m \times \text{global\_mean}}{n + m}$
Where n = category count, m = smoothing parameter. For Berlin (n=1, global_mean=0.5, m=10): $(1 × 1.0 + 10 × 0.5) / (1 + 10) = 0.545$ — pulled toward the global mean instead of the unreliable 1.0.
Cross-Validation Encoding (K-Fold)
| Fold | Training Data | Encoding Source | Test Data |
|---|---|---|---|
| Fold 1 as test | Compute means from Folds 2-5 | Means from Folds 2-5 | Apply to Fold 1 |
| Fold 2 as test | Compute means from Folds 1,3-5 | Means from Folds 1,3-5 | Apply to Fold 2 |
This ensures no data point's own target value contributes to its encoding — preventing leakage.
When to Use Target Encoding
| Scenario | Use Target Encoding? | Reason |
|---|---|---|
| High-cardinality (1000+ categories) | Yes ✓ | One-hot creates too many columns |
| Low-cardinality (3-10 categories) | Usually no | One-hot works fine and is simpler |
| Tree-based models (XGBoost, LightGBM) | Yes ✓ | Trees benefit from continuous signal |
| Linear models | Carefully | Must ensure proper smoothing |
Target Encoding is the most powerful encoding for high-cardinality categorical features — converting thousands of categories into a single informative numeric column that captures predictive signal, with the critical requirement that smoothing and cross-validation encoding must be used to prevent the data leakage that makes naive target encoding notoriously prone to overfitting.