data mixture
Pretraining Data Mixture and Curation is the strategic selection and weighting of training data domains that critically determines the capabilities, biases, and performance characteristics of large language models — where the composition of web text, books, code, scientific papers, dialogue, and multilingual content in the training mixture has a larger impact on model quality than architecture differences, making data curation one of the most important and closely guarded aspects of frontier LLM development.
Why Data Mixture Matters
- Same architecture + same compute + different data mixture → dramatically different models.
- Code data improves reasoning (even for non-code tasks).
- Math data enables quantitative reasoning.
- Book data improves long-range coherence.
- Web data provides breadth but includes noise.
Data Source Characteristics
| Source | Volume | Quality | What It Teaches |
|---|---|---|---|
| Common Crawl (web) | 100T+ tokens | Low-medium | Breadth, world knowledge |
| Wikipedia | ~4B tokens | High | Factual knowledge, structure |
| Books (BookCorpus, etc.) | ~5B tokens | High | Long-form coherence, reasoning |
| GitHub/StackOverflow | ~100B tokens | Medium-high | Code, structured thinking |
| ArXiv/PubMed | ~30B tokens | High | Scientific reasoning |
| Reddit/forums | ~50B tokens | Medium | Dialogue, opinions |
| Curated instruction data | ~1B tokens | Very high | Task following |
Known Model Mixtures
| Model | Web | Code | Books | Wiki | Other |
|---|---|---|---|---|---|
| Llama 1 | 67% | 4.5% | 4.5% | 4.5% | 19.5% (CC-cleaned) |
| Llama 2 | ~80% | ~10% | ~4% | ~3% | ~3% |
| Llama 3 | ~50% | ~25% | ~10% | ~5% | ~10% |
| GPT-3 | 60% | 0% | 16% | 3% | 21% |
| Phi-1.5 | 0% | 0% | 0% | 0% | 100% synthetic |
Data Filtering Pipeline
[Raw Common Crawl: ~300TB compressed]
↓
[Language identification] → Keep target languages
↓
[URL and domain filtering] → Remove known low-quality sites
↓
[Deduplication] → MinHash + exact dedup → removes 40-60%
↓
[Quality classifier] → FastText trained on curated vs. random → remove bottom 50%
↓
[Content filtering] → Remove toxic, PII, CSAM
↓
[Domain classification] → Tag and weight by domain
↓
[Final mixture: ~5-15T high-quality tokens]Data Mixing Strategies
| Strategy | Approach | Used By |
|---|---|---|
| Proportional | Sample proportional to domain size | Early models |
| Upsampled quality | Oversample high-quality domains (Wikipedia, books) | GPT-3, Llama 1 |
| DoReMi | Optimize domain weights via proxy model | |
| Data mixing laws | Predict performance from mixture via scaling laws | Research frontier |
| Curriculum | Start with easy/clean data, add harder data later | Some proprietary models |
Deduplication Impact
- Training on duplicated data: Memorization increases, generalization decreases.
- Exact dedup: Remove identical documents → easy, removes ~20%.
- Near-dedup (MinHash): Remove ~similar documents → removes additional 20-40%.
- Effect: Deduplication equivalent to 2-3× more unique training data.
Data Quality vs. Quantity
| Approach | Data | Model | Result |
|---|---|---|---|
| Llama 2 (70B) | 2T tokens (web-heavy) | 70B | Strong general |
| Phi-2 (2.7B) | 1.4T tokens (curated + synthetic) | 2.7B | ≈ Llama 2 7B quality |
| FineWeb-Edu | Web filtered for educational content | Various | Significant improvement |
Pretraining data curation is the most impactful yet least understood lever in LLM development — while architectural innovations yield marginal gains, the choice of which data to train on and in what proportions fundamentally determines a model's capabilities, with frontier labs investing millions of dollars and years of effort into data pipelines that are among their most carefully protected competitive advantages.