effect size
Effect size is a quantitative, standardized measure of the practical magnitude of an observed phenomenon or experimental difference, evaluated independently of sample size. While a p-value only assesses whether an observed difference is unlikely to have arisen under a null hypothesis of zero effect, effect size quantifies how large the difference actually is in standardized or physical units. Because statistical significance tests are directly confounded by sample size—where test statistics scale proportionally to √n, rendering arbitrarily microscopic, trivial fluctuations statistically significant (p < 0.001) in massive datasets, while masking game-changing physical effects (p > 0.05) in small pilot studies—effect size provides the true invariant scale needed to evaluate engineering importance, clinical relevance, and scientific reproducibility.
The foundational effect size for continuous outcomes is Cohen's d, which expresses the distance between two group means in units of pooled standard deviation. Given two groups with sample means X̄₁ and X̄₂ and pooled standard deviation s_pooled, Cohen's d is defined as:
Dividing by s_pooled standardizes the metric, removing all dependence on physical measurement units. Jacob Cohen (1988) established benchmark anchors widely used across science and engineering: small (d = 0.2), medium (d = 0.5), large (d = 0.8), and very large (d ≥ 1.2). An effect size of d = 0.8 means the treatment distribution's mean is shifted by 0.8 standard deviations relative to the control distribution; in terms of distributional separation, this corresponds to a Cohen's U₃ of 78.8% (meaning 78.8% of the treatment group scores above the control group mean) and a distribution overlap of only 68.9%. In small sample experiments (N < 50), Cohen's d exhibits a slight positive bias; Hedges' g applies an exact correction factor J(df) ≈ 1 − 3 / (4df − 1) to yield an unbiased population effect size estimate. When experimental intervention alters group variance, Glass's Δ standardizes solely against the unmanipulated control group standard deviation s_control.
Effect size metrics fall into three fundamental mathematical families depending on the underlying data structure and analytical objective. First, the d family (Standardized Mean Differences): Cohen's d, Hedges' g, and Glass's Δ, tailored for comparing continuous measurements between distinct cohorts. Second, the r and η² family (Strength of Association and Variance Explained): Pearson correlation r ([−1, 1]), Coefficient of Determination R² ([0, 1]), and Eta-squared (η² = SS_effect / SS_total) in ANOVA designs. Because sample η² overestimates population variance explained, Omega-squared (ω²) and Epsilon-squared (ε²) adjust for model degrees of freedom and mean square error, providing less biased population estimates. In multi-factor designs, Partial Eta-squared (η_p² = SS_effect / [SS_effect + SS_error]) isolates the proportion of variance explained by a specific factor after excluding other model effects. Third, the Risk and Categorical family: in binary 2×2 contingency tables, relative impact is measured by Relative Risk (RR = p₁ / p₂), Odds Ratio (OR = [p₁/(1 − p₁)] / [p₂/(1 − p₂)]), Risk Difference (RD = p₁ − p₂), and the Number Needed to Treat (NNT = 1 / |RD|), which translates abstract probabilities into concrete physical cohort requirements.
| Metric Family | Canonical Metric | Mathematical Formulation | Benchmark Anchors (Small / Medium / Large) | Primary Analytical Domain |
|---|---|---|---|---|
| Standardized Mean Difference | Cohen's d / Hedges' g | d = (X̄₁ − X̄₂) / s_pooled | 0.20 / 0.50 / 0.80 | Two-group comparison of continuous parameters |
| Control-Referenced Shift | Glass's Δ | Δ = (X̄_treatment − X̄_control) / s_control | 0.20 / 0.50 / 0.80 | Treatment alters variance; control variance is baseline |
| Variance Explained | Eta-squared (η²) | η² = SS_effect / SS_total | 0.01 / 0.06 / 0.14 | Multi-factor ANOVA; proportion of total sum-of-squares |
| Population Variance Explained | Omega-squared (ω²) | ω² = (SS_effect − df_effect × MS_error) / (SS_total + MS_error) | 0.01 / 0.06 / 0.14 | Unbiased population variance explained |
| Linear Association | Pearson r / R² | r = Cov(X, Y) / (σ_X × σ_Y) | r = 0.10 / 0.30 / 0.50 | Bivariate correlation & regression modeling |
| Binary Association | Odds Ratio (OR) | OR = (p₁ / (1 − p₁)) / (p₂ / (1 − p₂)) | 1.50 / 2.50 / 4.00 | Logistic regression & binary defect classification |
| Categorical Impact | Number Needed to Treat (NNT) | NNT = 1 ÷ abs(p₁ − p₂) | Context-specific (lower is stronger) | Intervention clinical/engineering effectiveness |
Effect size is the critical mathematical prerequisite for rigorous experimental design and statistical power calculation. Statistical power (1 − β, typically targeted at 80% or 90%) defines the probability of correctly rejecting a false null hypothesis. Power is an exact function of three variables: chosen significance threshold α (typically 0.05), sample size n, and the hypothesized population effect size d. For a balanced two-sample t-test, the required per-group sample size is approximated by:
Because sample size scales inversely with the square of effect size (n ∝ 1/d²), detecting subtle effects requires exponentially larger investments: detecting a large effect (d = 0.8) at α = 0.05 with 80% power requires only n ≈ 25 samples per group; detecting a medium effect (d = 0.5) requires n ≈ 64 per group; whereas detecting a small effect (d = 0.2) demands n ≈ 393 samples per group. Specifying a Minimum Detectable Effect (MDE) prior to data collection prevents underpowered experiments that waste capital and overpowered studies that detect meaningless trivialities.
In advanced semiconductor manufacturing and artificial intelligence, effect size serves as the indispensable filter against false alarms and low-yield process changes. In high-volume wafer fabs, automated metrology tools (CD-SEM, ellipsometry, scatterometry) sample tens of thousands of dies across production lots. With sample sizes exceeding n = 50,000, standard two-sample t-tests yield infinitesimal p-values (p < 10⁻¹²) for a 0.04 nm line-width variation between two etch chambers. However, if the process critical dimension (CD) standard deviation is σ = 0.8 nm, the standardized effect size is an imperceptible d = 0.04 / 0.8 = 0.05. Treating this statistical artifact as an actionable problem wastes millions in unnecessary chamber cleanings and tool downtime; fabs enforce strict Minimum Detectable Effect size thresholds (such as d ≥ 0.4 or process capability shift ΔC_pk ≥ 0.15) before qualifying tool modifications. Similarly, in artificial intelligence and machine learning benchmark evaluations (e.g. LLM inference latency, reasoning benchmarks, click-through-rate A/B tests), calculating standardized effect sizes and confidence intervals protects engineering teams from deploying complex model architectures that claim "statistically significant" improvements that are physically indistinguishable from evaluation noise.
Three recurring statistical traps compromise effect size interpretation. First, the p-value Confusion: conflating extreme statistical significance with substantial physical impact. Second, the Winner's Curse: small, underpowered studies (n < 25) that happen to achieve p < 0.05 systematically inflate observed effect sizes (Type M magnitude errors), producing effects that vanish upon replication. Third, Uncorrected Small-Sample Bias: failing to apply Hedges' g correction when evaluating pilot runs, leading to overconfident projections of production improvements.
Read effect size as the universal currency of practical impact: while the p-value merely checks whether noise can be ruled out, effect size measures the true physical distance between states, providing the scale-independent metric necessary to compare findings across tools, fabs, models, and scientific disciplines.