linear mode connectivity

Linear Mode Connectivity is a stronger form of mode connectivity where two trained networks are connected by a straight line (linear interpolation) in parameter space with no loss barrier — meaning $\mathcal{L}(\alpha \theta_1 + (1-\alpha) \theta_2) \leq \max(\mathcal{L}(\theta_1), \mathcal{L}(\theta_2))$ for all $\alpha \in [0, 1]$.

What Is Linear Mode Connectivity?

• Test: Interpolate weights: $\theta_\alpha = \alpha \theta_A + (1-\alpha) \theta_B$, evaluate loss at each $\alpha$.
• Connected: If no loss barrier exists along this line, the two solutions are linearly mode connected.
• Result: Models trained from the same initialization (or with shared early training) are typically linearly connected.

Why It Matters

• Model Merging: Linearly connected models can be averaged for free ensemble performance (model soups).
• Federated Learning: If local models are linearly connected, simple averaging works for aggregation.
• Git Re-Basin: Techniques like permutation alignment can make independently trained models linearly connected.

Linear Mode Connectivity is the alignment test for neural networks — two models that can be linearly interpolated without degradation live in the same loss basin.