polysemantic neurons

Polysemantic neurons is the neurons that respond to multiple unrelated features rather than a single interpretable concept - they complicate simple one-neuron-one-concept interpretations of model internals.

What Is Polysemantic neurons?

• Definition: A single neuron may activate for distinct patterns across different contexts.
• Representation Implication: Suggests compressed superposed coding in limited-dimensional spaces.
• Interpretability Challenge: Feature overlap makes direct semantic labeling ambiguous.
• Evidence: Observed through activation clustering and dictionary-based decomposition studies.

Why Polysemantic neurons Matters

• Method Design: Requires interpretability tools that go beyond single-neuron labels.
• Editing Risk: Changing one neuron can unintentionally affect multiple behaviors.
• Compression Insight: Polysemanticity reflects efficiency tradeoffs in representation capacity.
• Safety Relevance: Hidden feature overlap can mask risky behavior pathways.
• Theory Development: Motivates superposition and sparse-feature modeling frameworks.

How It Is Used in Practice

• Feature Decomposition: Use sparse autoencoders or dictionaries to split mixed neuron signals.
• Intervention Caution: Avoid direct neuron edits without downstream behavior audits.
• Cross-Context Analysis: Test activation meanings across diverse prompt domains.

Polysemantic neurons is a key phenomenon in understanding distributed transformer representations - polysemantic neurons show why robust interpretability must focus on feature spaces, not only individual units.