Why Does Robustness Reduce Superposition?

1. Background

1.1 Superposition

Superposition occurs when a model encodes more features than there are neurons (Elhage et al., 2022). We control it by intervening on sparsity when training toy models, following the setup of Elhage et al. (2022). Sparsity is governed by a threshold $S \in [0,1]$, which determines the activity of each component of the input $x \in \mathbb{R}^m$:

A feature $x_i$ is therefore active with probability $p(\mathcal{A}) = 1 - S$.

1.2 Interference

Elhage et al. (2022) define interference as the geometry of features in activation space, measured by the degree of non-orthogonality between feature directions. For two feature directions $W_{:,i}, W_{:,j} \in \mathbb{R}^n$, they are orthogonal when $W_{:,i}^\top W_{:,j} = 0$, and exhibit interference when $W_{:,i}^\top W_{:,j} \neq 0$. The more strictly positive this inner product, the more likely spurious activations are to corrupt the reconstruction of feature $i$, constituting harmful interference.

1.3 Adversarial Examples

An adversarial example is a perturbation of $x \in \mathbb{R}^m$, imperceptible to humans, that maximizes the MSE loss within an $\ell_2$ ball of radius $\varepsilon$:

Following Gorton & Lewis (2025), a small noise term is added to $x$ prior to attack generation to avoid gradient masking, after which a one-step $\ell_2$ gradient attack is applied.

2. Setup

2.1 Toy Models

We adopt the simplified toy model of Elhage et al. (2022) as employed by Gorton & Lewis (2025). Let $W \in \mathbb{R}^{n \times m}$, with $n = 20$ and $m = 100$. Given inputs $x \sim \mathcal{D}$, $\mathcal{D} \subset \mathbb{R}^m$, hidden representations are computed as $h = Wx \in \mathbb{R}^n$ and features are reconstructed via $\hat{x} = \text{ReLU}(W^\top h + b)$. The training objective is $\mathcal{L} = \|x - \hat{x}\|_2^2$. Higher sparsity leads the model to represent more features and, consequently, to exhibit more superposition.

2.2 Measuring Superposition

Elhage et al. (2022) quantify superposition as $n/\|W\|_F$, the number of dimensions per feature. We take the reciprocal to obtain the number of features per dimension, a more directly interpretable quantity:

$\Phi = 1$ indicates no superposition; $\Phi > 1$ indicates superposition, which increases monotonically with sparsity.

2.3 Representational Power

We define representational power as the squared Frobenius norm of $W$, measuring the total energy distributed across all feature directions $W_{:,i}$:

2.4 Feature Representation Threshold

To determine which features are represented and which are dropped, we measure the norm $\|W_{:,i}\|_2$ of each feature column following Elhage et al. (2022), applying the threshold $\tau = 0.01$:

2.5 Measuring Interference

Let $G = WW^\top \in \mathbb{R}^{n \times n}$ be the Gram matrix of $W$. The interference matrix is obtained by zeroing the diagonal, retaining only pairwise cross-feature terms:

2.6 Adversarial Training Protocol

Following Gorton & Lewis (2025), models are trained over a sweep of sparsity levels on a mixture of clean and adversarial examples:

We set $\alpha = 0.5$ to balance clean and robust accuracy and prevent collapse in either regime. Attacks are generated via (1) with $\varepsilon = \frac{0.1}{B}\sum_{x \in \mathcal{B}}\|x\|_2$, where $\mathcal{B}$ denotes the training batch of size $B$. Models are trained for $150{,}000$ steps at a learning rate of $10^{-3}$.

2.7 Sparsity Sweep

To cleanly isolate our theory empirically, we exclude both extremes of the sparsity range. At $S \leq 0.7$, models drop most features regardless of training regime, as sparsity is too low to motivate superposition and interference is easily avoided. At $S \geq 0.98$, both models represent all features at negligible interference cost. We therefore sweep over $\mathcal{S} = \{0.78,\, 0.80,\, 0.88,\, 0.90\}$, the range in which the adversarially trained model consistently drops more features than the standard model—an effect we attribute to the presence of distinct feature classes, rather than to the routine optimization of feature benefit against interference cost.

3. Representational Power and Feature Dropping

We train two models over $\mathcal{S}$—one under the adversarial training protocol (6) and one under standard training—with $x \sim \mathcal{U}([0,1]^m)$. We measure representational power (3) and the number of dropped features (4) for each.

Computing representational power across our sparsity sweep, we observe a key finding: adversarially trained models consistently exhibit lower representational power than their standardly trained counterparts.

To investigate why the adversarially trained model exhibits lower representational power, we compute $\|W_{:,i}\|_2$ for all $i$. Applying (4), we find that adversarially trained models drop more features than standardly trained models across all $S \in \mathcal{S}$.

We examine the dropped features through the lens of interference. Normalizing $W$ and deducing the interference matrix (5), we compute each column's total interference with the rest, then average across retained and dropped groups separately. We find that dropped features either exhibit higher interference than retained ones, or—when both groups are negative—the adversarial model preferentially retains those closest to $-1$.

These results establish that adversarially trained models reduce superposition by dropping more features than their standardly trained counterparts, and that the dropped features are those the model identifies as geometrically costly. The preference for retaining features with interference nearest to $-1$ is particularly telling: a value of $-1$ corresponds to antipodal feature directions, the configuration that maximally suppresses cross-feature activation. This aligns with the observation of Elhage et al. (2022) that models preferentially pack features into opposing directions to minimize spurious reconstruction. A key question remains: do the dropped features correspond to the robust or non-robust classes of Ilyas et al. (2019) and Li & Li (2025)?

4. Robust and Non-Robust Feature Recovery

4.1 Structured Data Construction

To answer this question, Section 3 must be reproduced under a data distribution where the ground-truth identity of robust and non-robust features is known a priori. We therefore construct a synthetic dataset that operationalizes the feature taxonomy of Ilyas et al. (2019) and Li & Li (2025) under the constraints of our toy model setting.

Robust features are broadly characterized as high-signal directions that encode stable, generalizable patterns, whereas non-robust features are low-amplitude, high-frequency directions that are predictive under clean inputs but easily disrupted by adversarial perturbations. More precisely, Ilyas et al. (2019) show that models depend substantially on non-robust features for classification, yet those features cease to correlate with the correct label after an attack. Li & Li (2025) further characterize this taxonomy by establishing that robust features carry greater signal amplitude than non-robust ones, while non-robust features are denser across the feature space.

We note, however, that our setting differs substantially from the classification context in which this taxonomy was originally defined. Our experiments are conducted on toy models optimizing a reconstruction loss (MSE), which bears structural similarities to linear regression but falls well short of the complexity of classification over real-world data with semantic labels. The instantiation of robust and non-robust features below is therefore necessarily an approximation made under these constraints.

We instantiate this taxonomy as follows. Let $x \sim \mathcal{U}([0,1]^m)$ with the sparsity mask of §1.1 applied. A uniformly random permutation $\pi$ of $\{1, \ldots, m\}$ partitions the feature indices into a robust set $\mathcal{R} = \{\pi(1), \ldots, \pi(n_r)\}$ and a non-robust set $\mathcal{N} = \{\pi(n_r + 1), \ldots, \pi(m)\}$, with $|\mathcal{R}| = n_r$ and $|\mathcal{N}| = m - n_r$. Each partition is then amplitude-scaled according to:

with $a_r = 6.0$ and $a_{nr} = 0.2$. Robust features thus carry high amplitude—strong, stable signal—while non-robust features carry low amplitude, making them more susceptible to erasure under $\ell_2$ perturbations. The partition $(\mathcal{R}, \mathcal{N})$ is retained at generation time and serves as the ground truth against which the model's dropped feature indices are evaluated. The full data generation procedure is given below.

def create_data(sparsity, num_samples, n_features=100, n_robust=30):
    values = torch.rand(num_samples, n_features)
    mask = values > sparsity
    x = values * mask.float()
    pi = torch.randperm(n_features)
    robust_idx     = pi[:n_robust]
    non_robust_idx = pi[n_robust:]
    x[:, robust_idx]     *= 6.0
    x[:, non_robust_idx] *= 0.2
    return x, x.clone(), robust_idx, non_robust_idx

To remain consistent with the definition of Li & Li (2025), we set $|\mathcal{N}| > |\mathcal{R}|$, making non-robust features denser than robust ones. We note that normal models can be sensitive to the amplitude ratio between robust and non-robust features; the specific values ($a_r = 6.0$, $a_{nr} = 0.2$) were chosen to reflect the signal-to-noise distinction while maintaining numerical stability during training.

4.2 Results

Training on data generated via (7), we obtain the same findings as in Section 3. The structured data, however, yields a substantially cleaner pattern: the adversarially trained model drops exactly $|\mathcal{N}| = 70$ features at every $S \in \mathcal{S}$, regardless of sparsity level. Notably, at $S \in \{0.88,\, 0.90\}$, the standardly trained model drops no features at all—yet the adversarially trained model still drops exactly $|\mathcal{N}|$—indicating that, despite the model having enough sparsity to represent all features, adversarial training remains highly sensitive to non-robust features. This sensitivity being sparsity-invariant suggests that those features directly interfere with the model's ultimate goal:

In order to determine whether or not the dropped features are primarily the non-robust ones, we map the ground truth indices of non-robust features retrieved at data creation time and compare them to the indices of the dropped features retrieved at the end of training. We find that they correspond exactly to $\mathcal{N}$ at every $S \in \mathcal{S}$, confirming that the adversarial model explicitly targets the non-robust feature set.

4.3 Why Non-Robust Features Are Dropped: A Theoretical Perspective

Our empirical findings raise a natural question: why do adversarially trained models systematically drop non-robust features? We offer a theoretical explanation grounded in the properties of non-robust features under adversarial attack. The argument is conceptual rather than a formal derivation, but it provides intuition for the observed behavior.

Let $\mathcal{D} = \{(x, y)\}$ with $y = x$, parameters $\theta$, loss $\mathcal{L}$, and adversarial examples $x_{\text{adv}}$ generated via an $\ell_2$-bounded attack. We conceptualize the adversarial training's primary training objective as:

Building on Ilyas et al. (2019), we define a feature $i$ as non-robust if it is highly sensitive to adversarial perturbations such that $x_{\text{adv},i} \not\approx x_i$, breaking the correspondence with its target $y_i = x_i$. Denote the set of non-robust features $\mathcal{N} \subset [m]$.

We hypothesize that features in $\mathcal{N}$ interfere with the adversarial training objective via two complementary mechanisms:

Direct loss inflation. Since $\mathcal{L} = \|y - \hat{x}\|_2^2 = \sum_i (y_i - \hat{x}_i)^2$ and $y$ is fixed under attack, each flipped feature $i \in \mathcal{N}$ contributes a non-zero penalty term to the loss. Even a single flipped feature increases $\mathcal{L}$, and the effect compounds: inputs with more non-robust features incur proportionally higher loss. Let $k$ denote the number of non-robust features flipped per adversarial example. This gives:

Dropping all features in $\mathcal{N}$ eliminates these penalty contributions entirely, and gives $P(\mathcal{N}_{\text{flipped}} \mid x_{\text{adv}})=0$—the probability that an adversarial example has flipped non-robust features—thus enabling the model to minimize (8) freely.

Costly interference. Non-robust features exhibit costly feature geometry, as evidenced by their mean total interference lying in the range $[0, 2)$ compared to retained features clustering near $-1$. This costly geometry induces spurious cross-feature activations that corrupt the reconstruction and further inflate $\mathcal{L}$.

Together, these two mechanisms explain why adversarially trained models prune non-robust features: they are the primary obstacle to minimizing the adversarial objective, and removing them reduces superposition while improving robustness.

5. Discussion

We have established a clean causal chain explaining why adversarial training reduces superposition: adversarially trained models drop more features than standard models, leaving fewer features to encode in the same dimensional space, and thus reducing superposition. We further demonstrated that the dropped features correspond precisely to non-robust features as defined by Ilyas et al. (2019).

These results answer our core question but raise new ones. First, why do adversarially trained models preserve the geometry of remaining features, keeping interference values largely intact (Gorton & Lewis, 2025)? Second, why does adversarial training preferentially align features in opposite directions, enabling antipodal superposition? Understanding these emergent behaviors would deepen our mechanistic account of adversarial robustness.

Our findings remain constrained by the toy model setting. The natural next step is validating our theory on real-world models using mechanistic tools like sparse autoencoders (SAEs), moving beyond controlled toy settings.

Recent work at the intersection of robustness and interpretability suggests a promising research direction. We are particularly interested in two questions: First, does superposition reduction in real adversarial models translate to meaningfully improved interpretability? Second, can we intentionally leverage the robust/non-robust feature trade-off to reduce polysemanticity in a controlled manner, without sacrificing critical representations?

References

Citation

Please cite this work as:

Or use the BibTeX citation: