Measured Pruning Damage Depends on the Evaluation Corpus: A Renaming Control for Mixture-of-Experts Expert Pruning
The same pruning looks 1.6× more damaging on stdlib than on an identifier-renamed copy of it.
At a glance
- Model
- 256-expert mixture-of-experts
- Intervention
- 32 experts pruned per layer
- Corpora
- CPython stdlib, and the same files with identifiers renamed
- Metrics
- Base NLL cost; flip rate at confidently predicted positions
- Headline
- 44% of base NLL on stdlib vs 16% renamed — 1.6× in absolute nats
- Control
- A rewrite that changes no program structure
- Mechanism
- Memorisation — hypothesised, not demonstrated
Abstract
Pruning 32 experts per layer from a 256-expert MoE costs 44% of base NLL on CPython stdlib and 16% on the same files with identifiers renamed: the reported damage figure moves 1.6× in absolute nats under a rewrite that changes no program structure. Flip rates at confidently predicted positions halve. An extractability probe confirms the corpora differ in memorisation as intended, but per-file extractability does not predict per-file damage, so memorisation is reported as the hypothesised mechanism rather than a demonstrated one.
Method
Prune 32 experts per layer from a 256-expert mixture-of-experts model, then measure the cost twice: once on CPython stdlib, and once on the same files with identifiers renamed.
The rename is the control. It changes no program structure, so any difference in measured damage cannot be attributed to the code doing something different — only to what the evaluation corpus is. An extractability probe checks that the two corpora do differ in memorisation, as intended.
What we found
The same pruning costs 44% of base NLL on stdlib and 16% on the renamed files. The reported damage figure moves 1.6× in absolute nats under a rewrite that changes no program structure.
Flip rates at confidently predicted positions halve.
What's uncertain
The extractability probe confirms the corpora differ in memorisation as intended, but per-file extractability does not predict per-file damage.
Memorisation is therefore reported as the hypothesised mechanism rather than a demonstrated one. Confidence: likely.