Measured Pruning Damage Depends on the Evaluation Corpus: A Renaming Control for Mixture-of-Experts Expert Pruning
The reported damage figure moves 1.6× under a rewrite that changes no program structure.
At a glance
- Model
- 256-expert mixture-of-experts
- Intervention
- 32 experts pruned per layer
- Corpora
- CPython stdlib, and the same files with identifiers renamed
- Metrics
- Base NLL cost; flip rate at confidently predicted positions
- Headline
- 44% of base NLL on stdlib vs 16% renamed — 1.6× in absolute nats
- Control
- A rewrite that changes no program structure
- Mechanism
- Memorisation — hypothesised, not demonstrated
Abstract
Pruning 32 experts per layer from a 256-expert MoE costs 44% of base NLL on CPython stdlib and 16% on the same files with identifiers renamed: the reported damage figure moves 1.6× in absolute nats under a rewrite that changes no program structure. Flip rates at confidently predicted positions halve. An extractability probe confirms the corpora differ in memorisation as intended, but per-file extractability does not predict per-file damage, so memorisation is reported as the hypothesised mechanism rather than a demonstrated one.
Method
Prune 32 experts per layer from a 256-expert mixture-of-experts model, then measure the cost twice: once on CPython stdlib, and once on the same files with identifiers renamed.
The rename is a rewrite that changes no program structure. An extractability probe checks whether the two corpora differ in memorisation.
What we found
The same pruning costs 44% of base NLL on CPython stdlib and 16% on the same files with identifiers renamed. The reported damage figure moves 1.6× in absolute nats under a rewrite that changes no program structure.
Flip rates at confidently predicted positions halve.
The extractability probe confirms the corpora differ in memorisation as intended.
What's uncertain
Per-file extractability does not predict per-file damage.
Memorisation is reported as the hypothesised mechanism rather than a demonstrated one.