The idea felt inevitable. Mixture-of-experts models route each token through a small subset of their experts, and domain-specific pruning is a solved problem: give an MoE nothing but Python and it concentrates onto a fraction of its experts, so you can discard the rest. The literature covers task-specific pruning thoroughly. What nobody had asked was the personal version: watch one person's actual usage for a while, find the experts they route to, and ship them a smaller model that is fully theirs.
For someone running models on a 12 GB card, that would be a meaningful win. It also had the shape of a good research question: mature technique, unasked variant, cheap to test. We tested it in one evening, and the answer is no, for a reason that generalises well beyond mixture-of-experts.
The preregistered gate
The gate matters because routing statistics offer many post-hoc escape routes: you can always find some layer or some threshold where the story looks alive. Writing the number down first removes the temptation.
Method
We took OLMoE-1B-7B (64 experts per layer, 16 MoE layers), and measured, for each corpus, how many experts are needed to cover 95% of routing probability mass at each layer. Corpora: three slices of one person's real usage (their code, their notes, their chat), the union of those three, and two controls: generic stories, and narrow single-domain Python.
Result
| Corpus | Experts for 95% mass (of 64) |
|---|---|
| personal — code only | 39.9 |
| narrow python (control) | 44.9 |
| generic stories (control) | 46.8 |
| personal — notes | 48.8 |
| personal — union (what you'd ship) | 50.5 |
| personal — chat | 51.8 |
Mean experts required per layer to cover 95% of routing mass. The gate required the union at ≤ 37.4 (20% under generic). It landed 8% over generic.
The single-domain result replicates the literature: code alone concentrates, needing 15% fewer experts than generic text. If a person were a domain, their union would concentrate too. It does the opposite. The union needed more experts than generic text, and was worse at every one of the 16 layers. There is no depth region where the idea survives.
Why it fails
Two mechanisms, one obvious in hindsight and one structural.
First, the hindsight one: a person is not a domain, a person is a union of domains. Their code concentrates, their notes concentrate somewhat, their chat barely concentrates, and each concentrates onto different experts. A union is at least as broad as its widest part and generally broader. "Personalisation" quietly imports the assumption "narrowing," and that assumption is false for anyone whose usage you would actually want to serve. The model that serves one whole person is closer to a general model than a domain model.
Second, the structural one: MoE training uses load-balancing auxiliary losses that actively push routing toward uniformity. Even the narrow Python control needed 70% of the experts, and normalised routing entropy sat at 0.80–0.94 everywhere. The architecture is trained to resist exactly the concentration this idea needs.
What this bought us
The negative result redirected real engineering. Follow-up work dropped pruning entirely and left the base model whole, because the evidence says the model that serves one person has to stay general. As it turns out, so do you.