Dianous

A person is not a domain

2026-07-18 · OLMoE-1B-7B · full writeup available

Prune a mixture-of-experts model to the experts you actually route to. Single domains concentrate (code alone needs 15% fewer experts than generic text), but a person's real usage is a union of domains, and the union needed 8% more experts than generic text, worse at all 16 layers.

Any scheme premised on "this user is narrow" must be tested against the union of what they actually do. Personalisation is not narrowing.

Local training objectives on pretrained models

2026-07-18 · Qwen2.5-0.5B · 80 runs · full writeup available

Training a pretrained model with per-block local losses instead of end-to-end backprop needed 3–4× the forgetting for the same knowledge absorption, replicated with and without rehearsal. Per-block losses force early layers to be immediately predictive, wrecking the division of labour pretraining built.

A method's benign behaviour from scratch says nothing about adapting a trained artifact.

Growing networks at their observed failure sites

2026-07-17 · 10 seeds · full writeup available

Grow capacity where a layer measurably conflates inputs it shouldn't. The pilot effect (+0.60pp, t=3.19) shrank to +0.34pp (t=1.61) on fresh seeds, and growth lost to dense-from-scratch at matched parameters in 10 of 10 seeds.

Where irrecoverable error is observed is not where capacity helps. And pilot effects shrink: never claim from one sweep.

The search gap at consumer scale

2026-07-18 · 5 sizes × 2 datasets · full writeup available

Is there headroom between what small models learn from scratch and what search or distillation could find for them? Mapped knowledge distillation against scratch training across five model sizes on CIFAR-10 and CIFAR-100. No cell opened; distillation was significantly negative in two cells, and at 835k params students beat their own teacher.

Measure whether a phenomenon exists at your scale before building methods to attack it. Scratch-trained small models are already at ceiling on consumer-reachable tasks.

Harness tricks measured on synthetic tasks

2026-07-19 · HotpotQA vs synthetic multi-hop · full writeup available

Context-shuffle majority voting cut errors 9× on synthetic multi-hop reasoning, then delivered ~7.5% relative on real HotpotQA (and nothing at all at 3B). Synthetic tasks are built so nearly all failures are positional; real tasks are dominated by capability failures.

Validate on naturalistic data before believing any harness result.

Scaffolding a 14B model past its ceiling

2026-07 · 14B Q4 · SWE-bench Lite

The hope that a good enough harness lets a quantized 14B model do real software-engineering work. It resolves approximately zero SWE-bench Lite tasks regardless of scaffold. The failures are capability failures, and no amount of orchestration around the model supplies capability the model doesn't have.

The ceiling is the model, not the scaffold.

Rules earned expensively

The cross-cutting methodology every kill above paid for: preregister the gate, confirm on fresh seeds, map before method, check the union, compare frontiers, validate naturalistically. Each rule lists its price.