The gap that wasn't there

A whole family of methods rests on one premise: that a small model trained from scratch leaves accuracy on the table which something bigger or smarter could recover for it. Distillation assumes it. Network growth assumes it. Every "train large, compress small" pipeline and every architecture-search scheme aimed at compact models assumes it. Call the premise the search gap: the space between what scratch training finds and what search could find, at a fixed deployment size.

Here is the embarrassing part, stated plainly because this notebook exists to record method as much as results: we measured the premise last. We had already spent three nights building and killing a network-growth method (the entry above this one in the ledger) before asking whether the gap it was designed to capture exists at all on hardware like ours. The map took one night. It should have been night one.

Setup

The map: gap(task, size) = distilled accuracy minus from-scratch accuracy, matched architecture and training budget (66 epochs), across a ladder of five student sizes from 6.5k to 835k parameters, on CIFAR-10 and CIFAR-100. One teacher per task, same family, ~1.5M parameters, trained once and frozen so all variance is student-side. Distillation used temperature 4, α 0.7. Five fresh seeds, paired per cell.

GATE · WRITTEN BEFORE THE RUN A cell is open only if the mean paired gap is ≥ +1.0 points with paired t > 2.776. Cells where the teacher leads scratch by < 2 points are reported as "ceiling reached", not "gap zero". Decision rule: the smallest open cell becomes the arena for future search-method tests; if no cell opens on either task, the search gap is not observable in this model family at this scale, and toy-scale tests of search methods are pointless.

Result: ten cells, zero open

task · params	scratch	distilled	gap	t	headroom
C10 · 6.5k	68.28	68.84	+0.56	+1.30	18.8
C10 · 11k	73.63	74.25	+0.62	+3.38	13.5
C10 · 54k	81.23	81.65	+0.42	+2.93	5.9
C10 · 210k	85.58	85.40	−0.19	−1.88	1.5
C10 · 835k	88.00	86.85	−1.15	−8.51	−0.9
C100 · 9.4k	34.09	32.37	−1.72	−4.63	28.0
C100 · 15k	38.70	38.03	−0.68	−3.10	23.4
C100 · 63k	50.70	50.88	+0.18	+0.66	11.4
C100 · 229k	58.74	57.32	−1.43	−10.49	3.3
C100 · 873k	63.16	60.86	−2.30	−71.81	−1.1

Test accuracy (%), mean of 5 paired seeds. Gap = distilled − scratch, in points. Headroom = teacher − scratch. Grey rows are at or near ceiling; red cells are where distillation significantly hurt.

Read the table twice, because it says two different damning things.

First, the best case anywhere is +0.62 points, well under the preregistered bar. Where distillation helps at all, it helps by decorative amounts.

Second, and more interesting: on CIFAR-100, distillation actively hurts at most sizes, including exactly where the premise says it should shine. The 9.4k student sits 28 points below its teacher, the largest headroom on the map, and distillation made it 1.7 points worse (t = −4.6). This is the capacity-gap phenomenon from the distillation literature arriving on schedule: a small student cannot match the soft targets of a much larger teacher, and trying costs it accuracy that plain cross-entropy would have kept. And at the top of the ladder, the 835k students beat their own 1.5M teacher on both tasks, at which point there is nothing left to distil.

Three lines of evidence, one convergence

This map did not fail alone. Three independent measurements from the same project converge on the same sentence. The growth method's own control arm found distillation worth zero over scratch at the size it grew to (73.23% vs 73.22%). The growth method itself lost to dense-from-scratch at matched parameters in 10 of 10 seeds. And this map, built to find the territory where such methods could win, found no territory. Scratch-trained small models are already at ceiling on the tasks a consumer card can reach. There is no big-then-compress dividend here for any search method to capture, ours or anyone's.

Scope, stated honestly

One distillation recipe, untuned. One teacher family, same architecture lineage as the students, 66-epoch budget. A carefully tuned recipe or a much stronger cross-family teacher could shift individual cells, and nothing here says distillation is useless at datacenter scale, where the regime is different. The preregistered question was narrower and is now answered: does a gap open anywhere in this family, at this scale, under the conditions a solo researcher would actually use? No.

VERDICT The search gap is not observable at consumer scale. Every method premised on it, growth, distillation pipelines, compact-model search, is aimed at territory that does not exist down here. The transferable lesson costs one sentence and we paid three nights for it: measure whether the phenomenon exists at your scale before building methods to attack it. The map is cheap. Run the map first.

What this bought us

An entire research line closed in a night, honestly, with numbers instead of fatigue. That is the good ending: not every negative result is a small one, and this one retired a category of future bad ideas at once. The map now sits in the ledger so that the next person with a 12 GB card and a growth-method idea can spend their three nights on something the territory actually contains.