For one evening we thought we had found a lever that beats the compounding law of multi-hop reasoning. The numbers were spectacular. They were also true, replicable, and mechanistically clean, and the lever is still nearly worthless. That combination is exactly why this entry exists.
The discovery, which was real
Setting: a small local model answering multi-hop questions over a context of facts, failing on some reasoning steps under greedy decoding. The obvious rescue is resampling at temperature: ask again, vote. On the steps the 3B model got wrong, we tried eight temperature resamples with the context held fixed, against eight tries with the context re-arranged, the same facts in a different order.
| lever, K=8 on failed steps | any correct | majority vote |
|---|---|---|
| temperature (context fixed) | 0.206 | 0.059 |
| context permutation | 1.000 | 0.941 |
3B model, steps failed under greedy decoding. Re-arranging the context recovers essentially everything; resampling recovers essentially nothing.
The controls sharpened it. Paraphrasing the question, same facts, different words, recovered almost nothing (0.116), so the sensitivity is specifically to context arrangement, not to input variation in general. Four permutations saturate (0.912 at K=4, no gain at K=8). And the arithmetic on synthetic chains was intoxicating: permutation voting lifts per-step accuracy roughly 0.85 → 0.99, and because errors compound multiplicatively over hops, a 4-hop chain goes from ~0.52 to ~0.96. A nine-fold error reduction, on our benchmark, from shuffling.
There was even a story about why this matters for people like us: permutation defeats prefix caching (each variant needs its own prefill), so it costs ~5× wall-clock, a price a single-user local box can pay and a per-token API business cannot. A lever tailor-made for the home lab. You can feel the narrative assembling itself.
The validation, which was fatal
Then we ran it on HotpotQA, real multi-hop questions over real Wikipedia paragraphs, instead of chains we generated ourselves.
| model · mode (K=4) | EM | F1 | wall |
|---|---|---|---|
| 3B · plain | 0.230 | 0.300 | 50s |
| 3B · permutation vote | 0.225 | 0.288 | 157s |
| 3B · + verification | 0.245 | 0.291 | 157s |
| 7B · plain | 0.535 | 0.657 | 661s |
| 7B · permutation vote | 0.565 | 0.675 | 1048s |
| 7B · + verification | 0.570 | 0.677 | 1048s |
HotpotQA, 200 questions per cell. At 3B the lever does nothing (slightly negative alone). At 7B: +3.5 EM points, a ~7.5% relative error reduction, for 1.6× the wall-clock.
Nine-fold on our synthetic chains. Seven and a half percent, relative, on the real thing, and only for the larger model. Not zero, but not a lever. The gap between those two numbers is the finding.
Why synthetic tasks flatter harnesses
The mechanism is almost tautological once seen. A synthetic multi-hop task is assembled from facts the model can individually handle; by construction, nearly every failure is a failure of attention arrangement, the model losing an entity in the middle of a context it could otherwise parse. Those are exactly the failures that re-arrangement fixes. Real tasks are not built that way. On HotpotQA the model fails because a paragraph is genuinely hard, an inference is genuinely beyond it, an answer needs knowledge it lacks. Those are capability failures, and no amount of shuffling, voting, or verifying supplies capability. A harness recovers the positional slice of failures, and synthetic benchmarks are, by construction, almost entirely positional slice.
The other humiliation, recorded on principle
That same session, we independently "discovered" six phenomena, the compositionality gap, the multiplicative compounding law, per-step retrieval, distractor filtering, the decomposition tax, position-order sensitivity, and every one of them already had a name and a paper, some going back to 2022. The instincts were good; each effect was real, which is why it had a name. But two minutes of literature search beats hours of GPU time and days of wrong framing, and we spent the hours first, six separate times, in one night. Search before build. It is the cheapest rule in this ledger and the one we broke most.
What this bought us
A working immune response. Every harness idea since has had to answer one question before any code was written: does this repair reading, or does it supply capability? If the honest answer is reading, the expected value is the HotpotQA column, not the synthetic one, and the idea has to be worth 7.5%, not 900%. Most aren't. The ones that survive that question are the ones worth a night of GPU.