Search before build
In one session we independently rediscovered six phenomena that already had names and papers. The instincts were good, every effect was real, and that is exactly the trap: real effects are the ones that already have literatures. Two minutes of search beats hours of GPU and days of wrong framing. Do the novelty check before the build, not after.
Preregister the gate, then obey it
Write the success threshold before the run, including what counts as void. This ledger contains two sweeps reported void by their own rules and one claim refused for missing a threshold by 0.05 of a t-statistic. Each of those moments felt pedantic and each one later proved to be the load-bearing wall. Post-hoc thresholds are how ledgers rot.
Paid in: the bridge test and collision growth
Pilot effects shrink; confirmation runs on fresh seeds
An effect at t = 3.19 on five seeds became t = 1.61 on ten fresh ones. Selection pressure inside a small pilot inflates whatever you noticed; re-using the pilot's seeds for confirmation is significance chasing with extra steps. The confirmation run is not bureaucracy, it is the experiment.
Measure whether the phenomenon exists at your scale before attacking it
We built a method for three nights before spending the one night that showed its target territory was empty at our scale. The map is always cheaper than the weapon. Run the map first.
Paid in: The gap that wasn't there
Check the union, not the part
Any premise of the form "this user / workload / deployment is narrow" must be tested against everything it actually spans, because a union is broader than its widest part. That one cheap measurement killed an entire project in an evening, before anything was built on the premise.
Paid in: A person is not a domain
Compare frontiers, not points
When two methods each have a knob that trades one metric against another, any single-setting comparison is gameable, by accident or otherwise. Sweep the knob, compare the curves. If your result reverses under a different learning rate, you did not have a result.
Never believe a harness result until it survives naturalistic data
Synthetic benchmarks are constructed from parts the model can handle, so their failures are almost entirely failures of arrangement, exactly the kind a harness fixes. A nine-fold improvement on an invented task became 7.5% relative on the real one. Ask of any scaffold: does it repair reading, or supply capability? Only the second survives contact with reality.
Keep the eval distribution out of the training mix
Rehearsal data drawn from the same distribution as the forgetting metric let one arm score better than the untouched base model, an impossible number that flattered the result and announced nothing. Contamination that helps you is the kind you won't look for. Disjointness is a design requirement, not a nicety.
Audit the measurements your conclusion stands on, they lean your way
Confounds are not symmetric noise: the ones you fail to notice are disproportionately the ones that favor your hypothesis, because those are the ones you had no urge to investigate. In one session, three separate measurement artifacts each pointed the hopeful direction; correcting one of them reversed the recommendation outright. Before believing a number you wanted, ask what besides the truth could have produced it.