Our winning result was the average result of looking
Yesterday we found a promising window in a strategy we had already retired, ran a control, and watched it pass. Today we ran the test that should have come first, and it turned out we had found nothing at all. This is that test, because it is more useful than the finding would have been.
The setup
Our bot's cron fires every 15 minutes through the trading day, which gives 23 entry slots with enough trades to measure. We had noticed one of them looked strong, run a random-entry control against it, and got the 97.5th percentile.
That number is not wrong. It is just not a p-value, because we had looked at all 23 slots before deciding which one to test. With 23 slots and a 5% threshold, you expect roughly one apparently-significant slot from pure noise. Finding one proves nothing.
The reality check
The correction for this is old and well-established (White, 2000; Hansen, 2005): build the null distribution of the best result rather than of a result.
- Score every control run in every slot, not just the slot you care about.
- Centre each slot on its own control mean, so slots that simply carry more market exposure cannot win by default.
- Take the maximum centred statistic from each control run. That gives you a distribution of what the best-looking slot looks like when there is nothing there.
- Compare your best slot against that distribution.
23 slots, 150 control runs, over 2022–2025 — deliberately excluding 2026, the year our parameters were fitted on.
The result
| Value | |
|---|---|
| Best slot | 19:45 |
| Its expectancy | +0.895 per trade |
| Its control mean | +0.130 |
| Delta | +0.765 |
| p, this slot alone | 0.020 |
| p, corrected for searching 23 slots | 0.467 |
The null distribution of the best slot has a mean of +0.750. Our best slot came in at +0.765.
A random search over these slots produces a winner averaging 0.750. We produced 0.765. We had not found an edge. We had found, to two decimal places, the average result of looking.
The detail that would have settled it earlier
The best slot in this window is 19:45. The best slot over 2022–2026 was 13:15. Different window, different winner.
A real effect keeps winning when you move the window. A noise one relocates. We could have caught this without any statistics at all, just by re-running the same search on a different period and noticing the answer moved.
Why this is worth publishing
Because 0.020 is the number that normally gets published. It is the standard output of reporting the winner of a search, it looks like a finding, and nothing about it announces that a search happened. The corrected number requires you to have kept every losing arm of the search, which is exactly what nobody does.
It also indicts our method rather than one result. Every slot-level claim this project has made came out of an uncorrected search. So does the weekly stop-width sweep that has been running since July, which crowns the best of ten widths every Friday and has never been corrected either. That one matters more, because the crowned width is what the live bot actually ran on.