In May 2026 we discovered that almost every backtest number we had ever published was wrong. Not slightly wrong — wrong by a factor of six. This is the full story of how it happened, what the honest numbers turned out to be, and the five specific ways a backtest lies that we now check for automatically. We publish it because every "proven strategy" you have ever been sold suffers from at least one of these.
We run a memecoin scanner that fires alerts when tokens look promising. To rank the detectors we backtested them against our own alert history — thousands of real alerts with recorded outcomes. The results looked spectacular, we quoted them in our marketing, and they were garbage. Three separate mechanisms inflated them, and none of them is exotic: they are the default failure modes of anyone backtesting anything.
Our outcome database stored each token's lifetime peak price. When a backtest asked "did this alert reach 10x?", it compared the entry against a peak that was often reached before the moment a live trader could have entered — or that was only knowable in hindsight. A signal that fires while the token is already collapsing from its top still counted as a 10x winner, because the peak was in the record.
When we re-measured with a forward-only tracker — peak ratcheted strictly from the alert timestamp onward, polled live every two minutes — the 10x rate of our flagship detector fell from the 45–55% we had been quoting to 6.8%.
Tokens that rugged in the first minutes often never made it into the tracking table that the backtest read — they died before the first snapshot. The pool the backtest saw was pre-filtered to survivors, which alone added roughly 30 percentage points to any "reaches 2x" rate computed from it. The fix was brutal but simple: log every alert at fire time with its live market data, before knowing anything about its future.
Our "consensus" analysis asked: when detectors A and B both fire, how often does the token moon? The bug: it took the maximum peak across all detectors' records — including detections that happened after the combo moment. A combo could get credit for a run that started hours before the second detector ever fired. Measured causally — outcome from the moment the combo actually completed — the headline combo's 2x rate fell from 78% to 17%.
The purge forced the architecture we should have started with:
- Forward-only outcome tracking. Every alert from every detector is logged at fire time with live entry data; peaks ratchet forward only. No record can ever know the future.
- An anti-bias backtest engine for user strategies: entries execute on the next bar after a signal, costs and slippage are always on, train/test splits and overfit flags are computed on every run — and the warnings print above the PnL.
- Honest marketing numbers. Our alert messages now quote the 6.8%, not the 45%. It reads worse. It is true.
- 1. Ask where the exit price comes from. If any exit references a high, a peak, or "the best price in the window", the test is reading the future. Real exits trigger on information available at that bar.
- 2. Ask what happened to the losers. If the dataset can't show assets that went to zero, delisted, or rugged, every rate in it is inflated by survivorship.
- 3. Re-run the same idea on a neighboring parameter. A real edge is a plateau: if TP 4% makes money but TP 3% and 5% lose, you have curve-fit noise. Our robustness sweep automates exactly this.
- 4. Split train and test by time. An edge discovered on 2023–2024 must still work on 2025–2026 untouched data. If the win rate halves out-of-sample, it was memorized, not discovered.
- 5. Compare against doing nothing. We tested 7 famous retail strategies on 3 years of BTC — not one beat simply holding. "Profitable" is meaningless without that baseline.
Because the alternative is trusting companies that have the same bugs and never looked. Every mechanism above is the default outcome of backtesting naively — we found ours because we went hunting for them, published the downgrade, and wired the checks into the product so they run on every strategy automatically, including yours.