A backtest is a machine for confirming what you want to believe. We know because we audited our own for months: strategies claiming +47% a year that honestly lost 6%, detectors "hitting 10x in 45% of cases" that really hit 6.8%, backtests with profit factor 1.55 whose live win rate was zero. Every single inflation came from one of the nine mechanisms below — which is why our engine now enforces a rule against each. Use this as a checklist against any backtest anyone shows you. Especially ours.
Positions still open at the end of the test get dropped. Delisted or rugged assets never enter the database. Either way the sample that remains is pre-filtered to winners — in our own memecoin data this alone inflated "reaches 2x" rates by about 30 percentage points. Check: ask how open positions and dead assets are handled. An honest engine force-closes every position at the last real price and reports it.
Backtests love filling you at the exact signal-bar close with zero fees. Live, you pay the spread, slippage, and fees on both sides — on majors that's roughly 0.2% per round trip, on memecoins several times more. A strategy earning 0.3% per trade in a frictionless backtest is a loser in production. Check: costs must be non-zero and stated; entries belong on the next bar after the signal.
"+120% total return" means nothing without max drawdown, losing streak, and expectancy. We once found a "+$59K winner" among our paper traders that was three lucky moonshots on top of hundreds of losers. Check: demand the full set — win rate, profit factor, max drawdown, longest losing streak, per-year breakdown — in one view.
Tune parameters until history looks great and you have curve-fitting, not an edge. Our S5 strategy backtested at profit factor 1.55 over 3 years; on forward data its win rate was 0–18% and every cell went red. Check: train/test split by time, and the test half must hold up. If test win rate is below ~70% of train, it's overfit.
If the test window ends while trades are still open, or outcomes are measured before they had time to mature, the tail of the distribution is missing. Check: the backtest should exclude the most recent bars entirely so every position has fully resolved.
With n=8, a coin flip produces "75% win rate" one run in ten. We refuse to render summary statistics below 10 trades and stamp everything under 30 as likely noise. Check: n ≥ 30 before believing any percentage.
We tested 7 famous retail strategies over 3 years of BTC — none beat holding, and several only made money in one bull stretch of 2024 that they rode like an index fund. Check: per-year and per-month breakdown; if the strategy loses in every year the asset fell, you've built directional beta, not an edge.
If a backtest can't show what data it scanned, what exact rules it compiled, and how many bars it processed, the errors above can't even be looked for. Check: the report should expose its query, data source, and bar counts. Ours prints them on every run.
The subtlest lie is ordering. Humans anchor on the first number they see; if that's "+$13,344", the overfit warning below it doesn't register. It's why our engine is hard-wired to print red warnings above the PnL — the frontend cannot reorder them. Check: if a tool shows profit before caveats, assume marketing, not measurement.
Our own "COMBO L7" strategy claimed +47.5% per year in backtests. The forensic found look-ahead in a trend filter (mechanism #4 wearing mechanism #1's coat): re-measured honestly it earned −6.4% per year, and the combined variant underperformed the simple one it was supposed to improve. We killed it and published the numbers — the same audit trail behind our full backtest confession.