Rethinking Backtesting
Why Backtesting Can Mislead!
In the world of quant investing, backtests are often seen as the gold standard for validating a strategy. But here’s the uncomfortable truth: most backtests lie. Not because of malice, but because of subtle pitfalls that can render even the most elegant models ineffective in live markets.
At Qode Advisors, we view backtesting as a tool - not a verdict. In this post, we’ll walk through common traps, show you how backtests can mislead, and highlight the principles we follow to ensure our models are built for reality, not fantasy.
1. Survivorship Bias: The Illusion of Robust Portfolios
A backtest that uses today's index constituents (say Nifty 50 or BSE 200) over the past decade assumes that these companies always existed in the index. That’s false. Many underperformers or delisted names are excluded, skewing results upward.
Qode Approach: We use point-in-time data to reconstruct the index as it existed at each historical timestamp. This way, we include companies that were part of the universe at that time, even if they were later removed or delisted.
2. Look-Ahead Bias: Knowing Tomorrow's News Today
This is when the strategy unknowingly uses future information to make present-day decisions. For example, using full-year financials in a January 1st rebalance is technically invalid if results were only declared in May.
Real Example: We tested a quality factor using ROE. Without adjusting for earnings announcement dates, the strategy showed a 23% CAGR. After correcting for actual declaration timelines, returns dropped to 14%.
The table below highlights how 10-year backtest returns using ROE as a quality factor vary depending on the rebalance month.
The variation is driven by look-ahead bias—where the strategy inadvertently selects stocks based on earnings results that hadn’t been announced yet at that point in time.
| Month of Rebalancing | CAGR Returns |
|---|---|
| March | 38.00% |
| April | 27.00% |
| May | 22.00% |
| June | 18.00% |
| July | 14.00% |
3. Data Snooping & Overfitting: Too Good to Be True
Backtests can be manipulated—intentionally or not—to fit historical data. You try different cut-offs for P/E, tweak rebalance frequencies, or add filters until the Sharpe ratio looks impressive. But you're now fitting noise, not signal.
Qode Approach: We use walk-forward validation and out-of-sample testing on different time periods and even different market regimes (pre-COVID vs post-COVID) to validate model generalizability.
Below table is an illustrative example that highlights problem of overfitting models.
| Period | In-Sample Sharpe | Out-of-Sample Sharpe |
|---|---|---|
| 2010-2015 | 1.54 | 0.88 |
| 2016-2020 | 1.62 | 0.91 |
| 2021-2024 | 1.29 | 1.22 |
Note the improvement in 2021-24 out-of-sample results as the model matured and became less overfit.
4. Liquidity & Slippage Ignorance: The Real Cost of Execution
Most backtests assume you can buy/sell at the closing price. In reality, slippage, bid-ask spread, and order size constraints eat into returns—especially in mid & small caps.
For strategies with high turnover, trading costs can materially erode returns—sometimes enough to eliminate the strategy’s edge altogether. Strategies with thinner trading edge are especially vulnerable.
This is why it’s critical to factor in slippage, impact costs, and liquidity constraints when moving from a backtest to real-world execution. Ignoring these can turn a promising model into an unviable one.
Qode Approach: We build liquidity-adjusted returns using a conservative estimate of average daily volume. We penalize stocks with less than Rs. 5 crore daily traded value, and apply a slippage haircut of 0.25% per trade on both entry and exit.
5. No Regime Testing: Assuming the Future Will Mirror the Past
A strategy that worked in a falling interest rate environment may not work in a rising rate regime. Many backtests overlook macro contexts.
Qode Approach: We segment our tests across macro regimes: high inflation vs low, credit growth booms vs slowdowns, global risk-on vs risk-off. Models that don’t work across these are either revised or discarded.
Nifty 50 performance across Inflation regimes:
| Inflation Regime | Period | NIFTY 50 CAGR |
|---|---|---|
| High Inflation | 2010–2013 | 6.30% |
| High Inflation | 2019–2020 | 7.10% |
| High Inflation | 2022-2022 | 4.30% |
| Low Inflation | 2016–2017 | 15.20% |
| Low Inflation | 2018–2019 | 12.80% |
| Low Inflation | 2023–2024 | 10.50% |
Note: These periods were selected based on annual CPI inflation data, categorizing years with CPI above 6% as high inflation and below 4% as low inflation.
6. Ignoring Taxes, Impact Cost & Rebalancing Friction
You might show a 20% CAGR in your backtest. But after accounting for short-term capital gains tax, transaction costs, and rebalance friction, actual net return could drop by 300-500 bps.
This is another significant cost to account for in high-churn strategies—once taxes are factored in, the investor’s alpha can quickly erode or even disappear entirely.
Qode Approach: We use a tax-impact module assuming 20% STCG for trades under a year, 0.10% transaction cost, and rebalance lag (1-day delay). This gives us a more conservative and realistic net return profile.
Backtest Humbly, Execute Boldly
A good backtest is like a wind tunnel for your aircraft design. It helps you test under idealized conditions. But markets are messy. At Qode, we focus on robustness, not perfection.
Our alpha doesn't come from curve-fitted strategies. It comes from blending data with economic intuition, testing under multiple stress scenarios, and keeping our assumptions rooted in execution realities.
So the next time you see a beautiful backtest, ask yourself: Would this survive the real world? If the answer is no, it might just be a mirage.
Share this article