Backtesting·Strategy Optimization·Intermediate
Overfitting Detection
Detect and quantify strategy overfitting using advanced statistical methods including the deflated Sharpe ratio test, probability of backtest overfitting metric, and performance degradation analysis across systematic parameter grid variations.
backtestingmodel-validationpattern-recognition
Backtesting Overfitting Detection
Comprehensive guide to detecting and analyzing overfitting in trading strategy backtests.
Introduction to Backtesting Overfitting
What is Backtesting Overfitting?
Backtesting overfitting occurs when a trading strategy is optimized too heavily on historical (in-sample) data, capturing market noise and specific conditions that are unlikely to repeat in live trading. The strategy performs exceptionally well on the data it was trained on but fails dramatically on new, unseen market data (out-of-sample).
Why Does It Happen?
- Parameter Curve-Fitting: Over-optimizing strategy parameters (moving average periods, thresholds, etc.) to maximize past returns
- Data Snooping: Running many strategy variations and selecting the best performing one by luck rather than robustness
- Look-Ahead Bias: Accidentally using future information in calculations
- Insufficient Data: Optimizing on too few trades or too short a time period
- Ignoring Transaction Costs: Over-trading without properly accounting for slippage and commissions
Key Indicators of Overfitting
- Large Sharpe Ratio Degradation: In-sample Sharpe >> Out-of-sample Sharpe
- Return Collapse: Strong returns in-sample, weak out-of-sample
- Increased Drawdown: Maximum drawdown increases significantly out-of-sample
- Win Rate Degradation: Win rate drops substantially out-of-sample
- Parameter Sensitivity: Strategy highly sensitive to small parameter changes
- Profit Factor Decline: Profit factor falls sharply out-of-sample
[23]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')
# Set random seed for reproducibility
np.random.seed(42)
plt.style.use('default')
plt.rcParams['figure.figsize'] = (14, 6)Part 1: Generate Synthetic Price Data
We'll create realistic synthetic OHLC (Open, High, Low, Close) price data with multiple market regimes to simulate real market conditions.
[24]
def generate_price_data(start_price=100, days=1000, trend=0.0001, volatility=0.02):
"""
Generate synthetic OHLC price data using geometric Brownian motion.
Args:
start_price: Initial price
days: Number of trading days
trend: Daily drift/trend
volatility: Daily volatility (standard deviation)
Returns:
DataFrame with OHLC data and dates
"""
dates = pd.date_range(start='2021-01-01', periods=days, freq='D')
# Generate daily returns using geometric Brownian motion
returns = np.random.normal(trend, volatility, days)
# Create price series
prices = start_price * np.exp(np.cumsum(returns))
# Generate intraday high and low
highs = prices * (1 + np.abs(np.random.normal(0, volatility/2, days)))
lows = prices * (1 - np.abs(np.random.normal(0, volatility/2, days)))
# Ensure low < close < high
closes = prices
opens = pd.Series(prices).shift(1).fillna(start_price).values
df = pd.DataFrame({
'Date': dates,
'Open': opens,
'High': np.maximum(highs, np.maximum(opens, closes)),
'Low': np.minimum(lows, np.minimum(opens, closes)),
'Close': closes,
'Volume': np.random.randint(1000000, 5000000, days)
})
df.set_index('Date', inplace=True)
return df
# Generate 1000 days of price data
df = generate_price_data(start_price=100, days=1000, trend=0.0003, volatility=0.015)
print(f"Data shape: {df.shape}")
print(f"\nFirst few rows:")
print(df.head())
print(f"\nLast few rows:")
print(df.tail())
print(f"\nPrice Statistics:")
print(df['Close'].describe())Data shape: (1000, 5)
First few rows:
Open High Low Close Volume
Date
2021-01-01 100.000000 101.835765 100.000000 100.778083 2635765
2021-01-02 100.778083 101.297097 100.490426 100.599464 3990811
2021-01-03 100.599464 101.657507 100.599464 101.612063 1724857
2021-01-04 101.612063 104.495917 101.612063 103.991349 3585181
2021-01-05 103.991349 104.200656 102.185673 103.657833 4787208
Last few rows:
Open High Low Close Volume
Date
2023-09-23 174.385529 175.098052 173.602943 173.703883 3293673
2023-09-24 173.703883 178.540633 173.703883 178.505127 4475488
2023-09-25 178.505127 181.475789 178.505127 180.283384 4020745
2023-09-26 180.283384 180.283384 178.350874 178.799002 1449764
2023-09-27 178.799002 181.403212 178.799002 180.395385 4302923
Price Statistics:
count 1000.000000
mean 118.028513
std 25.872106
min 84.248099
25% 103.072464
50% 112.381679
75% 124.571676
max 202.224838
Name: Close, dtype: float64
[25]
plt.figure(figsize=(14, 6))
plt.plot(df.index, df['Close'], label='Close Price', linewidth=2, color='blue')
plt.title('Synthetic Price Data (1000 Trading Days)', fontsize=14, fontweight='bold')
plt.xlabel('Date')
plt.ylabel('Price')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
Part 2: Define a Trading Strategy
We'll implement a simple Moving Average Crossover Strategy:
- Long signal: When short MA > long MA
- Short signal: When short MA < long MA
- Parameters: Fast MA period, Slow MA period, Stop-loss percentage
This strategy is prone to overfitting because its performance depends heavily on the choice of MA periods.
[26]
def calculate_moving_averages(data, fast_period, slow_period):
"""
Calculate fast and slow moving averages.
"""
data['MA_Fast'] = data['Close'].rolling(window=fast_period).mean()
data['MA_Slow'] = data['Close'].rolling(window=slow_period).mean()
return data
def generate_signals(data):
"""
Generate trading signals based on MA crossover.
Signal: 1 = Long, 0 = No position
"""
data['Signal'] = 0
data.loc[data['MA_Fast'] > data['MA_Slow'], 'Signal'] = 1
data['Position'] = data['Signal'].diff() # 1 = entry, -1 = exit
return data
def backtest_strategy(data, fast_period=10, slow_period=30, stop_loss=0.02, initial_capital=10000):
"""
Perform backtest of the MA crossover strategy.
Args:
data: DataFrame with OHLC data
fast_period: Fast MA period
slow_period: Slow MA period
stop_loss: Stop loss percentage (e.g., 0.02 = 2%)
initial_capital: Starting capital
Returns:
Dictionary with backtest results and metrics
"""
# Copy data to avoid modifying original
df = data.copy()
# Calculate moving averages
df = calculate_moving_averages(df, fast_period, slow_period)
df = generate_signals(df)
# Initialize portfolio tracking
portfolio_values = [] # This will store the daily portfolio values
current_capital = initial_capital # This tracks the compounding capital after closed trades
position = 0
entry_price = 0
capital_at_entry = 0 # Capital when a position was opened
trades = []
# Simulate trading
for i in range(len(df)):
current_price = df['Close'].iloc[i]
signal = df['Signal'].iloc[i]
# Entry logic
if signal == 1 and position == 0:
position = 1
entry_price = current_price
capital_at_entry = current_capital # Snapshot current capital when entering trade
# Exit logic: MA crossover or stop-loss
exit_triggered = False
if position == 1:
# Stop-loss exit
if current_price < entry_price * (1 - stop_loss):
exit_price = entry_price * (1 - stop_loss)
profit = (exit_price - entry_price) / entry_price # percentage return on trade
trades.append({'entry_price': entry_price, 'exit_price': exit_price, 'return': profit})
current_capital *= (1 + profit) # Compound the capital with closed trade profit
position = 0
entry_price = 0
capital_at_entry = 0
exit_triggered = True
# MA crossover exit (only if not already exited by stop-loss)
elif signal == 0 and not exit_triggered:
exit_price = current_price
profit = (exit_price - entry_price) / entry_price # percentage return on trade
trades.append({'entry_price': entry_price, 'exit_price': exit_price, 'return': profit})
current_capital *= (1 + profit) # Compound the capital with closed trade profit
position = 0
entry_price = 0
capital_at_entry = 0
exit_triggered = True
# Calculate and append daily portfolio value
if position == 1:
# If holding a position, current value is compounded capital at entry + unrealized PnL
unrealized_return = (current_price - entry_price) / entry_price
portfolio_values.append(capital_at_entry * (1 + unrealized_return))
else:
# If no position, portfolio value is just the current capital (cash)
portfolio_values.append(current_capital)
# Calculate metrics
df['Portfolio_Value'] = portfolio_values
total_return = (portfolio_values[-1] - initial_capital) / initial_capital
# Sharpe Ratio (assuming 252 trading days per year, 0% risk-free rate)
returns = pd.Series(portfolio_values).pct_change().dropna()
sharpe_ratio = (returns.mean() / returns.std()) * np.sqrt(252) if returns.std() > 0 else 0
# Maximum Drawdown
cumulative_returns = pd.Series(portfolio_values)
running_max = cumulative_returns.expanding().max()
drawdown = (cumulative_returns - running_max) / running_max
max_drawdown = drawdown.min()
# Win Rate
if len(trades) > 0:
winning_trades = len([t for t in trades if t['return'] > 0])
win_rate = winning_trades / len(trades)
avg_win = np.mean([t['return'] for t in trades if t['return'] > 0]) if winning_trades > 0 else 0
avg_loss = np.mean([t['return'] for t in trades if t['return'] < 0]) if (len(trades) - winning_trades) > 0 else 0
profit_factor = sum([t['return'] for t in trades if t['return'] > 0]) / abs(sum([t['return'] for t in trades if t['return'] < 0])) if sum([t['return'] for t in trades if t['return'] < 0]) != 0 else 0
else:
win_rate = 0
avg_win = 0
avg_loss = 0
profit_factor = 0
return {
'total_return': total_return,
'sharpe_ratio': sharpe_ratio,
'max_drawdown': max_drawdown,
'num_trades': len(trades),
'win_rate': win_rate,
'avg_win': avg_win,
'avg_loss': avg_loss,
'profit_factor': profit_factor,
'trades': trades,
'portfolio_values': portfolio_values,
'df': df
}
# Test the strategy with default parameters
print("Testing strategy with default parameters (fast=10, slow=30)...")
result_default = backtest_strategy(df, fast_period=10, slow_period=30)
print(f"\nDefault Parameters Results:")
print(f"Total Return: {result_default['total_return']:.2%}")
print(f"Sharpe Ratio: {result_default['sharpe_ratio']:.4f}")
print(f"Max Drawdown: {result_default['max_drawdown']:.2%}")
print(f"Number of Trades: {result_default['num_trades']}")
print(f"Win Rate: {result_default['win_rate']:.2%}")
print(f"Profit Factor: {result_default['profit_factor']:.2f}")Testing strategy with default parameters (fast=10, slow=30)... Default Parameters Results: Total Return: 96.25% Sharpe Ratio: 1.0924 Max Drawdown: -14.62% Number of Trades: 30 Win Rate: 40.00% Profit Factor: 3.55
Part 3: In-Sample vs Out-of-Sample Backtesting
We'll split the data:
- In-Sample: First 70% of data (used for optimization)
- Out-of-Sample: Last 30% of data (used to test if strategy generalizes)
We'll then optimize parameters on in-sample data and test them on out-of-sample data.
[27]
# Split data into in-sample and out-of-sample
split_idx = int(len(df) * 0.7)
df_in_sample = df.iloc[:split_idx].copy()
df_out_sample = df.iloc[split_idx:].copy()
print(f"Total data points: {len(df)}")
print(f"In-sample data points: {len(df_in_sample)} ({len(df_in_sample)/len(df)*100:.1f}%)")
print(f"Out-of-sample data points: {len(df_out_sample)} ({len(df_out_sample)/len(df)*100:.1f}%)")
print(f"\nIn-sample period: {df_in_sample.index[0].date()} to {df_in_sample.index[-1].date()}")
print(f"Out-of-sample period: {df_out_sample.index[0].date()} to {df_out_sample.index[-1].date()}")Total data points: 1000 In-sample data points: 700 (70.0%) Out-of-sample data points: 300 (30.0%) In-sample period: 2021-01-01 to 2022-12-01 Out-of-sample period: 2022-12-02 to 2023-09-27
Part 4: Parameter Optimization on In-Sample Data
We'll test different combinations of fast and slow MA periods to find the best performing parameters based on Sharpe Ratio. This is where overfitting happens - we're curve-fitting to historical data.
[28]
# Optimize parameters on in-sample data
fast_periods = range(5, 25, 2) # 5, 7, 9, 11, ..., 23
slow_periods = range(25, 55, 5) # 25, 30, 35, 40, 45, 50
optimization_results = []
for fast in fast_periods:
for slow in slow_periods:
if fast < slow: # Ensure fast MA is shorter than slow MA
result = backtest_strategy(df_in_sample, fast_period=fast, slow_period=slow)
optimization_results.append({
'fast_period': fast,
'slow_period': slow,
'sharpe_ratio': result['sharpe_ratio'],
'total_return': result['total_return'],
'max_drawdown': result['max_drawdown'],
'win_rate': result['win_rate'],
'profit_factor': result['profit_factor'],
'num_trades': result['num_trades']
})
# Convert to DataFrame and sort by Sharpe Ratio
optim_df = pd.DataFrame(optimization_results).sort_values('sharpe_ratio', ascending=False)
print("Top 10 Parameter Combinations (Optimized on In-Sample Data):")
print(optim_df.head(10).to_string())
# Get best parameters
best_fast = optim_df.iloc[0]['fast_period']
best_slow = optim_df.iloc[0]['slow_period']
best_sharpe_in_sample = optim_df.iloc[0]['sharpe_ratio']
print(f"\n✓ Optimal Parameters Found:")
print(f" Fast MA Period: {int(best_fast)}")
print(f" Slow MA Period: {int(best_slow)}")
print(f" In-Sample Sharpe Ratio: {best_sharpe_in_sample:.4f}")Top 10 Parameter Combinations (Optimized on In-Sample Data):
fast_period slow_period sharpe_ratio total_return max_drawdown win_rate profit_factor num_trades
18 11 25 0.608595 0.267300 -0.120491 0.476190 2.201745 21
8 7 35 0.599895 0.268625 -0.164280 0.400000 2.315334 20
12 9 25 0.592129 0.258091 -0.151806 0.363636 2.109605 22
3 5 40 0.575023 0.252684 -0.202738 0.350000 2.310852 20
24 13 25 0.547718 0.232831 -0.150758 0.400000 1.859695 25
6 7 25 0.532821 0.230580 -0.151708 0.347826 1.870026 23
7 7 30 0.492545 0.202142 -0.156667 0.391304 1.858277 23
1 5 30 0.463911 0.184051 -0.217192 0.363636 1.768934 22
59 23 50 0.451446 0.186138 -0.167313 0.350000 1.784366 20
15 9 40 0.429520 0.171194 -0.164761 0.409091 1.638251 22
✓ Optimal Parameters Found:
Fast MA Period: 11
Slow MA Period: 25
In-Sample Sharpe Ratio: 0.6086
Part 5: Test Optimized Parameters on Out-of-Sample Data
Now we apply the best parameters found on in-sample data to the out-of-sample data. If the strategy is overfit, we'll see significant performance degradation.
[29]
# Test best parameters on out-of-sample data
result_in_sample = backtest_strategy(df_in_sample, fast_period=int(best_fast), slow_period=int(best_slow))
result_out_sample = backtest_strategy(df_out_sample, fast_period=int(best_fast), slow_period=int(best_slow))
# Also test some other parameter combinations for comparison
comparison_results = []
for _, row in optim_df.head(5).iterrows():
fast = int(row['fast_period'])
slow = int(row['slow_period'])
is_result = backtest_strategy(df_in_sample, fast_period=fast, slow_period=slow)
oos_result = backtest_strategy(df_out_sample, fast_period=fast, slow_period=slow)
comparison_results.append({
'Fast': fast,
'Slow': slow,
'In-Sample Sharpe': is_result['sharpe_ratio'],
'Out-of-Sample Sharpe': oos_result['sharpe_ratio'],
'Sharpe Degradation': is_result['sharpe_ratio'] - oos_result['sharpe_ratio'],
'In-Sample Return': is_result['total_return'],
'Out-of-Sample Return': oos_result['total_return'],
'Return Degradation': is_result['total_return'] - oos_result['total_return'],
'In-Sample Max DD': is_result['max_drawdown'],
'Out-of-Sample Max DD': oos_result['max_drawdown']
})
comp_df = pd.DataFrame(comparison_results)
print("\n" + "="*120)
print("IN-SAMPLE vs OUT-OF-SAMPLE COMPARISON (Top 5 Parameter Combinations)")
print("="*120)
print(comp_df.to_string())
print("\n")======================================================================================================================== IN-SAMPLE vs OUT-OF-SAMPLE COMPARISON (Top 5 Parameter Combinations) ======================================================================================================================== Fast Slow In-Sample Sharpe Out-of-Sample Sharpe Sharpe Degradation In-Sample Return Out-of-Sample Return Return Degradation In-Sample Max DD Out-of-Sample Max DD 0 11 25 0.608595 2.040435 -1.431840 0.267300 0.502799 -0.235499 -0.120491 -0.097772 1 7 35 0.599895 2.442045 -1.842150 0.268625 0.662518 -0.393893 -0.164280 -0.097772 2 9 25 0.592129 2.058154 -1.466025 0.258091 0.508850 -0.250759 -0.151806 -0.097772 3 5 40 0.575023 2.302195 -1.727172 0.252684 0.614112 -0.361428 -0.202738 -0.097772 4 13 25 0.547718 2.149035 -1.601317 0.232831 0.541714 -0.308883 -0.150758 -0.088758
[30]
print("\n" + "="*100)
print("DETAILED METRICS: BEST STRATEGY (Fast={}, Slow={})".format(int(best_fast), int(best_slow)))
print("="*100)
metrics = [
('Total Return', lambda r: f"{r['total_return']:.2%}"),
('Sharpe Ratio', lambda r: f"{r['sharpe_ratio']:.4f}"),
('Max Drawdown', lambda r: f"{r['max_drawdown']:.2%}"),
('Number of Trades', lambda r: f"{r['num_trades']}"),
('Win Rate', lambda r: f"{r['win_rate']:.2%}"),
('Avg Win', lambda r: f"{r['avg_win']:.2%}"),
('Avg Loss', lambda r: f"{r['avg_loss']:.2%}"),
('Profit Factor', lambda r: f"{r['profit_factor']:.2f}")
]
print(f"\n{'Metric':<25} {'In-Sample':<20} {'Out-of-Sample':<20} {'Degradation':<15}")
print("-" * 80)
for metric_name, formatter in metrics:
is_val = formatter(result_in_sample)
oos_val = formatter(result_out_sample)
# Calculate degradation
if metric_name == 'Total Return':
deg = (result_in_sample['total_return'] - result_out_sample['total_return']) / result_in_sample['total_return'] if result_in_sample['total_return'] != 0 else 0
elif metric_name == 'Sharpe Ratio':
deg = result_in_sample['sharpe_ratio'] - result_out_sample['sharpe_ratio']
elif metric_name == 'Max Drawdown':
deg = result_in_sample['max_drawdown'] - result_out_sample['max_drawdown']
else:
deg = np.nan
deg_str = f"{deg:.2%}" if not np.isnan(deg) and metric_name in ['Total Return', 'Max Drawdown'] else \
f"{deg:.4f}" if not np.isnan(deg) and metric_name == 'Sharpe Ratio' else "-"
print(f"{metric_name:<25} {is_val:<20} {oos_val:<20} {deg_str:<15}")
print("\n")
print(" OVERFITTING INDICATORS:")
sharpe_degradation = (result_in_sample['sharpe_ratio'] - result_out_sample['sharpe_ratio']) / result_in_sample['sharpe_ratio'] if result_in_sample['sharpe_ratio'] != 0 else 0
return_degradation = (result_in_sample['total_return'] - result_out_sample['total_return']) / result_in_sample['total_return'] if result_in_sample['total_return'] != 0 else 0
print(f" • Sharpe Ratio degradation: {sharpe_degradation:.1%}")
if sharpe_degradation > 0.5:
print(f" SEVERE OVERFITTING DETECTED (degradation > 50%)")
elif sharpe_degradation > 0.3:
print(f" MODERATE OVERFITTING DETECTED (degradation > 30%)")
elif sharpe_degradation > 0.1:
print(f" MILD OVERFITTING DETECTED (degradation > 10%)")
else:
print(f" Acceptable overfitting level")
print(f"\n • Return degradation: {return_degradation:.1%}")
if return_degradation < 0:
print(f" Strategy actually improved out-of-sample (lucky!)")
elif return_degradation < 0.3:
print(f" Acceptable degradation")
else:
print(f" SIGNIFICANT PERFORMANCE DROP")
====================================================================================================
DETAILED METRICS: BEST STRATEGY (Fast=11, Slow=25)
====================================================================================================
Metric In-Sample Out-of-Sample Degradation
--------------------------------------------------------------------------------
Total Return 26.73% 50.28% -88.10%
Sharpe Ratio 0.6086 2.0404 -1.4318
Max Drawdown -12.05% -9.78% -2.27%
Number of Trades 21 9 -
Win Rate 47.62% 44.44% -
Avg Win 4.49% 14.17% -
Avg Loss -1.85% -1.68% -
Profit Factor 2.20 6.73 -
OVERFITTING INDICATORS:
• Sharpe Ratio degradation: -235.3%
Acceptable overfitting level
• Return degradation: -88.1%
Strategy actually improved out-of-sample (lucky!)
Part 6: Visualizations of Overfitting
[31]
# Visualization 1: Portfolio Value Over Time
fig, axes = plt.subplots(2, 1, figsize=(14, 10))
# In-sample
ax = axes[0]
ax.plot(df_in_sample.index, result_in_sample['portfolio_values'], label='Portfolio Value', linewidth=2, color='green')
ax.fill_between(df_in_sample.index, result_in_sample['portfolio_values'], alpha=0.2, color='green')
ax.set_title(f'In-Sample Performance (Fast MA={int(best_fast)}, Slow MA={int(best_slow)}) - Sharpe: {result_in_sample["sharpe_ratio"]:.4f}', fontweight='bold')
ax.set_ylabel('Portfolio Value ($)')
ax.legend(loc='upper left')
ax.grid(True, alpha=0.3)
# Out-of-sample
ax = axes[1]
ax.plot(df_out_sample.index, result_out_sample['portfolio_values'], label='Portfolio Value', linewidth=2, color='red')
ax.fill_between(df_out_sample.index, result_out_sample['portfolio_values'], alpha=0.2, color='red')
ax.set_title(f'Out-of-Sample Performance (Same Parameters) - Sharpe: {result_out_sample["sharpe_ratio"]:.4f}', fontweight='bold')
ax.set_ylabel('Portfolio Value ($)')
ax.set_xlabel('Date')
ax.legend(loc='upper left')
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
[32]
# Visualization 2: Sharpe Ratio Comparison
fig, ax = plt.subplots(figsize=(12, 6))
x_pos = np.arange(len(comp_df))
width = 0.35
bars1 = ax.bar(x_pos - width/2, comp_df['In-Sample Sharpe'], width, label='In-Sample', color='green', alpha=0.7)
bars2 = ax.bar(x_pos + width/2, comp_df['Out-of-Sample Sharpe'], width, label='Out-of-Sample', color='red', alpha=0.7)
ax.set_xlabel('Parameter Combination', fontweight='bold')
ax.set_ylabel('Sharpe Ratio', fontweight='bold')
ax.set_title('Sharpe Ratio: In-Sample vs Out-of-Sample (Top 5 Strategies)', fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels([f"Fast={int(row['Fast'])}, Slow={int(row['Slow'])}" for _, row in comp_df.iterrows()], rotation=45)
ax.legend()
ax.grid(True, alpha=0.3, axis='y')
# Add value labels on bars
for bars in [bars1, bars2]:
for bar in bars:
height = bar.get_height()
ax.text(bar.get_x() + bar.get_width()/2., height,
f'{height:.3f}', ha='center', va='bottom', fontsize=8)
plt.tight_layout()
plt.show()
[33]
# Visualization 3: Return Comparison
fig, ax = plt.subplots(figsize=(12, 6))
x_pos = np.arange(len(comp_df))
width = 0.35
bars1 = ax.bar(x_pos - width/2, comp_df['In-Sample Return'] * 100, width, label='In-Sample', color='blue', alpha=0.7)
bars2 = ax.bar(x_pos + width/2, comp_df['Out-of-Sample Return'] * 100, width, label='Out-of-Sample', color='orange', alpha=0.7)
ax.set_xlabel('Parameter Combination', fontweight='bold')
ax.set_ylabel('Total Return (%)', fontweight='bold')
ax.set_title('Total Return: In-Sample vs Out-of-Sample (Top 5 Strategies)', fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels([f"Fast={int(row['Fast'])}, Slow={int(row['Slow'])}" for _, row in comp_df.iterrows()], rotation=45)
ax.legend()
ax.grid(True, alpha=0.3, axis='y')
ax.axhline(y=0, color='black', linestyle='-', linewidth=0.5)
plt.tight_layout()
plt.show()
[34]
# Visualization 4: Degradation Analysis
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
# Sharpe Degradation
ax = axes[0, 0]
x_pos = np.arange(len(comp_df))
colors = ['red' if x > 0.3 else 'orange' if x > 0.1 else 'green' for x in comp_df['Sharpe Degradation']]
ax.bar(x_pos, comp_df['Sharpe Degradation'], color=colors, alpha=0.7)
ax.set_ylabel('Degradation (Points)', fontweight='bold')
ax.set_title('Sharpe Ratio Degradation', fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels([f"Fast={int(row['Fast'])}, Slow={int(row['Slow'])}" for _, row in comp_df.iterrows()], rotation=45)
ax.axhline(y=0.3, color='red', linestyle='--', linewidth=1, label='High Overfitting Threshold (0.3)')
ax.axhline(y=0.1, color='orange', linestyle='--', linewidth=1, label='Moderate Threshold (0.1)')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3, axis='y')
# Return Degradation
ax = axes[0, 1]
colors = ['red' if x > 0.3 else 'orange' if x > 0.1 else 'green' for x in comp_df['Return Degradation']]
ax.bar(x_pos, comp_df['Return Degradation'] * 100, color=colors, alpha=0.7)
ax.set_ylabel('Degradation (%)', fontweight='bold')
ax.set_title('Total Return Degradation', fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels([f"Fast={int(row['Fast'])}, Slow={int(row['Slow'])}" for _, row in comp_df.iterrows()], rotation=45)
ax.grid(True, alpha=0.3, axis='y')
# Max Drawdown Comparison
ax = axes[1, 0]
width = 0.35
ax.bar(x_pos - width/2, comp_df['In-Sample Max DD'] * 100, width, label='In-Sample', alpha=0.7)
ax.bar(x_pos + width/2, comp_df['Out-of-Sample Max DD'] * 100, width, label='Out-of-Sample', alpha=0.7)
ax.set_ylabel('Max Drawdown (%)', fontweight='bold')
ax.set_title('Maximum Drawdown Comparison', fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels([f"Fast={int(row['Fast'])}, Slow={int(row['Slow'])}" for _, row in comp_df.iterrows()], rotation=45)
ax.legend()
ax.grid(True, alpha=0.3, axis='y')
# Overfitting Heatmap
ax = axes[1, 1]
degradation_matrix = optim_df.pivot_table(values='sharpe_ratio', index='fast_period', columns='slow_period')
im = ax.imshow(degradation_matrix.values, cmap='RdYlGn', aspect='auto')
ax.set_xticks(range(len(degradation_matrix.columns)))
ax.set_yticks(range(len(degradation_matrix.index)))
ax.set_xticklabels(degradation_matrix.columns.astype(int))
ax.set_yticklabels(degradation_matrix.index.astype(int))
ax.set_xlabel('Slow MA Period', fontweight='bold')
ax.set_ylabel('Fast MA Period', fontweight='bold')
ax.set_title('In-Sample Sharpe Ratio Heatmap\n(Showing Parameter Sensitivity)', fontweight='bold')
plt.colorbar(im, ax=ax, label='Sharpe Ratio')
plt.tight_layout()
plt.show()
Part 7: Walk-Forward Analysis
A more robust way to detect overfitting is walk-forward analysis, where we:
- Use a rolling window of data
- Optimize on the in-sample portion
- Test on the immediate out-of-sample portion
- Move the window forward and repeat
- Compare results across windows
If performance is stable across windows, the strategy is robust. If it varies widely, it's overfit.
[35]
def walk_forward_analysis(data, window_size=200, step_size=50, n_windows=5):
"""
Perform walk-forward analysis.
"""
results = []
for window_idx in range(n_windows):
start_idx = window_idx * step_size
in_sample_end = start_idx + window_size
out_sample_end = min(in_sample_end + window_size // 2, len(data))
if out_sample_end >= len(data):
break
# Get in-sample and out-of-sample windows
is_window = data.iloc[start_idx:in_sample_end]
oos_window = data.iloc[in_sample_end:out_sample_end]
# Find best parameters on in-sample
best_params = None
best_sharpe = -np.inf
for fast in range(5, 20, 3):
for slow in range(25, 50, 5):
if fast < slow:
result = backtest_strategy(is_window, fast_period=fast, slow_period=slow)
if result['sharpe_ratio'] > best_sharpe:
best_sharpe = result['sharpe_ratio']
best_params = (fast, slow)
# Test on out-of-sample
if best_params:
oos_result = backtest_strategy(oos_window, fast_period=best_params[0], slow_period=best_params[1])
results.append({
'window': window_idx + 1,
'fast_period': best_params[0],
'slow_period': best_params[1],
'in_sample_sharpe': best_sharpe,
'out_sample_sharpe': oos_result['sharpe_ratio'],
'degradation': best_sharpe - oos_result['sharpe_ratio'],
'in_sample_return': best_sharpe, # Use Sharpe for comparison
'out_sample_return': oos_result['total_return']
})
return pd.DataFrame(results)
# Perform walk-forward analysis
print("Performing Walk-Forward Analysis...\n")
wf_results = walk_forward_analysis(df, window_size=200, step_size=50, n_windows=6)
print(wf_results.to_string())
print(f"\n\nWalk-Forward Analysis Summary:")
print(f"Average In-Sample Sharpe: {wf_results['in_sample_sharpe'].mean():.4f}")
print(f"Average Out-of-Sample Sharpe: {wf_results['out_sample_sharpe'].mean():.4f}")
print(f"Average Degradation: {wf_results['degradation'].mean():.4f}")
print(f"Degradation Std Dev: {wf_results['degradation'].std():.4f}")
if wf_results['degradation'].std() > 0.1:
print(f"\n HIGH INSTABILITY: Parameter performance varies significantly across windows")
print(f" This indicates OVERFITTING - parameters are not robust.")
else:
print(f"\n✓ STABLE PERFORMANCE: Parameter performance is consistent across windows")Performing Walk-Forward Analysis...
window fast_period slow_period in_sample_sharpe out_sample_sharpe degradation in_sample_return out_sample_return
0 1 11 25 0.835887 -0.615322 1.451210 0.835887 -0.028533
1 2 5 40 1.079565 1.510886 -0.431321 1.079565 0.077617
2 3 5 40 1.250415 1.265026 -0.014611 1.250415 0.075934
3 4 5 25 1.150208 0.112635 1.037572 1.150208 0.001388
4 5 5 30 1.585718 -2.294282 3.880000 1.585718 -0.103053
5 6 5 30 1.601367 -2.249767 3.851134 1.601367 -0.081099
Walk-Forward Analysis Summary:
Average In-Sample Sharpe: 1.2505
Average Out-of-Sample Sharpe: -0.3785
Average Degradation: 1.6290
Degradation Std Dev: 1.8619
HIGH INSTABILITY: Parameter performance varies significantly across windows
This indicates OVERFITTING - parameters are not robust.
[36]
# Visualization: Walk-Forward Results
fig, axes = plt.subplots(2, 1, figsize=(12, 8))
# Sharpe Ratio Trend
ax = axes[0]
ax.plot(wf_results['window'], wf_results['in_sample_sharpe'], marker='o', label='In-Sample Sharpe', linewidth=2, color='green')
ax.plot(wf_results['window'], wf_results['out_sample_sharpe'], marker='s', label='Out-of-Sample Sharpe', linewidth=2, color='red')
ax.fill_between(wf_results['window'], wf_results['in_sample_sharpe'], wf_results['out_sample_sharpe'], alpha=0.2, color='gray')
ax.set_ylabel('Sharpe Ratio', fontweight='bold')
ax.set_title('Walk-Forward Analysis: Sharpe Ratio Trends', fontweight='bold')
ax.legend()
ax.grid(True, alpha=0.3)
# Degradation Trend
ax = axes[1]
colors = ['red' if x > 0.3 else 'orange' if x > 0.1 else 'green' for x in wf_results['degradation']]
ax.bar(wf_results['window'], wf_results['degradation'], color=colors, alpha=0.7)
ax.axhline(y=wf_results['degradation'].mean(), color='black', linestyle='--', linewidth=2, label=f"Mean: {wf_results['degradation'].mean():.4f}")
ax.set_xlabel('Walk-Forward Window', fontweight='bold')
ax.set_ylabel('Sharpe Degradation', fontweight='bold')
ax.set_title('Walk-Forward Analysis: Degradation per Window', fontweight='bold')
ax.set_xticks(wf_results['window'])
ax.legend()
ax.grid(True, alpha=0.3, axis='y')
plt.tight_layout()
plt.show()
Conclusions: Detecting Backtesting Overfitting
Key Findings
From this analysis, we've demonstrated several critical aspects of backtesting overfitting:
1. In-Sample vs Out-of-Sample Performance Gap
The most obvious sign of overfitting is a large gap between in-sample and out-of-sample performance. Even with the same parameters, performance typically degrades significantly when tested on new data.
2. Parameter Sensitivity
When you optimize many different parameter combinations and find a "best" set, these parameters are likely overfit to the specific characteristics of the in-sample data. Their success may not generalize.
3. Consistent Degradation Pattern
If the degradation pattern is consistent across multiple walk-forward windows, it suggests the overfitting is systematic rather than random.
4. Multiple Metrics Tell the Story
Don't rely on a single metric (like Sharpe Ratio). Look at:
- Sharpe Ratio: Risk-adjusted returns
- Total Return: Absolute performance
- Maximum Drawdown: Risk during worst period
- Win Rate: Percentage of winning trades
- Profit Factor: Ratio of profits to losses
Red Flags for Overfitting
Severe (> 50% degradation in key metrics)
- Strategy has been heavily optimized to noise
- Likely to fail in live trading
- Simplify strategy or get more data
Moderate (20-50% degradation)
- Significant curve-fitting has occurred
- Use with caution; implement robust position sizing
- Monitor performance closely
Mild (< 20% degradation)
- Some overfitting present but acceptable
- Within typical optimization bounds
- Still requires real-world validation
How to Combat Backtesting Overfitting
- Use Out-of-Sample Testing: Always reserve data for testing
- Walk-Forward Analysis: Optimize on rolling windows to ensure robustness
- Parameter Stability: Check if similar parameters perform well across different periods
- Reduce Complexity: Simpler strategies generalize better
- Robust Metrics: Focus on Sharpe Ratio and drawdown, not just returns
- Multiple Asset Classes: Test on different instruments
- Account for Costs: Include slippage, commissions, and market impact
- Monte Carlo Simulation: Test parameter combinations randomly to see distribution
- Start Small: Use small position sizes when deploying new strategies
- Continuous Monitoring: Real-world results will reveal overfitting