Enter Platform
Resources
Back to Notebooks
Open in Colab
Research·Market Simulation·Advanced

Bootstrapped Backtest

Generate bootstrapped backtest performance metric sampling distributions by block-resampling historical strategy returns, comprehensively quantifying the plausible range of performance outcomes and establishing the statistical significance of observed backtest results beyond single-point estimates.

backtestingquant-researchsimulationstatistical-methods

Understanding Bootstrap Resampled Backtests

Introduction to Bootstrap Resampled Backtests

What is a Backtest?

A backtest is a simulation of a trading strategy using historical data. It helps assess the viability of a trading strategy by determining how it would have performed in the past. While essential, a single backtest run on a fixed historical dataset has limitations, primarily its dependence on that specific historical path.

What is Bootstrap Resampling?

Bootstrap resampling is a statistical technique that involves repeatedly resampling (with replacement) from a single observed sample to create many simulated samples. This allows for the estimation of the sampling distribution of a statistic (e.g., mean, standard deviation, Sharpe ratio) without making strong assumptions about the underlying population distribution.

Why Combine Bootstrap with Backtesting?

Traditional backtesting provides a single performance figure (e.g., Sharpe ratio, cumulative return) based on one historical path. This can be misleading, as the strategy's performance might be highly sensitive to the specific sequence of events in that history.

Bootstrap resampled backtesting addresses this limitation by generating multiple plausible alternative historical paths from the original data. By running the backtest on each resampled path, we obtain a distribution of performance metrics, rather than a single point estimate. This distribution provides valuable insights into:

  1. Robustness: How sensitive is the strategy's performance to different sequences of market events?
  2. Confidence Intervals: Estimating a range within which the true performance metric (e.g., Sharpe Ratio) likely lies.
  3. Risk Assessment: Understanding the variability and potential downside risks of the strategy under different market scenarios.

In essence, it helps to understand not just what the strategy achieved, but how reliably it achieved it, and what other outcomes were plausible given the observed data.

Setup: Imports and Data Generation

Before diving into the backtest, we need to import necessary libraries and create some mock financial time-series data to simulate a strategy's returns. For time-series data, standard bootstrapping (sampling individual observations with replacement) can destroy the temporal dependencies. Therefore, we will focus on block bootstrapping, which preserves some of these dependencies by sampling blocks of data rather than individual points.

[1]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Set a random seed for reproducibility
np.random.seed(42)

Generate Mock Daily Returns Data

We will create a synthetic time series of daily returns for a hypothetical trading strategy. This data will exhibit some characteristics typical of financial returns, such as a slight positive drift and some volatility. We will use a relatively long series to allow for meaningful resampling.

[2]
# Number of days for our mock historical data
num_days = 252 * 5  # 5 years of daily data (approx. 252 trading days per year)

# Generate daily returns
# Assume a mean daily return of 0.05% and a daily standard deviation of 1%
mean_daily_return = 0.0005
std_daily_return = 0.01

daily_returns = np.random.normal(loc=mean_daily_return, scale=std_daily_return, size=num_days)

# Convert to a pandas Series with a DateTime index
dates = pd.date_range(start='2010-01-01', periods=num_days, freq='B') # Business day frequency
df_returns = pd.Series(daily_returns, index=dates, name='Strategy Returns')

print(f"Generated {len(df_returns)} daily returns over {num_days} trading days.")
print("First 5 returns:")
display(df_returns.head())
print("Last 5 returns:")
display(df_returns.tail())

# Visualize the daily returns
plt.figure(figsize=(12, 6))
plt.plot(df_returns.index, df_returns.values, alpha=0.7)
plt.title('Mock Daily Strategy Returns Over Time')
plt.xlabel('Date')
plt.ylabel('Daily Return')
plt.grid(True)
plt.show()
Generated 1260 daily returns over 1260 trading days.
First 5 returns:
Strategy Returns
2010-01-01 0.005467
2010-01-04 -0.000883
2010-01-05 0.006977
2010-01-06 0.015730
2010-01-07 -0.001842 

Last 5 returns:
Strategy Returns
2014-10-24 -0.011596
2014-10-27 0.017226
2014-10-28 0.004690
2014-10-29 -0.006550
2014-10-30 -0.000058

cell output

Baseline Backtest

Before applying bootstrap resampling, let's perform a standard, single backtest on our mock data. This will serve as our baseline performance, which we will later compare against the distribution of performances from the bootstrapped backtests.

We will define a function to calculate common performance metrics:

  • Cumulative Returns: The total profit or loss from the initial investment.
  • Sharpe Ratio: A measure of risk-adjusted return, calculated as the excess return over the risk-free rate divided by the standard deviation of returns. (For simplicity, we'll assume a risk-free rate of 0 for this example).
  • Maximum Drawdown: The largest peak-to-trough decline in cumulative returns during a specific period. It's a measure of downside risk.
  • Annualized Return: The average return earned by an investment over a year.
  • Annualized Volatility: The standard deviation of the investment's returns over a year.
[3]
def calculate_backtest_metrics(returns: pd.Series, risk_free_rate: float = 0.0) -> dict:
    """
    Calculates common backtest performance metrics.

    Inputs:
    - returns (pd.Series): A series of daily returns.
    - risk_free_rate (float): The annual risk-free rate (default 0.0).

    Outputs:
    - dict: A dictionary containing the calculated metrics.

    Formulas used:
    - Cumulative Returns: (1 + r_1) * (1 + r_2) * ... * (1 + r_n) - 1
    - Sharpe Ratio: (Annualized Return - Annualized Risk-Free Rate) / Annualized Volatility
      (Here, we'll use daily returns to calculate daily Sharpe and then annualize)
    - Max Drawdown: max(peak_cumulative_return - cumulative_return) / peak_cumulative_return
    - Annualized Return: (1 + mean_daily_return)^252 - 1
    - Annualized Volatility: std_daily_return * sqrt(252)
    """
    if returns.empty:
        return {
            'Cumulative Returns': np.nan,
            'Annualized Return': np.nan,
            'Annualized Volatility': np.nan,
            'Sharpe Ratio': np.nan,
            'Max Drawdown': np.nan
        }

    # Cumulative Returns
    cumulative_returns = (1 + returns).cumprod() - 1

    # Annualized Return
    total_days = len(returns)
    trading_days_per_year = 252
    annualized_return = (1 + returns.mean()) ** trading_days_per_year - 1

    # Annualized Volatility
    annualized_volatility = returns.std() * np.sqrt(trading_days_per_year)

    # Sharpe Ratio (annualized)
    if annualized_volatility == 0:
        sharpe_ratio = np.nan
    else:
        sharpe_ratio = (annualized_return - risk_free_rate) / annualized_volatility

    # Maximum Drawdown
    if not cumulative_returns.empty:
        peak = cumulative_returns.expanding(min_periods=1).max()
        drawdown = (cumulative_returns - peak) / (1 + peak) # Note: Adding 1 to peak for correct percentage
        max_drawdown = drawdown.min() if not drawdown.empty else 0
    else:
        max_drawdown = 0.0

    return {
        'Cumulative Returns': cumulative_returns.iloc[-1] if not cumulative_returns.empty else np.nan,
        'Annualized Return': annualized_return,
        'Annualized Volatility': annualized_volatility,
        'Sharpe Ratio': sharpe_ratio,
        'Max Drawdown': max_drawdown
    }
[4]
print("### Running Baseline Backtest ###")
baseline_metrics = calculate_backtest_metrics(df_returns)

for metric, value in baseline_metrics.items():
    if isinstance(value, (float, np.float64)) and ('Returns' in metric or 'Drawdown' in metric or 'Ratio' in metric):
        print(f"{metric}: {value:.4f}")
    else:
        print(f"{metric}: {value:.4f}")
### Running Baseline Backtest ###
Cumulative Returns: 1.8528
Annualized Return: 0.2485
Annualized Volatility: 0.1571
Sharpe Ratio: 1.5822
Max Drawdown: -0.1818

Interpretation of Baseline Backtest

The baseline backtest provides a single set of performance metrics for our strategy based on the exact historical sequence of returns. For our mock data:

  • Cumulative Returns: The strategy achieved a total return over the 5-year period. A positive value indicates profitability.
  • Annualized Return: The average yearly return the strategy generated.
  • Annualized Volatility: A measure of the strategy's risk (fluctuation) on an annual basis.
  • Sharpe Ratio: This key metric indicates the return generated per unit of risk taken. A higher Sharpe ratio is generally better.
  • Max Drawdown: Represents the largest percentage drop from a peak in the equity curve to a subsequent trough. It's a crucial measure of capital preservation.

While these numbers are useful, they don't tell us how likely it is to achieve similar results in another similar 5-year period, or how sensitive these results are to slight variations in the return sequence. This is where bootstrap resampling comes in.

Visualization of Baseline Cumulative Returns

Let's plot the cumulative returns of our baseline strategy. This 'equity curve' visually represents the growth of an initial investment over time.

[5]
cumulative_returns_baseline = (1 + df_returns).cumprod() - 1

plt.figure(figsize=(12, 6))
plt.plot(cumulative_returns_baseline.index, cumulative_returns_baseline.values, label='Baseline Strategy Cumulative Returns', color='blue')
plt.title('Baseline Strategy Cumulative Returns Over 5 Years')
plt.xlabel('Date')
plt.ylabel('Cumulative Return')
plt.grid(True)
plt.legend()
plt.show()
cell output

Block Bootstrapping for Time Series Data

For time series data, simply resampling individual returns (standard bootstrapping) can break the inherent temporal dependencies (e.g., autocorrelation, volatility clustering) present in financial data. Block bootstrapping addresses this by resampling contiguous blocks of data rather than individual data points.

How Block Bootstrapping Works:

  1. Divide Data into Blocks: The original time series is divided into overlapping or non-overlapping blocks of a certain length.
  2. Sample Blocks: Blocks are sampled randomly with replacement.
  3. Concatenate Blocks: The sampled blocks are concatenated in the order they were drawn to form a new, 'resampled' time series.

By preserving the internal structure of these blocks, block bootstrapping helps maintain some of the original series' temporal characteristics.

Choosing Block Length:

The choice of block length L is crucial. A shorter block length might not capture longer-range dependencies, while a very long block length might make the resampled series too similar to the original, reducing the effectiveness of the bootstrap. There are statistical methods to determine optimal block length, but for this demonstration, we'll choose a reasonable fixed length (e.g., 20 trading days, or approximately one month).

[6]
def block_bootstrap(data: pd.Series, block_length: int, num_samples: int) -> list[pd.Series]:
    """
    Performs block bootstrapping on a time series.

    Inputs:
    - data (pd.Series): The original time series data.
    - block_length (int): The length of each block.
    - num_samples (int): The number of bootstrapped samples to generate.

    Outputs:
    - list[pd.Series]: A list of bootstrapped time series.

    Explanation:
    1. The data is conceptually divided into 'num_blocks_needed' blocks to reconstruct a series of the original length.
    2. A set of start indices for blocks is created (from 0 to len(data) - block_length).
    3. For each of the `num_samples` iterations:
       a. Randomly select `num_blocks_needed` start indices with replacement.
       b. For each selected start index, extract a block of `block_length` from the original data.
       c. Concatenate these blocks to form a new time series.
       d. Trim the new series to the original length to ensure consistency.
    """
    n = len(data)
    if n < block_length:
        raise ValueError("Data length must be greater than or equal to block length.")

    # Calculate the number of blocks needed to reconstruct a series of original length
    num_blocks_needed = int(np.ceil(n / block_length))

    # Generate possible starting indices for blocks
    start_indices = np.arange(n - block_length + 1)

    bootstrapped_series = []
    for _ in range(num_samples):
        # Randomly choose start indices for the blocks (with replacement)
        sampled_block_starts = np.random.choice(start_indices, size=num_blocks_needed, replace=True)

        # Concatenate the chosen blocks
        resampled_data = []
        for start_idx in sampled_block_starts:
            resampled_data.append(data.iloc[start_idx : start_idx + block_length])

        # Combine blocks and trim to original length
        combined_series = pd.concat(resampled_data).iloc[:n]
        # Ensure the index matches the original for consistency in plotting (optional but good practice)
        combined_series.index = data.index
        bootstrapped_series.append(combined_series)

    return bootstrapped_series

Performing Block Bootstrapping on Mock Returns

Now we'll apply the block_bootstrap function to our df_returns to generate multiple resampled return series. Each of these series represents a plausible alternative history for our strategy's performance, preserving some of the original's temporal structure.

[7]
block_length = 20  # Approximately one month of trading days
num_bootstrap_samples = 500 # Number of resampled historical paths

print(f"Generating {num_bootstrap_samples} bootstrapped samples with block length of {block_length} days...")
bootstrapped_returns_samples = block_bootstrap(df_returns, block_length, num_bootstrap_samples)

print(f"Generated {len(bootstrapped_returns_samples)} bootstrapped return series.")
print("First 5 elements of the first bootstrapped sample:")
display(bootstrapped_returns_samples[0].head())
Generating 500 bootstrapped samples with block length of 20 days...
Generated 500 bootstrapped return series.
First 5 elements of the first bootstrapped sample:
Strategy Returns
2010-01-01 -0.009357
2010-01-04 0.005540
2010-01-05 -0.004803
2010-01-06 -0.007429
2010-01-07 -0.000570

Running Backtests on Bootstrapped Samples

With our bootstrapped return series, we can now run our calculate_backtest_metrics function on each of them. This will give us a distribution of performance metrics (e.g., Sharpe Ratios, Max Drawdowns) rather than just a single point estimate. This distribution is key to understanding the robustness and statistical significance of our strategy's performance.

[8]
bootstrap_results = []
bootstrap_cumulative_returns = []

print(f"Running backtests on {num_bootstrap_samples} bootstrapped samples...")
for i, sample_returns in enumerate(bootstrapped_returns_samples):
    metrics = calculate_backtest_metrics(sample_returns)
    bootstrap_results.append(metrics)

    # Also store cumulative returns for visualization
    cumulative = (1 + sample_returns).cumprod() - 1
    bootstrap_cumulative_returns.append(cumulative)

# Convert results to a DataFrame for easier analysis
df_bootstrap_results = pd.DataFrame(bootstrap_results)

print("Done. Displaying first 5 rows of bootstrapped backtest metrics:")
display(df_bootstrap_results.head())
Running backtests on 500 bootstrapped samples...
Done. Displaying first 5 rows of bootstrapped backtest metrics:
Cumulative Returns Annualized Return Annualized Volatility Sharpe Ratio Max Drawdown
0 1.362299 0.201674 0.153760 1.311613 -0.163052
1 1.701437 0.235178 0.158042 1.488073 -0.131259
2 0.988265 0.161864 0.158780 1.019420 -0.172891
3 1.136692 0.177738 0.153463 1.158187 -0.239693
4 2.258753 0.281945 0.155868 1.808872 -0.147418

Visualization and Interpretation of Bootstrapped Results

Now we can analyze the distribution of the performance metrics obtained from our bootstrapped backtests. This will provide a more nuanced view than the single baseline backtest.

Distribution of Sharpe Ratios

A histogram of the Sharpe Ratios from all bootstrapped backtests allows us to see the range and most likely values for this key risk-adjusted performance metric. We'll also plot the baseline Sharpe Ratio for comparison.

[9]
baseline_sharpe = baseline_metrics['Sharpe Ratio']

plt.figure(figsize=(12, 7))
sns.histplot(df_bootstrap_results['Sharpe Ratio'].dropna(), kde=True, color='skyblue', label='Bootstrapped Sharpe Ratios')
plt.axvline(baseline_sharpe, color='red', linestyle='--', label=f'Baseline Sharpe Ratio: {baseline_sharpe:.4f}')

# Calculate and plot confidence interval for Sharpe Ratio
alpha = 0.05 # For 95% confidence interval
lower_bound = df_bootstrap_results['Sharpe Ratio'].quantile(alpha / 2)
upper_bound = df_bootstrap_results['Sharpe Ratio'].quantile(1 - alpha / 2)

plt.axvline(lower_bound, color='green', linestyle=':', label=f'95% CI: [{lower_bound:.4f}, {upper_bound:.4f}]')
plt.axvline(upper_bound, color='green', linestyle=':')

plt.title('Distribution of Bootstrapped Sharpe Ratios')
plt.xlabel('Sharpe Ratio')
plt.ylabel('Frequency')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.7)
plt.show()
cell output

Interpretation of Sharpe Ratio Distribution

  • Shape of the Histogram: Shows the spread of possible Sharpe Ratios. A wide spread indicates high variability in performance.
  • Location of Baseline Sharpe: If the baseline Sharpe Ratio falls within the densest part of the distribution, it suggests the original backtest result is a plausible outcome.
  • Confidence Interval: The green dotted lines represent a 95% confidence interval for the Sharpe Ratio. This range gives us an idea of the true Sharpe Ratio of the strategy, given our observed data. If the entire interval is above zero, it provides stronger evidence that the strategy has a positive risk-adjusted return.

This visualization helps answer: "If I had rerun this strategy on similar, but slightly different, historical data, what range of Sharpe Ratios might I have observed?"

Multiple Bootstrapped Cumulative Return Paths

Plotting several bootstrapped cumulative return paths alongside the baseline helps visualize the range of possible equity curves the strategy could have generated. This provides a dynamic view of performance robustness.

[10]
plt.figure(figsize=(14, 8))

# Plot a subset of bootstrapped paths
num_paths_to_plot = 50
for i in range(min(num_paths_to_plot, len(bootstrap_cumulative_returns))):
    plt.plot(bootstrap_cumulative_returns[i].index, bootstrap_cumulative_returns[i].values, color='grey', alpha=0.2)

# Plot the baseline cumulative returns prominently
plt.plot(cumulative_returns_baseline.index, cumulative_returns_baseline.values, color='red', linestyle='-', linewidth=2, label='Baseline Cumulative Returns')

plt.title('Baseline vs. Multiple Bootstrapped Cumulative Return Paths')
plt.xlabel('Date')
plt.ylabel('Cumulative Return')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.7)
plt.show()
cell output

Interpretation of Multiple Return Paths

  • Spread of Paths: The gray lines show how much the cumulative returns can vary across different resampled histories. A wide spread suggests less robust performance.
  • Consistency: If most bootstrapped paths show similar trends and end-points to the baseline, it indicates a more consistent strategy.
  • Worst-Case Scenarios: The paths also highlight potential worst-case scenarios, giving a visual representation of the downside risk under different market sequences.

This visualization complements the statistical distributions by showing the path-dependent nature of the strategy's performance, which is especially important for financial time series.

Conclusion

Bootstrap resampled backtesting is a powerful technique for assessing the robustness and statistical significance of trading strategies. By moving beyond a single historical backtest, it provides a distribution of potential outcomes, offering a more comprehensive understanding of a strategy's performance characteristics, risk, and reliability.

Key takeaways:

  • A single backtest is limited by its dependence on one historical path.
  • Block bootstrapping helps preserve temporal dependencies in time series data.
  • Analyzing the distribution of bootstrapped performance metrics (like Sharpe Ratio) provides confidence intervals and insights into robustness.
  • Visualizing multiple bootstrapped cumulative return paths illustrates the range of possible equity curves and potential worst-case scenarios.

Incorporating bootstrap resampling into your backtesting methodology can lead to more confident and well-informed investment decisions.

Bootstrapped Backtest · BitPredict