Enter Platform
Resources
Back to Notebooks
Open in Colab
Research·Strategy Hypothesis Testing·Advanced

Ab Test Strategy Variants

Design and execute statistically rigorous controlled A and B tests comparing two trading strategy variants with proper sample size and test duration determination, formal statistical significance hypothesis testing, and detailed analysis of performance differences and their drivers.

quant-researchrisk-controlstrading-strategies

A/B Test Strategy Variants — Research & Experimentation

Category: Research & Experimentation | Subcategory: Hypothesis

What This Notebook Does

A/B testing rigorously compares two strategy variants (A and B) to determine whether the difference in performance is statistically significant or just random chance. It prevents the most common research mistake: choosing variant B over A because B happened to perform better on this specific data, when the difference could easily be noise.

The challenge with financial time series A/B tests:

  • Returns are autocorrelated → standard t-test p-values are incorrect
  • Sample sizes are small (limited history) → need non-parametric methods
  • The effect size that matters is economic, not just statistical

This notebook uses:

  • Permutation test: non-parametric, correct for autocorrelated series
  • Block bootstrap: preserves time-series structure when resampling
  • Paired t-test: as a benchmark (assumes independence, often violated)

This notebook:

  1. Defines two competing strategy variants with one parameter changed
  2. Runs both variants on the same data using a simple momentum signal
  3. Tests whether the performance difference is significant using all three methods
  4. Reports the effect size with uncertainty quantification
  5. Recommends variant A or B based on the combined evidence
  6. Exports test results
[ ]
!pip install numpy pandas matplotlib seaborn scipy yfinance --quiet
[ ]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import warnings

warnings.filterwarnings('ignore')
%matplotlib inline
plt.rcParams['figure.figsize'] = (14, 5)
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.right'] = False
print('Imports ready.')
Imports ready.

Configuration

We test two momentum variants: Variant A uses a 7-day lookback; Variant B uses a 14-day lookback. Everything else is identical. The A/B test answers: does the extra 7 days of lookback statistically improve Sharpe ratio? N_PERMUTATIONS controls the power of the permutation test — 10,000 is sufficient for stable p-values.

[ ]
# ── Data Source Toggle ─────────────────────────────
USE_LIVE_DATA = False
TICKER        = 'BTC-USD'
START_DATE    = '2020-01-01'
END_DATE      = '2024-12-31'
# ──────────────────────────────────────────────────
VARIANT_A_LOOKBACK = 7    # short momentum window
VARIANT_B_LOOKBACK = 14   # longer momentum window
HOLD_DAYS          = 3    # both variants hold for 3 days after signal
N_PERMUTATIONS     = 10_000
SIGNIFICANCE       = 0.05
print('Config ready.')
Config ready.

Data Acquisition

We use a single price series for both variants — this is crucial for A/B testing: both strategies must be evaluated on the same data to ensure any performance difference is due to the parameter change, not data differences. The synthetic data includes a moderate autocorrelation so the momentum signal has some predictive power.

[ ]
if USE_LIVE_DATA:
    import yfinance as yf
    prices = yf.download(TICKER, start=START_DATE, end=END_DATE)['Close']
    print(f'Live data: {len(prices)} days')
else:
    rng = np.random.default_rng(42)
    n   = 1200
    raw = rng.standard_t(df=4, size=n) * 0.022
    rets = np.zeros(n)
    for i in range(1, n):
        rets[i] = 0.10 * rets[i-1] + raw[i]
    prices = pd.Series((1 + rets).cumprod() * 30000,
                        index=pd.date_range('2020-01-01', periods=n, freq='B'))
    print(f'Synthetic data: {n} days')

daily_rets = prices.pct_change().dropna()
print(f'Using same price series for both variants — fair A/B comparison')
Synthetic data: 1200 days
Using same price series for both variants — fair A/B comparison

Strategy Simulation

Both variants use an identical mechanism: compute the past N-day return as a signal; go long if positive, flat if negative. The only difference is the lookback window. We collect the daily return time series from each variant — this paired series is then used for all statistical tests.

[ ]
def momentum_strategy(prices: pd.Series, lookback: int, hold: int) -> pd.Series:
    """
    Simple momentum strategy: go long for `hold` days when past `lookback`-day return is positive.

    Parameters
    ----------
    prices   : pd.Series  Daily price series.
    lookback : int        Signal window in days.
    hold     : int        Number of days to hold after signal.

    Returns
    -------
    pd.Series  Daily strategy returns.
    """
    daily = prices.pct_change()
    signal = prices.pct_change(lookback).shift(1)  # no look-ahead
    position = (signal > 0).astype(float)
    return (position * daily).dropna()


rets_a = momentum_strategy(prices, VARIANT_A_LOOKBACK, HOLD_DAYS)
rets_b = momentum_strategy(prices, VARIANT_B_LOOKBACK, HOLD_DAYS)

# Align both series on the same dates
common = rets_a.index.intersection(rets_b.index)
rets_a = rets_a[common]
rets_b = rets_b[common]

sharpe_a = rets_a.mean() / (rets_a.std() + 1e-9) * np.sqrt(252)
sharpe_b = rets_b.mean() / (rets_b.std() + 1e-9) * np.sqrt(252)
print(f'Variant A (lookback={VARIANT_A_LOOKBACK}d): Sharpe = {sharpe_a:.3f}')
print(f'Variant B (lookback={VARIANT_B_LOOKBACK}d): Sharpe = {sharpe_b:.3f}')
print(f'Observed difference (B-A): {sharpe_b - sharpe_a:+.3f}')
Variant A (lookback=7d): Sharpe = -0.176
Variant B (lookback=14d): Sharpe = -0.004
Observed difference (B-A): +0.171

Statistical Tests

Three tests are applied to the paired daily return differences (rets_b - rets_a):

  1. Paired t-test: tests if mean difference ≠ 0. Assumes independence — likely violated but provides a reference
  2. Permutation test: randomly shuffles labels (A/B) many times to build a null distribution. No distributional assumption. The observed test statistic is compared to this null.
  3. Block bootstrap t-test: resamples blocks of consecutive days to preserve autocorrelation, then tests if the bootstrapped CI for the Sharpe difference excludes zero.
[ ]
diff = rets_b.values - rets_a.values
obs_stat = sharpe_b - sharpe_a

# 1. Paired t-test
t_stat, t_pval = stats.ttest_1samp(diff, 0)

# 2. Permutation test
rng_perm = np.random.default_rng(42)
perm_diffs = []
for _ in range(N_PERMUTATIONS):
    signs = rng_perm.choice([-1, 1], size=len(diff))
    perm_diff = diff * signs
    perm_sr = perm_diff.mean() / (perm_diff.std() + 1e-9) * np.sqrt(252)
    perm_diffs.append(perm_sr)
perm_pval = np.mean(np.abs(perm_diffs) >= abs(obs_stat))

# 3. Block bootstrap
block_size = 20
n_obs = len(diff)
boot_stats = []
for _ in range(N_PERMUTATIONS // 5):
    n_blocks = n_obs // block_size + 1
    starts = rng_perm.integers(0, n_obs - block_size, size=n_blocks)
    sample = np.concatenate([diff[s:s+block_size] for s in starts])[:n_obs]
    boot_stats.append(sample.mean() / (sample.std() + 1e-9) * np.sqrt(252))
boot_ci = np.percentile(boot_stats, [2.5, 97.5])

print('=== A/B TEST RESULTS ===')
print(f'Observed Sharpe difference (B-A): {obs_stat:+.4f}')
print(f'Paired t-test:      t={t_stat:.3f}, p={t_pval:.4f}')
print(f'Permutation test:   p={perm_pval:.4f} ({N_PERMUTATIONS} permutations)')
print(f'Block bootstrap CI: [{boot_ci[0]:+.4f}, {boot_ci[1]:+.4f}]')
sig_any = perm_pval < SIGNIFICANCE
print(f'\nConclusion: Variant B is {"SIGNIFICANTLY" if sig_any else "NOT SIGNIFICANTLY"} better than A (permutation p={perm_pval:.3f})')
=== A/B TEST RESULTS ===
Observed Sharpe difference (B-A): +0.1714
Paired t-test:      t=0.496, p=0.6202
Permutation test:   p=0.7085 (10000 permutations)
Block bootstrap CI: [-0.6493, +1.0892]

Conclusion: Variant B is NOT SIGNIFICANTLY better than A (permutation p=0.709)

Visualisation

The left panel shows equity curves for both variants — a clear visual check that variant B consistently outperforms or if the difference is sporadic. The right panel shows the permutation null distribution of Sharpe differences with the observed value marked. If the red line is far into the tail, the result is statistically significant.

[ ]
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
fig.suptitle('A/B Strategy Variant Test', fontsize=13, fontweight='bold')

ax1 = axes[0]
eq_a = (1 + rets_a).cumprod()
eq_b = (1 + rets_b).cumprod()
ax1.plot(eq_a.index, eq_a, color='#9e9e9e', lw=1.5, label=f'Variant A ({VARIANT_A_LOOKBACK}d, SR={sharpe_a:.2f})')
ax1.plot(eq_b.index, eq_b, color='#1976d2', lw=1.5, label=f'Variant B ({VARIANT_B_LOOKBACK}d, SR={sharpe_b:.2f})')
ax1.set_ylabel('Growth of $1'); ax1.legend(fontsize=9)
ax1.set_title('Equity Curves')

ax2 = axes[1]
ax2.hist(perm_diffs, bins=60, color='#9e9e9e', alpha=0.7, density=True,
          label='Null distribution (permuted)')
ax2.axvline(obs_stat, color='#e53935', lw=2, label=f'Observed Δ = {obs_stat:+.3f}')
ax2.axvline(-obs_stat, color='#e53935', lw=1, ls='--', alpha=0.5)
ax2.set_xlabel('Sharpe Difference (B-A)')
ax2.legend(fontsize=8)
ax2.set_title(f'Permutation Test (p={perm_pval:.3f})')

plt.tight_layout(); plt.show()
cell output

Export

Save the paired daily returns for both variants and the test summary. The test summary contains everything needed to reproduce the decision in a research log.

[ ]
pd.DataFrame({'variant_a': rets_a, 'variant_b': rets_b, 'diff': diff}).to_csv(
    'ab_test_strategy_variants.csv')
print('Saved: ab_test_strategy_variants.csv')
Saved: ab_test_strategy_variants.csv

Conclusion

Based on the A/B test, Variant B is not significantly better than Variant A. The permutation test yielded a p-value of 0.709, which is much higher than the significance level of 0.05. This indicates that the observed difference in Sharpe ratio (+0.171) between Variant B and Variant A could easily be due to random chance.

The block bootstrap confidence interval for the Sharpe difference includes zero, further supporting the conclusion that there is no statistically significant improvement from using Variant B over Variant A.