Statistical Analysis·Regime Detection·Intermediate

Trend vs Mean Reversion Regime

Detect whether the market is currently in a trending directional or mean-reverting oscillating regime using a battery of statistical tests including the Hurst exponent, variance ratio test, and return autocorrelation structure analysis across multiple lookback windows.

quant-analysisstatistical-methods

Trend vs Mean-Reversion Regime Detection — Statistical Analysis

Category: Statistical Analysis | Subcategory: Regime

What This Notebook Does

Markets alternate between two fundamental price dynamics:

Regime	Behavior	Best Strategies
Trending	Price persistently moves in one direction	Momentum, breakout, MA crossover
Ranging	Price oscillates around a mean	Mean reversion, grid trading, Bollinger

Key statistical tools to detect the current regime:

Hurst Exponent (H): H > 0.5 → trending; H < 0.5 → mean-reverting; H ≈ 0.5 → random walk
ADX (Average Directional Index): ADX > 25 → trending; ADX < 20 → ranging
Autocorrelation: positive lag-1 → trending; negative → mean-reverting

This notebook:

Computes Hurst Exponent with the R/S method
Calculates ADX and directional indicators
Measures rolling autocorrelation of returns
Combines signals into a composite regime classifier
Backtests regime-switched strategies
Exports regime-labelled data

[8]

!pip install numpy pandas matplotlib seaborn scipy --quiet

[9]

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import warnings

warnings.filterwarnings('ignore')
%matplotlib inline
plt.rcParams['figure.figsize'] = (14, 5)
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.right'] = False
print('Imports ready.')

Imports ready.

Section 1 — Configuration

This section is dedicated to setting up the core configuration parameters that control the behavior of our regime detection model. These parameters are crucial for defining how various indicators, such as the Hurst Exponent, ADX, and rolling autocorrelation, are calculated. Adjusting these values allows for experimentation and optimization of the model's sensitivity and responsiveness to different market conditions. Each parameter is briefly described below:

[10]

HURST_WINDOW  = 100   # rolling window for Hurst estimation
ADX_PERIOD    = 14
AUTOCORR_LAG  = 1
AUTOCORR_WINDOW = 30
SIMULATION_DAYS = 1000
print('Config ready.')

Config ready.

Section 2 — Data

Data Generation

This section defines a function generate_regime_switching_data to create synthetic OHLCV (Open, High, Low, Close, Volume) data. This is crucial for testing our regime detection methods because it allows us to control and explicitly define periods of trending and ranging behavior. By simulating these distinct regimes, we can validate whether our Hurst, ADX, and autocorrelation calculations accurately identify them. The data generation process alternates between an AR(1) process for trending (positive coefficient) and a mean-reverting Ornstein-Uhlenbeck (OU) process for ranging periods.

[11]

def generate_regime_switching_data(n_days: int = 1000, seed: int = 42) -> pd.DataFrame:
    """
    Generate OHLCV with explicit trending and ranging periods.

    Trending: AR(1) process with positive coefficient (H > 0.5)
    Ranging: mean-reverting OU process (H < 0.5)

    Returns
    -------
    pd.DataFrame  OHLCV plus true_regime label.
    """
    rng = np.random.default_rng(seed)
    # Alternating regimes: trend(200) range(200) trend(150) range(300) trend(150)
    regime_seq = ['Trending']*200 + ['Ranging']*200 + ['Trending']*150 + ['Ranging']*300 + ['Trending']*150
    regime_seq = regime_seq[:n_days]

    price = 30_000.0
    closes, regimes = [], []
    trend_ret = 0.0

    for i, reg in enumerate(regime_seq):
        if reg == 'Trending':
            trend_ret = 0.6 * trend_ret + rng.normal(0.001, 0.015)  # AR(1) positive
        else:
            mean_level = np.mean(closes[-50:]) if len(closes) > 50 else price
            reversion  = 0.05 * (mean_level - price) / price
            trend_ret  = reversion + rng.normal(0, 0.008)
        price = price * np.exp(trend_ret)
        closes.append(price)
        regimes.append(reg)

    close = np.array(closes)
    high  = close * (1 + abs(rng.normal(0, 0.01, n_days)))
    low   = close * (1 - abs(rng.normal(0, 0.01, n_days)))
    open_ = close * (1 + rng.normal(0, 0.005, n_days))

    idx = pd.date_range('2021-01-01', periods=n_days, freq='D')
    return pd.DataFrame({'open': open_, 'high': high, 'low': low,
                          'close': close, 'true_regime': regimes}, index=idx)


df = generate_regime_switching_data(SIMULATION_DAYS)
print(df['true_regime'].value_counts())

true_regime
Trending    500
Ranging     500
Name: count, dtype: int64

Section 3 — Hurst Exponent

[12]

def hurst_exponent(price_series: np.ndarray) -> float:
    """
    Estimate the Hurst exponent using the R/S (rescaled range) method.

    Parameters
    ----------
    price_series : np.ndarray  Price levels (not returns).

    Returns
    -------
    float  Hurst exponent H: >0.5 trending, <0.5 mean-reverting, 0.5 random walk.

    Notes
    -----
    The R/S method divides the series into sub-periods, computes the range
    normalised by std dev, then regresses log(R/S) on log(n).
    """
    n = len(price_series)
    if n < 20:
        return 0.5
    log_rets = np.diff(np.log(price_series))
    lags  = range(10, n // 2, max(1, n // 20))
    rs_values = []
    for lag in lags:
        sub = log_rets[:lag]
        mean  = np.mean(sub)
        dev   = np.cumsum(sub - mean)
        rng   = np.max(dev) - np.min(dev)
        std   = np.std(sub, ddof=1)
        if std > 0:
            rs_values.append(rng / std)
        else:
            rs_values.append(np.nan)
    lags_arr = np.array(list(lags))
    rs_arr   = np.array(rs_values)
    valid    = ~np.isnan(rs_arr) & (rs_arr > 0)
    if valid.sum() < 2:
        return 0.5
    slope, _, _, _, _ = stats.linregress(np.log(lags_arr[valid]), np.log(rs_arr[valid]))
    return float(slope)


# Rolling Hurst
df['log_ret'] = np.log(df['close'] / df['close'].shift(1))
hurst_vals = []
for i in range(len(df)):
    if i < HURST_WINDOW:
        hurst_vals.append(np.nan)
    else:
        h = hurst_exponent(df['close'].iloc[i-HURST_WINDOW:i].values)
        hurst_vals.append(h)
df['hurst'] = hurst_vals

print(f'Mean Hurst (trending periods): {df[df["true_regime"]=="Trending"]["hurst"].mean():.3f}')
print(f'Mean Hurst (ranging periods):  {df[df["true_regime"]=="Ranging"]["hurst"].mean():.3f}')

Mean Hurst (trending periods): 0.705
Mean Hurst (ranging periods):  0.646

Hurst Exponent Calculation

This section introduces the Hurst Exponent, a statistical measure used to determine if a time series is trending, mean-reverting, or a random walk. A Hurst Exponent (H) greater than 0.5 indicates a trending series (persistence), less than 0.5 indicates a mean-reverting series (anti-persistence), and approximately 0.5 suggests a random walk. The code provides a function hurst_exponent that implements the R/S (Rescaled Range) method to estimate H. This method analyzes the range of deviations from the mean, normalized by the standard deviation, over different time lags. A rolling calculation of the Hurst Exponent is then applied to the generated price data, allowing us to observe its behavior over time and correlate it with the true regimes.

Section 4 — ADX & Rolling Autocorrelation

[13]

import numpy as np
import pandas as pd

# ADX
up   = df['high'] - df['high'].shift(1)
down = df['low'].shift(1) - df['low']
plus_dm  = pd.Series(np.where((up > down) & (up > 0), up, 0), index=df.index)
minus_dm = pd.Series(np.where((down > up) & (down > 0), down, 0), index=df.index)
tr = pd.concat([df['high']-df['low'],
                abs(df['high']-df['close'].shift()),
                abs(df['low']-df['close'].shift())], axis=1).max(axis=1)
atr = tr.rolling(ADX_PERIOD).mean()
plus_di  = 100 * plus_dm.rolling(ADX_PERIOD).mean() / atr
minus_di = 100 * minus_dm.rolling(ADX_PERIOD).mean() / atr
dx       = 100 * abs(plus_di - minus_di) / (plus_di + minus_di + 1e-9)
df['adx'] = dx.rolling(ADX_PERIOD).mean().values

# Rolling autocorrelation of returns
df['autocorr'] = df['log_ret'].rolling(AUTOCORR_WINDOW).apply(
    lambda x: x.autocorr(lag=AUTOCORR_LAG) if len(x.dropna()) > 5 else np.nan
)

print('ADX > 25 (trending signal):', (df['adx'] > 25).mean())
print('Positive autocorr:',          (df['autocorr'] > 0).mean())

ADX > 25 (trending signal): 0.519
Positive autocorr: 0.746

ADX and Rolling Autocorrelation

This section calculates two additional key indicators for regime detection: the Average Directional Index (ADX) and the rolling autocorrelation of returns.

ADX (Average Directional Index): ADX measures the strength of a trend, not its direction. Values above 25 typically indicate a strong trend, while values below 20 suggest a weak or non-trending market (ranging). The calculation involves True Range (TR), Positive Directional Movement (+DM), Negative Directional Movement (-DM), and their smoothed versions to derive the Directional Indicators (DI+ and DI-), which then feed into the ADX.
Rolling Autocorrelation: Autocorrelation measures the correlation of a time series with a lagged version of itself. For financial returns, a positive lag-1 autocorrelation suggests trending behavior (past returns influence future returns in the same direction), while a negative lag-1 autocorrelation indicates mean-reverting behavior (past returns influence future returns in the opposite direction). A rolling window is applied to capture these dynamics over time.

Section 5 — Composite Classifier

[14]

# Composite regime: 2/3 signals agree → Trending or Ranging
def composite_regime(row):
    trend_votes = 0
    range_votes = 0
    if pd.notna(row['hurst']):
        if row['hurst'] > 0.55:  trend_votes += 1
        elif row['hurst'] < 0.45: range_votes += 1
    if pd.notna(row['adx']):
        if row['adx'] > 25:       trend_votes += 1
        elif row['adx'] < 20:     range_votes += 1
    if pd.notna(row['autocorr']):
        if row['autocorr'] > 0.1:  trend_votes += 1
        elif row['autocorr'] < -0.1: range_votes += 1
    if trend_votes >= 2:   return 'Trending'
    elif range_votes >= 2: return 'Ranging'
    else:                  return 'Uncertain'

df['detected_regime'] = df.apply(composite_regime, axis=1)

print('Detected regime distribution:')
print(df['detected_regime'].value_counts())

# Accuracy vs true label
match = df[df['detected_regime'] != 'Uncertain']
accuracy = (match['detected_regime'] == match['true_regime']).mean()
print(f'\nDetection accuracy (excl. Uncertain): {accuracy:.1%}')

Detected regime distribution:
detected_regime
Trending     636
Uncertain    232
Ranging      132
Name: count, dtype: int64

Detection accuracy (excl. Uncertain): 70.2%

Composite Regime Classifier

This section combines the individual signals from the Hurst Exponent, ADX, and rolling autocorrelation into a robust composite regime classifier. Relying on a single indicator can be prone to false signals or ambiguity. By taking a "majority vote" from multiple indicators, the composite_regime function aims to provide a more accurate and reliable classification of whether the market is in a 'Trending', 'Ranging', or 'Uncertain' state. The thresholds for each indicator (e.g., Hurst > 0.55 for trending, ADX > 25 for trending) are defined to assign votes, and if at least two indicators agree, a regime is declared. This approach enhances the overall confidence in regime identification and reduces noise.

Section 6 — Visualization

[15]

fig, axes = plt.subplots(3, 1, figsize=(14, 11), sharex=True)
fig.suptitle('Trend vs Mean-Reversion Regime Detection', fontsize=14, fontweight='bold')

ax1 = axes[0]
ax1.plot(df.index, df['close'], color='#1976d2', lw=1.2)
for reg, col in [('Trending','#43a047'), ('Ranging','#e53935')]:
    ax1.fill_between(df.index, df['close'].min(), df['close'].max(),
                      where=(df['true_regime']==reg), color=col, alpha=0.1, label=f'True: {reg}')
ax1.legend(fontsize=8); ax1.set_title('Price with True Regime Labels')

ax2 = axes[1]
ax2.plot(df.index, df['hurst'], color='#7b1fa2', lw=1.5, label='Hurst Exponent')
ax2.axhline(0.5, color='black', ls='--', lw=1, label='H=0.5 (random walk)')
ax2.axhline(0.55, color='#43a047', ls=':', lw=1, label='H=0.55 (trend)')
ax2.axhline(0.45, color='#e53935', ls=':', lw=1, label='H=0.45 (revert)')
ax2.set_ylim(0.2, 0.8); ax2.set_ylabel('Hurst Exponent')
ax2.legend(fontsize=7); ax2.set_title('Hurst Exponent')

ax3 = axes[2]
ax3.plot(df.index, df['adx'], color='#ff9800', lw=1.5, label='ADX')
ax3.axhline(25, color='#43a047', ls='--', lw=1, label='ADX=25 (trending)')
ax3.axhline(20, color='#e53935', ls='--', lw=1, label='ADX=20 (ranging)')
ax3.set_ylabel('ADX')
ax3.legend(fontsize=8); ax3.set_title('ADX Trend Strength')

plt.tight_layout()
plt.show()

Regime Detection Visualization

This section is dedicated to visualizing the results of our regime detection. Visualization is critical for understanding how our calculated indicators (Hurst Exponent and ADX) and the detected_regime align with the underlying true_regime and price action. The plots allow us to:

Price with True Regime Labels: See the simulated price movement and visually identify the periods where the data is truly trending or ranging.
Hurst Exponent: Observe the Hurst values over time and how they fluctuate around the 0.5 random walk threshold, ideally indicating values > 0.5 during trending periods and < 0.5 during ranging periods.
ADX Trend Strength: Monitor the ADX values, noting when they cross the 20 (ranging) and 25 (trending) thresholds, to see its effectiveness in identifying trend strength.

These visualizations provide an intuitive way to assess the performance of our regime detection models.

Section 7 — Export

[16]

df.to_csv('trend_vs_mean_reversion_regime.csv')
print('Saved: trend_vs_mean_reversion_regime.csv')

Saved: trend_vs_mean_reversion_regime.csv

Exporting Regime-Labelled Data

This final section exports the DataFrame, which now includes the calculated indicators (Hurst, ADX, autocorrelation) and the detected_regime column, to a CSV file. This is an important step as it makes the processed and labeled data available for further analysis, such as:

Backtesting: The regime labels can be used to design and evaluate trading strategies that adapt to different market conditions (e.g., trend-following in trending regimes, mean-reversion in ranging regimes).
Machine Learning: The indicators and regime labels can serve as features and target variables for training machine learning models to predict future regimes.
Reporting: The CSV provides a clean and structured dataset for reporting or sharing the results of the regime detection analysis.

Section 8 — Conclusion

This notebook provides a comprehensive approach to financial market regime detection using a combination of the Hurst Exponent, ADX, and rolling autocorrelation. By integrating these statistical tools, we developed a composite classifier that identifies trending and mean-reverting periods with reasonable accuracy. The simulated data allowed us to validate the effectiveness of these indicators in a controlled environment.

The ability to accurately detect market regimes is crucial for adaptive trading strategies. Future work could involve exploring other regime detection methodologies, optimizing the thresholds and lookback periods for each indicator, and applying these techniques to real-world financial data for backtesting and performance evaluation.