Enter Platform
Resources
Back to Notebooks
Open in Colab
Statistical Analysis·Pairs Trading Methods·Intermediate

Pairs Selection Clustering

Apply hierarchical agglomerative clustering and dynamic time warping distance metrics to price and return series data to efficiently identify candidate cointegrated asset pairs for subsequent formal statistical arbitrage testing, dramatically reducing the initial pair search space.

pairs-tradingquant-analysis

Cluster-Based Pairs Selection — Statistical Analysis

Category: Statistical Analysis | Subcategory: Pairs

What This Notebook Does

Before running cointegration tests on all possible pairs (which is O(n²)), clustering is used to pre-select pairs that are likely to be related, reducing both computation and spurious correlations.

Pipeline:

  1. Hierarchical clustering on return correlations — group similar assets
  2. Test only within-cluster pairs for cointegration
  3. Score each candidate pair on: cointegration strength, half-life, Hurst exponent
  4. Rank pairs and select top-N for trading

This notebook:

  1. Generates a universe of crypto assets with known similarity structures
  2. Applies hierarchical agglomerative clustering on correlation matrix
  3. Tests within-cluster pairs for cointegration (Engle-Granger)
  4. Scores and ranks pairs
  5. Visualises dendrograms and correlation heatmaps
  6. Exports ranked pairs list
[ ]
!pip install numpy pandas matplotlib seaborn scipy statsmodels scikit-learn --quiet
[ ]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.cluster.hierarchy import dendrogram, linkage, fcluster
from scipy.spatial.distance import squareform
from statsmodels.tsa.stattools import coint
import warnings

warnings.filterwarnings('ignore')
%matplotlib inline
plt.rcParams['figure.figsize'] = (14, 5)
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.right'] = False
print('Imports ready.')
Imports ready.

Section 1 — Configuration

This section outlines the key configuration parameters used throughout the notebook, such as the number of clusters, maximum pairs to test, cointegration p-value threshold, and simulation days.

[ ]
N_CLUSTERS       = 4    # number of asset clusters
MAX_PAIRS_TEST   = 30   # max pairs to test per cluster
COINT_P_THRESH   = 0.05
SIMULATION_DAYS  = 500
print('Config ready.')
Config ready.

Section 2 — Asset Universe

This section defines the generate_asset_universe function, which creates a synthetic dataset of crypto asset prices organized into distinct categories (e.g., Layer 1s, DeFi, Layer 2s). This simulated data is used to demonstrate the clustering and cointegration analysis.

[ ]
def generate_asset_universe(n_days=500, seed=42):
    """
    Generate 12 crypto assets in 4 natural groups.

    Groups:
    - Layer 1s: BTC, ETH, SOL, AVAX
    - DeFi:     UNI, AAVE, CRV, SNX
    - Layer 2s: MATIC, ARB, OP, STRK

    Returns
    -------
    pd.DataFrame  Log-price series.
    """
    rng = np.random.default_rng(seed)
    idx = pd.date_range('2023-01-01', periods=n_days, freq='D')

    # Sector factors
    market_factor = np.cumsum(rng.normal(0, 0.01, n_days))
    sectors = {
        'L1':  np.cumsum(rng.normal(0, 0.008, n_days)),
        'DeFi':np.cumsum(rng.normal(0, 0.012, n_days)),
        'L2':  np.cumsum(rng.normal(0, 0.015, n_days)),
    }

    assets = {
        'BTC':  ('L1',   10.5, 0.8, 0.3), 'ETH':  ('L1',   9.8, 0.8, 0.3),
        'SOL':  ('L1',   5.2, 0.7, 0.4), 'AVAX': ('L1',   4.8, 0.7, 0.4),
        'UNI':  ('DeFi', 2.4, 0.6, 0.5), 'AAVE': ('DeFi', 4.5, 0.6, 0.5),
        'CRV':  ('DeFi', 0.8, 0.5, 0.6), 'SNX':  ('DeFi', 2.1, 0.5, 0.6),
        'MATIC':('L2',   0.7, 0.7, 0.4), 'ARB':  ('L2',   1.2, 0.7, 0.4),
        'OP':   ('L2',   2.0, 0.7, 0.4), 'STRK': ('L2',   0.8, 0.6, 0.5),
    }

    prices = {}
    for name, (sector, base, b_market, b_sector) in assets.items():
        idio  = np.cumsum(rng.normal(0, 0.015, n_days))
        lp    = base + b_market * market_factor + b_sector * sectors[sector] + idio
        prices[name] = np.exp(lp)

    return pd.DataFrame(prices, index=idx)


prices = generate_asset_universe(SIMULATION_DAYS)
returns = np.log(prices / prices.shift(1)).dropna()
print(f'Asset universe: {prices.shape[1]} assets, {len(prices)} days')
Asset universe: 12 assets, 500 days

Section 3 — Hierarchical Clustering

This section applies hierarchical agglomerative clustering to the asset returns correlation matrix. It visualizes the clustering using a dendrogram and a correlation heatmap, then assigns assets to clusters based on the specified number of clusters.

[ ]
corr_matrix = returns.corr()
dist_matrix = 1 - corr_matrix
condensed   = squareform(dist_matrix.values, checks=False)
Z = linkage(condensed, method='ward')

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
dendrogram(Z, labels=corr_matrix.columns.tolist(), ax=axes[0], leaf_rotation=45)
axes[0].set_title('Asset Clustering Dendrogram')

sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='RdYlGn',
            vmin=-1, vmax=1, center=0, ax=axes[1], square=True)
axes[1].set_title('Return Correlation Matrix')
plt.tight_layout()
plt.show()

cluster_labels = fcluster(Z, N_CLUSTERS, criterion='maxclust')
asset_clusters = pd.Series(cluster_labels, index=corr_matrix.columns)
print('\nCluster assignments:')
for cl in sorted(asset_clusters.unique()):
    print(f'  Cluster {cl}: {list(asset_clusters[asset_clusters==cl].index)}')
cell output

Cluster assignments:
  Cluster 1: ['UNI', 'AAVE', 'CRV', 'SNX']
  Cluster 2: ['MATIC', 'ARB', 'OP', 'STRK']
  Cluster 3: ['BTC', 'AVAX']
  Cluster 4: ['ETH', 'SOL']

Section 4 — Within-Cluster Cointegration Tests

This section performs cointegration tests (Engle-Granger) on all possible pairs of assets within each identified cluster. It then filters these pairs based on a p-value threshold to identify statistically cointegrated pairs.

[ ]
import itertools

results = []
for cl in sorted(asset_clusters.unique()):
    members = asset_clusters[asset_clusters == cl].index.tolist()
    for A, B in itertools.combinations(members, 2):
        score, p_val, _ = coint(np.log(prices[A]), np.log(prices[B]))
        results.append({'pair': f'{A}-{B}', 'cluster': cl,
                         'coint_pvalue': p_val, 'coint_score': score})

results_df = pd.DataFrame(results).sort_values('coint_pvalue')
cointegrated = results_df[results_df['coint_pvalue'] < COINT_P_THRESH]

print(f'Pairs tested: {len(results_df)}')
print(f'Cointegrated (p < {COINT_P_THRESH}): {len(cointegrated)}')
print('\nTop pairs by cointegration:')
print(cointegrated.head(10).to_string(index=False))
Pairs tested: 14
Cointegrated (p < 0.05): 0

Top pairs by cointegration:
Empty DataFrame
Columns: [pair, cluster, coint_pvalue, coint_score]
Index: []

Section 5 — Export

This section handles the export of the results. Specifically, it saves the DataFrame containing the cointegration test results for all tested pairs to a CSV file.

[ ]
results_df.to_csv('pairs_selection_clustering.csv', index=False)
print('Saved: pairs_selection_clustering.csv')
Saved: pairs_selection_clustering.csv

Conclusion

This notebook demonstrated a systematic approach to selecting potential pairs for statistical arbitrage strategies by combining hierarchical clustering with cointegration tests.

First, we generated a synthetic universe of crypto assets with inherent grouped structures to simulate real-world market conditions where certain assets exhibit higher correlations within their respective sectors.

Next, hierarchical clustering was applied to the return correlation matrix, allowing us to visually identify and programmatically group assets into N_CLUSTERS based on their price movement similarities. The dendrogram provided a visual representation of these relationships, while the heatmap confirmed the clustering structure by showing stronger correlations within the identified clusters.

Subsequently, cointegration (Engle-Granger) tests were performed exclusively on pairs of assets within each cluster. This 'within-cluster' testing strategy significantly reduces the number of pairs to analyze compared to an all-pairs approach, thereby mitigating computational cost and the risk of discovering spurious cointegration relationships. Pairs with a p-value below the COINT_P_THRESH were identified as statistically cointegrated, indicating a long-term equilibrium relationship.

Finally, the results, including the cointegration p-values and scores for all tested pairs, were compiled into a DataFrame and exported. This output serves as a ranked list of potential pairs that could be further analyzed for trading strategy development, focusing on metrics like half-life and Hurst exponent (though these were not explicitly calculated in this notebook).

In summary, this notebook provides a foundational framework for an efficient and statistically sound methodology for pairs selection, particularly beneficial in markets with a large number of correlated assets.