ML Model Training Pipeline

The ML training pipeline transforms raw market data into validated, deployable prediction models through a sequence of 18 rigorously designed stages. Unlike the strategy optimization pipeline, which searches for optimal parameters of hand-crafted rules, the ML pipeline learns predictive patterns directly from data. This makes it both more powerful and more susceptible to overfitting, requiring a substantially more elaborate validation framework.

The pipeline operates in five conceptual stages:

1. Setup (Steps 1–4): Model selection, data sourcing, train/walk-forward/holdout splitting, and feature selection.
2. Feature Engineering (Steps 5–7): Fractional differentiation, winsorization, IC-based filtering, triple-barrier labeling, and sample weighting.
3. Training & Optimization (Steps 8–10): Combinatorial purged cross-validation, TPE hyperparameter search, model calibration, and feature importance analysis.
4. Backtesting & Validation (Steps 11–16): Signal generation, backtest execution, walk-forward analysis, Monte Carlo simulation, overfitting diagnostics, and holdout evaluation.
5. Selection & Deployment (Steps 17–18): Multi-gate model selection and ensemble construction.

The pipeline supports both classification (predicting discrete market states: long, short, or neutral) and regression (predicting continuous returns). With 45 classifiers and 58 regressors available, including GPU-accelerated CatBoost, LightGBM, and XGBoost, the model space is vast, making rigorous validation essential.

The first step selects the model architecture and configures its hyperparameters. The system exposes 45 scikit-learn-compatible classifiers and 58 regressors through a unified interface built on ml_registry.json.

Classifier vs. Regressor

Classifiers predict discrete labels: typically long (+1), short (−1), or neutral (0). They are the natural choice when the labeling method (Step 6) produces categorical outputs, such as triple-barrier labels. The predicted class probabilities can be thresholded to control signal frequency and confidence.

Regressors predict continuous values: expected return, volatility, or any numeric target. They are appropriate when using regression-based labeling (Step 6) and produce signals proportional to the predicted magnitude, naturally scaling position sizes to conviction.

GPU-Accelerated Models

CatBoost, LightGBM, and XGBoost, available in both classifier and regressor variants, support GPU acceleration when use_gpu is enabled (Step 9). For datasets exceeding 10,000 bars, GPU training can reduce wall-clock time by 5–20× depending on model complexity and feature count. GPU models use histogram-based tree construction, which parallelises efficiently across CUDA cores.

Hyperparameter Optimization

When optimize_params is enabled, the system searches the model's hyperparameter space using the same TPE sampler described in the Strategy Optimization pipeline (Step 2 of that document). Each model in ml_registry.json defines its own parameter ranges with three optimization presets:

• Low (Narrow): 75–125% of default: fast, conservative search near defaults.
• Medium (Balanced): 50–150% of default: recommended for most use cases.
• High (Wide): 0–200% of default: broad exploration at higher overfitting risk.

Each parameter's min, max, and step can be individually overridden for fine-grained control. The total combination count, the product of all enabled parameter ranges, determines the minimum n_trials needed for adequate coverage.

The data source configuration specifies the trading pair, bar type, timeframe, and historical date range. Data is sourced from Binance. These choices fundamentally determine what patterns the model can learn and whether those patterns generalise.

Bar Type & Timeframe Selection

The choice of bar type (time, dollar, volume, hybrid, renko, range) and timeframe (1m through 1w) determines the information density and statistical properties of the input data. Time bars at 12h (the default) provide a balance of statistical sample size and signal-to-noise ratio for most strategies. Dollar and volume bars produce more stationary return distributions, important for ML models that assume IID data. For a detailed treatment of bar types, see the Data Quality documentation.

Structural Break Detection

Financial time series are non-stationary: the data-generating process changes at unpredictable points due to regulatory shifts, market structure evolution, or macroeconomic regime changes. When enabled, structural break detection uses the Chow test to identify points where the statistical properties of the series change significantly:

where RSS is residual sum of squares from regressions on each segment and the pooled data. The threshold parameter controls the F-statistic significance level. Breaks above the threshold are flagged; the model can then be trained only on data since the most recent break, avoiding contamination from structurally different historical regimes.

The dataset is partitioned into three temporal segments: train (70%), walk-forward (20%), and holdout (10%). As with the strategy optimization pipeline, splits are strictly temporal, no random shuffling, to prevent look-ahead bias.

Percentage vs. Date-Based Splits

Percentage mode allocates fixed fractions of the total date range. Simple and reproducible. Date-based mode allows explicit cut points: useful when you know specific dates where market regimes changed and want the walk-forward period to cover a particular event.

Purge Gap Multiplier

The purge gap is an embargo period inserted between the train and test sets within each cross-validation fold. It prevents data leakage from overlapping labels: a trade whose holding period spans the train/test boundary would leak future information into the training set. The gap is calculated as:

where horizon is the labeling horizon from Step 6. With horizon=15 and multiplier=1, a 15-bar gap is inserted between each train and test fold. Higher multipliers provide conservative protection against leakage at the cost of reducing effective training data.

BitPredict provides 29 pre-computed financial features spanning multiple domains: momentum and trend (returns, EMA, supertrend, risk-adjusted momentum), volatility and dispersion (skewness, volume ROC, entropy collapse rate), market regime metadata (transition pressure, regime scores, trend acceleration Z-scores), information theory (transfer entropy between timeframes, information geometry distance), and structural features (VWAP, overnight gap, PV divergence, recovery slope).

Feature Categories

• Momentum/trend (8): returns_4hr, returns_arith, returns_abs, ema_20, supertrend, risk_adj_momentum_50, score_accelerating, mr_trend_acceleration_z
• Volatility/dispersion (5): skewness_60, volume_roc, entropy_collapse_rate, mr_volatility_skew_chg, log_return_50
• Regime metadata (7): transition_pressure, mr_transition_pressure_chg, score_bear, mr_score_bear_chg, mr_score_transition_z, mr_score_accelerating_z, recovery_slope
• Information theory (2): transfer_entropy_5m_15m, information_geometry_dist
• Structural (7): vwap, overnight_gap, pv_divergence, return_autocorr_5, time_sine, ...

The default selection, "all", passes all 29 features to the feature pipeline (Step 5), which then performs automated filtering. A custom subset can be specified to restrict the model to domain-specific features or to test hypotheses about which feature families drive performance.

The feature pipeline is the most mathematically sophisticated preprocessing step. It transforms raw features into stationary, decorrelated, high-signal inputs through a sequence of operations: outlier treatment, fractional differentiation, feature selection via Information Coefficient (IC), adversarial validation, cluster-based redundancy removal, and predictive decay analysis.

Winsorization

Extreme outliers, common in financial data due to flash crashes, exchange errors, or data feed anomalies, can dominate model training. Winsorization clips feature values at specified percentiles:

The default clips at the 1st and 99th percentiles, preserving the central 98% of the distribution unchanged. This is a one-sided operation on each feature independently.

Fractional Differentiation (FFD)

Financial time series are typically I(1): integrated of order 1, meaning they contain a unit root and are non-stationary. Standard ML models assume I(0) stationarity. Integer differentiation (first differences) achieves stationarity but destroys all memory: the differenced series retains no information about past levels.

Fractional differentiation provides a continuum between levels (d = 0, maximum memory, non-stationary) and first differences (d = 1, minimum memory, stationary). The optimal order d* is found by grid search over [d_min, d_max] at step granularity, testing each candidate with the Augmented Dickey-Fuller (ADF) test:

where the weights for fractional differentiation are computed via the binomial expansion:

The result is a stationary series that preserves as much memory as possible, critical for financial ML where persistent trends carry predictive signal.

Information Coefficient (IC) Filtering

The IC measures the cross-sectional correlation between a feature's values and forward returns. For each feature f:

where Spearman (rank) correlation is used for robustness to outliers. Features with |IC| < ic_threshold (default 0.02) are dropped: they carry negligible predictive signal. The ICIR (Information Coefficient Information Ratio) adds a consistency dimension:

Features with ICIR < icir_threshold (default 0.5) are dropped: their predictive power is inconsistent over time, making them unreliable for training.

Adversarial Validation

A subtle form of overfitting occurs when features differ systematically between train and test periods, not in their predictive relationship with the target, but in their marginal distributions. The model learns to exploit these distributional differences rather than genuine predictive patterns. Adversarial validation detects this by training a classifier to distinguish train from test samples using the features. Features with AUC > adversarial_auc_threshold (default 0.55) are flagged as “too predictive of time” and removed.

Cluster-Based Decorrelation

Redundant features inflate dimensionality without adding information, increasing overfitting risk. Features are clustered by their pairwise correlation distance; within each cluster closer than cluster_distance_threshold (default 0.5), only the feature with the highest IC is retained. This reduces the effective feature count while preserving the information content of each independent signal family.

Predictive Decay Analysis

A feature that predicts well at horizon 1 but degrades at horizon 20 may be useful for short-term models but harmful for longer-horizon strategies. Decay analysis measures IC at multiple horizons [1, 2, 5, 10, 20] and computes the decay ratio:

Features with decay_ratio < decay_min_ic_ratio (default 0.75) are flagged for potential removal: their signal fades too quickly for the chosen labeling horizon.

Pipeline Modes

Four aggressiveness modes control how aggressively features are filtered:

• Auto (default): Balances feature retention against signal quality using adaptive thresholds.
• Balanced: Moderately aggressive, suitable for most use cases.
• Strict: Aggressively removes features with marginal IC, high correlation, or temporal instability. Fewer features survive but those that do are high-confidence. Risk of underfitting.
• Lenient: Retains most features. Useful for initial exploration or when the feature count is already small. Higher overfitting risk.

Labeling transforms raw price data into supervised learning targets. This is arguably the most consequential modeling decision: the label definition determines what the model learns to predict, and errors in labeling propagate through every downstream step.

Triple Barrier Method (López de Prado)

The triple barrier method defines three boundaries for each observation at time t:

1. Profit-Taking (PT) barrier: upper threshold at price × (1 + pt_mult × σ). If hit first, label = +1 (long) or −1 (short) depending on direction.
2. Stop-Loss (SL) barrier: lower threshold at price × (1 − sl_mult × σ). If hit first, label = opposite sign.
3. Vertical (time) barrier: horizon bars after entry. If neither PT nor SL is hit, label = 0 or the sign of the return at the vertical barrier.

The volatility σ is either the Average True Range (ATR, default) or a fixed percentage (pct_base). ATR adapts barrier distances to current volatility, producing more consistent hit rates across different market conditions:

Fixed Percentage

Simpler alternative: label = +1 if the return over the horizon exceeds threshold, −1 if it falls below −threshold, 0 otherwise. Suitable when volatility is stable or when you want uniform thresholds across all observations.

Regression

Produces continuous labels: log returns, simple returns, or raw prices. Used with regressor models (Step 1). The risk_adjusted option normalises returns by realised volatility, making labels comparable across volatility regimes.

Label Quality Filter

Labels are tested for statistical significance: for each label class, a t-test compares the mean forward return conditional on that label against zero. Labels with |t| < min_tvalue (default 1.0) are considered uninformative and can be dropped or down-weighted.

Meta-Labeling

Meta-labeling trains a secondary binary classifier to predict whether the primary model's prediction will be correct. This “model of the model” filters out low-confidence predictions, improving precision at the cost of recall. Particularly effective when the primary model has high recall but low precision: the meta-model suppresses false positives.

Financial labels are not IID: labels generated from overlapping time windows share information, violating the independence assumption of standard ML loss functions. Sample weights correct for this by assigning lower weight to samples with higher overlap.

Uniqueness Weights

Each observation's weight is proportional to the fraction of its return that is not overlapped by other observations' labels. An observation whose entire return window is covered by other labels receives zero weight; it contributes no independent information:

where T_i is the length of label i's return window and the inner sum counts how many labels cover each time point.

Time Decay Weights

Older observations receive exponentially lower weight, reflecting the intuition that recent market behaviour is more relevant for predicting near-future returns:

With decay_factor = 0.95 and 100 bars, the oldest observation receives 0.95¹⁰⁰ ≈ 0.006, effectively zero weight. Recent observations dominate.

Sequential Bootstrap

Standard bootstrap resampling assumes IID data and produces unrealistic resampled datasets for financial time series. The sequential bootstrap modifies the sampling probability to favour observations with lower average uniqueness, those that are less overlapped with already-sampled observations. This produces bootstrap samples that better preserve the dependency structure of the original data.

Standard k-fold cross-validation, the workhorse of general ML, is invalid for financial data because it randomly shuffles observations across folds, leaking future information into training. Financial cross-validation must respect temporal ordering and purge overlapping labels.

Purged K-Fold

Splits data into n_splits sequential folds. Each fold's training set consists of all data before the test fold, with a purge gap (Step 3) inserted before the test boundary. This eliminates look-ahead bias but provides only one train/test path; the performance estimate can be sensitive to the particular split boundaries.

Combinatorial Purged Cross-Validation (CPCV)

CPCV generates multiple train/test paths by combinatorially selecting groups of folds for testing while using the remaining folds for training. With n_splits = 5 and oos_groups = 2:

Each path provides an independent performance estimate. The distribution of these estimates, not just the mean, reveals how sensitive the model is to the particular historical period used for testing. A model with Sharpe ranging from 0.8 to 2.4 across paths is far less trustworthy than one with Sharpe consistently at 1.5 ± 0.2, even though both have the same mean.

CPCV also enables the Probability of Backtest Overfitting (PBO) calculation in Step 15: by comparing the rank of the selected model across all paths, PBO estimates the probability that a different model would have been selected under different market conditions.

When optimize_params is enabled (Step 1), this step controls the search study configuration. Unlike the strategy optimization pipeline which always runs the search, ML training can run with fixed hyperparameters, useful for rapid iteration when you already know good parameter ranges.

Composite Score

The default objective function is a weighted composite of four metrics, each clipped to prevent individual extremes from dominating:

where PF = Profit Factor, DD = Max Drawdown (normalised). Sortino Ratio receives the highest weight (0.4) because it penalises only downside volatility, the kind that matters for trading. A custom formula string can override these weights entirely.

Median Pruner

The MedianPruner terminates unpromising trials early. After n_startup_trials (default 10) complete, each subsequent trial's intermediate fold scores are compared to the median of completed trials at the same step. If a trial consistently underperforms the median, it is pruned - saving compute for more promising configurations.

After hyperparameter optimization, the top_k models are trained on the full training set with two optional post-training analyses.

Probability Calibration

Many classifiers produce biased probability estimates: a predicted probability of 0.8 may correspond to an empirical frequency of only 0.6. Calibration adjusts predicted probabilities to match empirical frequencies using cross-validated isotonic regression or Platt scaling:

where f is the calibration function learned on held-out folds. With cv = 3, the model is trained on 2/3 of the data, calibrated on the remaining 1/3, and the process is repeated 3 times. Well-calibrated probabilities are essential for position sizing; if the model says “80% confident” but is actually only 60% correct, your position sizes will be systematically too large.

Feature Importance (SHAP)

When enabled, feature importance is computed either natively (model built-in) or via SHAP (SHapley Additive exPlanations). SHAP values decompose each prediction into additive feature contributions using cooperative game theory:

where φ_j is the Shapley value for feature j, F is the set of all features, and the sum is over all possible feature subsets. SHAP values satisfy three desirable properties, local accuracy, missingness, and consistency, making them the most theoretically grounded feature importance method.

The stability_threshold measures the consistency of feature importance rankings across cross-validation folds. Features whose importance rank fluctuates significantly across folds (stability < 0.5) are flagged as unreliable.

Raw model predictions, class probabilities or regression values, must be converted into actionable trading signals. This step defines the thresholds and filters that control signal frequency, confidence, and expected return.

Probability Threshold

proba_threshold (default 0.0) sets the minimum predicted probability to generate any signal. At 0.0, all predictions generate signals. At 0.7, only high-confidence predictions generate signals, reducing trade frequency but increasing the win rate.

Confidence Margin

For multi-class classifiers, the confidence margin is the minimum difference between the highest and second-highest class probabilities to take a position:

A margin of 0.05 means the model must be at least 5% more confident in “long” than “short” to generate a long signal. This suppresses ambiguous predictions where the model is uncertain between classes.

Minimum Return Threshold

min_return_threshold (default 0.01 = 1%) suppresses signals whose predicted return is below this value. This avoids generating signals for tiny expected moves that would be consumed by transaction costs.

The backtest step simulates trading the model's signals through historical data with realistic execution assumptions: starting balance, position sizing, leverage, transaction fees, slippage, and a full risk management stack.

Risk Management Stack

Four independent risk controls, each configurable:

• Stop Loss (SL): Static (fixed %), ATR-based (dynamic volatility-adjusted), or none. Default: static at 1%.
• Take Profit (TP): Static (2%), ATR-based, risk-reward ratio (RR), or none. RR mode sets TP = SL × ratio (default 2.0), ensuring asymmetric risk-reward.
• Trailing Stop Loss (TSL): Trailing (locks in gains at trail_pct from peak, default 0.5%), Chandelier (ATR-based), or none. Activation threshold (default 1.0%) prevents trailing from triggering on noise.
• Time Stop: Closes positions after max_duration (default 12h) regardless of P&L. Prevents capital from being locked in stagnant positions.

Cost Sensitivity

When enabled, the backtest is repeated at multiple fee tiers (default: 0.0% and 0.2%) to assess the model's sensitivity to trading costs. A model that is profitable at 0.0% but loses money at 0.2% fees is fragile; small increases in exchange fees or spread widenings will erase its edge. Models are filtered at min_fee_tier (default 0.1%).

Walk-forward analysis extends out-of-sample testing across multiple sequential time windows, simulating how the model would perform if periodically retrained. This is the closest backtesting approximation to live model deployment.

Rolling Window Design

The walk-forward period is divided into n_windows (default 5) sequential windows. For each window:

1. The model is trained on all data before the window start (if retrain=true).
2. Signals are generated for the window period.
3. Performance metrics are computed for the window.
4. The window advances and the process repeats.

Two aggregate metrics gate this phase:

• Profitable Ratio: fraction of windows with positive return. Must exceed min_profitable_ratio (default 0.6).
• Consistency Score: measures the stability of window-level Sharpe ratios. Must exceed min_consistency_score (default 0.5).

Monte Carlo simulation reshuffles the sequence of trade returns to estimate the distribution of possible strategy outcomes. This is the same methodology as the strategy optimization pipeline's Step 10, applied to ML model trades.

ML-Specific Considerations

ML models introduce additional Monte Carlo considerations beyond those for rule-based strategies. Model predictions are conditional on feature values: reshuffling trade returns implicitly assumes that trade outcomes are exchangeable, which may not hold if the model's edge varies systematically with market conditions.

The stationary method (Politis-Romano, default) is recommended for ML models because it preserves short-range dependencies in trade outcomes while randomising long-range structure. With n_simulations = 500 and worst_case_percentile = 5, the bottom 5th percentile of simulated outcomes must pass the profitability and worst-case thresholds.

Three complementary overfitting tests form the core model validation framework. A model must pass all enabled tests to proceed to holdout evaluation and deployment.

Probability of Backtest Overfitting (PBO)

PBO uses the CPCV path structure (Step 8) to estimate the probability that the model selected as “best” is overfit. For each CPCV path p, the model with the highest in-sample performance is identified. If this “best” model consistently underperforms out-of-sample, relative to other models that ranked lower in-sample, the selection process is overfit:

A PBO of 0.5 means the “best” model performs no better than the median model out-of-sample; the selection process adds no value. A PBO of 0.05 means the selected model is genuinely superior. The default rejection threshold is max_pbo = 0.2.

Deflated Sharpe Ratio (DSR)

As in the strategy optimization pipeline (Step 3), DSR adjusts the observed Sharpe ratio for the expected maximum Sharpe from N trials under the null of no predictive power. For ML models, N is the product of search trials (Step 9) and the number of CPCV paths, typically 15 × 10 = 150. The resulting deflation is substantial:

With N = 150 trials, the expected maximum Sharpe under the null is approximately 2.5 - meaning a reported Sharpe of 2.0 may not be statistically significant. DSR must exceed min_dsr (default 1.0).

Multiple Testing Correction

When enabled, applies Bonferroni-style correction across all tested models. The effective number of tests accounts for both the number of search trials and the CPCV path count.

The holdout set, reserved since Step 3 and never seen by the optimizer, the feature pipeline, or any intermediate processing, provides the final, unbiased performance estimate. This is the closest approximation to future unseen data.

Primary Metric

The default evaluation metric is Sharpe Ratio, but any of six metrics can serve as the primary gate: Sharpe, Sortino, Calmar, Win Rate, Profit Factor, or Max Drawdown %. The choice should align with the optimization objective (Step 9); evaluating on a different metric than you optimised for creates an inconsistency.

Model selection applies a configurable set of mandatory and soft gates to filter the pool of trained models. Only models that pass all mandatory gates are eligible for deployment; soft gate failures produce warnings but do not block.

Mandatory Gates (all must pass)

Optional Mandatory Gates

Walk-forward profitable ratio, walk-forward consistency, and cost sensitivity can be promoted from optional to mandatory via the multi-select configuration.

Soft Gates (warnings)

Feature importance stability: flags models whose important features vary significantly across folds, suggesting the model relies on features that are not consistently predictive.

Ensemble methods combine multiple models to reduce variance, improve robustness, and capture complementary predictive signals. When enabled, models that individually pass the selection gates (Step 17) are combined into a single predictor.

Weighted Vote

Each model's prediction is weighted by its validation performance (e.g., Sharpe ratio). The ensemble prediction is the weighted average of individual model probabilities. Weights are normalised to sum to 1.

Meta-Model

A secondary “meta-model” is trained to combine individual model predictions, learning which models to trust under which market conditions. More powerful than weighted voting but requires additional training data and is itself susceptible to overfitting.

Signal Correlation Constraint

Models with pairwise signal correlation above max_signal_correlation (default 0.5) are not combined: they provide redundant information. This constraint ensures ensemble diversity: the benefit of ensembling comes from combining models that make different mistakes, not from averaging identical predictions.