Combinatorial Purged CV
Implement combinatorial purged cross-validation from Advances in Financial Machine Learning that prevents information leakage between training and testing sets through purging overlapping observations and embargoing adjacent time periods.
Combinatorial Purged Cross-Validation (CPCV)
1. Introduction: What is Combinatorial Purged Cross-Validation?
Combinatorial Purged Cross-Validation (CPCV) is a robust backtesting methodology designed specifically for financial time series and other serially correlated data. It addresses critical issues of data leakage that arise when applying traditional cross-validation techniques to time-dependent datasets. CPCV combines three key concepts:
- Cross-Validation: Splitting the data into multiple train/test sets to evaluate model performance.
- Purging: Removing training observations that overlap with the test set's feature and label generation windows, preventing look-ahead bias.
- Embargoing: Excluding data points immediately following a test set from subsequent training sets, preventing information leakage from predictions to future training data.
- Combinatorial Sampling: Generating multiple, distinct train/test splits that respect purging and embargoing, allowing for a more thorough and less optimistic assessment of model robustness than a single, sequential split.
Why is CPCV Important?
In financial machine learning, observations are often correlated over time. For example, a target variable (e.g., future price movement) might be derived from a window of historical data, and features might also be constructed over a window. If standard cross-validation is used, a training sample could contain information that is directly used to form a test sample, leading to:
- Look-ahead bias: The model implicitly 'sees' future information.
- Over-optimistic performance estimates: The model appears to perform better than it would on truly unseen data.
CPCV mitigates these issues, providing a more realistic and reliable assessment of a model's out-of-sample performance and generalizability, which is crucial for deployment in live trading environments.
2. Understanding Purging and Embargoing
Let's first understand the core mechanisms that prevent data leakage in time series cross-validation: purging and embargoing.
Data Generation Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import itertools
# Set random seed for reproducibility
np.random.seed(42)
# Generate mock timestamps
n_samples = 100
start_date = pd.to_datetime('2020-01-01')
time_series_data = pd.DataFrame({
'timestamp': [start_date + pd.Timedelta(days=i) for i in range(n_samples)],
'feature_1': np.random.rand(n_samples),
'target': np.random.randint(0, 2, n_samples) # Binary target
})
print(f"Generated {n_samples} data points from {time_series_data['timestamp'].min().date()} to {time_series_data['timestamp'].max().date()}.")
Generated 100 data points from 2020-01-01 to 2020-04-09.
2.1 Purging
What it is: Purging refers to the process of removing training set observations that are too close to test set observations. This proximity is determined by the look-forward and look-back windows used to construct features and labels.
Why it matters: If a test observation (e.g., t_i) has a target label y_i derived from data up to t_i + k and a feature vector X_i derived from data from t_i - l to t_i, then any training observation t_j that falls within the [t_i - l, t_i + k] interval (or similar overlapping intervals) would leak information. Purging ensures that the training set does not contain information that was used to define the test set.
How it works: For a given test sample's start time t_0 and end time t_1, we define a 'purge' window [t_0 - T_purge, t_1 + T_purge]. Any training sample falling within this window is removed.
2.2 Embargoing
What it is: Embargoing is the process of preventing any training observations that follow a test set from appearing in a subsequent training set, up to a certain period. This is to avoid data leakage from the predictions made on the test set back into future training sets.
Why it matters: If you train a model, make predictions on a test set, and then immediately use data following that test set for another training set, the information from your model's predictions (e.g., an implicit market impact) could subtly leak back into the training data. This can create an artificial advantage for the model.
How it works: After a test set ends at time t_1, an 'embargo' period [t_1, t_1 + T_embargo] is introduced, during which no training samples are allowed.
3. Implementation of Purging and Embargoing in K-Fold Cross-Validation
Let's create a custom cross-validation split generator that incorporates purging and embargoing for a standard K-Fold-like structure. This will form the basis for our combinatorial approach.
class PurgedKFold:
"""
Generates purged and embargoed K-Fold cross-validation splits.
This class extends traditional K-Fold by incorporating purging and embargoing
to prevent data leakage in time series data, especially in financial contexts.
"""
def __init__(self, n_splits=3, t_series=None, purge_pct=0.05, embargo_pct=0.01):
"""
Initializes the PurgedKFold cross-validator.
Inputs:
- n_splits (int): The number of splits for cross-validation.
- t_series (pd.Series): A pandas Series of timestamps or a column that can be converted
to timestamps, representing the observation times for the data.
This is crucial for determining purge and embargo windows.
- purge_pct (float): The percentage of the test set's length to purge from the training set
before and after the test set. For example, if a test set has 100 observations
and `purge_pct=0.05`, then 5 observations immediately preceding and
following the test set will be excluded from the training set.
- embargo_pct (float): The percentage of the test set's length to embargo from subsequent
training sets. This window starts *after* the current test set ends.
For example, if a test set has 100 observations and `embargo_pct=0.01`,
then 1 observation immediately after the test set will be excluded
from any subsequent training set.
Outputs:
- None: Initializes the object to be used with the `split` method.
"""
if not isinstance(t_series, pd.Series):
raise ValueError("t_series must be a pandas Series of timestamps.")
self.n_splits = n_splits
self.t_series = pd.to_datetime(t_series).sort_values().reset_index(drop=True)
self.purge_pct = purge_pct
self.embargo_pct = embargo_pct
self.n_samples = len(self.t_series)
def split(self, X, y=None, groups=None):
"""
Generates indices to split data into training and test sets.
Inputs:
- X (array-like): The data to be split (features). The length should match `n_samples`.
- y (array-like, optional): The target variable. Not directly used for splitting but
included for scikit-learn compatibility.
- groups (array-like, optional): Group labels for the samples. Not directly used for
splitting but included for scikit-learn compatibility.
Outputs:
- Yields (train_indices, test_indices) tuples for each split:
- train_indices (np.array): Array of integer indices for the training set.
- test_indices (np.array): Array of integer indices for the test set.
Formulas Used:
- `purge_window = int(test_set_length * self.purge_pct)`
This calculates the number of observations to purge based on the test set size.
- `embargo_window = int(test_set_length * self.embargo_pct)`
This calculates the number of observations to embargo based on the test set size.
"""
if len(X) != self.n_samples:
raise ValueError("Length of X must match length of t_series.")
indices = np.arange(self.n_samples)
fold_sizes = np.full(self.n_splits, self.n_samples // self.n_splits, dtype=int)
fold_sizes[:self.n_samples % self.n_splits] += 1
current = 0
for fold_size in fold_sizes:
# Define test set indices and their time boundaries
test_start = current
test_end = current + fold_size
test_indices = indices[test_start:test_end]
if len(test_indices) == 0: # Handle cases where a fold might be empty after division
current += fold_size
continue
test_start_time = self.t_series.iloc[test_indices[0]]
test_end_time = self.t_series.iloc[test_indices[-1]]
test_duration = test_end_time - test_start_time
# Calculate purge and embargo durations based on test duration
purge_duration = test_duration * self.purge_pct / (1 - self.purge_pct) if self.purge_pct > 0 else pd.Timedelta(seconds=0)
embargo_duration = test_duration * self.embargo_pct / (1 - self.embargo_pct) if self.embargo_pct > 0 else pd.Timedelta(seconds=0)
# Define purge boundaries
purge_start_time = test_start_time - purge_duration
purge_end_time = test_end_time + purge_duration
# Define embargo boundary (starts after test set ends)
embargo_end_time = test_end_time + embargo_duration
# Initialize valid training indices (all indices not in test set)
train_indices_candidate = np.setdiff1d(indices, test_indices)
# Apply Purging: remove training indices that fall within the purge window
# Create boolean mask for times within purge window
purge_mask = (
(self.t_series.iloc[train_indices_candidate] >= purge_start_time) &
(self.t_series.iloc[train_indices_candidate] <= purge_end_time)
)
train_indices_purged = train_indices_candidate[~purge_mask]
# Apply Embargoing: remove training indices that fall within the embargo window (after current test set)
# This specifically applies to subsequent folds in a sequential CV, but for a general KFold,
# it means removing anything that comes *after* the test_end_time + embargo_duration relative to the current test.
# For a sequential KFold, this typically means removing future observations from current train set.
# For a combinatorial KFold, we need to ensure that *any* future test set doesn't use data too close to previous predictions.
# Here, we ensure that the *current* train set doesn't extend into the embargo zone of the *current* test set.
embargo_mask = (
(self.t_series.iloc[train_indices_purged] > test_end_time) &
(self.t_series.iloc[train_indices_purged] <= embargo_end_time)
)
train_indices = train_indices_purged[~embargo_mask]
yield train_indices, test_indices
current += fold_size
3.1 Visualization of a Single Purged and Embargoed Split
Let's visualize how purging and embargoing affect a single train-test split. We'll mark the training data, test data, and the regions that are purged and embargoed.
def plot_splits(t_series, splits, title):
"""
Visualizes train, test, purged, and embargoed regions for cross-validation splits.
Inputs:
- t_series (pd.Series): The timestamps for the data points.
- splits (list of tuples): A list where each tuple contains (train_indices, test_indices)
for a single split.
- title (str): The title for the plot.
Outputs:
- None: Displays a matplotlib plot.
"""
plt.figure(figsize=(15, 6))
plt.title(title)
n_splits = len(splits)
for i, (train_idx, test_idx) in enumerate(splits):
# Get original indices and corresponding timestamps
all_indices = np.arange(len(t_series))
# Plotting for each split (as horizontal lines)
y_pos = i * 2 # Offset for plotting multiple splits
# Mark all data points as grey initially (potential data)
plt.hlines(y_pos, t_series.iloc[0], t_series.iloc[-1], linewidth=10, color='lightgray', label='All Data' if i == 0 else "")
# Plot training data
if len(train_idx) > 0:
train_start = t_series.iloc[train_idx[0]]
train_end = t_series.iloc[train_idx[-1]]
plt.hlines(y_pos, train_start, train_end, linewidth=8, color='blue', label='Train Set' if i == 0 else "", alpha=0.6)
# Plot test data
if len(test_idx) > 0:
test_start = t_series.iloc[test_idx[0]]
test_end = t_series.iloc[test_idx[-1]]
plt.hlines(y_pos, test_start, test_end, linewidth=8, color='red', label='Test Set' if i == 0 else "")
# Calculate and plot purge/embargo regions for visualization purposes
# These are illustrative based on the test set of the current split
test_duration = test_end - test_start
purge_duration = test_duration * purger.purge_pct / (1 - purger.purge_pct) if purger.purge_pct > 0 else pd.Timedelta(seconds=0)
embargo_duration = test_duration * purger.embargo_pct / (1 - purger.embargo_pct) if purger.embargo_pct > 0 else pd.Timedelta(seconds=0)
purge_start_vis = test_start - purge_duration
purge_end_vis = test_end + purge_duration
embargo_start_vis = test_end # Embargo starts after test
embargo_end_vis = test_end + embargo_duration
# Plot purged region (around test set, affecting training data)
plt.hlines(y_pos, purge_start_vis, purge_end_vis, linewidth=10, color='orange', alpha=0.4, label='Purged Region' if i == 0 else "")
# Plot embargoed region (after test set, affecting future training data)
# Note: This is an illustrative visualization of the *concept* of embargoing
# relative to the current test set. The actual embargoing logic prevents *future*
# training sets from using this region.
plt.hlines(y_pos, embargo_start_vis, embargo_end_vis, linewidth=10, color='green', alpha=0.4, label='Embargoed Region' if i == 0 else "")
plt.xlabel('Timestamp')
plt.ylabel('Split Index')
plt.yticks(np.arange(n_splits) * 2, [f'Split {j+1}' for j in range(n_splits)])
plt.ylim(-1, n_splits * 2 - 1 + 2) # Adjust y-limits
plt.legend(loc='upper left', bbox_to_anchor=(1, 1))
plt.grid(True, linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Example with PurgedKFold
purger = PurgedKFold(n_splits=3, t_series=time_series_data['timestamp'], purge_pct=0.1, embargo_pct=0.02)
single_splits = list(purger.split(time_series_data))
plot_splits(time_series_data['timestamp'], single_splits, 'Visualization of Single Purged and Embargoed Splits')
Interpretation of Visualization 3.1:
This plot shows how a time series dataset is divided into training and test sets by the PurgedKFold class.
- Blue regions represent the training data for each split.
- Red regions represent the test data for each split.
- Orange regions indicate the purged zones. These are periods around the test set that are excluded from the current training set to prevent data leakage due to overlapping feature/label computation windows.
- Green regions indicate the embargoed zones. These are periods immediately following the test set that are excluded from the current training set and, more critically, from subsequent training sets to prevent leakage from model predictions.
4. The Combinatorial Aspect
What it is: In standard cross-validation, splits are often contiguous or simple, sequential folds. Combinatorial cross-validation takes this a step further by generating all valid combinations of train/test folds, given a certain number of splits.
Why it matters: A single sequential split might by chance perform well or poorly due to specific market conditions or data characteristics within that split. By testing the model across many different combinations of folds, we get a more robust estimate of its performance and can assess its stability across various data partitions. This helps to reduce the variance of the performance estimate.
How it works: Instead of just N folds, we consider k distinct (purged and embargoed) folds. Then, we can create combinations of these k folds to form multiple train/test sets. For example, if we have 3 folds (Fold 1, Fold 2, Fold 3), we could have test sets as:
- (Fold 1, Folds 2+3 as train)
- (Fold 2, Folds 1+3 as train)
- (Fold 3, Folds 1+2 as train)
This is just a simple example; the 'combinatorial' aspect can involve more complex selections of multiple folds as test sets.
class CombinatorialPurgedKFold:
"""
Generates Combinatorial Purged K-Fold cross-validation splits.
This class builds upon PurgedKFold to generate multiple valid combinations
of train/test sets, enhancing the robustness of backtesting for time series.
"""
def __init__(self, n_folds=3, n_test_folds=1, t_series=None, purge_pct=0.05, embargo_pct=0.01):
"""
Initializes the CombinatorialPurgedKFold cross-validator.
Inputs:
- n_folds (int): The total number of initial sub-folds to divide the data into.
These sub-folds will then be combined.
- n_test_folds (int): The number of these sub-folds to use as the test set in each combination.
The remaining `n_folds - n_test_folds` will be used for training.
- t_series (pd.Series): A pandas Series of timestamps for the data points.
- purge_pct (float): Percentage of test set duration to purge from the training set.
- embargo_pct (float): Percentage of test set duration to embargo from subsequent training sets.
Outputs:
- None: Initializes the object.
"""
if not isinstance(t_series, pd.Series):
raise ValueError("t_series must be a pandas Series of timestamps.")
if not (1 <= n_test_folds < n_folds):
raise ValueError("n_test_folds must be less than n_folds and at least 1.")
self.n_folds = n_folds
self.n_test_folds = n_test_folds
self.t_series = pd.to_datetime(t_series).sort_values().reset_index(drop=True)
self.purge_pct = purge_pct
self.embargo_pct = embargo_pct
self.n_samples = len(self.t_series)
def split(self, X, y=None, groups=None):
"""
Generates indices for Combinatorial Purged K-Fold splits.
Inputs:
- X (array-like): The data to be split.
- y (array-like, optional): Target variable.
- groups (array-like, optional): Group labels.
Outputs:
- Yields (train_indices, test_indices) tuples for each combinatorial split.
Formulas Used:
- The `PurgedKFold` logic is applied internally to generate the initial `n_folds` splits.
- Combinations are generated using `itertools.combinations`:
`combinations(range(self.n_folds), self.n_test_folds)`
"""
if len(X) != self.n_samples:
raise ValueError("Length of X must match length of t_series.")
# Step 1: Generate initial purged K-folds
# We will treat each of these as 'base' folds that can be combined.
base_purger = PurgedKFold(n_splits=self.n_folds, t_series=self.t_series,
purge_pct=self.purge_pct, embargo_pct=self.embargo_pct)
base_splits = []
for train_idx, test_idx in base_purger.split(X):
base_splits.append({'train': train_idx, 'test': test_idx})
# Step 2: Generate combinatorial splits
# We select n_test_folds to be the test set, and the rest as train.
for test_fold_indices_combo in itertools.combinations(range(self.n_folds), self.n_test_folds):
current_test_indices = np.array([], dtype=int)
current_train_indices = np.array([], dtype=int)
# Identify indices for the current combinatorial test set
for fold_idx in test_fold_indices_combo:
current_test_indices = np.union1d(current_test_indices, base_splits[fold_idx]['test'])
# Identify indices for the current combinatorial training set
# This is all indices NOT in the current test set, AND respecting initial purging/embargoing
all_indices = np.arange(self.n_samples)
candidate_train_indices = np.setdiff1d(all_indices, current_test_indices)
# Re-apply purging and embargoing based on the combined test set
# For simplicity, we define a combined test time window for purging/embargoing
# A more rigorous implementation would track individual purge/embargo windows of base folds
if len(current_test_indices) == 0: continue
combined_test_start_time = self.t_series.iloc[current_test_indices.min()]
combined_test_end_time = self.t_series.iloc[current_test_indices.max()]
combined_test_duration = combined_test_end_time - combined_test_start_time
# Calculate purge and embargo durations for this combined test
purge_duration = combined_test_duration * self.purge_pct / (1 - self.purge_pct) if self.purge_pct > 0 else pd.Timedelta(seconds=0)
embargo_duration = combined_test_duration * self.embargo_pct / (1 - self.embargo_pct) if self.embargo_pct > 0 else pd.Timedelta(seconds=0)
purge_start_time = combined_test_start_time - purge_duration
purge_end_time = combined_test_end_time + purge_duration
embargo_end_time = combined_test_end_time + embargo_duration
# Apply Purging to the candidate training set
purge_mask = (
(self.t_series.iloc[candidate_train_indices] >= purge_start_time) &
(self.t_series.iloc[candidate_train_indices] <= purge_end_time)
)
train_indices_purged = candidate_train_indices[~purge_mask]
# Apply Embargoing to the purged training set
embargo_mask = (
(self.t_series.iloc[train_indices_purged] > combined_test_end_time) &
(self.t_series.iloc[train_indices_purged] <= embargo_end_time)
)
final_train_indices = train_indices_purged[~embargo_mask]
yield final_train_indices, current_test_indices
4.1 Visualization of Combinatorial Purged K-Fold Splits
Now let's visualize how the CombinatorialPurgedKFold generates multiple unique train/test splits, each adhering to the purging and embargoing rules. We'll set n_folds=5 and n_test_folds=1 to create 5 distinct splits where one fold acts as the test set.
combo_purger = CombinatorialPurgedKFold(n_folds=5, n_test_folds=1,
t_series=time_series_data['timestamp'],
purge_pct=0.1, embargo_pct=0.02)
combo_splits = list(combo_purger.split(time_series_data))
print(f"Generated {len(combo_splits)} combinatorial splits.")
plot_splits(time_series_data['timestamp'], combo_splits, 'Visualization of Combinatorial Purged K-Fold Splits')
Generated 5 combinatorial splits.
Interpretation of Visualization 4.1:
This plot demonstrates the combinatorial aspect. Each horizontal bar represents a unique train-test split generated by CombinatorialPurgedKFold.
- Notice how the red test sets shift across different parts of the timeline for each split.
- For each red test set, corresponding orange purged regions and green embargoed regions are created, ensuring that the blue training sets are free from data leakage.
- The key takeaway is that the model is being evaluated on different combinations of test data, providing a more comprehensive and statistically robust assessment of its performance compared to a single sequential or simple k-fold approach.
5. Practical Example: Evaluating a Simple Model with CPCV
Let's demonstrate how to use CombinatorialPurgedKFold to evaluate a simple machine learning model (e.g., Logistic Regression) on our mock time series data. We'll compare its performance across the different combinatorial splits.
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
# Prepare data
X = time_series_data[['feature_1']].values
y = time_series_data['target'].values
# Initialize CPCV
cpcv = CombinatorialPurgedKFold(n_folds=5, n_test_folds=1,
t_series=time_series_data['timestamp'],
purge_pct=0.1, embargo_pct=0.02)
# Store results
accuracies = []
precisions = []
recalls = []
f1_scores = []
# Iterate through splits
for i, (train_idx, test_idx) in enumerate(cpcv.split(X, y)):
print(f"\n--- Combinatorial Split {i+1} ---")
print(f"Train set size: {len(train_idx)}, Test set size: {len(test_idx)}")
if len(train_idx) == 0 or len(test_idx) == 0:
print("Skipping split due to empty train or test set after purging/embargoing.")
continue
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# Train a simple Logistic Regression model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate performance
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, zero_division=0) # Handle no positive predictions
recall = recall_score(y_test, y_pred, zero_division=0)
f1 = f1_score(y_test, y_pred, zero_division=0)
accuracies.append(accuracy)
precisions.append(precision)
recalls.append(recall)
f1_scores.append(f1)
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")
# Aggregate and report overall performance
if accuracies:
print("\n--- Overall CPCV Performance ---")
print(f"Average Accuracy: {np.mean(accuracies):.4f} (+/- {np.std(accuracies):.4f})")
print(f"Average Precision: {np.mean(precisions):.4f} (+/- {np.std(precisions):.4f})")
print(f"Average Recall: {np.mean(recalls):.4f} (+/- {np.std(recalls):.4f})")
print(f"Average F1-Score: {np.mean(f1_scores):.4f} (+/- {np.std(f1_scores):.4f})")
else:
print("No valid splits were processed.")
--- Combinatorial Split 1 --- Train set size: 78, Test set size: 20 Accuracy: 0.3500 Precision: 0.3500 Recall: 1.0000 F1-Score: 0.5185 --- Combinatorial Split 2 --- Train set size: 76, Test set size: 20 Accuracy: 0.3000 Precision: 0.0000 Recall: 0.0000 F1-Score: 0.0000 --- Combinatorial Split 3 --- Train set size: 76, Test set size: 20 Accuracy: 0.3500 Precision: 0.3500 Recall: 1.0000 F1-Score: 0.5185 --- Combinatorial Split 4 --- Train set size: 76, Test set size: 20 Accuracy: 0.4500 Precision: 0.0000 Recall: 0.0000 F1-Score: 0.0000 --- Combinatorial Split 5 --- Train set size: 78, Test set size: 20 Accuracy: 0.4000 Precision: 0.2500 Recall: 0.1000 F1-Score: 0.1429 --- Overall CPCV Performance --- Average Accuracy: 0.3700 (+/- 0.0510) Average Precision: 0.1900 (+/- 0.1594) Average Recall: 0.4200 (+/- 0.4750) Average F1-Score: 0.2360 (+/- 0.2365)
Interpretation of Model Evaluation:
The output above shows the performance metrics (Accuracy, Precision, Recall, F1-Score) for each combinatorial split and then provides an average and standard deviation across all splits.
- Per-split metrics: Observing the metrics for each individual split gives insight into how robust the model is to different data partitions.
- Average metrics: The mean of the metrics across all splits provides a more reliable estimate of the model's expected out-of-sample performance.
- Standard Deviation: The standard deviation indicates the variability of the model's performance across different splits. A high standard deviation might suggest that the model's performance is highly dependent on the specific data it's tested on, potentially indicating instability or a lack of generalization, even with purging and embargoing in place.
This practical example demonstrates that CPCV provides a more rigorous backtesting framework than traditional cross-validation methods by explicitly accounting for serial correlation and potential data leakage in time series datasets.
6. Conclusion
Combinatorial Purged Cross-Validation (CPCV) is an essential technique for robust model evaluation in fields dealing with time-dependent and serially correlated data, such as quantitative finance. By integrating purging, embargoing, and combinatorial sampling, CPCV effectively mitigates look-ahead bias and over-optimistic performance estimates that plague standard cross-validation methods.
Key Takeaways:
- Prevents Data Leakage: Purging and embargoing are critical for ensuring that training data does not implicitly contain information from the test set.
- Robust Performance Estimation: The combinatorial approach provides a more stable and reliable assessment of model performance across varied data segments, reducing the risk of overfitting to specific market regimes.
- Increased Confidence: By rigorously testing a model under conditions that simulate truly unseen data, CPCV enhances confidence in the model's out-of-sample generalization capabilities.
While more computationally intensive than simpler cross-validation schemes, the benefits of CPCV in providing a realistic evaluation of predictive models in time series applications often outweigh the additional cost.