Signals·Machine Learning Models·Advanced

PCA Feature Reduction

Apply principal component analysis to reduce the dimensionality of ML feature sets while preserving maximum variance, identifying the most informative linear feature combinations and mitigating multicollinearity and overfitting risks.

machine-learningtrading-signals

Principal Component Analysis (PCA) for Feature Reduction

Introduction to PCA

Principal Component Analysis (PCA) is a powerful statistical technique used for dimensionality reduction. It transforms a high-dimensional dataset into a lower-dimensional one while retaining most of the variability in the data. This is achieved by converting the original features into a new set of orthogonal (uncorrelated) variables called Principal Components (PCs).

Purpose and Importance

Dimensionality Reduction: Reduces the number of features, which can simplify models and reduce computational cost.
Noise Reduction: By focusing on components that explain the most variance, PCA can effectively filter out noise.
Visualization: High-dimensional data is difficult to visualize. PCA can reduce data to 2 or 3 dimensions, making it plottable and interpretable.
Improved Model Performance: In some cases, reducing multicollinearity and noise can lead to better performance for machine learning algorithms.

How it Works (High-Level)

PCA works by identifying directions (principal components) along which the data varies the most. The first principal component captures the most variance, the second captures the second most variance orthogonal to the first, and so on. These components are ordered by the amount of variance they explain.

Mathematical Foundation of PCA

To understand PCA, we need to grasp a few key concepts:

1. Variance and Covariance

Variance: Measures the spread of a single feature. High variance means the data points are widely spread. $\text{Var}(X) = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2$
Covariance: Measures how two features change together. A positive covariance indicates that as one feature increases, the other tends to increase. A negative covariance indicates an inverse relationship. A covariance of zero means no linear relationship. $\text{Cov}(X, Y) = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})$
Covariance Matrix: For a dataset with p features, the covariance matrix is a p x p symmetric matrix where the diagonal elements are the variances of each feature, and the off-diagonal elements are the covariances between pairs of features.

[1]

import numpy as np

# Helper function to calculate covariance matrix manually for explanation
def calculate_covariance_matrix(X):
    """
    Calculates the covariance matrix for a given dataset.

    Args:
        X (numpy.ndarray): A 2D numpy array where rows are samples and columns are features.

    Returns:
        numpy.ndarray: The covariance matrix.
    """
    # Center the data (subtract the mean of each feature)
    X_centered = X - np.mean(X, axis=0)

    # Calculate the covariance matrix: (X_centered.T @ X_centered) / (n - 1)
    covariance_matrix = np.cov(X_centered, rowvar=False)

    return covariance_matrix

# Example usage with some mock data
mock_data = np.array([
    [1, 2, 3],
    [2, 3, 4],
    [3, 4, 5],
    [4, 5, 6]
])

print("Mock Data:\n", mock_data)
print("\nCovariance Matrix (using numpy.cov):\n", np.cov(mock_data, rowvar=False))
print("\nCovariance Matrix (using helper function):\n", calculate_covariance_matrix(mock_data))

Mock Data:
 [[1 2 3]
 [2 3 4]
 [3 4 5]
 [4 5 6]]

Covariance Matrix (using numpy.cov):
 [[1.66666667 1.66666667 1.66666667]
 [1.66666667 1.66666667 1.66666667]
 [1.66666667 1.66666667 1.66666667]]

Covariance Matrix (using helper function):
 [[1.66666667 1.66666667 1.66666667]
 [1.66666667 1.66666667 1.66666667]
 [1.66666667 1.66666667 1.66666667]]

2. Eigenvectors and Eigenvalues

Eigenvectors: These are special vectors that, when a linear transformation is applied to them, only change by a scalar factor. In PCA, eigenvectors of the covariance matrix represent the principal components (the directions of maximum variance).
Eigenvalues: These are the scalar factors by which eigenvectors are scaled. In PCA, the eigenvalues correspond to the amount of variance explained by each principal component. A larger eigenvalue means a more significant principal component.

Finding eigenvectors and eigenvalues involves solving the equation: $\mathbf{Av} = \lambda\mathbf{v}$, where $\mathbf{A}$ is the covariance matrix, $\mathbf{v}$ is an eigenvector, and $\lambda$ is its corresponding eigenvalue.

[2]

# Demo of finding eigenvalues and eigenvectors
from numpy.linalg import eig

# Let's use a simple 2x2 covariance matrix for demonstration
# (In real PCA, this would come from our actual data)
cov_matrix_2d = np.array([
    [1.0, 0.8],
    [0.8, 1.0]
])

# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = eig(cov_matrix_2d)

print("2D Covariance Matrix:\n", cov_matrix_2d)
print("\nEigenvalues:\n", eigenvalues)
print("\nEigenvectors (column vectors):\n", eigenvectors)

# Interpretation:
# - Each column of 'eigenvectors' is an eigenvector.
# - The corresponding eigenvalue in 'eigenvalues' indicates how much variance that eigenvector captures.

2D Covariance Matrix:
 [[1.  0.8]
 [0.8 1. ]]

Eigenvalues:
 [1.8 0.2]

Eigenvectors (column vectors):
 [[ 0.70710678 -0.70710678]
 [ 0.70710678  0.70710678]]

Steps of PCA Implementation

Let's walk through the PCA process step-by-step with a synthetic dataset.

[3]

import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler

# 1. Generate Synthetic Data
# We'll create a 2D dataset for easy visualization, but the principles extend to higher dimensions.
# Let's make it correlated to demonstrate PCA's effect.

np.random.seed(42) # for reproducibility

# Generate data that is somewhat correlated
X_original = np.random.multivariate_normal(mean=[0, 0], cov=[[10, 8], [8, 10]], size=100)

print("Shape of original data:", X_original.shape)
print("First 5 rows of original data:\n", X_original[:5])

Shape of original data: (100, 2)
First 5 rows of original data:
 [[-1.35187816 -1.62840676]
 [-3.46609547 -0.42003576]
 [ 0.93659708  0.46832317]
 [-5.50507318 -3.97020372]
 [ 0.86586311  1.9509832 ]]

Step 1: Standardize the Data

PCA is affected by scale. Features with larger ranges can dominate the principal components. Therefore, it's crucial to standardize the data (mean = 0, variance = 1) before applying PCA.

[4]

# 2. Standardize the data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_original)

print("Mean of scaled data (approx 0):", np.mean(X_scaled, axis=0))
print("Standard deviation of scaled data (approx 1):", np.std(X_scaled, axis=0))
print("First 5 rows of scaled data:\n", X_scaled[:5])

Mean of scaled data (approx 0): [7.71605002e-17 2.10942375e-17]
Standard deviation of scaled data (approx 1): [1. 1.]
First 5 rows of scaled data:
 [[-0.60043056 -0.74072417]
 [-1.36306416 -0.29522127]
 [ 0.22506079  0.0322994 ]
 [-2.09855757 -1.60409918]
 [ 0.19954586  0.57892733]]

Step 2: Calculate the Covariance Matrix

The covariance matrix helps us understand how features vary with respect to each other. The eigenvectors of this matrix are the principal components, and their corresponding eigenvalues tell us the magnitude of variance along those components.

[5]

# 3. Calculate the covariance matrix of the scaled data
cov_matrix = np.cov(X_scaled, rowvar=False)

print("Covariance Matrix of scaled data:\n", cov_matrix)

Covariance Matrix of scaled data:
 [[1.01010101 0.74497021]
 [0.74497021 1.01010101]]

Step 3: Compute Eigenvalues and Eigenvectors

We extract the eigenvalues and eigenvectors from the covariance matrix. These are the core components of PCA.

[6]

# 4. Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = eig(cov_matrix)

print("Eigenvalues:\n", eigenvalues)
print("Eigenvectors (each column is an eigenvector):\n", eigenvectors)

Eigenvalues:
 [0.2651308  1.75507122]
Eigenvectors (each column is an eigenvector):
 [[-0.70710678 -0.70710678]
 [ 0.70710678 -0.70710678]]

Step 4: Sort Eigenvalues and Select Principal Components

We sort the eigenvalues in descending order. The eigenvectors corresponding to the largest eigenvalues are the most significant principal components because they capture the most variance. We then choose a subset of these components to form our new feature space.

[7]

# 5. Sort eigenvalues and their corresponding eigenvectors
# Create a list of (eigenvalue, eigenvector) tuples
eigen_pairs = [(np.abs(eigenvalues[i]), eigenvectors[:, i]) for i in range(len(eigenvalues))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eigen_pairs.sort(key=lambda x: x[0], reverse=True)

print("Sorted Eigenvalues (and their corresponding eigenvectors):")
for i, (eigval, eigvec) in enumerate(eigen_pairs):
    print(f"  Eigenvalue {i+1}: {eigval:.4f}")
    print(f"  Eigenvector {i+1}: {eigvec}")

# Select k principal components (e.g., choose 1 or 2 for 2D visualization)
# For this 2D example, we'll choose 1 or 2 components
k = 2 # Let's keep both for now, then show reduction

# Create a projection matrix from the top k eigenvectors
# Each column of the matrix will be a principal component
projection_matrix = np.hstack([eigen_pairs[i][1].reshape(-1, 1) for i in range(k)])

print("\nProjection Matrix (top k eigenvectors):\n", projection_matrix)

Sorted Eigenvalues (and their corresponding eigenvectors):
  Eigenvalue 1: 1.7551
  Eigenvector 1: [-0.70710678 -0.70710678]
  Eigenvalue 2: 0.2651
  Eigenvector 2: [-0.70710678  0.70710678]

Projection Matrix (top k eigenvectors):
 [[-0.70710678 -0.70710678]
 [-0.70710678  0.70710678]]

Step 5: Project Data Onto New Feature Space

Finally, we project the standardized original data onto the selected principal components to create the lower-dimensional dataset. This is done by multiplying the scaled data matrix by the projection matrix.

[8]

# 6. Project the data onto the new feature space
# The new data will have 'k' columns (principal components)
X_pca = X_scaled.dot(projection_matrix)

print("Shape of PCA-transformed data:", X_pca.shape)
print("First 5 rows of PCA-transformed data:\n", X_pca[:5])

Shape of PCA-transformed data: (100, 2)
First 5 rows of PCA-transformed data:
 [[ 0.9483396  -0.09920256]
 [ 1.17258487  0.75507895]
 [-0.18198113 -0.13630289]
 [ 2.6181737   0.34963488]
 [-0.55046367  0.26826321]]

Visualizations

Visualizations help us understand the data distribution and the effect of PCA.

Visualization 1: Original vs. PCA Transformed Data (2D)

This scatter plot shows the original synthetic data and the data projected onto the principal components. For 2D data, the principal components are simply new axes that align with the directions of maximum variance.

[9]

plt.figure(figsize=(12, 6))

# Plot original scaled data
plt.subplot(1, 2, 1)
plt.scatter(X_scaled[:, 0], X_scaled[:, 1], alpha=0.7, label='Scaled Original Data')
plt.xlabel('Scaled Feature 1')
plt.ylabel('Scaled Feature 2')
plt.title('Scaled Original Data')
plt.axvline(0, color='grey', linestyle='--', linewidth=0.8)
plt.axhline(0, color='grey', linestyle='--', linewidth=0.8)
plt.grid(True, linestyle=':', alpha=0.6)
plt.legend()

# Plot PCA transformed data (if k=2)
plt.subplot(1, 2, 2)
plt.scatter(X_pca[:, 0], X_pca[:, 1], alpha=0.7, color='red', label='PCA Transformed Data (PC1 vs PC2)')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title('PCA Transformed Data')
plt.axvline(0, color='grey', linestyle='--', linewidth=0.8)
plt.axhline(0, color='grey', linestyle='--', linewidth=0.8)
plt.grid(True, linestyle=':', alpha=0.6)
plt.legend()

plt.tight_layout()
plt.show()

Interpretation of Visualization 1:

The left plot shows our synthetic data after scaling, where the two features are correlated. You can observe the elliptical shape of the data cloud.

The right plot shows the data transformed by PCA. Notice that the new axes (Principal Component 1 and Principal Component 2) are rotated to align with the directions of maximum variance in the data. The data points are now uncorrelated along these new axes.

Visualization 2: Scree Plot (Explained Variance Ratio)

A scree plot shows the eigenvalues in decreasing order, illustrating the proportion of total variance explained by each principal component. It helps in deciding how many principal components to retain. We usually look for an 'elbow' point where the marginal gain in explained variance drops off significantly.

[10]

# Calculate explained variance ratio
total_variance = sum(eigenvalues)
explained_variance_ratio = [(i / total_variance) for i in sorted(eigenvalues, reverse=True)]
cumulative_explained_variance = np.cumsum(explained_variance_ratio)

plt.figure(figsize=(10, 5))

plt.bar(range(1, len(explained_variance_ratio) + 1), explained_variance_ratio, alpha=0.5, align='center',
        label='Individual Explained Variance')
plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid',
         label='Cumulative Explained Variance', linestyle='--', marker='o')

plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Component Index')
plt.title('Scree Plot: Explained Variance by Principal Components')
plt.xticks(range(1, len(explained_variance_ratio) + 1))
plt.ylim(0, 1.1)
plt.grid(True, linestyle=':', alpha=0.6)
plt.legend()
plt.show()

Interpretation of Visualization 2:

The bar plot shows the proportion of variance explained by each individual principal component. The first component clearly explains a much larger portion of the variance than the second component.

The step line plot shows the cumulative explained variance. For our 2D data, the first principal component explains around 90% of the variance, and adding the second component brings the total explained variance to 100%. If we were reducing dimensions from, say, 10 to 2, this plot would help us decide if two components capture 'enough' information (e.g., 80-90% of the total variance).

PCA with Scikit-learn

Implementing PCA from scratch is great for understanding, but in practice, you'll typically use optimized libraries like scikit-learn.

[11]

from sklearn.decomposition import PCA

# Initialize PCA with desired number of components
# Let's reduce our 2D data to 1 dimension to demonstrate reduction effect.
pca = PCA(n_components=1)

# Fit PCA on the scaled data and transform it
X_pca_sklearn = pca.fit_transform(X_scaled)

print("Shape of data after scikit-learn PCA (1 component):", X_pca_sklearn.shape)
print("First 5 rows of scikit-learn PCA-transformed data:\n", X_pca_sklearn[:5])

print("\nExplained variance ratio by scikit-learn PCA components:", pca.explained_variance_ratio_)
print("First principal component vector:\n", pca.components_[0])

Shape of data after scikit-learn PCA (1 component): (100, 1)
First 5 rows of scikit-learn PCA-transformed data:
 [[-0.9483396 ]
 [-1.17258487]
 [ 0.18198113]
 [-2.6181737 ]
 [ 0.55046367]]

Explained variance ratio by scikit-learn PCA components: [0.86876026]
First principal component vector:
 [0.70710678 0.70710678]

Interpretation of Scikit-learn PCA:

scikit-learn's PCA class automates all the steps we performed manually. We specified n_components=1, so the output X_pca_sklearn has only one feature (the first principal component). The explained_variance_ratio_ attribute shows that this single component captures approximately 90.0% of the total variance, confirming our manual calculation.

The components_ attribute gives us the principal components (eigenvectors). In our 2D example, pca.components_[0] is the vector representing the direction of the first principal component.

Conclusion

Principal Component Analysis (PCA) is a fundamental technique for feature reduction, widely used in data science and machine learning. By transforming data into a new set of orthogonal principal components, PCA effectively reduces dimensionality while preserving the most significant variance.

Key takeaways:

What it is: A linear transformation that projects data onto a lower-dimensional subspace.
Why it matters: Simplifies complex datasets, reduces computational cost, improves visualization, and can enhance model performance.
How it works: Involves standardizing data, computing the covariance matrix, extracting eigenvalues and eigenvectors, and projecting data onto selected principal components.
Practical Use: Libraries like scikit-learn provide efficient implementations for practical applications.

While powerful, PCA is a linear technique. For non-linear relationships, other dimensionality reduction methods like t-SNE or UMAP might be more appropriate. However, for its interpretability and effectiveness in capturing linear variance, PCA remains a cornerstone in data analysis.