Signals·Machine Learning Models·Advanced

Isolation Forest Anomaly

Use isolation forest anomaly detection on price, volume, and derived feature data to identify statistically unusual market behavior that may indicate manipulation, regime transitions, or high-impact trading opportunities.

machine-learningtrading-signals

Isolation Forest Anomaly Detection

Introduction to Isolation Forest

Isolation Forest is an unsupervised machine learning algorithm used for anomaly detection. Unlike many other anomaly detection algorithms that try to model normal points, Isolation Forest explicitly identifies anomalies (outliers) as points that are 'easy to isolate'. It is particularly effective for high-dimensional datasets and works by building an ensemble of isolation trees.

Purpose

The main purpose of Isolation Forest is to detect anomalies efficiently and effectively. Anomalies are data points that deviate significantly from the majority of the data, often indicating critical incidents, errors, or rare events.

Importance

Anomaly detection is crucial in various fields:

Fraud Detection: Identifying unusual transactions.
Cybersecurity: Detecting network intrusions or malware.
Manufacturing: Spotting defective products.
Healthcare: Finding unusual patient data that might indicate a disease.
IT Operations: Monitoring system performance for unusual behavior.

Isolation Forest's efficiency and ability to handle high-dimensional data make it a valuable tool in these domains.

How Isolation Forest Works

The core idea behind Isolation Forest is that anomalies are few and different, making them more susceptible to isolation than normal points. It constructs a collection of isolation trees (iTrees) based on random sub-samples of the data.

Isolation Trees (iTrees)

An iTree is a binary tree where each node is split by randomly selecting a feature and then randomly selecting a split value within the range of that feature. The partitioning continues recursively until each instance is isolated or a maximum tree depth is reached.

Anomaly Score

The anomaly score for an instance is calculated based on the number of splits required to isolate it. Normal instances require more splits (longer paths in the tree) to be isolated, while anomalies, being 'different', typically require fewer splits (shorter paths).

Short path length: Indicates an anomaly.
Long path length: Indicates a normal data point.

By averaging the path lengths across all iTrees in the forest, the algorithm assigns an anomaly score to each data point. A higher score signifies a higher likelihood of being an anomaly.

Code Demonstration: Implementing Isolation Forest

[1]

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.ensemble import IsolationForest
from sklearn.datasets import make_blobs

# Set random seed for reproducibility
np.random.seed(42)

1. Generating Sample Data

We will create a synthetic 2D dataset with two clusters of normal data points and a few scattered anomaly points. This will help visualize how Isolation Forest identifies these outliers.

[2]

# Generate 'normal' data points (two clusters)
X, _ = make_blobs(n_samples=300, centers=[[2, 2], [-2, -2]], cluster_std=0.8, random_state=42)

# Generate 'anomaly' data points (scattered)
X_anomalies = np.random.uniform(low=-5, high=5, size=(30, 2))

# Combine normal and anomaly data
X_combined = np.vstack([X, X_anomalies])

df = pd.DataFrame(X_combined, columns=['Feature_1', 'Feature_2'])

print("Shape of the combined dataset:", df.shape)
display(df.head())

Shape of the combined dataset: (330, 2)

	Feature_1	Feature_2
0	1.046957	2.525243
1	1.564494	2.088738
2	-1.903763	-1.588449
3	1.214793	2.369683
4	2.046567	1.085624

2. Training the Isolation Forest Model

We will initialize the IsolationForest model. Key parameters include:

n_estimators: The number of trees in the forest.
contamination: The proportion of outliers in the dataset. This is used to determine the threshold for anomaly scores. It's often estimated or set based on domain knowledge.
random_state: For reproducibility.

The model is trained on the combined dataset.

[3]

# Initialize Isolation Forest model
# contamination is the proportion of outliers in the dataset (estimated)
model = IsolationForest(n_estimators=100, contamination=0.1, random_state=42)

# Train the model
model.fit(df)

print("Isolation Forest model trained successfully.")

Isolation Forest model trained successfully.

3. Calculating Anomaly Scores and Predictions

After training, we can use the model to:

decision_function(): Compute the anomaly score for each sample. Lower values indicate more anomalous samples.
predict(): Predict if a sample is an outlier (returns -1) or an inlier (returns 1). This prediction is based on the contamination parameter.

[4]

# Get anomaly scores
df['anomaly_score'] = model.decision_function(df[['Feature_1', 'Feature_2']])

# Predict anomalies (-1 for outliers, 1 for inliers)
df['is_anomaly'] = model.predict(df[['Feature_1', 'Feature_2']])

print("Anomaly scores and predictions added to the DataFrame.")
display(df.head())

Anomaly scores and predictions added to the DataFrame.

	Feature_1	Feature_2	anomaly_score	is_anomaly
0	1.046957	2.525243	0.092761	1
1	1.564494	2.088738	0.124135	1
2	-1.903763	-1.588449	0.132242	1
3	1.214793	2.369683	0.098781	1
4	2.046567	1.085624	0.114338	1

Visualizations

Visualization 1: Data Distribution with Detected Anomalies

This scatter plot shows the original data points, colored according to whether they were classified as normal (inliers) or anomalies by the Isolation Forest model. This allows us to visually inspect how well the model separated the synthetic anomalies from the normal clusters.

[5]

plt.figure(figsize=(10, 7))

# Plot normal points
plt.scatter(df[df['is_anomaly'] == 1]['Feature_1'],
            df[df['is_anomaly'] == 1]['Feature_2'],
            c='blue', label='Normal (Inlier)', s=50, alpha=0.7, edgecolors='w')

# Plot anomaly points
plt.scatter(df[df['is_anomaly'] == -1]['Feature_1'],
            df[df['is_anomaly'] == -1]['Feature_2'],
            c='red', label='Anomaly (Outlier)', s=100, alpha=0.8, edgecolors='k', marker='X')

plt.title('Isolation Forest Anomaly Detection (2D Data)')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()

# Interpretation:
# The blue points represent the data identified as normal, forming distinct clusters.
# The red 'X' marks represent the data identified as anomalies. As expected, these are scattered
# away from the main clusters, demonstrating the model's ability to isolate them.

Visualization 2: Distribution of Anomaly Scores

This histogram displays the distribution of anomaly scores assigned by the Isolation Forest. Anomalies are expected to have lower decision_function values (or higher raw anomaly scores, depending on the implementation's sign convention). This visualization helps understand the separation between normal and anomalous scores and confirms that detected anomalies indeed have distinct scores.

[6]

plt.figure(figsize=(10, 6))

# Plot histogram of anomaly scores
sns.histplot(df['anomaly_score'], bins=50, kde=True, color='purple')

# Highlight the threshold based on contamination
# Note: IsolationForest doesn't have threshold_ attribute directly; we can compute approximate threshold
# or visualize using the contamination percentile
threshold = np.percentile(df['anomaly_score'], 10)  # 10th percentile since contamination=0.1
plt.axvline(x=threshold, color='red', linestyle='--', label=f'Anomaly Threshold ({threshold:.2f})')

plt.title('Distribution of Anomaly Scores')
plt.xlabel('Anomaly Score (lower = more anomalous)')
plt.ylabel('Frequency')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()

# Interpretation:
# The histogram shows a bimodal-like distribution. The majority of points (normal) have higher anomaly scores,
# while a smaller group of points (anomalies) have significantly lower scores. The red dashed line indicates the approximate threshold
# used by the model to classify points as anomalies or normal based on the `contamination` parameter. Points to the left
# of this line are classified as anomalies (-1).

Advantages and Disadvantages of Isolation Forest

Advantages

Efficiency: It has a low linear time complexity, making it suitable for large datasets.
Scalability: Performs well in high-dimensional datasets without requiring density or distance measures.
Effectiveness: Does not require a distance metric, which can be challenging in high dimensions.
Handles Irregular Shapes: Unlike distance-based methods, Isolation Forest can handle non-spherical clusters.
Explicit Anomaly Detection: It directly isolates anomalies instead of profiling normal data, which can be more effective when anomalies are rare.

Disadvantages

Sensitivity to Contamination Parameter: The contamination parameter significantly impacts the results and often requires prior knowledge or careful tuning.
Randomness: Due to its random nature, results can vary slightly between runs if random_state is not set.
Not Ideal for Cluster-Based Anomalies: If anomalies form their own small, dense clusters, Isolation Forest might not perform as well, as they would still require many splits to be isolated.
Interpretation: While anomaly scores are provided, understanding why a point is anomalous beyond its short path length can sometimes be less intuitive than with other methods.

Conclusion

Isolation Forest is a powerful and efficient algorithm for unsupervised anomaly detection. Its unique approach of explicitly isolating anomalies by leveraging their distinct characteristics makes it a valuable tool, especially for large and high-dimensional datasets. By understanding its principles, implementation, and interpreting its results, practitioners can effectively identify unusual patterns across various applications.