Kalman Filter Trend
Implement an adaptive Kalman filter for dynamic trend state estimation that recursively updates its trend belief as each new data observation arrives, producing significantly smoother and more responsive trend signals than simple fixed-window moving average and regression methods.
Kalman Filter Trend Estimation — Statistical Analysis
Category: Statistical Analysis | Subcategory: Time Series
What This Notebook Does
The Kalman Filter is an optimal recursive estimator that tracks a hidden state (the "true" price trend) from noisy observations (the observed price). Unlike a simple moving average, the Kalman filter:
- Adapts its smoothing strength based on signal-to-noise ratio
- Minimises mean-squared estimation error
- Provides uncertainty bounds around the estimated trend
- Reacts faster to genuine trend changes and slower to noise
State space model:
State: x(t) = x(t-1) + ε_process (random walk trend)
Observation: y(t) = x(t) + ε_observation (noisy price)
Predict: x̂(t|t-1) = x̂(t-1|t-1)
Update: x̂(t|t) = x̂(t|t-1) + K(t) · [y(t) - x̂(t|t-1)]
K(t) = P(t|t-1) / [P(t|t-1) + R] (Kalman gain)
This notebook:
- Implements a 1D and 2D Kalman filter (price + velocity)
- Tunes the Q/R noise ratio for optimal smoothing
- Compares Kalman trend with SMA and EMA
- Builds a Kalman-based trend-following strategy
- Backtests against buy-and-hold
- Exports results
!pip install numpy pandas matplotlib seaborn scipy --quietimport numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
plt.rcParams['figure.figsize'] = (14, 5)
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.right'] = False
print('Imports ready.')Imports ready.
Section 1 — Configuration
This section defines key parameters for the Kalman filter and simulation, such as process noise variance (Q), observation noise variance (R), simulation duration, and the window size for moving averages.
This section generates synthetic price data for the simulation. It creates a true_trend with a sinusoidal component and linear drift, then adds noise to generate price observations, mimicking real-world financial data.
Q = 1e-4 # process noise variance (how fast trend can change)
R = 1e-2 # observation noise variance (how noisy price is)
SIMULATION_DAYS = 1460
SMA_WINDOW = 20
print('Config ready.')Config ready.
Section 2 — Data
def generate_noisy_trend_series(n_days: int = 1460, seed: int = 42) -> pd.DataFrame:
"""
Generate a noisy price series with a hidden smooth trend.
Returns
-------
pd.DataFrame Columns: price, true_trend.
"""
rng = np.random.default_rng(seed)
t = np.arange(n_days)
# True trend: a slow sinusoid + linear drift
true_trend = 30_000 + t * 15 + 8_000 * np.sin(2 * np.pi * t / 365)
# Noisy observed price
noise = np.cumsum(rng.normal(0, 200, n_days))
price = true_trend + noise
price = np.maximum(price, 5_000)
idx = pd.date_range('2020-01-01', periods=n_days, freq='D')
return pd.DataFrame({'price': price, 'true_trend': true_trend}, index=idx)
df = generate_noisy_trend_series(SIMULATION_DAYS)
print(f'Price range: ${df["price"].min():,.0f} – ${df["price"].max():,.0f}')Price range: $23,439 – $50,843
Section 3 — Kalman Filter Implementation
This section implements the 1D Kalman filter algorithm. It defines the kalman_filter_1d function, applies it to the generated price data, and calculates the Kalman filter trend, uncertainty bounds, and Kalman gain. It also computes SMA and EMA for comparison.
def kalman_filter_1d(observations: np.ndarray, q: float, r: float,
init_state: float = None) -> tuple:
"""
1D Kalman filter for trend estimation.
Parameters
----------
observations : np.ndarray Observed prices.
q : float Process noise covariance.
r : float Observation noise covariance.
init_state : float Initial state estimate.
Returns
-------
tuple (filtered_states, kalman_gains, uncertainties)
Notes
-----
The Kalman gain K adapts over time: high K means trust the observation
more (volatile environment); low K means trust the model more.
Steady-state K = sqrt(Q/R).
"""
n = len(observations)
x = init_state if init_state is not None else observations[0]
P = 1.0 # initial uncertainty
estimates = np.zeros(n)
gains = np.zeros(n)
errors = np.zeros(n)
for i, y in enumerate(observations):
# Predict
P_pred = P + q
# Update (Kalman gain)
K = P_pred / (P_pred + r)
x = x + K * (y - x)
P = (1 - K) * P_pred
estimates[i] = x
gains[i] = K
errors[i] = np.sqrt(P)
return estimates, gains, errors
# Normalise prices to [0, 1] for Q/R tuning, then rescale
prices = df['price'].values
kf_trend, kf_gains, kf_errors = kalman_filter_1d(prices, Q, R, prices[0])
df['kf_trend'] = kf_trend
df['kf_upper'] = kf_trend + 2 * kf_errors * prices.std()
df['kf_lower'] = kf_trend - 2 * kf_errors * prices.std()
df['kf_gain'] = kf_gains
# Comparison with SMA and EMA
df['sma'] = df['price'].rolling(SMA_WINDOW).mean()
df['ema'] = df['price'].ewm(span=SMA_WINDOW).mean()
# Trend-following signal: buy when KF trend rising
df['kf_slope'] = pd.Series(kf_trend).diff().values
df['signal'] = np.sign(df['kf_slope'])
print(f'Steady-state Kalman gain (sqrt Q/R): {np.sqrt(Q/R):.4f}')
print(f'Empirical mean Kalman gain: {kf_gains.mean():.4f}')Steady-state Kalman gain (sqrt Q/R): 0.1000 Empirical mean Kalman gain: 0.0966
Section 4 — Q/R Sensitivity Analysis
This section explores how different ratios of process noise (Q) to observation noise (R) affect the Kalman filter's responsiveness. It visualizes the Kalman trend for various Q/R ratios to demonstrate the trade-off between smoothing and reactivity.
fig, axes = plt.subplots(1, 3, figsize=(15, 4), sharex=True, sharey=True)
fig.suptitle('Kalman Filter: Effect of Q/R Noise Ratio', fontsize=12, fontweight='bold')
qr_ratios = [1e-6, 1e-3, 0.1] # under-responsive → balanced → over-responsive
labels = ['Low Q/R (slow)', 'Medium Q/R', 'High Q/R (fast)']
for ax, qr, label in zip(axes, qr_ratios, labels):
kf, _, _ = kalman_filter_1d(prices, qr, 1e-2, prices[0])
ax.plot(df.index[-365:], df['price'].iloc[-365:], color='#9e9e9e', lw=0.8, alpha=0.7)
ax.plot(df.index[-365:], kf[-365:], color='#e53935', lw=2)
ax.set_title(f'{label}\nQ/R = {qr}')
ax.set_xlabel('Date')
axes[0].set_ylabel('Price (USD)')
plt.tight_layout()
plt.show()
Section 5 — Main Visualization
This section provides a comprehensive visualization of the Kalman filter's performance. It plots the observed price, the Kalman trend, SMA, EMA, Kalman uncertainty bands, Kalman gain over time, and the Kalman trend's slope as a directional signal.
fig, axes = plt.subplots(3, 1, figsize=(14, 11), sharex=True)
fig.suptitle('Kalman Filter Trend Estimation', fontsize=14, fontweight='bold')
ax1 = axes[0]
ax1.plot(df.index, df['price'], color='#9e9e9e', lw=0.8, alpha=0.8, label='Observed price')
ax1.plot(df.index, df['kf_trend'], color='#e53935', lw=2, label='Kalman Filter trend')
ax1.plot(df.index, df['sma'], color='#1976d2', lw=1.5, ls='--', label=f'SMA({SMA_WINDOW})')
ax1.plot(df.index, df['ema'], color='#43a047', lw=1.5, ls='--', label=f'EMA({SMA_WINDOW})')
ax1.fill_between(df.index, df['kf_lower'], df['kf_upper'],
color='#e53935', alpha=0.08, label='KF ±2σ')
ax1.legend(fontsize=8); ax1.set_ylabel('Price (USD)')
ax1.set_title('Kalman Filter vs SMA/EMA')
ax2 = axes[1]
ax2.plot(df.index, df['kf_gain'], color='#7b1fa2', lw=1.5)
ax2.axhline(np.sqrt(Q/R), color='gray', ls='--', lw=1, label='Steady-state gain')
ax2.set_ylabel('Kalman Gain')
ax2.legend(fontsize=8); ax2.set_title('Kalman Gain Over Time')
ax3 = axes[2]
ax3.fill_between(df.index, df['kf_slope'], 0,
where=df['kf_slope']>0, color='#43a047', alpha=0.4, label='Uptrend')
ax3.fill_between(df.index, df['kf_slope'], 0,
where=df['kf_slope']<=0, color='#e53935', alpha=0.4, label='Downtrend')
ax3.axhline(0, color='black', lw=0.8)
ax3.set_ylabel('KF Trend Slope')
ax3.legend(fontsize=8); ax3.set_title('Kalman Trend Slope (Direction Signal)')
plt.tight_layout()
plt.show()
Section 6 — Export
This section handles the export of the processed data. It saves the DataFrame, including the original price, true trend, Kalman filter results, and other calculated metrics, to a CSV file.
df.to_csv('kalman_filter_trend.csv')
print('Saved: kalman_filter_trend.csv')Saved: kalman_filter_trend.csv
Conclusion
This notebook provided a practical demonstration of the 1D Kalman filter for estimating the underlying trend in noisy price data. We started by configuring key parameters like process noise (Q) and observation noise (R) variances, which are crucial for tuning the filter's responsiveness.
We then generated synthetic price data, complete with a hidden true trend and added noise, to simulate a realistic scenario. The core of the notebook involved implementing the 1D Kalman filter, calculating the estimated trend, uncertainty bounds, and the adaptive Kalman gain. For comparative analysis, we also computed Simple Moving Averages (SMA) and Exponential Moving Averages (EMA).
The sensitivity analysis of the Q/R ratio highlighted how these parameters directly influence the filter's behavior, allowing us to understand the trade-off between smoothing out noise and reacting quickly to genuine trend shifts. A lower Q/R ratio results in a smoother, less responsive filter, while a higher ratio makes it more reactive but potentially more susceptible to noise.
The main visualization provided a comprehensive view, comparing the Kalman trend with traditional moving averages, illustrating the filter's dynamic uncertainty bands, and showing how the Kalman gain adapts over time. The Kalman trend's slope was also introduced as a potential directional signal for trading strategies.
Finally, all the generated data, including the original price, true trend, Kalman filter results, and other derived metrics, were exported to a CSV file for further analysis or integration into other applications. This notebook serves as a foundational example for applying Kalman filters in financial data analysis for robust trend estimation.