Live Trading·Trading Infrastructure·Intermediate

Heartbeat Monitor

Build a comprehensive system-wide heartbeat monitoring framework that continuously checks all critical system components are alive and responsive with configurable health check probes for each service, automatically alerting on any detected component failure or degradation for immediate operations response.

live-tradingmonitoring

Monitoring System Heartbeat/Health

Introduction: What is System Heartbeat/Health Monitoring?

System heartbeat/health monitoring is a critical practice in maintaining the reliability and availability of software systems and infrastructure. It involves periodically checking the operational status of a system component (e.g., a server, a service, an application process) to ensure it is alive, responsive, and functioning as expected.

Definition

A heartbeat is a periodic signal generated by a system component to indicate that it is still running. If a monitoring system fails to receive a heartbeat within an expected timeframe, it assumes the component is unhealthy or has failed.

Purpose

The primary purposes of heartbeat monitoring are:

  1. Early Detection of Failures: Identify system outages or degradations promptly.
  2. Proactive Alerts: Notify administrators or automated systems when issues arise.
  3. System Recovery: Trigger automated recovery mechanisms (e.g., restarting a service).
  4. Performance Baseline: Provide data for understanding normal system behavior and identifying anomalies.

Importance

In today's interconnected and always-on environments, system downtime can lead to significant financial losses, reputational damage, and user dissatisfaction. Heartbeat monitoring acts as a vital early warning system, enabling rapid response to failures and contributing to higher system uptime and overall reliability.

Basic Concept: How Heartbeat Signals Work

At its core, a heartbeat mechanism involves two main parts:

  1. The Sender (Monitored System): Periodically sends a message (the heartbeat) indicating its liveness.
  2. The Receiver (Monitoring System): Waits for heartbeats from registered senders. If a heartbeat is not received within a predefined timeout, the sender is marked as unhealthy or offline.

Let's simulate a simple heartbeat sender.

[22]
import time
import datetime
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

print('All imports successful.')
All imports successful.
[23]
def generate_heartbeat(system_id: str, timestamp: datetime.datetime = None) -> dict:
    """
    Simulates generating a single heartbeat signal.

    Inputs:
    - system_id (str): A unique identifier for the system sending the heartbeat.
    - timestamp (datetime, optional): The time to record for the heartbeat. If None, uses datetime.datetime.now().

    Outputs:
    - dict: A dictionary containing the heartbeat data.
      - 'timestamp' (datetime): The time the heartbeat was generated.
      - 'system_id' (str): The ID of the system.
      - 'status' (str): The current status, typically 'ALIVE'.
    """
    if timestamp is None:
        timestamp = datetime.datetime.now()

    return {
        'timestamp': timestamp,
        'system_id': system_id,
        'status': 'ALIVE'
    }

# Demonstrate generating a heartbeat
sample_heartbeat = generate_heartbeat(system_id="AuthService-01")
print(f"Generated Heartbeat: {sample_heartbeat}")
Generated Heartbeat: {'timestamp': datetime.datetime(2026, 6, 18, 8, 24, 59, 406229), 'system_id': 'AuthService-01', 'status': 'ALIVE'}

Explanation of generate_heartbeat Function

  • Inputs:
    • system_id: A string that uniquely identifies the system component. This is crucial for the monitoring system to know who is sending the heartbeat.
    • interval_seconds: The expected period between consecutive heartbeats. While not directly used in generating a single heartbeat, it's vital for the monitoring component to establish a timeout.
  • Outputs:
    • A Python dictionary containing the current timestamp, the system's ID, and a 'status' indicating it's alive. This structure represents the minimal information a heartbeat signal might carry.
  • Formula/Logic: The function simply captures the current time using datetime.datetime.now() and packages it with the provided system_id and a static 'ALIVE' status.

Implementing a Heartbeat Monitoring System

Now, let's build the other side: a monitoring system that receives these heartbeats and determines the health of the monitored systems. This involves:

  1. Storing last known heartbeat times.
  2. Periodically checking if any system's last heartbeat is older than a predefined timeout.

We'll simulate multiple systems sending heartbeats and a central monitor checking their health.

[24]
class HeartbeatMonitor:
    """
    A class to simulate a central heartbeat monitoring system.
    It keeps track of the last received heartbeat for each system_id.
    """
    def __init__(self, heartbeat_timeout_seconds: int = 10):
        """
        Inputs:
        - heartbeat_timeout_seconds (int): Max time allowed without a heartbeat
          before a system is considered DOWN.
        """
        self.last_heartbeats = {}
        self.heartbeat_timeout = datetime.timedelta(seconds=heartbeat_timeout_seconds)
        print(f"Heartbeat Monitor initialized with timeout: {heartbeat_timeout_seconds} seconds.")

    def record_heartbeat(self, heartbeat_data: dict):
        """Records a received heartbeat."""
        self.last_heartbeats[heartbeat_data['system_id']] = heartbeat_data['timestamp']

    def get_system_status(self, system_id: str, check_time: datetime.datetime = None) -> str:
        """
        Returns 'UP', 'DOWN', or 'UNKNOWN' for a given system_id.
        Optionally, check_time can be provided to evaluate status at a specific point in time.
        """
        if check_time is None:
            check_time = datetime.datetime.now() # Use real-time if not provided

        if system_id not in self.last_heartbeats:
            return 'UNKNOWN'
        elapsed = check_time - self.last_heartbeats[system_id]
        return 'DOWN' if elapsed > self.heartbeat_timeout else 'UP'

    def get_all_system_statuses(self, check_time: datetime.datetime = None) -> dict:
        """
        Returns status of all monitored systems.
        Optionally, check_time can be provided to evaluate status at a specific point in time.
        """
        if check_time is None:
            check_time = datetime.datetime.now() # Use real-time if not provided
        return {sid: self.get_system_status(sid, check_time=check_time) for sid in self.last_heartbeats}

print('HeartbeatMonitor class defined.')
HeartbeatMonitor class defined.
[25]
# Initialize the monitor with a 5-second timeout
monitor = HeartbeatMonitor(heartbeat_timeout_seconds=5)

# Define mock systems
systems = ['WebServer-01', 'Database-01', 'PaymentService-01']

# Use a single clean status_log with ONE row per (tick, system).
status_log = []  # One row per (tick, system_id)

print("--- Simulating Heartbeats (20 ticks × 1s) ---\n")

# Establish a fixed start time for the simulation to control 'current_time'
simulated_start_time = datetime.datetime.now().replace(microsecond=0) # Round to second for cleaner output
simulated_tick_duration = datetime.timedelta(seconds=1)

for i in range(20): # Increased simulation duration to 20 ticks
    real_time_sleep = 0.5 # A small real-world delay to see output incrementally
    time.sleep(real_time_sleep)

    # Calculate the current simulated time for this tick
    current_sim_time = simulated_start_time + (i * simulated_tick_duration)
    print(f"Simulated Time: {current_sim_time.strftime('%H:%M:%S')}")

    # Step 1: Send heartbeats (with intentional failures)
    for system_id in systems:
        # Database-01 goes offline during simulated ticks 7–12 (6 seconds)
        if system_id == 'Database-01' and 7 <= i <= 12:
            pass
        # PaymentService-01 goes offline from simulated tick 14 onward
        elif system_id == 'PaymentService-01' and i >= 14:
            pass
        else:
            # Pass current_sim_time to generate_heartbeat for consistent timestamps
            monitor.record_heartbeat(generate_heartbeat(system_id, timestamp=current_sim_time))

    # Step 2: Record ONE status row per system AFTER all heartbeats processed
    for system_id in systems:
        # Pass the current_sim_time to get_system_status for accurate historical status
        status = monitor.get_system_status(system_id, check_time=current_sim_time)
        status_log.append({
            'time': current_sim_time,
            'system_id': system_id,
            'monitor_status': status
        })
        print(f"  {system_id}: {status}")

print("\n--- Simulation Complete ---")

# Build clean DataFrame — no mixed rows, no ffill needed
status_df = pd.DataFrame(status_log)

print(f"\nDataFrame shape: {status_df.shape}")
print(status_df.head(9))

print("\nLast recorded heartbeats:")
for sys_id, last_hb_time in monitor.last_heartbeats.items():
    print(f"  {sys_id}: {last_hb_time}")

print("\nFinal System Statuses:")
# Pass the last current_sim_time from the loop to get the final status at the end of the simulation
print(monitor.get_all_system_statuses(check_time=current_sim_time))
Heartbeat Monitor initialized with timeout: 5 seconds.
--- Simulating Heartbeats (20 ticks × 1s) ---

Simulated Time: 08:24:59
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:00
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:01
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:02
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:03
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:04
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:05
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:06
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:07
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:08
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:09
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:10
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:11
  WebServer-01: UP
  Database-01: DOWN
  PaymentService-01: UP
Simulated Time: 08:25:12
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:13
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:14
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:15
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:16
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:17
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: UP
Simulated Time: 08:25:18
  WebServer-01: UP
  Database-01: UP
  PaymentService-01: DOWN

--- Simulation Complete ---

DataFrame shape: (60, 3)
                 time          system_id monitor_status
0 2026-06-18 08:24:59       WebServer-01             UP
1 2026-06-18 08:24:59        Database-01             UP
2 2026-06-18 08:24:59  PaymentService-01             UP
3 2026-06-18 08:25:00       WebServer-01             UP
4 2026-06-18 08:25:00        Database-01             UP
5 2026-06-18 08:25:00  PaymentService-01             UP
6 2026-06-18 08:25:01       WebServer-01             UP
7 2026-06-18 08:25:01        Database-01             UP
8 2026-06-18 08:25:01  PaymentService-01             UP

Last recorded heartbeats:
  WebServer-01: 2026-06-18 08:25:18
  Database-01: 2026-06-18 08:25:18
  PaymentService-01: 2026-06-18 08:25:12

Final System Statuses:
{'WebServer-01': 'UP', 'Database-01': 'UP', 'PaymentService-01': 'DOWN'}

Explanation of HeartbeatMonitor Class and Demonstration

  • Inputs:
    • heartbeat_timeout_seconds (for __init__): Defines how long a system can go without sending a heartbeat before being declared 'DOWN'.
    • heartbeat_data (for record_heartbeat): The dictionary generated by generate_heartbeat.
    • system_id (for get_system_status): The identifier of the system to check.
  • Outputs:
    • record_heartbeat: Updates the internal last_heartbeats dictionary.
    • get_system_status: Returns 'UP', 'DOWN', or 'UNKNOWN' for a given system.
    • get_all_system_statuses: Returns a dictionary of all monitored systems' statuses.
  • Formulas/Logic:
    • Timeout Calculation: self.heartbeat_timeout = datetime.timedelta(seconds=heartbeat_timeout_seconds) converts the integer seconds into a timedelta object for easy comparison with time differences.
    • Status Determination: time_since_last_hb = datetime.datetime.now() - last_hb_time calculates the elapsed time since the last heartbeat. If time_since_last_hb is greater than self.heartbeat_timeout, the system is considered 'DOWN'.

Visualizing System Health

Visualizations are crucial for quickly understanding the state of multiple systems over time or at a glance. They can highlight trends, identify outages, and provide a dashboard-like view of overall system health.

Visualization 1: Heartbeat Status Timeline for a Single System

This plot shows the monitored status ('UP' or 'DOWN') of a specific system over the simulated time. It helps to visually identify periods when a system was considered unhealthy by the monitor.

[27]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(15, 8))

# Get unique system IDs from the status_df
all_systems = status_df['system_id'].unique()

# Define a color palette for different systems using plt.cm.get_cmap
colors_palette = plt.cm.get_cmap('tab10', len(all_systems))

for i, system_id in enumerate(all_systems):
    single_system_df = status_df[status_df['system_id'] == system_id].copy()

    # Map status to numeric for plotting
    single_system_df['status_numeric'] = single_system_df['monitor_status'].map(
        {'UP': 1, 'DOWN': 0, 'UNKNOWN': np.nan}
    )

    # Plot each system's status timeline
    ax.plot(
        single_system_df['time'],
        single_system_df['status_numeric'],
        drawstyle='steps-post',
        marker='o', markersize=4, linewidth=2,
        color=colors_palette(i), label=f'{system_id} Status'
    )

    # Shade DOWN periods in red for each system
    down_rows = single_system_df[single_system_df['monitor_status'] == 'DOWN']
    for _, row in down_rows.iterrows():
        ax.axvspan(
            row['time'],
            row['time'] + pd.Timedelta(seconds=1), # Shade for 1 second duration
            color='red', alpha=0.15 # Use a lighter alpha for overlaying multiple systems
        )

ax.set_title('Monitored Heartbeat Status Over Time for All Systems', fontsize=14)
ax.set_xlabel('Simulated Time')
ax.set_ylabel('Status')
ax.set_yticks([0, 1])
ax.set_yticklabels(['DOWN (0)', 'UP (1)'], fontsize=11)
ax.set_ylim(-0.3, 1.4)
ax.grid(True, linestyle='--', alpha=0.6)
ax.legend(loc='upper right', bbox_to_anchor=(1.25, 1))
plt.xticks(rotation=20)
plt.tight_layout()
plt.show()
/tmp/ipykernel_4705/383987717.py:11: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.
  colors_palette = plt.cm.get_cmap('tab10', len(all_systems))
cell output

Interpretation of Visualization 1

This line plot clearly shows the state of Database-01 as perceived by the monitoring system. We can observe:

  • The periods where the system is 'UP' (value 1) and 'DOWN' (value 0).
  • The red shaded areas highlight the exact time intervals when the monitor declared Database-01 as 'DOWN' due to missed heartbeats. This corresponds to our simulation where Database-01 intentionally stopped sending heartbeats between 7 and 10 seconds into the simulation.

This type of visualization is essential for incident response, allowing engineers to quickly pinpoint when an issue started and potentially when it was resolved.

Visualization 2: Current System Health Dashboard (Bar Chart)

This visualization provides a snapshot of the health status of all monitored systems at a specific point in time (the end of our simulation). It's useful for a quick overview of the overall system landscape.

[28]
import pandas as pd
import matplotlib.pyplot as plt

# Get the final status for each system from the status_df
# This ensures we use the status as recorded at the last simulated time point
# The .last() method on a groupby object correctly retrieves the last non-null entry for each group
final_statuses = status_df.groupby('system_id')['monitor_status'].last()

plt.figure(figsize=(10, 6))
colors = {'UP': 'green', 'DOWN': 'red', 'UNKNOWN': 'gray'}

# Ensure systems are sorted for consistent plotting if desired
final_statuses = final_statuses.sort_index()

bar_colors = [colors[status] for status in final_statuses.values]

# Create a Series with numeric values for plotting, keeping the system IDs as index
# The height of each bar will be 1, and the color will indicate the status.
plot_data = pd.Series([1] * len(final_statuses), index=final_statuses.index)

plot_data.plot(kind='bar', color=bar_colors)

plt.title('Current Health Status of All Monitored Systems (at end of simulation)')
plt.xlabel('System ID')
plt.ylabel('Status')
plt.yticks([]) # Hide y-axis ticks as labels are on bars
plt.xticks(rotation=45, ha='right')
plt.grid(axis='y', linestyle='--', alpha=0.6)

# Add text labels for status on top of bars
for index, system_id in enumerate(final_statuses.index):
    status_value = final_statuses.loc[system_id]
    # Position the text slightly above the bar (height 1)
    plt.text(index, 0.5, status_value, ha='center', va='center', color='white', fontweight='bold', fontsize=10)

plt.ylim(0, 1.2) # Adjust y-limit to fit the bar and text
plt.tight_layout()
plt.show()
cell output

Interpretation of Visualization 2

This bar chart acts as a simple dashboard. Each bar represents a system, and its color and text label immediately convey its health status:

  • Green ('UP'): The system is currently sending heartbeats within the expected timeout.
  • Red ('DOWN'): The system has failed to send a heartbeat within the configured timeout period.
  • Gray ('UNKNOWN'): No heartbeats have ever been received from this system (not explicitly shown in this demo with 'UNKNOWN' status, but would be an important addition for real-world scenarios where systems might not have reported yet).

From this chart, we can quickly see that WebServer-01 is 'UP', while Database-01 and PaymentService-01 are 'DOWN' at the end of the simulation, aligning with our intentional failures in the simulation.

Beyond Heartbeats: Other Health Metrics

While heartbeats are excellent for detecting liveness, a comprehensive system health monitor often includes other metrics to understand why a system might be struggling even if it's technically 'ALIVE'. These can include:

  • CPU Utilization: High CPU might indicate a bottleneck or runaway process.
  • Memory Usage: Excessive memory consumption can lead to slowdowns or crashes.
  • Disk I/O: Slow disk operations can affect application performance.
  • Network Latency/Throughput: Network issues can make a healthy service appear unresponsive.
  • Application-Specific Metrics: e.g., request per second, error rates, queue lengths.

These metrics are typically collected by agents on the monitored systems and sent to a centralized monitoring platform for analysis and alerting.

Conclusion

System heartbeat/health monitoring is a foundational aspect of robust system operations. By implementing mechanisms to regularly check the liveness of components and visualize their status, organizations can significantly improve their ability to detect, diagnose, and respond to incidents, ultimately leading to more stable and reliable services.

This notebook demonstrated the basic principles of generating and monitoring heartbeats, along with simple visualizations to interpret system health. Real-world monitoring systems are far more complex, incorporating distributed architectures, sophisticated alerting, and integration with incident management tools, but the core concept remains the same: know when your systems are alive and well, and react quickly when they are not.