Alerts·Alert Type Implementations·Intermediate

System Error Alert

Implement comprehensive system error alerting that triggers immediate multi-channel notifications on unhandled application exceptions, exchange API communication failures, WebSocket stream disconnections, and data quality anomalies with full error diagnostic context and suggested remediation steps for rapid operations response.

alertsnotifications

Notifications and Alerts: Alert on System Errors

This notebook demonstrates how to implement a system for monitoring simulated system metrics and triggering alerts when predefined thresholds are breached. The core idea is to continuously observe operational data (e.g., CPU usage, memory consumption, error rates) and notify stakeholders when anomalies or critical conditions occur. This helps in proactive identification and resolution of potential system issues, ensuring reliability and performance.

Core Concepts

Concept	Description	Key Mechanism
Metrics Collection	Gathering quantitative data about system performance and health (e.g., CPU, Memory, Disk I/O).	Simulated data generation, time-series data.
Alert Rules	Predefined conditions or thresholds that, when met or exceeded by metrics, indicate a problem.	Dictionary-based rule definition.
Alert Evaluation	The process of continuously checking collected metrics against defined alert rules.	Iterative checking, stateful evaluation.
Notification System	Mechanism to deliver alerts to relevant personnel or systems (e.g., email, SMS, logging).	Simulated logging, print statements.
State Management	Maintaining the current status of the system and alerts (e.g., alert triggered, alert resolved).	Dictionary-based state for metrics and alerts.
Debouncing/Throttling	Preventing excessive notifications for fluctuating metrics or recurring issues.	Cooldown periods, event aggregation.
Exponential Backoff	Strategy for retrying failed operations with progressively longer delays.	`time.sleep` with increasing intervals.

Dependency Installation

We'll install faker for generating realistic-looking data and loguru for advanced logging capabilities.

[1]

# Install necessary libraries
!pip install faker loguru pandas matplotlib seaborn --quiet

[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m2.0/2.0 MB[0m [31m4.6 MB/s[0m eta [36m0:00:00[0m
[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m61.6/61.6 kB[0m [31m1.8 MB/s[0m eta [36m0:00:00[0m
[?25h

Library Imports

This section includes all necessary library imports, organized by standard libraries first, followed by third-party libraries.

[2]

# Standard library imports
import collections
import datetime
import math
import random
import time

# Third-party library imports
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from faker import Faker
from loguru import logger

Core Functions

This section defines the core functions required for metric generation, rule definition, alert evaluation, and notification handling. Each function is presented in its own block with a comprehensive docstring, type hints, and logger statements.

Function Name: `create_system_state`

This function initializes the system's operational state, including current metrics and a history for tracking. It sets up initial values for CPU usage, memory usage, and a counter for system errors. It also initializes a deque to maintain a rolling window of recent metrics for trend analysis.

Parameters:

buffer_size (int): The maximum number of historical metric entries to keep.

Returns:

dict: An initialized dictionary representing the system's current state.

[3]

def create_system_state(buffer_size: int = 10) -> dict:
    """
    Initializes the system's operational state dictionary.

    Parameters
    ----------
    buffer_size : int, optional
        The maximum number of historical metric entries to keep in the rolling window,
        defaults to 10.

    Returns
    -------
    dict
        An initialized dictionary representing the system's current state with
        metrics, history, and alert status.

    Examples
    --------
    >>> state = create_system_state(buffer_size=5)
    >>> assert 'metrics' in state
    >>> assert 'history' in state
    >>> assert 'active_alerts' in state
    """
    state = {
        'metrics': {
            'cpu_usage': 0.0,  # Percentage
            'memory_usage': 0.0, # Percentage
            'error_rate': 0.0, # Percentage of operations failing
            'response_time': 0, # Milliseconds
            'network_latency': 0, # Milliseconds
        },
        'history': collections.deque(maxlen=buffer_size),
        'active_alerts': {},
        'alert_cooldowns': {},
        'last_metric_time': None
    }
    logger.info(f"System state initialized with buffer size: {buffer_size}")
    return state

Function Name: `simulate_metrics`

This function simulates new system metrics, adding a small amount of random jitter to previous values to mimic real-world fluctuations. It updates CPU usage, memory usage, error rate, response time, and network latency based on the provided state and current time. The jitter ensures that metrics are not perfectly smooth, making the simulation more realistic.

Parameters:

state (dict): The current system state dictionary.
current_time (datetime.datetime): The timestamp for the new metrics.

Returns:

dict: The updated system state dictionary with new simulated metrics.

[4]

def simulate_metrics(state: dict, current_time: datetime.datetime) -> dict:
    """
    Simulates new system metrics with random fluctuations.

    Parameters
    ----------
    state : dict
        The current system state dictionary.
    current_time : datetime.datetime
        The timestamp for the new metrics.

    Returns
    -------
    dict
        The updated system state dictionary with new simulated metrics.

    Examples
    --------
    >>> state = create_system_state()
    >>> current_time = datetime.datetime.now()
    >>> updated_state = simulate_metrics(state, current_time)
    >>> assert updated_state['last_metric_time'] == current_time
    """
    logger.debug("Simulating new metrics...")

    # Add random jitter to previous metrics
    previous_metrics = state['metrics']
    jitter = lambda val: max(0, min(100, val + random.uniform(-5, 5)))
    time_jitter = lambda val: max(0, val + random.randint(-20, 20))

    new_cpu = jitter(previous_metrics['cpu_usage'])
    new_memory = jitter(previous_metrics['memory_usage'])
    new_error_rate = max(0.0, min(100.0, previous_metrics['error_rate'] + random.uniform(-1, 2)))
    new_response_time = time_jitter(previous_metrics['response_time'])
    new_network_latency = time_jitter(previous_metrics['network_latency'])

    state['metrics'] = {
        'cpu_usage': new_cpu,
        'memory_usage': new_memory,
        'error_rate': new_error_rate,
        'response_time': new_response_time,
        'network_latency': new_network_latency,
    }
    state['last_metric_time'] = current_time
    state['history'].append({'time': current_time, **state['metrics']})
    logger.info(f"Metrics simulated: {state['metrics']}")
    return state

Function Name: `define_alert_rules`

This function creates a dictionary of alert rules. Each rule specifies a metric, a threshold, a comparison operator, a severity level, and a cooldown period. This structured approach allows for flexible and easily configurable alert conditions.

Parameters:

None

Returns:

dict: A dictionary where keys are rule names and values are rule definitions.

[5]

def define_alert_rules() -> dict:
    """
    Defines a set of alert rules for various system metrics.

    Each rule includes:
    - `metric`: The name of the metric to monitor.
    - `threshold`: The value against which the metric is compared.
    - `operator`: The comparison operator (e.g., '>', '<', '>=', '<=').
    - `severity`: The severity level of the alert (e.g., 'WARNING', 'CRITICAL').
    - `cooldown`: Time in seconds before the same alert can be re-triggered.

    Returns
    -------
    dict
        A dictionary where keys are rule names and values are rule definitions.

    Examples
    --------
    >>> rules = define_alert_rules()
    >>> assert 'High CPU Usage' in rules
    >>> assert rules['High CPU Usage']['metric'] == 'cpu_usage'
    """
    rules = {
        'High CPU Usage': {
            'metric': 'cpu_usage',
            'threshold': 80.0,
            'operator': '>',
            'severity': 'CRITICAL',
            'cooldown': 300  # 5 minutes
        },
        'High Memory Usage': {
            'metric': 'memory_usage',
            'threshold': 90.0,
            'operator': '>',
            'severity': 'CRITICAL',
            'cooldown': 300
        },
        'Elevated Error Rate': {
            'metric': 'error_rate',
            'threshold': 5.0,
            'operator': '>',
            'severity': 'WARNING',
            'cooldown': 180 # 3 minutes
        },
        'Critical Error Rate': {
            'metric': 'error_rate',
            'threshold': 15.0,
            'operator': '>',
            'severity': 'CRITICAL',
            'cooldown': 300
        },
        'High Response Time': {
            'metric': 'response_time',
            'threshold': 500, # ms
            'operator': '>',
            'severity': 'WARNING',
            'cooldown': 120 # 2 minutes
        },
        'Critical Response Time': {
            'metric': 'response_time',
            'threshold': 1000, # ms
            'operator': '>',
            'severity': 'CRITICAL',
            'cooldown': 300
        },
        'High Network Latency': {
            'metric': 'network_latency',
            'threshold': 200, # ms
            'operator': '>',
            'severity': 'WARNING',
            'cooldown': 120
        }
    }
    logger.info(f"Defined {len(rules)} alert rules.")
    return rules

Function Name: `evaluate_alert_rules`

This function evaluates the current system metrics against a set of predefined alert rules. It checks each metric against its corresponding threshold using the specified operator. If a rule is triggered and not currently in a cooldown period, an alert is activated. If an active alert's condition is no longer met, it is resolved. This function manages the lifecycle of alerts.

Parameters:

state (dict): The current system state dictionary, which will be updated with active and resolved alerts.
rules (dict): A dictionary of alert rule definitions.
current_time (datetime.datetime): The current timestamp for alert evaluation.

Returns:

dict: The updated system state dictionary, including changes to active_alerts and alert_cooldowns.

[6]

def evaluate_alert_rules(state: dict, rules: dict, current_time: datetime.datetime) -> dict:
    """
    Evaluates current metrics against defined alert rules and manages alert states.

    Parameters
    ----------
    state : dict
        The current system state dictionary, which will be updated with active and resolved alerts.
    rules : dict
        A dictionary of alert rule definitions.
    current_time : datetime.datetime
        The current timestamp for alert evaluation.

    Returns
    -------
    dict
        The updated system state dictionary, including changes to `active_alerts` and `alert_cooldowns`.

    Examples
    --------
    >>> state = create_system_state()
    >>> rules = define_alert_rules()
    >>> current_time = datetime.datetime.now()
    >>> state['metrics']['cpu_usage'] = 90.0 # Simulate high CPU
    >>> updated_state = evaluate_alert_rules(state, rules, current_time)
    >>> assert 'High CPU Usage' in updated_state['active_alerts']
    """
    logger.debug("Evaluating alert rules...")
    current_metrics = state['metrics']
    triggered_alerts = []

    # Resolve alerts that are no longer active
    alerts_to_resolve = []
    for alert_name, alert_info in state['active_alerts'].items():
        rule = rules.get(alert_name)
        if not rule:
            logger.warning(f"Active alert '{alert_name}' has no corresponding rule. Resolving it.")
            alerts_to_resolve.append(alert_name)
            continue

        metric_value = current_metrics.get(rule['metric'])
        if metric_value is None:
            logger.warning(f"Metric '{rule['metric']}' for alert '{alert_name}' not found. Resolving it.")
            alerts_to_resolve.append(alert_name)
            continue

        # Check if the condition is still met. If not, resolve the alert.
        is_condition_met = False
        operator = rule['operator']
        threshold = rule['threshold']
        if operator == '>':
            is_condition_met = metric_value > threshold
        elif operator == '<':
            is_condition_met = metric_value < threshold
        elif operator == '>=':
            is_condition_met = metric_value >= threshold
        elif operator == '<=':
            is_condition_met = metric_value <= threshold
        elif operator == '==':
            is_condition_met = metric_value == threshold
        elif operator == '!=':
            is_condition_met = metric_value != threshold

        if not is_condition_met:
            alerts_to_resolve.append(alert_name)
            logger.info(f"Alert '{alert_name}' (severity: {rule['severity']}) resolved at {current_time}.")

    for alert_name in alerts_to_resolve:
        if alert_name in state['active_alerts']:
            del state['active_alerts'][alert_name]
        # Optionally, you could reset cooldown here if an alert resolving implies cooldown ends
        # Or, maintain cooldown to prevent immediate re-triggering after brief resolution

    # Check for new alerts
    for alert_name, rule in rules.items():
        metric_value = current_metrics.get(rule['metric'])

        if metric_value is None:
            logger.warning(f"Metric '{rule['metric']}' for rule '{alert_name}' not found. Skipping evaluation.")
            continue

        is_triggered = False
        operator = rule['operator']
        threshold = rule['threshold']

        if operator == '>':
            is_triggered = metric_value > threshold
        elif operator == '<':
            is_triggered = metric_value < threshold
        elif operator == '>=':
            is_triggered = metric_value >= threshold
        elif operator == '<=':
            is_triggered = metric_value <= threshold
        elif operator == '==':
            is_triggered = metric_value == threshold
        elif operator == '!=':
            is_triggered = metric_value != threshold

        if is_triggered:
            # Check cooldown
            last_triggered_time = state['alert_cooldowns'].get(alert_name, datetime.datetime.min)
            if (current_time - last_triggered_time).total_seconds() > rule['cooldown']:
                if alert_name not in state['active_alerts']:
                    alert_info = {
                        'time': current_time,
                        'metric': rule['metric'],
                        'value': metric_value,
                        'threshold': threshold,
                        'operator': operator,
                        'severity': rule['severity']
                    }
                    state['active_alerts'][alert_name] = alert_info
                    state['alert_cooldowns'][alert_name] = current_time
                    triggered_alerts.append((alert_name, alert_info))
                    logger.warning(f"ALERT TRIGGERED: {alert_name} (Severity: {rule['severity']}, Value: {metric_value}) at {current_time}")
                else:
                    logger.debug(f"Alert '{alert_name}' already active. No re-triggering.")
            else:
                logger.debug(f"Alert '{alert_name}' is in cooldown. Next trigger possible after {last_triggered_time + datetime.timedelta(seconds=rule['cooldown'])}")

    return state

Function Name: `send_notification`

This function simulates sending a notification for a triggered alert. In a real-world scenario, this would integrate with external services like email, SMS, or Slack. For this demonstration, it simply logs the alert details to the console, formatted according to its severity.

Parameters:

alert_name (str): The name of the alert.
alert_info (dict): A dictionary containing details about the triggered alert.

Returns:

None

[7]

def send_notification(alert_name: str, alert_info: dict) -> None:
    """
    Simulates sending a notification for a triggered alert.

    In a real system, this function would integrate with actual notification services
    like email, SMS, or a messaging platform.

    Parameters
    ----------
    alert_name : str
        The name of the alert that was triggered.
    alert_info : dict
        A dictionary containing details about the triggered alert, e.g., time, metric,
        value, threshold, operator, severity.

    Returns
    -------
    None

    Examples
    --------
    >>> alert_details = {
    ...    'time': datetime.datetime.now(),
    ...    'metric': 'cpu_usage',
    ...    'value': 95.0,
    ...    'threshold': 80.0,
    ...    'operator': '>',
    ...    'severity': 'CRITICAL'
    ... }
    >>> send_notification('High CPU Alert', alert_details)
    """
    severity = alert_info['severity']
    message = (
        f"[{severity}] Alert: {alert_name} - "
        f"Metric '{alert_info['metric']}' has value {alert_info['value']:.2f} "
        f"which {alert_info['operator']} {alert_info['threshold']:.2f} at {alert_info['time']}."
    )

    if severity == 'CRITICAL':
        logger.critical(message)
    elif severity == 'WARNING':
        logger.warning(message)
    else:
        logger.info(message)
    logger.debug(f"Notification sent for alert: {alert_name}")

Function Name: `run_monitor_cycle_with_retry`

This function executes a single monitoring cycle, including simulating metrics, evaluating rules, and sending notifications. It incorporates a retry mechanism with exponential backoff for robustness against transient errors. If any step within the cycle fails, it will reattempt the operation up to a maximum number of retries, increasing the delay between attempts with random jitter.

Parameters:

state (dict): The current system state dictionary.
rules (dict): A dictionary of alert rule definitions.
max_retries (int): The maximum number of retry attempts.
base_delay (float): The base delay in seconds for exponential backoff.

Returns:

dict: The updated system state dictionary after the monitoring cycle.

Raises:

Exception: If the operation fails after all retries.

[8]

def run_monitor_cycle_with_retry(state: dict, rules: dict, max_retries: int = 3, base_delay: float = 1.0) -> dict:
    """
    Executes a single monitoring cycle with a retry mechanism using exponential backoff.

    This function combines metric simulation, rule evaluation, and notification sending.
    It attempts to run these steps and retries on failure with increasing delays.

    Parameters
    ----------
    state : dict
        The current system state dictionary.
    rules : dict
        A dictionary of alert rule definitions.
    max_retries : int, optional
        The maximum number of retry attempts, defaults to 3.
    base_delay : float, optional
        The base delay in seconds for exponential backoff, defaults to 1.0.

    Returns
    -------
    dict
        The updated system state dictionary after the monitoring cycle.

    Raises
    -------
    Exception
        If the operation fails after all retries.

    Examples
    --------
    >>> state = create_system_state()
    >>> rules = define_alert_rules()
    >>> try:
    >>>     state = run_monitor_cycle_with_retry(state, rules)
    >>> except Exception as e:
    >>>     print(f"Monitoring cycle failed: {e}")
    """
    for attempt in range(max_retries):
        try:
            logger.info(f"Starting monitoring cycle (Attempt {attempt + 1}/{max_retries})...")
            current_time = datetime.datetime.now()

            # Simulate metrics
            state = simulate_metrics(state, current_time)

            # Introduce a random chance of internal error for demonstration
            if random.random() < 0.05: # 5% chance of failure
                if attempt < max_retries - 1:
                    logger.error("Simulated internal error during metric processing! Retrying...")
                    raise IOError("Simulated temporary service outage")
                else:
                    logger.critical("Simulated internal error during metric processing! Max retries reached.")
                    raise IOError("Simulated persistent service outage")

            # Evaluate alert rules
            state = evaluate_alert_rules(state, rules, current_time)

            # Send notifications for newly triggered alerts
            for alert_name, alert_info in state['active_alerts'].items():
                # Only send notification if it was newly triggered in this cycle or recently became active
                # For simplicity, we resend for all active. A real system would track 'last_notified_time'
                send_notification(alert_name, alert_info)

            logger.info("Monitoring cycle completed successfully.")
            return state

        except Exception as e:
            logger.error(f"Monitoring cycle failed on attempt {attempt + 1}: {e}")
            if attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5) # Exponential backoff with jitter
                logger.info(f"Retrying in {delay:.2f} seconds...")
                time.sleep(delay)
            else:
                logger.critical(f"All {max_retries} attempts failed for monitoring cycle.")
                raise # Re-raise the last exception if all retries fail
    return state # Should not be reached

Demonstration and Visualization

This section demonstrates the complete alert monitoring system in action. We will simulate metric generation over several time steps, evaluate alerts, and visualize the metric trends along with the occurrences of triggered alerts. This will provide a clear understanding of how the system responds to changing conditions and identifies critical events.

[9]

# Initialize logging to display messages correctly in Colab
logger.add(lambda msg: print(msg.strip()), colorize=True, format="<level>{level: <8}</level> | <green>{time:YYYY-MM-DD HH:mm:ss}</green> | {message}")

# Configuration
SIMULATION_DURATION_MINUTES = 2
METRIC_INTERVAL_SECONDS = 10
BUFFER_SIZE = 60 # Keep 10 minutes of history at 10-second intervals

# Initialize state and rules
system_state = create_system_state(buffer_size=BUFFER_SIZE)
alert_rules = define_alert_rules()

alert_log = [] # To store details of triggered and resolved alerts for visualization

start_time = datetime.datetime.now()
end_time = start_time + datetime.timedelta(minutes=SIMULATION_DURATION_MINUTES)

current_sim_time = start_time

logger.info(f"Starting simulation for {SIMULATION_DURATION_MINUTES} minutes...")

while current_sim_time < end_time:
    try:
        # Introduce a deliberate spike in CPU/Memory/Error for demonstration
        if 5 <= (current_sim_time - start_time).total_seconds() / 60 < 10: # Between 5 and 10 minutes
            system_state['metrics']['cpu_usage'] = min(95.0, system_state['metrics']['cpu_usage'] + random.uniform(5, 10))
            system_state['metrics']['memory_usage'] = min(98.0, system_state['metrics']['memory_usage'] + random.uniform(3, 7))
            system_state['metrics']['response_time'] = min(1200, system_state['metrics']['response_time'] + random.randint(100, 300))

        # Introduce a deliberate spike in error rate
        if 15 <= (current_sim_time - start_time).total_seconds() / 60 < 20: # Between 15 and 20 minutes
            system_state['metrics']['error_rate'] = min(20.0, system_state['metrics']['error_rate'] + random.uniform(2, 5))
            system_state['metrics']['network_latency'] = min(300, system_state['metrics']['network_latency'] + random.randint(50, 100))

        system_state = run_monitor_cycle_with_retry(system_state, alert_rules, max_retries=3, base_delay=0.5)

        # Log active alerts for visualization
        for alert_name, alert_info in system_state['active_alerts'].items():
            alert_log.append({
                'time': alert_info['time'],
                'alert': alert_name,
                'severity': alert_info['severity'],
                'status': 'TRIGGERED'
            })

    except Exception as e:
        logger.critical(f"Simulation halted due to unrecoverable error: {e}")
        break

    current_sim_time += datetime.timedelta(seconds=METRIC_INTERVAL_SECONDS + random.uniform(-2, 2)) # Add some jitter to interval
    time.sleep(max(0, METRIC_INTERVAL_SECONDS + random.uniform(-1, 1) - ((datetime.datetime.now() - system_state['last_metric_time']).total_seconds() if system_state['last_metric_time'] else 0))) # Keep consistent average interval

logger.info("Simulation finished.")

[32m2026-06-16 09:31:13.365[0m | [1mINFO    [0m | [36m__main__[0m:[36mcreate_system_state[0m:[36m37[0m - [1mSystem state initialized with buffer size: 60[0m
[32m2026-06-16 09:31:13.366[0m | [1mINFO    [0m | [36m__main__[0m:[36mdefine_alert_rules[0m:[36m74[0m - [1mDefined 7 alert rules.[0m
[32m2026-06-16 09:31:13.368[0m | [1mINFO    [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m20[0m - [1mStarting simulation for 2 minutes...[0m
[32m2026-06-16 09:31:13.370[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:31:13.371[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:31:13.372[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 0, 'memory_usage': 3.786536031142532, 'error_rate': 1.4960513311521515, 'response_time': 6, 'network_latency': 2}[0m
[32m2026-06-16 09:31:13.373[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:31:13.374[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | System state initialized with buffer size: 60
[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | Defined 7 alert rules.
[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | Starting simulation for 2 minutes...
[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:13[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | Metrics simulated: {'cpu_usage': 0, 'memory_usage': 3.786536031142532, 'error_rate': 1.4960513311521515, 'response_time': 6, 'network_latency': 2}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:13[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:31:13[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:31:22.933[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:31:22.934[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:31:22.937[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 0, 'memory_usage': 8.744271012792126, 'error_rate': 1.8164319231635466, 'response_time': 24, 'network_latency': 16}[0m
[32m2026-06-16 09:31:22.938[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:31:22.939[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:31:22[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:22[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:31:22[0m | Metrics simulated: {'cpu_usage': 0, 'memory_usage': 8.744271012792126, 'error_rate': 1.8164319231635466, 'response_time': 24, 'network_latency': 16}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:22[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:31:22[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:31:33.194[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:31:33.195[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:31:33.197[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 0.855850970468337, 'memory_usage': 5.178281173710089, 'error_rate': 3.4976528566941383, 'response_time': 36, 'network_latency': 13}[0m
[32m2026-06-16 09:31:33.198[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:31:33.200[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:31:33[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:33[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:31:33[0m | Metrics simulated: {'cpu_usage': 0.855850970468337, 'memory_usage': 5.178281173710089, 'error_rate': 3.4976528566941383, 'response_time': 36, 'network_latency': 13}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:33[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:31:33[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:31:42.858[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:31:42.860[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:31:42.861[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 5.688391489189836, 'memory_usage': 5.559927414890499, 'error_rate': 3.45558732001198, 'response_time': 21, 'network_latency': 20}[0m
[32m2026-06-16 09:31:42.862[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:31:42.864[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:31:42[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:42[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:31:42[0m | Metrics simulated: {'cpu_usage': 5.688391489189836, 'memory_usage': 5.559927414890499, 'error_rate': 3.45558732001198, 'response_time': 21, 'network_latency': 20}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:42[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:31:42[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:31:52.737[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:31:52.739[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:31:52.741[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 4.7455987080316335, 'memory_usage': 10.455199704611166, 'error_rate': 3.1214472020141333, 'response_time': 22, 'network_latency': 26}[0m
[32m2026-06-16 09:31:52.743[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:31:52.744[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:31:52[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:52[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:31:52[0m | Metrics simulated: {'cpu_usage': 4.7455987080316335, 'memory_usage': 10.455199704611166, 'error_rate': 3.1214472020141333, 'response_time': 22, 'network_latency': 26}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:31:52[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:31:52[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:01.942[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:01.944[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:01.946[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 4.96211250520544, 'memory_usage': 13.950798789675046, 'error_rate': 3.1942876438170584, 'response_time': 15, 'network_latency': 24}[0m
[32m2026-06-16 09:32:01.946[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:01.947[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:01[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:01[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:01[0m | Metrics simulated: {'cpu_usage': 4.96211250520544, 'memory_usage': 13.950798789675046, 'error_rate': 3.1942876438170584, 'response_time': 15, 'network_latency': 24}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:01[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:01[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:12.219[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:12.221[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:12.223[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 8.218221536569896, 'memory_usage': 15.733625011273674, 'error_rate': 2.781953525592616, 'response_time': 13, 'network_latency': 17}[0m
[32m2026-06-16 09:32:12.224[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:12.225[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:12[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:12[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:12[0m | Metrics simulated: {'cpu_usage': 8.218221536569896, 'memory_usage': 15.733625011273674, 'error_rate': 2.781953525592616, 'response_time': 13, 'network_latency': 17}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:12[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:12[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:22.406[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:22.407[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:22.408[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 11.020843136446619, 'memory_usage': 10.952629069485363, 'error_rate': 4.6324621019898435, 'response_time': 23, 'network_latency': 10}[0m
[32m2026-06-16 09:32:22.410[0m | [31m[1mERROR   [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m49[0m - [31m[1mSimulated internal error during metric processing! Retrying...[0m
[32m2026-06-16 09:32:22.411[0m | [31m[1mERROR   [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m68[0m - [31m[1mMonitoring cycle failed on attempt 1: Simulated temporary service outage[0m
[32m2026-06-16 09:32:22.411[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m71[0m - [1mRetrying in 0.68 seconds...[0m

[1mINFO    [0m | [32m2026-06-16 09:32:22[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:22[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:22[0m | Metrics simulated: {'cpu_usage': 11.020843136446619, 'memory_usage': 10.952629069485363, 'error_rate': 4.6324621019898435, 'response_time': 23, 'network_latency': 10}
[31m[1mERROR   [0m | [32m2026-06-16 09:32:22[0m | Simulated internal error during metric processing! Retrying...
[31m[1mERROR   [0m | [32m2026-06-16 09:32:22[0m | Monitoring cycle failed on attempt 1: Simulated temporary service outage
[1mINFO    [0m | [32m2026-06-16 09:32:22[0m | Retrying in 0.68 seconds...

[32m2026-06-16 09:32:23.096[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 2/3)...[0m
[32m2026-06-16 09:32:23.097[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:23.099[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 9.923031482399225, 'memory_usage': 13.156498933939115, 'error_rate': 3.790225031167655, 'response_time': 5, 'network_latency': 19}[0m
[32m2026-06-16 09:32:23.100[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:23.101[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:23[0m | Starting monitoring cycle (Attempt 2/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:23[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:23[0m | Metrics simulated: {'cpu_usage': 9.923031482399225, 'memory_usage': 13.156498933939115, 'error_rate': 3.790225031167655, 'response_time': 5, 'network_latency': 19}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:23[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:23[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:33.141[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:33.142[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:33.144[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 13.866161777727367, 'memory_usage': 9.740555871747196, 'error_rate': 4.434486300029587, 'response_time': 0, 'network_latency': 35}[0m
[32m2026-06-16 09:32:33.146[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:33.146[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:33[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:33[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:33[0m | Metrics simulated: {'cpu_usage': 13.866161777727367, 'memory_usage': 9.740555871747196, 'error_rate': 4.434486300029587, 'response_time': 0, 'network_latency': 35}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:33[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:33[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:43.042[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:43.043[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:43.044[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 9.804835538169531, 'memory_usage': 5.355654237511748, 'error_rate': 3.8208565670957038, 'response_time': 0, 'network_latency': 41}[0m
[32m2026-06-16 09:32:43.046[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:43.047[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:43[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:43[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:43[0m | Metrics simulated: {'cpu_usage': 9.804835538169531, 'memory_usage': 5.355654237511748, 'error_rate': 3.8208565670957038, 'response_time': 0, 'network_latency': 41}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:43[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:43[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:32:53.314[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:32:53.315[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:32:53.316[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 10.114014999387047, 'memory_usage': 7.922927045790666, 'error_rate': 2.999857947337267, 'response_time': 0, 'network_latency': 29}[0m
[32m2026-06-16 09:32:53.317[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:32:53.318[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:32:53[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:53[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:32:53[0m | Metrics simulated: {'cpu_usage': 10.114014999387047, 'memory_usage': 7.922927045790666, 'error_rate': 2.999857947337267, 'response_time': 0, 'network_latency': 29}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:32:53[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:32:53[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:33:02.521[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m40[0m - [1mStarting monitoring cycle (Attempt 1/3)...[0m
[32m2026-06-16 09:33:02.523[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m24[0m - [34m[1mSimulating new metrics...[0m
[32m2026-06-16 09:33:02.525[0m | [1mINFO    [0m | [36m__main__[0m:[36msimulate_metrics[0m:[36m46[0m - [1mMetrics simulated: {'cpu_usage': 11.652304033976485, 'memory_usage': 6.9466263724691855, 'error_rate': 3.791424310081976, 'response_time': 18, 'network_latency': 34}[0m
[32m2026-06-16 09:33:02.527[0m | [34m[1mDEBUG   [0m | [36m__main__[0m:[36mevaluate_alert_rules[0m:[36m28[0m - [34m[1mEvaluating alert rules...[0m
[32m2026-06-16 09:33:02.528[0m | [1mINFO    [0m | [36m__main__[0m:[36mrun_monitor_cycle_with_retry[0m:[36m64[0m - [1mMonitoring cycle completed successfully.[0m

[1mINFO    [0m | [32m2026-06-16 09:33:02[0m | Starting monitoring cycle (Attempt 1/3)...
[34m[1mDEBUG   [0m | [32m2026-06-16 09:33:02[0m | Simulating new metrics...
[1mINFO    [0m | [32m2026-06-16 09:33:02[0m | Metrics simulated: {'cpu_usage': 11.652304033976485, 'memory_usage': 6.9466263724691855, 'error_rate': 3.791424310081976, 'response_time': 18, 'network_latency': 34}
[34m[1mDEBUG   [0m | [32m2026-06-16 09:33:02[0m | Evaluating alert rules...
[1mINFO    [0m | [32m2026-06-16 09:33:02[0m | Monitoring cycle completed successfully.

[32m2026-06-16 09:33:13.463[0m | [1mINFO    [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m53[0m - [1mSimulation finished.[0m

[1mINFO    [0m | [32m2026-06-16 09:33:13[0m | Simulation finished.

Visualize System Metrics and Alerts

We will now visualize the simulated system metrics over time, highlighting when alerts were triggered. This helps in understanding the correlation between metric values and alert states.

[10]

# Convert history to DataFrame for easy plotting
metrics_df = pd.DataFrame(list(system_state['history']))
metrics_df.set_index('time', inplace=True)

alert_log_df = pd.DataFrame(alert_log)
if not alert_log_df.empty:
    alert_log_df.drop_duplicates(subset=['time', 'alert', 'status'], inplace=True)
    alert_log_df.sort_values(by='time', inplace=True)

# Prepare plots
fig, axes = plt.subplots(nrows=3, ncols=2, figsize=(18, 15), sharex=True)
axes = axes.flatten()

metric_cols = ['cpu_usage', 'memory_usage', 'error_rate', 'response_time', 'network_latency']
titles = {
    'cpu_usage': 'CPU Usage (%)',
    'memory_usage': 'Memory Usage (%)',
    'error_rate': 'Error Rate (%)',
    'response_time': 'Response Time (ms)',
    'network_latency': 'Network Latency (ms)'
}

# Plot each metric
for i, col in enumerate(metric_cols):
    sns.lineplot(ax=axes[i], x=metrics_df.index, y=metrics_df[col], label=col, color=f'C{i}')
    axes[i].set_title(titles[col])
    axes[i].set_ylabel(titles[col].split('(')[0].strip())
    axes[i].grid(True, linestyle='--', alpha=0.7)

    # Plot alert triggers on the respective metric subplots
    if not alert_log_df.empty:
        for _, alert_row in alert_log_df[alert_log_df['status'] == 'TRIGGERED'].iterrows():
            alert_name = alert_row['alert']
            # Find which rule this alert corresponds to and its metric
            for r_name, rule_def in alert_rules.items():
                if r_name == alert_name and rule_def['metric'] == col:
                    axes[i].axvline(alert_row['time'], color='red', linestyle=':', lw=2, label=f'Alert: {alert_name}')
                    # Add text annotation for the alert
                    axes[i].text(alert_row['time'], axes[i].get_ylim()[1] * 0.9, '🚨', color='red', fontsize=12, ha='center')
                    break

# Additional plot for combined alerts (e.g., severity over time)
if not alert_log_df.empty:
    alert_severity_map = {'WARNING': 1, 'CRITICAL': 2}
    alert_log_df['severity_num'] = alert_log_df['severity'].map(alert_severity_map)

    axes[5].plot(alert_log_df['time'], alert_log_df['severity_num'], 'o', markersize=8, color='purple', label='Alert Events')
    axes[5].set_title('Alert Severity Over Time')
    axes[5].set_ylabel('Severity (1=Warning, 2=Critical)')
    axes[5].set_yticks([1, 2])
    axes[5].set_yticklabels(['WARNING', 'CRITICAL'])
    axes[5].grid(True, linestyle='--', alpha=0.7)
    axes[5].legend()
else:
    axes[5].set_visible(False) # Hide if no alerts

fig.autofmt_xdate()
plt.tight_layout()
plt.show()

Production Considerations

Implementing a robust alerting system in a production environment requires careful consideration of various factors beyond just metric monitoring. Here are some best practices:

Aspect	Best Practice
Metric Granularity	Collect metrics at an appropriate frequency (e.g., every 5-10 seconds for critical systems) but aggregate over longer periods for historical analysis to manage storage and processing load.
Threshold Tuning	Continuously review and adjust alert thresholds based on historical data, seasonality, and observed system behavior. Avoid alert fatigue by setting realistic and actionable thresholds.
Notification Channels	Utilize multiple, redundant notification channels (e.g., email, SMS, PagerDuty, Slack, incident management platforms) to ensure critical alerts reach the right people promptly. Consider escalation policies.
Debouncing & Grouping	Implement debouncing (e.g., only alert if a metric stays above a threshold for N consecutive samples) and alert grouping to prevent a flood of notifications during widespread issues, reducing noise and focusing on root causes.
Monitoring Redundancy	Deploy monitoring agents and the alerting system itself with high availability and redundancy. A monitoring system failing to monitor is a critical blind spot.
Runbooks & Remediation	For each critical alert, define clear runbooks or standard operating procedures that guide responders through diagnosis and initial remediation steps. Automate remediation where possible.
Security	Ensure that metric data is secured (encryption in transit and at rest) and that access to the alerting system and notification channels is properly authenticated and authorized.
Testing Alerts	Regularly test your alerting system (e.g., by deliberately triggering alerts in a staging environment) to ensure that notifications are delivered, and escalation paths are functional.
Cost Management	Be mindful of the cost associated with metric storage, processing, and notification services, especially in cloud environments. Implement intelligent data retention policies.
Distributed Tracing & Logs	Integrate with distributed tracing and centralized logging systems. Alerts indicate a problem, but logs and traces provide the context and details needed for effective debugging and root cause analysis.
Configuration Management	Manage alert rules and notification settings as code, stored in a version control system (e.g., Git). This allows for easier review, auditing, and automated deployment of changes.

Conclusion

This notebook has provided a comprehensive framework for building a system error notification and alerting mechanism. We've covered:

State Management: Using dictionaries to maintain system metrics, historical data, and active alerts.
Metric Simulation: Generating realistic, fluctuating system metrics with random jitter.
Rule Definition: Creating flexible alert rules based on metrics, thresholds, operators, and severities.
Alert Evaluation: A mechanism to continuously check metrics against rules, activate new alerts, and resolve old ones, incorporating cooldown periods to prevent alert fatigue.
Notification: A simulated notification system to inform stakeholders about critical events.
Robustness: Implementing retry logic with exponential backoff for resilience against transient failures.
Visualization: Plotting metric trends alongside alert occurrences to gain insights into system behavior.

By following these principles and practices, developers and operations teams can implement effective monitoring and alerting solutions that contribute significantly to the reliability and stability of their systems.

System Error Alert

Notifications and Alerts: Alert on System Errors

Core Concepts

Dependency Installation

Library Imports

Core Functions

Function Name: create_system_state

Function Name: simulate_metrics

Function Name: define_alert_rules

Function Name: evaluate_alert_rules

Function Name: send_notification

Function Name: run_monitor_cycle_with_retry

Demonstration and Visualization

Visualize System Metrics and Alerts

Production Considerations

Conclusion

Function Name: `create_system_state`

Function Name: `simulate_metrics`

Function Name: `define_alert_rules`

Function Name: `evaluate_alert_rules`

Function Name: `send_notification`

Function Name: `run_monitor_cycle_with_retry`