Infrastructure·System Monitoring·Intermediate

Grafana Alerting Setup

Configure comprehensive Grafana dashboards and multi-severity alerting rules for production trading system monitoring with real-time panels displaying PnL, net exposure, drawdown depth, system component health status, and exchange connectivity with threshold-based alert routing to notification channels.

infrastructuremonitoringnotifications

Setup Grafana Trading Alerts

This notebook outlines a systematic approach to designing and implementing a robust system for generating trading alerts, with a focus on principles that can be integrated or consumed by a monitoring tool like Grafana. While direct Grafana API interaction isn't demonstrated due to environment constraints, the core logic for data fetching, indicator calculation, condition evaluation, and notification triggering is covered.

Concepts

Concept	Description	Relevance to Grafana Alerts
Trading Data	Time-series financial data (e.g., stock prices, volume, indicators).	Source data that Grafana would visualize and on which alerts would be based.
Technical Indicators	Mathematical transformations of trading data to identify patterns (e.g., SMA, RSI, MACD).	Conditions in Grafana alerts are often based on these calculated values.
Alert Conditions	Specific criteria that, when met, trigger an alert (e.g., price crosses SMA, RSI overbought).	These are the rules configured within Grafana's alerting engine.
Notification Channels	Methods for delivering alerts (e.g., email, Slack, PagerDuty, webhooks).	Grafana supports various notification channels for alert delivery.
State Management	Keeping track of variables and parameters throughout the alert system's lifecycle.	Essential for persistent alert states (e.g., alert acknowledged, cooldown periods).
Backoff/Retry Logic	Strategies for handling transient failures when interacting with external services or APIs.	Crucial for robust systems that interact with data sources or notification services.
Logging	Recording events, errors, and significant actions within the system for debugging and auditing.	Provides visibility into alert generation and delivery, useful for troubleshooting in Grafana dashboards.
Data Visualization	Graphical representation of data and alerts for understanding trends and system behavior.	Grafana excels at visualizing data and alert states.

Dependency Installation

This section installs all necessary Python packages required for the notebook. We'll use pandas for data manipulation, numpy for numerical operations, matplotlib and seaborn for plotting, and requests for simulating external API calls (e.g., for notifications).

[ ]

import sys
!{sys.executable} -m pip install pandas numpy matplotlib seaborn requests

Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (2.0.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0)
Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2)
Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (2.32.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.2)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.2)
Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests) (3.18)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests) (2026.5.20)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)

Library Imports

This cell imports all required libraries, organizing them with standard libraries first, followed by third-party libraries.

[ ]

import logging
import time
import random
from collections import deque
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import requests

Core Functions

This section defines the core functions for our Grafana trading alert system. Each function is presented in its own code block with a detailed markdown header, comprehensive docstrings, type hints, and logger statements.

Function Name: create_logger

This helper function creates and configures a Python logger instance. It sets up the logger with a specified name, level, and format, ensuring that all subsequent logging messages from this logger are consistently formatted and displayed.

Parameters: logger_name (str): The name for the logger. level (int): The logging level (e.g., logging.INFO, logging.DEBUG).

Returns: (logging.Logger): A configured logger instance.

[ ]

def create_logger(logger_name: str, level: int = logging.INFO) -> logging.Logger:
    """
    Creates and configures a Python logger instance.

    Parameters
    ----------
    logger_name : str
        The name for the logger.
    level : int, optional
        The logging level (e.g., logging.INFO, logging.DEBUG), defaults to logging.INFO.

    Returns
    -------
    logging.Logger
        A configured logger instance.
    """
    logger = logging.getLogger(logger_name)
    logger.setLevel(level)

    # Prevent duplicate handlers if the cell is run multiple times
    if not logger.handlers:
        handler = logging.StreamHandler(sys.stdout)
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        handler.setFormatter(formatter)
        logger.addHandler(handler)
    return logger

alerts_logger = create_logger('alerts_system')
alerts_logger.info('Logger initialized for alerts_system.')

2026-06-11 06:17:32,901 - alerts_system - INFO - Logger initialized for alerts_system.

INFO:alerts_system:Logger initialized for alerts_system.

Function Name: create_trading_state

This function initializes the state dictionary for managing trading data and alert configurations. It sets up default values for various parameters, including data storage, indicator settings, and alert thresholds. Using a dictionary ensures flexible state management without relying on classes.

Parameters: initial_price (float): The starting price for simulated data. simulation_steps (int): The number of data points to simulate. sma_period (int): The period for the Simple Moving Average (SMA) indicator. rsi_period (int): The period for the Relative Strength Index (RSI) indicator.

Returns: (dict): An initialized state dictionary.

[ ]

from google.colab import userdata

def create_trading_state(initial_price: float = 100.0,
                        simulation_steps: int = 200,
                        sma_period: int = 20,
                        rsi_period: int = 14) -> dict:
    """
    Initializes the state dictionary for managing trading data and alert configurations.

    Parameters
    ----------
    initial_price : float, optional
        The starting price for simulated data, defaults to 100.0.
    simulation_steps : int, optional
        The number of data points to simulate, defaults to 200.
    sma_period : int, optional
        The period for the Simple Moving Average (SMA) indicator, defaults to 20.
    rsi_period : int, optional
        The period for the Relative Strength Index (RSI) indicator, defaults to 14.

    Returns
    -------
    dict
        An initialized state dictionary.
    """
    alerts_logger.info(f"Creating trading state with initial_price={initial_price}, steps={simulation_steps}")

    # Retrieve webhook URL from Colab Secrets
    grafana_webhook_url = userdata.get('GRAFANA_WEBHOOK_URL')
    if not grafana_webhook_url:
        alerts_logger.warning("GRAFANA_WEBHOOK_URL not found in Colab secrets. Using placeholder.")
        grafana_webhook_url = 'https://example.com/grafana-webhook-endpoint'

    state = {
        'data': pd.DataFrame(columns=['timestamp', 'price', 'sma', 'rsi', 'alert_signal']),
        'price_history': deque(maxlen=rsi_period + 1), # Maxlen for RSI calculation
        'sma_period': sma_period,
        'rsi_period': rsi_period,
        'initial_price': initial_price,
        'simulation_steps': simulation_steps,
        'alert_config': {
            'sma_cross_threshold': 0.01, # Percentage difference for SMA cross alert
            'rsi_overbought': 70,
            'rsi_oversold': 30,
            'notification_webhook': grafana_webhook_url
        },
        'current_alerts': {},
        'last_alert_time': {}
    }
    alerts_logger.debug('Trading state created successfully.')
    return state

Function Name: simulate_trading_data

This function simulates new trading price data points. It generates a random walk with some drift, mimicking realistic price movements. This data will be used to calculate indicators and evaluate alert conditions.

Parameters: state (dict): The current state dictionary. current_step (int): The current simulation step number.

Returns: (dict): The updated state dictionary with new simulated price data.

[ ]

def simulate_trading_data(state: dict, current_step: int) -> dict:
    """
    Simulates a new trading price data point using a random walk.

    Parameters
    ----------
    state : dict
        Current state dictionary.
    current_step : int
        The current simulation step number.

    Returns
    -------
    dict
        Updated state with new simulated price data.
    """
    alerts_logger.debug(f"Simulating data for step {current_step}")
    last_price = state['price_history'][-1] if state['price_history'] else state['initial_price']
    price_change = np.random.normal(0, 0.5) # small random change
    new_price = max(1.0, last_price + price_change) # Ensure price doesn't go below 1
    state['price_history'].append(new_price)
    state['data'].loc[current_step, 'timestamp'] = pd.Timestamp.now() + pd.Timedelta(minutes=current_step)
    state['data'].loc[current_step, 'price'] = new_price
    alerts_logger.debug(f"Generated price: {new_price:.2f} at step {current_step}")
    return state

Function Name: calculate_indicators

This function calculates technical indicators, specifically Simple Moving Average (SMA) and Relative Strength Index (RSI), based on the price_history stored in the state. It uses deque for efficient rolling window calculations for RSI and pandas.Series.rolling for SMA.

Parameters: state (dict): The current state dictionary.

Returns: (dict): The updated state dictionary with calculated SMA and RSI values for the latest data point.

[ ]

def calculate_indicators(state: dict) -> dict:
    """
    Calculates technical indicators (SMA and RSI) for the latest data point.

    Parameters
    ----------
    state : dict
        Current state dictionary.

    Returns
    -------
    dict
        Updated state with calculated SMA and RSI values.
    """
    current_step = state['data'].index[-1]
    prices = pd.Series(list(state['price_history']))

    # Calculate SMA
    if len(prices) >= state['sma_period']:
        sma = prices.iloc[-state['sma_period']:].mean()
        state['data'].loc[current_step, 'sma'] = sma
        alerts_logger.debug(f"Calculated SMA: {sma:.2f}")
    else:
        state['data'].loc[current_step, 'sma'] = np.nan
        alerts_logger.debug('Not enough data for SMA calculation.')

    # Calculate RSI
    if len(prices) > state['rsi_period']:
        # Ensure we have enough data points for RSI period + 1 for diff
        if len(state['price_history']) > state['rsi_period']:
            diff = prices.diff().dropna()
            gains = diff.where(diff > 0, 0)
            losses = -diff.where(diff < 0, 0)

            avg_gain = gains.iloc[-state['rsi_period']:].mean()
            avg_loss = losses.iloc[-state['rsi_period']:].mean()

            if avg_loss == 0:
                rsi = 100 if avg_gain > 0 else 50
            else:
                rs = avg_gain / avg_loss
                rsi = 100 - (100 / (1 + rs))
            state['data'].loc[current_step, 'rsi'] = rsi
            alerts_logger.debug(f"Calculated RSI: {rsi:.2f}")
        else:
            state['data'].loc[current_step, 'rsi'] = np.nan
            alerts_logger.debug('Not enough data for RSI calculation.')
    else:
        state['data'].loc[current_step, 'rsi'] = np.nan
        alerts_logger.debug('Not enough data for RSI calculation.')

    return state

Function Name: evaluate_alert_condition

This function evaluates whether any predefined alert conditions are met for the latest data point. It checks for SMA crossovers and RSI overbought/oversold levels. If an alert condition is met, it updates the state accordingly.

Parameters: state (dict): The current state dictionary.

Returns: (dict): The updated state dictionary with alert_signal set if an alert is triggered.

[ ]

def evaluate_alert_condition(state: dict) -> dict:
    """
    Evaluates predefined alert conditions for the latest data point.

    Parameters
    ----------
    state : dict
        Current state dictionary.

    Returns
    -------
    dict
        Updated state with `alert_signal` set if an alert is triggered.
    """
    current_step = state['data'].index[-1]
    latest_row = state['data'].loc[current_step]
    price = latest_row['price']
    sma = latest_row['sma']
    rsi = latest_row['rsi']

    alert_triggered = False
    alert_message = []

    # SMA Crossover Alert (simple check if price is significantly different from SMA)
    if pd.notna(sma):
        sma_cross_threshold = state['alert_config']['sma_cross_threshold']
        if price > sma * (1 + sma_cross_threshold):
            alert_message.append(f"Price ({price:.2f}) is significantly above SMA ({sma:.2f}).")
            alert_triggered = True
        elif price < sma * (1 - sma_cross_threshold):
            alert_message.append(f"Price ({price:.2f}) is significantly below SMA ({sma:.2f}).")
            alert_triggered = True

    # RSI Overbought/Oversold Alert
    if pd.notna(rsi):
        if rsi > state['alert_config']['rsi_overbought']:
            alert_message.append(f"RSI ({rsi:.2f}) is overbought.")
            alert_triggered = True
        elif rsi < state['alert_config']['rsi_oversold']:
            alert_message.append(f"RSI ({rsi:.2f}) is oversold.")
            alert_triggered = True

    state['data'].loc[current_step, 'alert_signal'] = 1 if alert_triggered else 0

    if alert_triggered:
        alerts_logger.warning(f"Alert triggered at step {current_step}: {'; '.join(alert_message)}")
        state['current_alerts'][current_step] = {'message': '; '.join(alert_message), 'status': 'new'}
    else:
        alerts_logger.debug(f"No alert triggered at step {current_step}.")

    return state

Function Name: send_grafana_alert_notification

This function simulates sending an alert notification, potentially to a Grafana webhook or another notification service. It includes retry logic with exponential backoff and random jitter to handle transient network issues or API rate limits gracefully.

Parameters: state (dict): The current state dictionary. alert_id (int): The identifier for the triggered alert. message (str): The alert message to be sent. max_retries (int): Maximum number of retries for sending the notification.

Returns: (dict): The updated state dictionary, reflecting the outcome of the notification attempt.

[ ]

def send_grafana_alert_notification(state: dict, alert_id: int, message: str, max_retries: int = 3) -> dict:
    """
    Simulates sending an alert notification, with retry logic and exponential backoff.

    Parameters
    ----------
    state : dict
        Current state dictionary.
    alert_id : int
        The identifier for the triggered alert.
    message : str
        The alert message to be sent.
    max_retries : int, optional
        Maximum number of retries for sending the notification, defaults to 3.

    Returns
    -------
    dict
        Updated state reflecting the outcome of the notification attempt.
    """
    webhook_url = state['alert_config']['notification_webhook']
    payload = {
        'alertId': alert_id,
        'ruleName': 'Trading Alert',
        'state': 'alerting',
        'message': message,
        'evalMatches': [
            {'value': state['data'].loc[alert_id, 'price'], 'metric': 'price', 'tags': {}},
            {'value': state['data'].loc[alert_id, 'sma'], 'metric': 'sma', 'tags': {}} if pd.notna(state['data'].loc[alert_id, 'sma']) else None,
            {'value': state['data'].loc[alert_id, 'rsi'], 'metric': 'rsi', 'tags': {}} if pd.notna(state['data'].loc[alert_id, 'rsi']) else None
        ]
    }
    payload['evalMatches'] = [item for item in payload['evalMatches'] if item is not None]

    for i in range(max_retries):
        try:
            alerts_logger.info(f"Attempt {i+1}/{max_retries}: Sending notification for alert {alert_id}.")
            # Simulate a successful or failed request
            if random.random() < 0.9: # 90% chance of success
                response = requests.post(webhook_url, json=payload, timeout=5)
                response.raise_for_status() # Raise an exception for HTTP errors (4xx or 5xx)
                alerts_logger.info(f"Notification for alert {alert_id} sent successfully.")
                state['current_alerts'][alert_id]['status'] = 'sent'
                state['last_alert_time'][alert_id] = time.time()
                return state
            else:
                raise requests.exceptions.RequestException("Simulated network error")

        except requests.exceptions.Timeout:
            alerts_logger.warning(f"Timeout sending notification for alert {alert_id}. Retrying...")
        except requests.exceptions.ConnectionError:
            alerts_logger.warning(f"Connection error sending notification for alert {alert_id}. Retrying...")
        except requests.exceptions.RequestException as e:
            alerts_logger.warning(f"Error sending notification for alert {alert_id}: {e}. Retrying...")
        except Exception as e:
            alerts_logger.error(f"Unexpected error sending notification for alert {alert_id}: {e}.")
            break # Don't retry for unexpected errors

        # Exponential backoff with jitter
        sleep_time = (2 ** i) + random.uniform(0, 1)
        alerts_logger.debug(f"Waiting {sleep_time:.2f} seconds before retry.")
        time.sleep(sleep_time)

    alerts_logger.error(f"Failed to send notification for alert {alert_id} after {max_retries} retries.")
    state['current_alerts'][alert_id]['status'] = 'failed'
    return state

Function Name: update_alert_status

This function allows for manual or programmatic updating of an alert's status within the state. This is useful for marking alerts as acknowledged, resolved, or for implementing cooldown periods to prevent alert storming.

Parameters: state (dict): The current state dictionary. alert_id (int): The identifier of the alert to update. new_status (str): The new status to set for the alert (e.g., 'acknowledged', 'resolved').

Returns: (dict): The updated state dictionary.

[ ]

def update_alert_status(state: dict, alert_id: int, new_status: str) -> dict:
    """
    Updates the status of a specific alert in the state.

    Parameters
    ----------
    state : dict
        Current state dictionary.
    alert_id : int
        The identifier of the alert to update.
    new_status : str
        The new status to set for the alert (e.g., 'acknowledged', 'resolved').

    Returns
    -------
    dict
        Updated state dictionary.
    """
    if alert_id in state['current_alerts']:
        old_status = state['current_alerts'][alert_id]['status']
        state['current_alerts'][alert_id]['status'] = new_status
        alerts_logger.info(f"Alert {alert_id} status updated from '{old_status}' to '{new_status}'.")
    else:
        alerts_logger.warning(f"Attempted to update non-existent alert {alert_id}.")
    return state

Function Name: track_metrics

This function is designed to track and summarize various performance metrics of the alert system, such as the total number of alerts, number of sent notifications, or alerts that failed to send. It takes a dictionary of metrics to update and the current state.

Parameters: metrics_state (dict): A dictionary to store and update metrics. state (dict): The current trading system state.

Returns: (dict): The updated metrics state dictionary.

[ ]

def track_metrics(metrics_state: dict, state: dict) -> dict:
    """
    Tracks and summarizes various performance metrics of the alert system.

    Parameters
    ----------
    metrics_state : dict
        A dictionary to store and update metrics.
    state : dict
        The current trading system state.

    Returns
    -------
    dict
        The updated metrics state dictionary.
    """
    metrics_state['total_simulated_steps'] = state['simulation_steps']
    metrics_state['total_alerts_triggered'] = len(state['current_alerts'])

    sent_alerts = sum(1 for alert_info in state['current_alerts'].values() if alert_info['status'] == 'sent')
    metrics_state['total_notifications_sent'] = sent_alerts

    failed_alerts = sum(1 for alert_info in state['current_alerts'].values() if alert_info['status'] == 'failed')
    metrics_state['total_notifications_failed'] = failed_alerts

    alerts_logger.debug('Metrics updated.')
    return metrics_state

Demonstration and Visualization

This section demonstrates the end-to-end workflow of the alert system. It simulates trading data, calculates indicators, evaluates alert conditions, and simulates sending notifications. Visualizations are provided to illustrate price movements, indicator values, and alert triggers.

[ ]

alerts_logger.info('Starting demonstration of the trading alert system.')

# 1. Initialize state
trading_state = create_trading_state(initial_price=100, simulation_steps=250, sma_period=20, rsi_period=14)
metrics_state = {
    'total_simulated_steps': 0,
    'total_alerts_triggered': 0,
    'total_notifications_sent': 0,
    'total_notifications_failed': 0
}

# 2. Simulate data and evaluate alerts in a loop
for i in range(trading_state['simulation_steps']):
    trading_state = simulate_trading_data(trading_state, i)
    trading_state = calculate_indicators(trading_state)
    trading_state = evaluate_alert_condition(trading_state)

    # If an alert was triggered and not yet sent/failed
    if i in trading_state['current_alerts'] and trading_state['current_alerts'][i]['status'] == 'new':
        trading_state = send_grafana_alert_notification(trading_state, i, trading_state['current_alerts'][i]['message'])

alerts_logger.info('Demonstration data simulation complete.')

# 3. Process and display data
df_results = trading_state['data'].copy()
df_results.set_index('timestamp', inplace=True)

display(df_results.head())
display(df_results.tail())

2026-06-11 06:17:37,770 - alerts_system - INFO - Starting demonstration of the trading alert system.

INFO:alerts_system:Starting demonstration of the trading alert system.

2026-06-11 06:17:37,781 - alerts_system - INFO - Creating trading state with initial_price=100, steps=250

INFO:alerts_system:Creating trading state with initial_price=100, steps=250

2026-06-11 06:17:37,994 - alerts_system - WARNING - Alert triggered at step 26: RSI (25.75) is oversold.

WARNING:alerts_system:Alert triggered at step 26: RSI (25.75) is oversold.

2026-06-11 06:17:37,999 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 26.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 26.

2026-06-11 06:17:38,100 - alerts_system - WARNING - Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:39,756 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 26.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 26.

2026-06-11 06:17:39,859 - alerts_system - WARNING - Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:42,677 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 26.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 26.

2026-06-11 06:17:42,746 - alerts_system - WARNING - Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 26: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:47,316 - alerts_system - ERROR - Failed to send notification for alert 26 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 26 after 3 retries.

2026-06-11 06:17:47,325 - alerts_system - WARNING - Alert triggered at step 28: RSI (29.16) is oversold.

WARNING:alerts_system:Alert triggered at step 28: RSI (29.16) is oversold.

2026-06-11 06:17:47,329 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 28.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 28.

2026-06-11 06:17:47,382 - alerts_system - WARNING - Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:48,608 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 28.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 28.

2026-06-11 06:17:48,668 - alerts_system - WARNING - Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:51,025 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 28.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 28.

2026-06-11 06:17:51,084 - alerts_system - WARNING - Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 28: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:55,304 - alerts_system - ERROR - Failed to send notification for alert 28 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 28 after 3 retries.

2026-06-11 06:17:55,313 - alerts_system - WARNING - Alert triggered at step 29: RSI (27.93) is oversold.

WARNING:alerts_system:Alert triggered at step 29: RSI (27.93) is oversold.

2026-06-11 06:17:55,316 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 29.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 29.

2026-06-11 06:17:55,369 - alerts_system - WARNING - Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:56,895 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 29.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 29.

2026-06-11 06:17:56,948 - alerts_system - WARNING - Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:17:59,553 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 29.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 29.

2026-06-11 06:17:59,606 - alerts_system - WARNING - Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 29: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:03,617 - alerts_system - ERROR - Failed to send notification for alert 29 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 29 after 3 retries.

2026-06-11 06:18:03,633 - alerts_system - WARNING - Alert triggered at step 30: RSI (23.97) is oversold.

WARNING:alerts_system:Alert triggered at step 30: RSI (23.97) is oversold.

2026-06-11 06:18:03,636 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 30.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 30.

2026-06-11 06:18:03,692 - alerts_system - WARNING - Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:05,115 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 30.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 30.

2026-06-11 06:18:05,180 - alerts_system - WARNING - Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:07,193 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 30.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 30.

2026-06-11 06:18:07,248 - alerts_system - WARNING - Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 30: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:11,825 - alerts_system - ERROR - Failed to send notification for alert 30 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 30 after 3 retries.

2026-06-11 06:18:11,832 - alerts_system - WARNING - Alert triggered at step 31: RSI (23.82) is oversold.

WARNING:alerts_system:Alert triggered at step 31: RSI (23.82) is oversold.

2026-06-11 06:18:11,834 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 31.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 31.

2026-06-11 06:18:11,907 - alerts_system - WARNING - Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:13,639 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 31.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 31.

2026-06-11 06:18:13,720 - alerts_system - WARNING - Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:16,609 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 31.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 31.

2026-06-11 06:18:16,662 - alerts_system - WARNING - Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 31: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:21,607 - alerts_system - ERROR - Failed to send notification for alert 31 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 31 after 3 retries.

2026-06-11 06:18:21,615 - alerts_system - WARNING - Alert triggered at step 32: RSI (23.35) is oversold.

WARNING:alerts_system:Alert triggered at step 32: RSI (23.35) is oversold.

2026-06-11 06:18:21,617 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 32.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 32.

2026-06-11 06:18:21,673 - alerts_system - WARNING - Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:22,904 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 32.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 32.

2026-06-11 06:18:22,959 - alerts_system - WARNING - Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:25,479 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 32.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 32.

2026-06-11 06:18:25,532 - alerts_system - WARNING - Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 32: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:30,360 - alerts_system - ERROR - Failed to send notification for alert 32 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 32 after 3 retries.

2026-06-11 06:18:30,369 - alerts_system - WARNING - Alert triggered at step 33: RSI (18.03) is oversold.

WARNING:alerts_system:Alert triggered at step 33: RSI (18.03) is oversold.

2026-06-11 06:18:30,372 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 33.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 33.

2026-06-11 06:18:30,431 - alerts_system - WARNING - Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:31,651 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 33.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 33.

2026-06-11 06:18:31,705 - alerts_system - WARNING - Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:34,248 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 33.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 33.

2026-06-11 06:18:34,302 - alerts_system - WARNING - Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 33: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:39,034 - alerts_system - ERROR - Failed to send notification for alert 33 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 33 after 3 retries.

2026-06-11 06:18:39,044 - alerts_system - WARNING - Alert triggered at step 34: RSI (19.13) is oversold.

WARNING:alerts_system:Alert triggered at step 34: RSI (19.13) is oversold.

2026-06-11 06:18:39,046 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 34.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 34.

2026-06-11 06:18:39,106 - alerts_system - WARNING - Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:40,722 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 34.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 34.

2026-06-11 06:18:40,779 - alerts_system - WARNING - Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:43,079 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 34.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 34.

2026-06-11 06:18:43,140 - alerts_system - WARNING - Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 34: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:47,276 - alerts_system - ERROR - Failed to send notification for alert 34 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 34 after 3 retries.

2026-06-11 06:18:47,285 - alerts_system - WARNING - Alert triggered at step 35: RSI (17.09) is oversold.

WARNING:alerts_system:Alert triggered at step 35: RSI (17.09) is oversold.

2026-06-11 06:18:47,287 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 35.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 35.

2026-06-11 06:18:47,291 - alerts_system - WARNING - Error sending notification for alert 35: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 35: Simulated network error. Retrying...

2026-06-11 06:18:48,983 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 35.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 35.

2026-06-11 06:18:49,056 - alerts_system - WARNING - Error sending notification for alert 35: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 35: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:51,782 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 35.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 35.

2026-06-11 06:18:51,840 - alerts_system - WARNING - Error sending notification for alert 35: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 35: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:56,677 - alerts_system - ERROR - Failed to send notification for alert 35 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 35 after 3 retries.

2026-06-11 06:18:56,713 - alerts_system - WARNING - Alert triggered at step 50: RSI (29.79) is oversold.

WARNING:alerts_system:Alert triggered at step 50: RSI (29.79) is oversold.

2026-06-11 06:18:56,716 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 50.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 50.

2026-06-11 06:18:56,788 - alerts_system - WARNING - Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:18:58,015 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 50.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 50.

2026-06-11 06:18:58,070 - alerts_system - WARNING - Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:00,448 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 50.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 50.

2026-06-11 06:19:00,513 - alerts_system - WARNING - Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 50: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:04,861 - alerts_system - ERROR - Failed to send notification for alert 50 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 50 after 3 retries.

2026-06-11 06:19:04,873 - alerts_system - WARNING - Alert triggered at step 52: RSI (24.80) is oversold.

WARNING:alerts_system:Alert triggered at step 52: RSI (24.80) is oversold.

2026-06-11 06:19:04,875 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 52.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 52.

2026-06-11 06:19:04,877 - alerts_system - WARNING - Error sending notification for alert 52: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 52: Simulated network error. Retrying...

2026-06-11 06:19:06,092 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 52.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 52.

2026-06-11 06:19:06,149 - alerts_system - WARNING - Error sending notification for alert 52: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 52: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:08,885 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 52.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 52.

2026-06-11 06:19:08,942 - alerts_system - WARNING - Error sending notification for alert 52: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 52: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:13,362 - alerts_system - ERROR - Failed to send notification for alert 52 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 52 after 3 retries.

2026-06-11 06:19:13,395 - alerts_system - WARNING - Alert triggered at step 66: RSI (80.67) is overbought.

WARNING:alerts_system:Alert triggered at step 66: RSI (80.67) is overbought.

2026-06-11 06:19:13,399 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 66.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 66.

2026-06-11 06:19:13,455 - alerts_system - WARNING - Error sending notification for alert 66: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 66: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:14,769 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 66.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 66.

2026-06-11 06:19:14,772 - alerts_system - WARNING - Error sending notification for alert 66: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 66: Simulated network error. Retrying...

2026-06-11 06:19:17,292 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 66.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 66.

2026-06-11 06:19:17,348 - alerts_system - WARNING - Error sending notification for alert 66: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 66: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:22,018 - alerts_system - ERROR - Failed to send notification for alert 66 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 66 after 3 retries.

2026-06-11 06:19:22,025 - alerts_system - WARNING - Alert triggered at step 67: RSI (78.26) is overbought.

WARNING:alerts_system:Alert triggered at step 67: RSI (78.26) is overbought.

2026-06-11 06:19:22,027 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 67.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 67.

2026-06-11 06:19:22,095 - alerts_system - WARNING - Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:23,905 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 67.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 67.

2026-06-11 06:19:23,961 - alerts_system - WARNING - Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:26,108 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 67.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 67.

2026-06-11 06:19:26,164 - alerts_system - WARNING - Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 67: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:30,646 - alerts_system - ERROR - Failed to send notification for alert 67 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 67 after 3 retries.

2026-06-11 06:19:30,653 - alerts_system - WARNING - Alert triggered at step 68: RSI (84.51) is overbought.

WARNING:alerts_system:Alert triggered at step 68: RSI (84.51) is overbought.

2026-06-11 06:19:30,656 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 68.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 68.

2026-06-11 06:19:30,717 - alerts_system - WARNING - Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:32,531 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 68.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 68.

2026-06-11 06:19:32,587 - alerts_system - WARNING - Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:34,990 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 68.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 68.

2026-06-11 06:19:35,053 - alerts_system - WARNING - Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 68: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:39,261 - alerts_system - ERROR - Failed to send notification for alert 68 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 68 after 3 retries.

2026-06-11 06:19:39,270 - alerts_system - WARNING - Alert triggered at step 69: RSI (81.28) is overbought.

WARNING:alerts_system:Alert triggered at step 69: RSI (81.28) is overbought.

2026-06-11 06:19:39,272 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 69.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 69.

2026-06-11 06:19:39,327 - alerts_system - WARNING - Error sending notification for alert 69: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 69: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:40,599 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 69.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 69.

2026-06-11 06:19:40,602 - alerts_system - WARNING - Error sending notification for alert 69: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 69: Simulated network error. Retrying...

2026-06-11 06:19:43,113 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 69.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 69.

2026-06-11 06:19:43,174 - alerts_system - WARNING - Error sending notification for alert 69: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 69: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:48,121 - alerts_system - ERROR - Failed to send notification for alert 69 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 69 after 3 retries.

2026-06-11 06:19:48,130 - alerts_system - WARNING - Alert triggered at step 70: RSI (70.83) is overbought.

WARNING:alerts_system:Alert triggered at step 70: RSI (70.83) is overbought.

2026-06-11 06:19:48,133 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 70.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 70.

2026-06-11 06:19:48,195 - alerts_system - WARNING - Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:49,602 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 70.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 70.

2026-06-11 06:19:49,663 - alerts_system - WARNING - Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:52,404 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 70.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 70.

2026-06-11 06:19:52,478 - alerts_system - WARNING - Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 70: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:56,750 - alerts_system - ERROR - Failed to send notification for alert 70 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 70 after 3 retries.

2026-06-11 06:19:56,780 - alerts_system - WARNING - Alert triggered at step 79: RSI (21.98) is oversold.

WARNING:alerts_system:Alert triggered at step 79: RSI (21.98) is oversold.

2026-06-11 06:19:56,783 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 79.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 79.

2026-06-11 06:19:56,843 - alerts_system - WARNING - Error sending notification for alert 79: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 79: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:19:58,309 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 79.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 79.

2026-06-11 06:19:58,312 - alerts_system - WARNING - Error sending notification for alert 79: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 79: Simulated network error. Retrying...

2026-06-11 06:20:00,358 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 79.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 79.

2026-06-11 06:20:00,419 - alerts_system - WARNING - Error sending notification for alert 79: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 79: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:05,196 - alerts_system - ERROR - Failed to send notification for alert 79 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 79 after 3 retries.

2026-06-11 06:20:05,204 - alerts_system - WARNING - Alert triggered at step 80: RSI (24.30) is oversold.

WARNING:alerts_system:Alert triggered at step 80: RSI (24.30) is oversold.

2026-06-11 06:20:05,208 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 80.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 80.

2026-06-11 06:20:05,264 - alerts_system - WARNING - Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:06,515 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 80.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 80.

2026-06-11 06:20:06,569 - alerts_system - WARNING - Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:09,517 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 80.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 80.

2026-06-11 06:20:09,574 - alerts_system - WARNING - Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 80: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:14,182 - alerts_system - ERROR - Failed to send notification for alert 80 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 80 after 3 retries.

2026-06-11 06:20:14,190 - alerts_system - WARNING - Alert triggered at step 81: RSI (28.65) is oversold.

WARNING:alerts_system:Alert triggered at step 81: RSI (28.65) is oversold.

2026-06-11 06:20:14,192 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 81.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 81.

2026-06-11 06:20:14,252 - alerts_system - WARNING - Error sending notification for alert 81: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 81: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:16,191 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 81.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 81.

2026-06-11 06:20:16,194 - alerts_system - WARNING - Error sending notification for alert 81: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 81: Simulated network error. Retrying...

2026-06-11 06:20:18,210 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 81.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 81.

2026-06-11 06:20:18,267 - alerts_system - WARNING - Error sending notification for alert 81: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 81: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:23,056 - alerts_system - ERROR - Failed to send notification for alert 81 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 81 after 3 retries.

2026-06-11 06:20:23,066 - alerts_system - WARNING - Alert triggered at step 82: RSI (26.09) is oversold.

WARNING:alerts_system:Alert triggered at step 82: RSI (26.09) is oversold.

2026-06-11 06:20:23,070 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 82.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 82.

2026-06-11 06:20:23,151 - alerts_system - WARNING - Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:24,850 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 82.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 82.

2026-06-11 06:20:24,908 - alerts_system - WARNING - Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:27,008 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 82.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 82.

2026-06-11 06:20:27,062 - alerts_system - WARNING - Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 82: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:31,495 - alerts_system - ERROR - Failed to send notification for alert 82 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 82 after 3 retries.

2026-06-11 06:20:31,504 - alerts_system - WARNING - Alert triggered at step 83: RSI (25.94) is oversold.

WARNING:alerts_system:Alert triggered at step 83: RSI (25.94) is oversold.

2026-06-11 06:20:31,508 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 83.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 83.

2026-06-11 06:20:31,564 - alerts_system - WARNING - Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:32,722 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 83.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 83.

2026-06-11 06:20:32,780 - alerts_system - WARNING - Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:35,450 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 83.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 83.

2026-06-11 06:20:35,512 - alerts_system - WARNING - Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 83: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:40,028 - alerts_system - ERROR - Failed to send notification for alert 83 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 83 after 3 retries.

2026-06-11 06:20:40,048 - alerts_system - WARNING - Alert triggered at step 88: RSI (74.14) is overbought.

WARNING:alerts_system:Alert triggered at step 88: RSI (74.14) is overbought.

2026-06-11 06:20:40,049 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 88.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 88.

2026-06-11 06:20:40,107 - alerts_system - WARNING - Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:41,864 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 88.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 88.

2026-06-11 06:20:41,922 - alerts_system - WARNING - Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:44,814 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 88.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 88.

2026-06-11 06:20:44,873 - alerts_system - WARNING - Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 88: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:49,771 - alerts_system - ERROR - Failed to send notification for alert 88 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 88 after 3 retries.

2026-06-11 06:20:49,781 - alerts_system - WARNING - Alert triggered at step 89: RSI (78.90) is overbought.

WARNING:alerts_system:Alert triggered at step 89: RSI (78.90) is overbought.

2026-06-11 06:20:49,784 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 89.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 89.

2026-06-11 06:20:49,843 - alerts_system - WARNING - Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:51,810 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 89.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 89.

2026-06-11 06:20:51,865 - alerts_system - WARNING - Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:54,187 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 89.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 89.

2026-06-11 06:20:54,265 - alerts_system - WARNING - Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 89: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:20:59,003 - alerts_system - ERROR - Failed to send notification for alert 89 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 89 after 3 retries.

2026-06-11 06:20:59,022 - alerts_system - WARNING - Alert triggered at step 91: RSI (72.43) is overbought.

WARNING:alerts_system:Alert triggered at step 91: RSI (72.43) is overbought.

2026-06-11 06:20:59,026 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 91.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 91.

2026-06-11 06:20:59,083 - alerts_system - WARNING - Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:00,308 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 91.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 91.

2026-06-11 06:21:00,363 - alerts_system - WARNING - Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:02,406 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 91.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 91.

2026-06-11 06:21:02,462 - alerts_system - WARNING - Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 91: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:07,254 - alerts_system - ERROR - Failed to send notification for alert 91 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 91 after 3 retries.

2026-06-11 06:21:07,262 - alerts_system - WARNING - Alert triggered at step 92: RSI (76.22) is overbought.

WARNING:alerts_system:Alert triggered at step 92: RSI (76.22) is overbought.

2026-06-11 06:21:07,264 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 92.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 92.

2026-06-11 06:21:07,315 - alerts_system - WARNING - Error sending notification for alert 92: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 92: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:09,308 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 92.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 92.

2026-06-11 06:21:09,310 - alerts_system - WARNING - Error sending notification for alert 92: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 92: Simulated network error. Retrying...

2026-06-11 06:21:11,690 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 92.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 92.

2026-06-11 06:21:11,693 - alerts_system - WARNING - Error sending notification for alert 92: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 92: Simulated network error. Retrying...

2026-06-11 06:21:16,303 - alerts_system - ERROR - Failed to send notification for alert 92 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 92 after 3 retries.

2026-06-11 06:21:16,311 - alerts_system - WARNING - Alert triggered at step 93: RSI (77.22) is overbought.

WARNING:alerts_system:Alert triggered at step 93: RSI (77.22) is overbought.

2026-06-11 06:21:16,313 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 93.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 93.

2026-06-11 06:21:16,316 - alerts_system - WARNING - Error sending notification for alert 93: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 93: Simulated network error. Retrying...

2026-06-11 06:21:17,554 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 93.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 93.

2026-06-11 06:21:17,555 - alerts_system - WARNING - Error sending notification for alert 93: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 93: Simulated network error. Retrying...

2026-06-11 06:21:20,308 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 93.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 93.

2026-06-11 06:21:20,362 - alerts_system - WARNING - Error sending notification for alert 93: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 93: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:25,122 - alerts_system - ERROR - Failed to send notification for alert 93 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 93 after 3 retries.

2026-06-11 06:21:25,132 - alerts_system - WARNING - Alert triggered at step 94: RSI (78.81) is overbought.

WARNING:alerts_system:Alert triggered at step 94: RSI (78.81) is overbought.

2026-06-11 06:21:25,135 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 94.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 94.

2026-06-11 06:21:25,192 - alerts_system - WARNING - Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:26,937 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 94.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 94.

2026-06-11 06:21:26,991 - alerts_system - WARNING - Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:29,097 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 94.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 94.

2026-06-11 06:21:29,170 - alerts_system - WARNING - Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 94: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:33,959 - alerts_system - ERROR - Failed to send notification for alert 94 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 94 after 3 retries.

2026-06-11 06:21:33,970 - alerts_system - WARNING - Alert triggered at step 95: RSI (73.49) is overbought.

WARNING:alerts_system:Alert triggered at step 95: RSI (73.49) is overbought.

2026-06-11 06:21:33,974 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 95.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 95.

2026-06-11 06:21:34,031 - alerts_system - WARNING - Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:35,937 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 95.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 95.

2026-06-11 06:21:35,990 - alerts_system - WARNING - Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:38,557 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 95.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 95.

2026-06-11 06:21:38,611 - alerts_system - WARNING - Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 95: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:43,040 - alerts_system - ERROR - Failed to send notification for alert 95 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 95 after 3 retries.

2026-06-11 06:21:43,049 - alerts_system - WARNING - Alert triggered at step 96: RSI (77.24) is overbought.

WARNING:alerts_system:Alert triggered at step 96: RSI (77.24) is overbought.

2026-06-11 06:21:43,051 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 96.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 96.

2026-06-11 06:21:43,104 - alerts_system - WARNING - Error sending notification for alert 96: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 96: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:44,632 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 96.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 96.

2026-06-11 06:21:44,634 - alerts_system - WARNING - Error sending notification for alert 96: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 96: Simulated network error. Retrying...

2026-06-11 06:21:46,906 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 96.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 96.

2026-06-11 06:21:46,909 - alerts_system - WARNING - Error sending notification for alert 96: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 96: Simulated network error. Retrying...

2026-06-11 06:21:50,920 - alerts_system - ERROR - Failed to send notification for alert 96 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 96 after 3 retries.

2026-06-11 06:21:50,927 - alerts_system - WARNING - Alert triggered at step 97: RSI (70.63) is overbought.

WARNING:alerts_system:Alert triggered at step 97: RSI (70.63) is overbought.

2026-06-11 06:21:50,930 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 97.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 97.

2026-06-11 06:21:50,982 - alerts_system - WARNING - Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:52,516 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 97.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 97.

2026-06-11 06:21:52,568 - alerts_system - WARNING - Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:21:55,518 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 97.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 97.

2026-06-11 06:21:55,593 - alerts_system - WARNING - Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 97: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:00,188 - alerts_system - ERROR - Failed to send notification for alert 97 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 97 after 3 retries.

2026-06-11 06:22:00,232 - alerts_system - WARNING - Alert triggered at step 114: RSI (26.04) is oversold.

WARNING:alerts_system:Alert triggered at step 114: RSI (26.04) is oversold.

2026-06-11 06:22:00,236 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 114.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 114.

2026-06-11 06:22:00,295 - alerts_system - WARNING - Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:01,484 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 114.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 114.

2026-06-11 06:22:01,547 - alerts_system - WARNING - Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:03,583 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 114.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 114.

2026-06-11 06:22:03,653 - alerts_system - WARNING - Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 114: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:08,144 - alerts_system - ERROR - Failed to send notification for alert 114 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 114 after 3 retries.

2026-06-11 06:22:08,151 - alerts_system - WARNING - Alert triggered at step 115: RSI (24.02) is oversold.

WARNING:alerts_system:Alert triggered at step 115: RSI (24.02) is oversold.

2026-06-11 06:22:08,153 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 115.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 115.

2026-06-11 06:22:08,209 - alerts_system - WARNING - Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:09,851 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 115.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 115.

2026-06-11 06:22:09,912 - alerts_system - WARNING - Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:12,801 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 115.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 115.

2026-06-11 06:22:12,854 - alerts_system - WARNING - Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 115: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:17,209 - alerts_system - ERROR - Failed to send notification for alert 115 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 115 after 3 retries.

2026-06-11 06:22:17,225 - alerts_system - WARNING - Alert triggered at step 118: RSI (27.88) is oversold.

WARNING:alerts_system:Alert triggered at step 118: RSI (27.88) is oversold.

2026-06-11 06:22:17,228 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 118.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 118.

2026-06-11 06:22:17,231 - alerts_system - WARNING - Error sending notification for alert 118: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 118: Simulated network error. Retrying...

2026-06-11 06:22:18,687 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 118.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 118.

2026-06-11 06:22:18,743 - alerts_system - WARNING - Error sending notification for alert 118: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 118: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:21,073 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 118.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 118.

2026-06-11 06:22:21,132 - alerts_system - WARNING - Error sending notification for alert 118: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 118: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:25,421 - alerts_system - ERROR - Failed to send notification for alert 118 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 118 after 3 retries.

2026-06-11 06:22:25,454 - alerts_system - WARNING - Alert triggered at step 130: RSI (25.62) is oversold.

WARNING:alerts_system:Alert triggered at step 130: RSI (25.62) is oversold.

2026-06-11 06:22:25,457 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 130.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 130.

2026-06-11 06:22:25,538 - alerts_system - WARNING - Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:26,937 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 130.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 130.

2026-06-11 06:22:26,996 - alerts_system - WARNING - Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:29,645 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 130.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 130.

2026-06-11 06:22:29,700 - alerts_system - WARNING - Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 130: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:34,190 - alerts_system - ERROR - Failed to send notification for alert 130 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 130 after 3 retries.

2026-06-11 06:22:34,200 - alerts_system - WARNING - Alert triggered at step 131: RSI (13.00) is oversold.

WARNING:alerts_system:Alert triggered at step 131: RSI (13.00) is oversold.

2026-06-11 06:22:34,203 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 131.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 131.

2026-06-11 06:22:34,206 - alerts_system - WARNING - Error sending notification for alert 131: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 131: Simulated network error. Retrying...

2026-06-11 06:22:36,198 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 131.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 131.

2026-06-11 06:22:36,260 - alerts_system - WARNING - Error sending notification for alert 131: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 131: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:38,409 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 131.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 131.

2026-06-11 06:22:38,464 - alerts_system - WARNING - Error sending notification for alert 131: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 131: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:43,260 - alerts_system - ERROR - Failed to send notification for alert 131 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 131 after 3 retries.

2026-06-11 06:22:43,271 - alerts_system - WARNING - Alert triggered at step 132: RSI (12.80) is oversold.

WARNING:alerts_system:Alert triggered at step 132: RSI (12.80) is oversold.

2026-06-11 06:22:43,275 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 132.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 132.

2026-06-11 06:22:43,327 - alerts_system - WARNING - Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:44,527 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 132.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 132.

2026-06-11 06:22:44,584 - alerts_system - WARNING - Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:46,641 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 132.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 132.

2026-06-11 06:22:46,697 - alerts_system - WARNING - Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 132: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:50,744 - alerts_system - ERROR - Failed to send notification for alert 132 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 132 after 3 retries.

2026-06-11 06:22:50,753 - alerts_system - WARNING - Alert triggered at step 133: RSI (13.68) is oversold.

WARNING:alerts_system:Alert triggered at step 133: RSI (13.68) is oversold.

2026-06-11 06:22:50,757 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 133.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 133.

2026-06-11 06:22:50,817 - alerts_system - WARNING - Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:52,795 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 133.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 133.

2026-06-11 06:22:52,850 - alerts_system - WARNING - Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:55,223 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 133.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 133.

2026-06-11 06:22:55,298 - alerts_system - WARNING - Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 133: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:22:59,756 - alerts_system - ERROR - Failed to send notification for alert 133 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 133 after 3 retries.

2026-06-11 06:22:59,764 - alerts_system - WARNING - Alert triggered at step 134: RSI (9.40) is oversold.

WARNING:alerts_system:Alert triggered at step 134: RSI (9.40) is oversold.

2026-06-11 06:22:59,767 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 134.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 134.

2026-06-11 06:22:59,769 - alerts_system - WARNING - Error sending notification for alert 134: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 134: Simulated network error. Retrying...

2026-06-11 06:23:01,375 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 134.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 134.

2026-06-11 06:23:01,430 - alerts_system - WARNING - Error sending notification for alert 134: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 134: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:04,426 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 134.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 134.

2026-06-11 06:23:04,481 - alerts_system - WARNING - Error sending notification for alert 134: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 134: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:09,100 - alerts_system - ERROR - Failed to send notification for alert 134 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 134 after 3 retries.

2026-06-11 06:23:09,109 - alerts_system - WARNING - Alert triggered at step 135: RSI (11.26) is oversold.

WARNING:alerts_system:Alert triggered at step 135: RSI (11.26) is oversold.

2026-06-11 06:23:09,112 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 135.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 135.

2026-06-11 06:23:09,185 - alerts_system - WARNING - Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:10,954 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 135.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 135.

2026-06-11 06:23:11,011 - alerts_system - WARNING - Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:13,580 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 135.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 135.

2026-06-11 06:23:13,638 - alerts_system - WARNING - Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 135: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:18,306 - alerts_system - ERROR - Failed to send notification for alert 135 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 135 after 3 retries.

2026-06-11 06:23:18,319 - alerts_system - WARNING - Alert triggered at step 136: RSI (11.43) is oversold.

WARNING:alerts_system:Alert triggered at step 136: RSI (11.43) is oversold.

2026-06-11 06:23:18,322 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 136.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 136.

2026-06-11 06:23:18,381 - alerts_system - WARNING - Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:19,617 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 136.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 136.

2026-06-11 06:23:19,672 - alerts_system - WARNING - Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:21,683 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 136.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 136.

2026-06-11 06:23:21,746 - alerts_system - WARNING - Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 136: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:26,253 - alerts_system - ERROR - Failed to send notification for alert 136 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 136 after 3 retries.

2026-06-11 06:23:26,262 - alerts_system - WARNING - Alert triggered at step 137: RSI (14.26) is oversold.

WARNING:alerts_system:Alert triggered at step 137: RSI (14.26) is oversold.

2026-06-11 06:23:26,263 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 137.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 137.

2026-06-11 06:23:26,340 - alerts_system - WARNING - Error sending notification for alert 137: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 137: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:27,857 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 137.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 137.

2026-06-11 06:23:27,919 - alerts_system - WARNING - Error sending notification for alert 137: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 137: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:30,424 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 137.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 137.

2026-06-11 06:23:30,426 - alerts_system - WARNING - Error sending notification for alert 137: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 137: Simulated network error. Retrying...

2026-06-11 06:23:34,583 - alerts_system - ERROR - Failed to send notification for alert 137 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 137 after 3 retries.

2026-06-11 06:23:34,594 - alerts_system - WARNING - Alert triggered at step 138: RSI (4.93) is oversold.

WARNING:alerts_system:Alert triggered at step 138: RSI (4.93) is oversold.

2026-06-11 06:23:34,599 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 138.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 138.

2026-06-11 06:23:34,603 - alerts_system - WARNING - Error sending notification for alert 138: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 138: Simulated network error. Retrying...

2026-06-11 06:23:35,848 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 138.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 138.

2026-06-11 06:23:35,901 - alerts_system - WARNING - Error sending notification for alert 138: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 138: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:38,037 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 138.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 138.

2026-06-11 06:23:38,089 - alerts_system - WARNING - Error sending notification for alert 138: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 138: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:43,052 - alerts_system - ERROR - Failed to send notification for alert 138 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 138 after 3 retries.

2026-06-11 06:23:43,060 - alerts_system - WARNING - Alert triggered at step 139: RSI (4.43) is oversold.

WARNING:alerts_system:Alert triggered at step 139: RSI (4.43) is oversold.

2026-06-11 06:23:43,062 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 139.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 139.

2026-06-11 06:23:43,122 - alerts_system - WARNING - Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:44,751 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 139.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 139.

2026-06-11 06:23:44,813 - alerts_system - WARNING - Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:47,769 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 139.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 139.

2026-06-11 06:23:47,822 - alerts_system - WARNING - Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 139: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:52,157 - alerts_system - ERROR - Failed to send notification for alert 139 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 139 after 3 retries.

2026-06-11 06:23:52,166 - alerts_system - WARNING - Alert triggered at step 140: RSI (15.33) is oversold.

WARNING:alerts_system:Alert triggered at step 140: RSI (15.33) is oversold.

2026-06-11 06:23:52,168 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 140.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 140.

2026-06-11 06:23:52,231 - alerts_system - WARNING - Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:54,133 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 140.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 140.

2026-06-11 06:23:54,214 - alerts_system - WARNING - Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:23:56,283 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 140.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 140.

2026-06-11 06:23:56,342 - alerts_system - WARNING - Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 140: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:00,920 - alerts_system - ERROR - Failed to send notification for alert 140 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 140 after 3 retries.

2026-06-11 06:24:00,928 - alerts_system - WARNING - Alert triggered at step 141: RSI (14.63) is oversold.

WARNING:alerts_system:Alert triggered at step 141: RSI (14.63) is oversold.

2026-06-11 06:24:00,930 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 141.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 141.

2026-06-11 06:24:00,932 - alerts_system - WARNING - Error sending notification for alert 141: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 141: Simulated network error. Retrying...

2026-06-11 06:24:02,735 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 141.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 141.

2026-06-11 06:24:02,817 - alerts_system - WARNING - Error sending notification for alert 141: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 141: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:05,787 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 141.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 141.

2026-06-11 06:24:05,790 - alerts_system - WARNING - Error sending notification for alert 141: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 141: Simulated network error. Retrying...

2026-06-11 06:24:10,198 - alerts_system - ERROR - Failed to send notification for alert 141 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 141 after 3 retries.

2026-06-11 06:24:10,209 - alerts_system - WARNING - Alert triggered at step 142: RSI (27.07) is oversold.

WARNING:alerts_system:Alert triggered at step 142: RSI (27.07) is oversold.

2026-06-11 06:24:10,211 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 142.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 142.

2026-06-11 06:24:10,267 - alerts_system - WARNING - Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:12,185 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 142.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 142.

2026-06-11 06:24:12,240 - alerts_system - WARNING - Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:14,382 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 142.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 142.

2026-06-11 06:24:14,441 - alerts_system - WARNING - Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 142: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:19,258 - alerts_system - ERROR - Failed to send notification for alert 142 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 142 after 3 retries.

2026-06-11 06:24:19,271 - alerts_system - WARNING - Alert triggered at step 143: RSI (24.98) is oversold.

WARNING:alerts_system:Alert triggered at step 143: RSI (24.98) is oversold.

2026-06-11 06:24:19,275 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 143.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 143.

2026-06-11 06:24:19,344 - alerts_system - WARNING - Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:20,855 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 143.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 143.

2026-06-11 06:24:20,920 - alerts_system - WARNING - Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:23,502 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 143.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 143.

2026-06-11 06:24:23,556 - alerts_system - WARNING - Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 143: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:27,612 - alerts_system - ERROR - Failed to send notification for alert 143 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 143 after 3 retries.

2026-06-11 06:24:27,621 - alerts_system - WARNING - Alert triggered at step 144: RSI (27.28) is oversold.

WARNING:alerts_system:Alert triggered at step 144: RSI (27.28) is oversold.

2026-06-11 06:24:27,623 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 144.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 144.

2026-06-11 06:24:27,682 - alerts_system - WARNING - Error sending notification for alert 144: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 144: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:29,033 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 144.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 144.

2026-06-11 06:24:29,036 - alerts_system - WARNING - Error sending notification for alert 144: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 144: Simulated network error. Retrying...

2026-06-11 06:24:31,777 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 144.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 144.

2026-06-11 06:24:31,837 - alerts_system - WARNING - Error sending notification for alert 144: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 144: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:36,373 - alerts_system - ERROR - Failed to send notification for alert 144 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 144 after 3 retries.

2026-06-11 06:24:36,380 - alerts_system - WARNING - Alert triggered at step 145: RSI (28.39) is oversold.

WARNING:alerts_system:Alert triggered at step 145: RSI (28.39) is oversold.

2026-06-11 06:24:36,382 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 145.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 145.

2026-06-11 06:24:36,452 - alerts_system - WARNING - Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:37,505 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 145.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 145.

2026-06-11 06:24:37,560 - alerts_system - WARNING - Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:40,054 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 145.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 145.

2026-06-11 06:24:40,116 - alerts_system - WARNING - Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 145: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:45,119 - alerts_system - ERROR - Failed to send notification for alert 145 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 145 after 3 retries.

2026-06-11 06:24:45,127 - alerts_system - WARNING - Alert triggered at step 146: RSI (29.47) is oversold.

WARNING:alerts_system:Alert triggered at step 146: RSI (29.47) is oversold.

2026-06-11 06:24:45,131 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 146.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 146.

2026-06-11 06:24:45,190 - alerts_system - WARNING - Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:46,701 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 146.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 146.

2026-06-11 06:24:46,762 - alerts_system - WARNING - Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:49,011 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 146.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 146.

2026-06-11 06:24:49,066 - alerts_system - WARNING - Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 146: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:53,904 - alerts_system - ERROR - Failed to send notification for alert 146 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 146 after 3 retries.

2026-06-11 06:24:53,971 - alerts_system - WARNING - Alert triggered at step 172: RSI (75.61) is overbought.

WARNING:alerts_system:Alert triggered at step 172: RSI (75.61) is overbought.

2026-06-11 06:24:53,975 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 172.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 172.

2026-06-11 06:24:54,034 - alerts_system - WARNING - Error sending notification for alert 172: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 172: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:24:55,167 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 172.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 172.

2026-06-11 06:24:55,170 - alerts_system - WARNING - Error sending notification for alert 172: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 172: Simulated network error. Retrying...

2026-06-11 06:24:57,999 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 172.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 172.

2026-06-11 06:24:58,050 - alerts_system - WARNING - Error sending notification for alert 172: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 172: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:02,148 - alerts_system - ERROR - Failed to send notification for alert 172 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 172 after 3 retries.

2026-06-11 06:25:02,160 - alerts_system - WARNING - Alert triggered at step 174: RSI (78.70) is overbought.

WARNING:alerts_system:Alert triggered at step 174: RSI (78.70) is overbought.

2026-06-11 06:25:02,163 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 174.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 174.

2026-06-11 06:25:02,217 - alerts_system - WARNING - Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:03,892 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 174.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 174.

2026-06-11 06:25:03,945 - alerts_system - WARNING - Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:06,737 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 174.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 174.

2026-06-11 06:25:06,797 - alerts_system - WARNING - Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 174: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:11,519 - alerts_system - ERROR - Failed to send notification for alert 174 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 174 after 3 retries.

2026-06-11 06:25:11,528 - alerts_system - WARNING - Alert triggered at step 175: RSI (79.08) is overbought.

WARNING:alerts_system:Alert triggered at step 175: RSI (79.08) is overbought.

2026-06-11 06:25:11,533 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 175.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 175.

2026-06-11 06:25:11,606 - alerts_system - WARNING - Error sending notification for alert 175: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 175: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:12,959 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 175.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 175.

2026-06-11 06:25:13,015 - alerts_system - WARNING - Error sending notification for alert 175: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 175: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:15,800 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 175.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 175.

2026-06-11 06:25:15,803 - alerts_system - WARNING - Error sending notification for alert 175: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 175: Simulated network error. Retrying...

2026-06-11 06:25:20,743 - alerts_system - ERROR - Failed to send notification for alert 175 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 175 after 3 retries.

2026-06-11 06:25:20,752 - alerts_system - WARNING - Alert triggered at step 176: RSI (78.85) is overbought.

WARNING:alerts_system:Alert triggered at step 176: RSI (78.85) is overbought.

2026-06-11 06:25:20,754 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 176.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 176.

2026-06-11 06:25:20,809 - alerts_system - WARNING - Error sending notification for alert 176: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 176: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:22,077 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 176.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 176.

2026-06-11 06:25:22,135 - alerts_system - WARNING - Error sending notification for alert 176: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 176: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:24,880 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 176.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 176.

2026-06-11 06:25:24,882 - alerts_system - WARNING - Error sending notification for alert 176: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 176: Simulated network error. Retrying...

2026-06-11 06:25:28,899 - alerts_system - ERROR - Failed to send notification for alert 176 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 176 after 3 retries.

2026-06-11 06:25:28,907 - alerts_system - WARNING - Alert triggered at step 177: RSI (79.59) is overbought.

WARNING:alerts_system:Alert triggered at step 177: RSI (79.59) is overbought.

2026-06-11 06:25:28,910 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 177.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 177.

2026-06-11 06:25:28,965 - alerts_system - WARNING - Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:30,903 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 177.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 177.

2026-06-11 06:25:30,957 - alerts_system - WARNING - Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:33,437 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 177.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 177.

2026-06-11 06:25:33,493 - alerts_system - WARNING - Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 177: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:37,661 - alerts_system - ERROR - Failed to send notification for alert 177 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 177 after 3 retries.

2026-06-11 06:25:37,683 - alerts_system - WARNING - Alert triggered at step 182: RSI (70.67) is overbought.

WARNING:alerts_system:Alert triggered at step 182: RSI (70.67) is overbought.

2026-06-11 06:25:37,686 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 182.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 182.

2026-06-11 06:25:37,743 - alerts_system - WARNING - Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:39,314 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 182.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 182.

2026-06-11 06:25:39,387 - alerts_system - WARNING - Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:41,487 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 182.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 182.

2026-06-11 06:25:41,562 - alerts_system - WARNING - Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 182: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:45,574 - alerts_system - ERROR - Failed to send notification for alert 182 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 182 after 3 retries.

2026-06-11 06:25:45,582 - alerts_system - WARNING - Alert triggered at step 183: RSI (76.95) is overbought.

WARNING:alerts_system:Alert triggered at step 183: RSI (76.95) is overbought.

2026-06-11 06:25:45,584 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 183.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 183.

2026-06-11 06:25:45,646 - alerts_system - WARNING - Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:46,788 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 183.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 183.

2026-06-11 06:25:46,842 - alerts_system - WARNING - Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:48,895 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 183.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 183.

2026-06-11 06:25:48,948 - alerts_system - WARNING - Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 183: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:53,231 - alerts_system - ERROR - Failed to send notification for alert 183 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 183 after 3 retries.

2026-06-11 06:25:53,239 - alerts_system - WARNING - Alert triggered at step 184: RSI (73.74) is overbought.

WARNING:alerts_system:Alert triggered at step 184: RSI (73.74) is overbought.

2026-06-11 06:25:53,242 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 184.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 184.

2026-06-11 06:25:53,299 - alerts_system - WARNING - Error sending notification for alert 184: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 184: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:25:54,607 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 184.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 184.

2026-06-11 06:25:54,610 - alerts_system - WARNING - Error sending notification for alert 184: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 184: Simulated network error. Retrying...

2026-06-11 06:25:57,252 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 184.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 184.

2026-06-11 06:25:57,307 - alerts_system - WARNING - Error sending notification for alert 184: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 184: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:02,127 - alerts_system - ERROR - Failed to send notification for alert 184 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 184 after 3 retries.

2026-06-11 06:26:02,144 - alerts_system - WARNING - Alert triggered at step 187: RSI (70.35) is overbought.

WARNING:alerts_system:Alert triggered at step 187: RSI (70.35) is overbought.

2026-06-11 06:26:02,147 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 187.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 187.

2026-06-11 06:26:02,200 - alerts_system - WARNING - Error sending notification for alert 187: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 187: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:03,237 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 187.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 187.

2026-06-11 06:26:03,239 - alerts_system - WARNING - Error sending notification for alert 187: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 187: Simulated network error. Retrying...

2026-06-11 06:26:05,925 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 187.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 187.

2026-06-11 06:26:05,985 - alerts_system - WARNING - Error sending notification for alert 187: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 187: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:10,460 - alerts_system - ERROR - Failed to send notification for alert 187 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 187 after 3 retries.

2026-06-11 06:26:10,467 - alerts_system - WARNING - Alert triggered at step 188: RSI (70.10) is overbought.

WARNING:alerts_system:Alert triggered at step 188: RSI (70.10) is overbought.

2026-06-11 06:26:10,471 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 188.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 188.

2026-06-11 06:26:10,551 - alerts_system - WARNING - Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:11,807 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 188.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 188.

2026-06-11 06:26:11,878 - alerts_system - WARNING - Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:13,973 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 188.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 188.

2026-06-11 06:26:14,030 - alerts_system - WARNING - Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 188: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:18,655 - alerts_system - ERROR - Failed to send notification for alert 188 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 188 after 3 retries.

2026-06-11 06:26:18,669 - alerts_system - WARNING - Alert triggered at step 190: RSI (71.42) is overbought.

WARNING:alerts_system:Alert triggered at step 190: RSI (71.42) is overbought.

2026-06-11 06:26:18,673 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 190.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 190.

2026-06-11 06:26:18,731 - alerts_system - WARNING - Error sending notification for alert 190: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 190: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:19,818 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 190.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 190.

2026-06-11 06:26:19,821 - alerts_system - WARNING - Error sending notification for alert 190: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 190: Simulated network error. Retrying...

2026-06-11 06:26:22,558 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 190.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 190.

2026-06-11 06:26:22,614 - alerts_system - WARNING - Error sending notification for alert 190: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 190: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:26,866 - alerts_system - ERROR - Failed to send notification for alert 190 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 190 after 3 retries.

2026-06-11 06:26:26,876 - alerts_system - WARNING - Alert triggered at step 191: RSI (72.62) is overbought.

WARNING:alerts_system:Alert triggered at step 191: RSI (72.62) is overbought.

2026-06-11 06:26:26,878 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 191.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 191.

2026-06-11 06:26:26,936 - alerts_system - WARNING - Error sending notification for alert 191: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 191: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:28,140 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 191.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 191.

2026-06-11 06:26:28,143 - alerts_system - WARNING - Error sending notification for alert 191: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 191: Simulated network error. Retrying...

2026-06-11 06:26:30,296 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 191.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 191.

2026-06-11 06:26:30,299 - alerts_system - WARNING - Error sending notification for alert 191: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 191: Simulated network error. Retrying...

2026-06-11 06:26:35,122 - alerts_system - ERROR - Failed to send notification for alert 191 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 191 after 3 retries.

2026-06-11 06:26:35,131 - alerts_system - WARNING - Alert triggered at step 192: RSI (85.21) is overbought.

WARNING:alerts_system:Alert triggered at step 192: RSI (85.21) is overbought.

2026-06-11 06:26:35,133 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 192.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 192.

2026-06-11 06:26:35,187 - alerts_system - WARNING - Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:36,351 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 192.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 192.

2026-06-11 06:26:36,405 - alerts_system - WARNING - Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:39,179 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 192.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 192.

2026-06-11 06:26:39,228 - alerts_system - WARNING - Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 192: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:43,361 - alerts_system - ERROR - Failed to send notification for alert 192 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 192 after 3 retries.

2026-06-11 06:26:43,371 - alerts_system - WARNING - Alert triggered at step 193: RSI (81.43) is overbought.

WARNING:alerts_system:Alert triggered at step 193: RSI (81.43) is overbought.

2026-06-11 06:26:43,375 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 193.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 193.

2026-06-11 06:26:43,452 - alerts_system - WARNING - Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:44,863 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 193.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 193.

2026-06-11 06:26:44,922 - alerts_system - WARNING - Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:47,591 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 193.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 193.

2026-06-11 06:26:47,663 - alerts_system - WARNING - Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 193: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:51,787 - alerts_system - ERROR - Failed to send notification for alert 193 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 193 after 3 retries.

2026-06-11 06:26:51,798 - alerts_system - WARNING - Alert triggered at step 194: RSI (82.16) is overbought.

WARNING:alerts_system:Alert triggered at step 194: RSI (82.16) is overbought.

2026-06-11 06:26:51,800 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 194.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 194.

2026-06-11 06:26:51,858 - alerts_system - WARNING - Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:53,616 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 194.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 194.

2026-06-11 06:26:53,680 - alerts_system - WARNING - Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:26:56,565 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 194.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 194.

2026-06-11 06:26:56,623 - alerts_system - WARNING - Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 194: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:01,023 - alerts_system - ERROR - Failed to send notification for alert 194 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 194 after 3 retries.

2026-06-11 06:27:01,032 - alerts_system - WARNING - Alert triggered at step 195: RSI (84.37) is overbought.

WARNING:alerts_system:Alert triggered at step 195: RSI (84.37) is overbought.

2026-06-11 06:27:01,034 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 195.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 195.

2026-06-11 06:27:01,088 - alerts_system - WARNING - Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:02,535 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 195.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 195.

2026-06-11 06:27:02,595 - alerts_system - WARNING - Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:04,652 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 195.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 195.

2026-06-11 06:27:04,714 - alerts_system - WARNING - Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 195: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:09,155 - alerts_system - ERROR - Failed to send notification for alert 195 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 195 after 3 retries.

2026-06-11 06:27:09,165 - alerts_system - WARNING - Alert triggered at step 196: RSI (86.30) is overbought.

WARNING:alerts_system:Alert triggered at step 196: RSI (86.30) is overbought.

2026-06-11 06:27:09,167 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 196.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 196.

2026-06-11 06:27:09,226 - alerts_system - WARNING - Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:10,494 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 196.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 196.

2026-06-11 06:27:10,551 - alerts_system - WARNING - Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:12,721 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 196.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 196.

2026-06-11 06:27:12,773 - alerts_system - WARNING - Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 196: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:16,795 - alerts_system - ERROR - Failed to send notification for alert 196 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 196 after 3 retries.

2026-06-11 06:27:16,805 - alerts_system - WARNING - Alert triggered at step 197: RSI (82.91) is overbought.

WARNING:alerts_system:Alert triggered at step 197: RSI (82.91) is overbought.

2026-06-11 06:27:16,808 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 197.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 197.

2026-06-11 06:27:16,872 - alerts_system - WARNING - Error sending notification for alert 197: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 197: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:18,011 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 197.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 197.

2026-06-11 06:27:18,090 - alerts_system - WARNING - Error sending notification for alert 197: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 197: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:20,323 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 197.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 197.

2026-06-11 06:27:20,325 - alerts_system - WARNING - Error sending notification for alert 197: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 197: Simulated network error. Retrying...

2026-06-11 06:27:25,241 - alerts_system - ERROR - Failed to send notification for alert 197 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 197 after 3 retries.

2026-06-11 06:27:25,250 - alerts_system - WARNING - Alert triggered at step 198: RSI (77.47) is overbought.

WARNING:alerts_system:Alert triggered at step 198: RSI (77.47) is overbought.

2026-06-11 06:27:25,253 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 198.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 198.

2026-06-11 06:27:25,310 - alerts_system - WARNING - Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:27,062 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 198.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 198.

2026-06-11 06:27:27,121 - alerts_system - WARNING - Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:29,155 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 198.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 198.

2026-06-11 06:27:29,214 - alerts_system - WARNING - Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 198: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:34,144 - alerts_system - ERROR - Failed to send notification for alert 198 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 198 after 3 retries.

2026-06-11 06:27:34,155 - alerts_system - WARNING - Alert triggered at step 199: RSI (82.08) is overbought.

WARNING:alerts_system:Alert triggered at step 199: RSI (82.08) is overbought.

2026-06-11 06:27:34,161 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 199.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 199.

2026-06-11 06:27:34,236 - alerts_system - WARNING - Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:35,920 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 199.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 199.

2026-06-11 06:27:35,977 - alerts_system - WARNING - Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:38,288 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 199.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 199.

2026-06-11 06:27:38,341 - alerts_system - WARNING - Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 199: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:42,509 - alerts_system - ERROR - Failed to send notification for alert 199 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 199 after 3 retries.

2026-06-11 06:27:42,519 - alerts_system - WARNING - Alert triggered at step 200: RSI (74.30) is overbought.

WARNING:alerts_system:Alert triggered at step 200: RSI (74.30) is overbought.

2026-06-11 06:27:42,521 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 200.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 200.

2026-06-11 06:27:42,523 - alerts_system - WARNING - Error sending notification for alert 200: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 200: Simulated network error. Retrying...

2026-06-11 06:27:44,489 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 200.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 200.

2026-06-11 06:27:44,545 - alerts_system - WARNING - Error sending notification for alert 200: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 200: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:46,600 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 200.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 200.

2026-06-11 06:27:46,654 - alerts_system - WARNING - Error sending notification for alert 200: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 200: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:50,940 - alerts_system - ERROR - Failed to send notification for alert 200 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 200 after 3 retries.

2026-06-11 06:27:50,950 - alerts_system - WARNING - Alert triggered at step 201: RSI (74.42) is overbought.

WARNING:alerts_system:Alert triggered at step 201: RSI (74.42) is overbought.

2026-06-11 06:27:50,952 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 201.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 201.

2026-06-11 06:27:51,015 - alerts_system - WARNING - Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:52,657 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 201.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 201.

2026-06-11 06:27:52,723 - alerts_system - WARNING - Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:55,288 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 201.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 201.

2026-06-11 06:27:55,386 - alerts_system - WARNING - Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 201: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:27:59,601 - alerts_system - ERROR - Failed to send notification for alert 201 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 201 after 3 retries.

2026-06-11 06:27:59,609 - alerts_system - WARNING - Alert triggered at step 202: RSI (74.10) is overbought.

WARNING:alerts_system:Alert triggered at step 202: RSI (74.10) is overbought.

2026-06-11 06:27:59,612 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 202.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 202.

2026-06-11 06:27:59,615 - alerts_system - WARNING - Error sending notification for alert 202: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 202: Simulated network error. Retrying...

2026-06-11 06:28:01,563 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 202.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 202.

2026-06-11 06:28:01,617 - alerts_system - WARNING - Error sending notification for alert 202: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 202: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:04,077 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 202.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 202.

2026-06-11 06:28:04,137 - alerts_system - WARNING - Error sending notification for alert 202: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 202: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:08,438 - alerts_system - ERROR - Failed to send notification for alert 202 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 202 after 3 retries.

2026-06-11 06:28:08,446 - alerts_system - WARNING - Alert triggered at step 203: RSI (74.00) is overbought.

WARNING:alerts_system:Alert triggered at step 203: RSI (74.00) is overbought.

2026-06-11 06:28:08,450 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 203.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 203.

2026-06-11 06:28:08,526 - alerts_system - WARNING - Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:09,574 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 203.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 203.

2026-06-11 06:28:09,629 - alerts_system - WARNING - Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:12,370 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 203.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 203.

2026-06-11 06:28:12,430 - alerts_system - WARNING - Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 203: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:17,325 - alerts_system - ERROR - Failed to send notification for alert 203 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 203 after 3 retries.

2026-06-11 06:28:17,367 - alerts_system - WARNING - Alert triggered at step 213: RSI (28.26) is oversold.

WARNING:alerts_system:Alert triggered at step 213: RSI (28.26) is oversold.

2026-06-11 06:28:17,371 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 213.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 213.

2026-06-11 06:28:17,432 - alerts_system - WARNING - Error sending notification for alert 213: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 213: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:18,908 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 213.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 213.

2026-06-11 06:28:18,959 - alerts_system - WARNING - Error sending notification for alert 213: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 213: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:21,412 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 213.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 213.

2026-06-11 06:28:21,416 - alerts_system - WARNING - Error sending notification for alert 213: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 213: Simulated network error. Retrying...

2026-06-11 06:28:26,382 - alerts_system - ERROR - Failed to send notification for alert 213 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 213 after 3 retries.

2026-06-11 06:28:26,389 - alerts_system - WARNING - Alert triggered at step 214: RSI (26.92) is oversold.

WARNING:alerts_system:Alert triggered at step 214: RSI (26.92) is oversold.

2026-06-11 06:28:26,391 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 214.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 214.

2026-06-11 06:28:26,454 - alerts_system - WARNING - Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:28,238 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 214.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 214.

2026-06-11 06:28:28,297 - alerts_system - WARNING - Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:30,777 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 214.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 214.

2026-06-11 06:28:30,834 - alerts_system - WARNING - Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 214: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:35,776 - alerts_system - ERROR - Failed to send notification for alert 214 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 214 after 3 retries.

2026-06-11 06:28:35,784 - alerts_system - WARNING - Alert triggered at step 215: RSI (29.55) is oversold.

WARNING:alerts_system:Alert triggered at step 215: RSI (29.55) is oversold.

2026-06-11 06:28:35,787 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 215.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 215.

2026-06-11 06:28:35,846 - alerts_system - WARNING - Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:36,989 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 215.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 215.

2026-06-11 06:28:37,041 - alerts_system - WARNING - Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:39,123 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 215.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 215.

2026-06-11 06:28:39,181 - alerts_system - WARNING - Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 215: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:43,710 - alerts_system - ERROR - Failed to send notification for alert 215 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 215 after 3 retries.

2026-06-11 06:28:43,717 - alerts_system - WARNING - Alert triggered at step 216: RSI (25.73) is oversold.

WARNING:alerts_system:Alert triggered at step 216: RSI (25.73) is oversold.

2026-06-11 06:28:43,719 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 216.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 216.

2026-06-11 06:28:43,775 - alerts_system - WARNING - Error sending notification for alert 216: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 216: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:45,294 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 216.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 216.

2026-06-11 06:28:45,353 - alerts_system - WARNING - Error sending notification for alert 216: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 216: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:48,177 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 216.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 216.

2026-06-11 06:28:48,180 - alerts_system - WARNING - Error sending notification for alert 216: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 216: Simulated network error. Retrying...

2026-06-11 06:28:52,607 - alerts_system - ERROR - Failed to send notification for alert 216 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 216 after 3 retries.

2026-06-11 06:28:52,620 - alerts_system - WARNING - Alert triggered at step 217: RSI (23.86) is oversold.

WARNING:alerts_system:Alert triggered at step 217: RSI (23.86) is oversold.

2026-06-11 06:28:52,625 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 217.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 217.

2026-06-11 06:28:52,709 - alerts_system - WARNING - Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:54,349 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 217.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 217.

2026-06-11 06:28:54,403 - alerts_system - WARNING - Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:28:57,269 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 217.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 217.

2026-06-11 06:28:57,341 - alerts_system - WARNING - Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 217: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:01,438 - alerts_system - ERROR - Failed to send notification for alert 217 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 217 after 3 retries.

2026-06-11 06:29:01,450 - alerts_system - WARNING - Alert triggered at step 218: RSI (24.53) is oversold.

WARNING:alerts_system:Alert triggered at step 218: RSI (24.53) is oversold.

2026-06-11 06:29:01,452 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 218.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 218.

2026-06-11 06:29:01,510 - alerts_system - WARNING - Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:02,618 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 218.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 218.

2026-06-11 06:29:02,675 - alerts_system - WARNING - Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:05,572 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 218.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 218.

2026-06-11 06:29:05,629 - alerts_system - WARNING - Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 218: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:10,020 - alerts_system - ERROR - Failed to send notification for alert 218 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 218 after 3 retries.

2026-06-11 06:29:10,029 - alerts_system - WARNING - Alert triggered at step 219: RSI (21.77) is oversold.

WARNING:alerts_system:Alert triggered at step 219: RSI (21.77) is oversold.

2026-06-11 06:29:10,033 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 219.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 219.

2026-06-11 06:29:10,103 - alerts_system - WARNING - Error sending notification for alert 219: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 219: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:11,510 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 219.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 219.

2026-06-11 06:29:11,512 - alerts_system - WARNING - Error sending notification for alert 219: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 219: Simulated network error. Retrying...

2026-06-11 06:29:14,417 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 219.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 219.

2026-06-11 06:29:14,475 - alerts_system - WARNING - Error sending notification for alert 219: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 219: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:18,481 - alerts_system - ERROR - Failed to send notification for alert 219 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 219 after 3 retries.

2026-06-11 06:29:18,491 - alerts_system - WARNING - Alert triggered at step 220: RSI (26.86) is oversold.

WARNING:alerts_system:Alert triggered at step 220: RSI (26.86) is oversold.

2026-06-11 06:29:18,494 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 220.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 220.

2026-06-11 06:29:18,496 - alerts_system - WARNING - Error sending notification for alert 220: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 220: Simulated network error. Retrying...

2026-06-11 06:29:20,361 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 220.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 220.

2026-06-11 06:29:20,416 - alerts_system - WARNING - Error sending notification for alert 220: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 220: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:23,108 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 220.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 220.

2026-06-11 06:29:23,169 - alerts_system - WARNING - Error sending notification for alert 220: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 220: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:27,767 - alerts_system - ERROR - Failed to send notification for alert 220 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 220 after 3 retries.

2026-06-11 06:29:27,835 - alerts_system - WARNING - Alert triggered at step 237: RSI (29.80) is oversold.

WARNING:alerts_system:Alert triggered at step 237: RSI (29.80) is oversold.

2026-06-11 06:29:27,838 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 237.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 237.

2026-06-11 06:29:27,902 - alerts_system - WARNING - Error sending notification for alert 237: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 237: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:29,220 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 237.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 237.

2026-06-11 06:29:29,279 - alerts_system - WARNING - Error sending notification for alert 237: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 237: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:32,085 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 237.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 237.

2026-06-11 06:29:32,087 - alerts_system - WARNING - Error sending notification for alert 237: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 237: Simulated network error. Retrying...

2026-06-11 06:29:36,580 - alerts_system - ERROR - Failed to send notification for alert 237 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 237 after 3 retries.

2026-06-11 06:29:36,587 - alerts_system - WARNING - Alert triggered at step 238: RSI (24.36) is oversold.

WARNING:alerts_system:Alert triggered at step 238: RSI (24.36) is oversold.

2026-06-11 06:29:36,590 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 238.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 238.

2026-06-11 06:29:36,662 - alerts_system - WARNING - Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:37,680 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 238.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 238.

2026-06-11 06:29:37,733 - alerts_system - WARNING - Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:40,314 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 238.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 238.

2026-06-11 06:29:40,373 - alerts_system - WARNING - Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 238: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:45,001 - alerts_system - ERROR - Failed to send notification for alert 238 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 238 after 3 retries.

2026-06-11 06:29:45,009 - alerts_system - WARNING - Alert triggered at step 239: RSI (19.55) is oversold.

WARNING:alerts_system:Alert triggered at step 239: RSI (19.55) is oversold.

2026-06-11 06:29:45,013 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 239.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 239.

2026-06-11 06:29:45,070 - alerts_system - WARNING - Error sending notification for alert 239: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 239: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:46,348 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 239.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 239.

2026-06-11 06:29:46,351 - alerts_system - WARNING - Error sending notification for alert 239: Simulated network error. Retrying...

WARNING:alerts_system:Error sending notification for alert 239: Simulated network error. Retrying...

2026-06-11 06:29:49,265 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 239.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 239.

2026-06-11 06:29:49,321 - alerts_system - WARNING - Error sending notification for alert 239: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 239: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:54,279 - alerts_system - ERROR - Failed to send notification for alert 239 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 239 after 3 retries.

2026-06-11 06:29:54,287 - alerts_system - WARNING - Alert triggered at step 240: RSI (24.12) is oversold.

WARNING:alerts_system:Alert triggered at step 240: RSI (24.12) is oversold.

2026-06-11 06:29:54,289 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 240.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 240.

2026-06-11 06:29:54,351 - alerts_system - WARNING - Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:55,598 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 240.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 240.

2026-06-11 06:29:55,654 - alerts_system - WARNING - Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:29:58,202 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 240.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 240.

2026-06-11 06:29:58,261 - alerts_system - WARNING - Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 240: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:02,788 - alerts_system - ERROR - Failed to send notification for alert 240 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 240 after 3 retries.

2026-06-11 06:30:02,794 - alerts_system - WARNING - Alert triggered at step 241: RSI (27.03) is oversold.

WARNING:alerts_system:Alert triggered at step 241: RSI (27.03) is oversold.

2026-06-11 06:30:02,797 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 241.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 241.

2026-06-11 06:30:02,855 - alerts_system - WARNING - Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:04,449 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 241.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 241.

2026-06-11 06:30:04,516 - alerts_system - WARNING - Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:06,664 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 241.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 241.

2026-06-11 06:30:06,723 - alerts_system - WARNING - Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 241: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:11,442 - alerts_system - ERROR - Failed to send notification for alert 241 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 241 after 3 retries.

2026-06-11 06:30:11,450 - alerts_system - WARNING - Alert triggered at step 242: RSI (17.80) is oversold.

WARNING:alerts_system:Alert triggered at step 242: RSI (17.80) is oversold.

2026-06-11 06:30:11,454 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 242.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 242.

2026-06-11 06:30:11,521 - alerts_system - WARNING - Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:13,178 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 242.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 242.

2026-06-11 06:30:13,233 - alerts_system - WARNING - Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:16,217 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 242.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 242.

2026-06-11 06:30:16,303 - alerts_system - WARNING - Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 242: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:21,262 - alerts_system - ERROR - Failed to send notification for alert 242 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 242 after 3 retries.

2026-06-11 06:30:21,268 - alerts_system - WARNING - Alert triggered at step 243: RSI (21.26) is oversold.

WARNING:alerts_system:Alert triggered at step 243: RSI (21.26) is oversold.

2026-06-11 06:30:21,270 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 243.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 243.

2026-06-11 06:30:21,354 - alerts_system - WARNING - Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:22,596 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 243.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 243.

2026-06-11 06:30:22,648 - alerts_system - WARNING - Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:25,460 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 243.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 243.

2026-06-11 06:30:25,514 - alerts_system - WARNING - Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 243: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:30,040 - alerts_system - ERROR - Failed to send notification for alert 243 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 243 after 3 retries.

2026-06-11 06:30:30,051 - alerts_system - WARNING - Alert triggered at step 244: RSI (21.82) is oversold.

WARNING:alerts_system:Alert triggered at step 244: RSI (21.82) is oversold.

2026-06-11 06:30:30,055 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 244.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 244.

2026-06-11 06:30:30,114 - alerts_system - WARNING - Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:31,461 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 244.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 244.

2026-06-11 06:30:31,516 - alerts_system - WARNING - Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:34,299 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 244.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 244.

2026-06-11 06:30:34,358 - alerts_system - WARNING - Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 244: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:38,887 - alerts_system - ERROR - Failed to send notification for alert 244 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 244 after 3 retries.

2026-06-11 06:30:38,906 - alerts_system - WARNING - Alert triggered at step 246: RSI (24.05) is oversold.

WARNING:alerts_system:Alert triggered at step 246: RSI (24.05) is oversold.

2026-06-11 06:30:38,910 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 246.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 246.

2026-06-11 06:30:38,970 - alerts_system - WARNING - Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:40,564 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 246.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 246.

2026-06-11 06:30:40,622 - alerts_system - WARNING - Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:42,876 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 246.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 246.

2026-06-11 06:30:42,932 - alerts_system - WARNING - Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 246: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:47,926 - alerts_system - ERROR - Failed to send notification for alert 246 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 246 after 3 retries.

2026-06-11 06:30:47,938 - alerts_system - WARNING - Alert triggered at step 247: RSI (19.99) is oversold.

WARNING:alerts_system:Alert triggered at step 247: RSI (19.99) is oversold.

2026-06-11 06:30:47,941 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 247.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 247.

2026-06-11 06:30:48,000 - alerts_system - WARNING - Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:49,746 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 247.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 247.

2026-06-11 06:30:49,824 - alerts_system - WARNING - Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:52,672 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 247.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 247.

2026-06-11 06:30:52,727 - alerts_system - WARNING - Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 247: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:57,680 - alerts_system - ERROR - Failed to send notification for alert 247 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 247 after 3 retries.

2026-06-11 06:30:57,689 - alerts_system - WARNING - Alert triggered at step 248: RSI (17.00) is oversold.

WARNING:alerts_system:Alert triggered at step 248: RSI (17.00) is oversold.

2026-06-11 06:30:57,691 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 248.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 248.

2026-06-11 06:30:57,751 - alerts_system - WARNING - Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:30:59,325 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 248.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 248.

2026-06-11 06:30:59,379 - alerts_system - WARNING - Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:31:01,507 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 248.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 248.

2026-06-11 06:31:01,564 - alerts_system - WARNING - Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 248: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:31:06,262 - alerts_system - ERROR - Failed to send notification for alert 248 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 248 after 3 retries.

2026-06-11 06:31:06,271 - alerts_system - WARNING - Alert triggered at step 249: RSI (16.89) is oversold.

WARNING:alerts_system:Alert triggered at step 249: RSI (16.89) is oversold.

2026-06-11 06:31:06,274 - alerts_system - INFO - Attempt 1/3: Sending notification for alert 249.

INFO:alerts_system:Attempt 1/3: Sending notification for alert 249.

2026-06-11 06:31:06,332 - alerts_system - WARNING - Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:31:07,960 - alerts_system - INFO - Attempt 2/3: Sending notification for alert 249.

INFO:alerts_system:Attempt 2/3: Sending notification for alert 249.

2026-06-11 06:31:08,024 - alerts_system - WARNING - Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:31:10,202 - alerts_system - INFO - Attempt 3/3: Sending notification for alert 249.

INFO:alerts_system:Attempt 3/3: Sending notification for alert 249.

2026-06-11 06:31:10,258 - alerts_system - WARNING - Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

WARNING:alerts_system:Error sending notification for alert 249: 405 Client Error: Method Not Allowed for url: https://example.com/grafana-webhook-endpoint. Retrying...

2026-06-11 06:31:15,063 - alerts_system - ERROR - Failed to send notification for alert 249 after 3 retries.

ERROR:alerts_system:Failed to send notification for alert 249 after 3 retries.

2026-06-11 06:31:15,067 - alerts_system - INFO - Demonstration data simulation complete.

INFO:alerts_system:Demonstration data simulation complete.
/usr/local/lib/python3.12/dist-packages/pandas/core/indexes/base.py:7588: FutureWarning: Dtype inference on a pandas object (Series, Index, ExtensionArray) is deprecated. The Index constructor will keep the original dtype in the future. Call `infer_objects` on the result to get the old behavior.
  return Index(sequences[0], name=names)

	price	sma	rsi	alert_signal
timestamp
2026-06-11 06:17:37.805442	99.995509	NaN	NaN	0
2026-06-11 06:18:37.823476	99.772118	NaN	NaN	0
2026-06-11 06:19:37.832763	100.023414	NaN	NaN	0
2026-06-11 06:20:37.838418	99.748019	NaN	NaN	0
2026-06-11 06:21:37.841789	99.721288	NaN	NaN	0

	price	sma	rsi	alert_signal
timestamp
2026-06-11 10:35:38.890463	93.858024	NaN	31.754604	0
2026-06-11 10:36:38.900136	92.708123	NaN	24.051017	1
2026-06-11 10:37:47.931632	92.078348	NaN	19.991328	1
2026-06-11 10:38:57.684487	92.339019	NaN	17.003624	1
2026-06-11 10:40:06.264775	91.939624	NaN	16.893239	1

Visualization: Price, SMA, RSI, and Alert Signals

This plot visualizes the simulated price data, the calculated Simple Moving Average (SMA), the Relative Strength Index (RSI), and highlights points where alerts were triggered. This helps in understanding the relationship between indicators and alert conditions.

[ ]

fig, axes = plt.subplots(3, 1, figsize=(15, 18), sharex=True)

# Plot 1: Price and SMA
axes[0].plot(df_results.index, df_results['price'], label='Price', color='blue')
axes[0].plot(df_results.index, df_results['sma'], label=f'SMA({trading_state["sma_period"]})', color='orange', linestyle='--')
alert_points = df_results[df_results['alert_signal'] == 1]
axes[0].scatter(alert_points.index, alert_points['price'], color='red', marker='^', s=100, label='Alert Trigger', zorder=5)
axes[0].set_title('Price with SMA and Alert Triggers')
axes[0].set_ylabel('Price')
axes[0].legend()
axes[0].grid(True)

# Plot 2: RSI
axes[1].plot(df_results.index, df_results['rsi'], label=f'RSI({trading_state["rsi_period"]})', color='purple')
axes[1].axhline(trading_state['alert_config']['rsi_overbought'], color='red', linestyle=':', label='RSI Overbought')
axes[1].axhline(trading_state['alert_config']['rsi_oversold'], color='green', linestyle=':', label='RSI Oversold')
axes[1].fill_between(df_results.index, trading_state['alert_config']['rsi_overbought'], 100, color='red', alpha=0.1)
axes[1].fill_between(df_results.index, 0, trading_state['alert_config']['rsi_oversold'], color='green', alpha=0.1)
alert_points_rsi = alert_points[pd.notna(alert_points['rsi'])]
axes[1].scatter(alert_points_rsi.index, alert_points_rsi['rsi'], color='red', marker='^', s=100, zorder=5)
axes[1].set_title('Relative Strength Index (RSI) with Alert Levels')
axes[1].set_ylabel('RSI Value')
axes[1].legend()
axes[1].grid(True)

# Plot 3: Alert Signals (Binary)
axes[2].plot(df_results.index, df_results['alert_signal'], label='Alert Signal', color='grey', drawstyle='steps-post')
axes[2].set_title('Binary Alert Signals Over Time')
axes[2].set_xlabel('Timestamp')
axes[2].set_ylabel('Alert (1=True)')
axes[2].set_yticks([0, 1])
axes[2].legend()
axes[2].grid(True)

plt.tight_layout()
plt.show()

alerts_logger.info('Visualization complete.')

2026-06-11 06:31:18,009 - alerts_system - INFO - Visualization complete.

INFO:alerts_system:Visualization complete.

Summary of Alert System Metrics

This section displays a summary of key metrics from the alert system's operation, providing insights into how many alerts were triggered, successfully sent, or failed.

[ ]

# Update and display metrics
metrics_state = track_metrics(metrics_state, trading_state)
metrics_df = pd.DataFrame.from_dict(metrics_state, orient='index', columns=['Value'])
metrics_df.index.name = 'Metric'
display(metrics_df)
alerts_logger.info('Metrics displayed.')

	Value
Metric
total_simulated_steps	250
total_alerts_triggered	94
total_notifications_sent	0
total_notifications_failed	94

2026-06-11 06:31:20,130 - alerts_system - INFO - Metrics displayed.

INFO:alerts_system:Metrics displayed.

Production Considerations

When deploying a trading alert system in a production environment, several factors must be carefully considered to ensure reliability, scalability, and security. Integrating with tools like Grafana, Prometheus, and external notification services requires robust practices.

Aspect	Best Practices
API Keys & Credentials	Never hardcode. Use environment variables, secret management services (e.g., Google Secret Manager, AWS Secrets Manager, HashiCorp Vault), or Colab's `userdata` for API keys and sensitive credentials (e.g., Grafana API tokens, notification webhook URLs).
Error Handling & Retries	Implement comprehensive `try/except` blocks. Use exponential backoff with jitter for retries when making external API calls (data sources, notification services) to handle transient failures and prevent overwhelming services.
Logging & Monitoring	Utilize structured logging (e.g., JSON format) for easy parsing and analysis. Integrate with centralized logging systems (e.g., Stackdriver, ELK stack). Monitor alert system health with metrics (e.g., number of alerts triggered, notification success/failure rates) using tools like Prometheus and visualize in Grafana. Set up alerts on the monitoring system itself for critical failures of the alert system.
Alert Cooldowns	Implement cooldown periods to prevent alert storms for recurring conditions. An alert should only be re-sent after a defined interval or when the condition has genuinely resolved and re-triggered.
Scalability	Design the system to handle increasing data volumes and alert complexity. Consider message queues (e.g., Kafka, RabbitMQ) for decoupling components, and stateless processing units for easy horizontal scaling.
Testing	Implement unit tests for individual functions and integration tests for the entire alert pipeline. Simulate various market conditions, including edge cases, to ensure alerts trigger correctly and notifications are sent reliably.
Security	Ensure all communications are encrypted (HTTPS). Validate incoming data to prevent injection attacks. Adhere to the principle of least privilege for any service accounts or APIs used.
Configuration Management	Externalize configurations (e.g., alert thresholds, indicator parameters) to avoid code changes for parameter adjustments. Use configuration files (e.g., YAML, JSON) or environment variables, allowing for easy updates without redeployment.
Time Synchronization	Ensure all components of the system (data sources, alert engine, notification services) are synchronized to a common time source (e.g., NTP) to prevent discrepancies in alert triggering and analysis.
Rollback Strategy	Have a clear plan for rolling back changes if a new deployment introduces critical issues, minimizing downtime and impact on trading operations.

Conclusion

This notebook has provided a foundational framework for building a trading alert system, emphasizing modularity, robust error handling, and clear state management. We've covered the simulation of trading data, calculation of key technical indicators (SMA, RSI), evaluation of alert conditions, and the simulation of sending notifications with retry logic. The visualizations demonstrate how to track and understand alert triggers in relation to market data and indicators.

By adhering to the outlined production considerations and integrating with powerful monitoring platforms like Grafana, this framework can be extended to create sophisticated and reliable trading alert solutions that enhance decision-making and operational efficiency.