Live Trading·Advanced Techniques·Advanced
Kill Switch
Implement a production hard kill switch mechanism that instantaneously stops all trading system activity, cancels every open order across all exchanges, and optionally liquidates all open positions when triggered by configurable severe risk limit breaches or critical operational failure conditions.
live-tradingsafety
Hard Kill Switch for All Trading
A "hard kill switch" in trading refers to an automated system designed to immediately halt all trading activities under specific, predefined critical conditions. Its primary purpose is to prevent catastrophic losses or uncontrolled behavior from automated trading systems (bots) in the event of extreme market volatility, system malfunctions, connectivity issues, or other unforeseen circumstances. It acts as a last line of defense, ensuring that trading operations can be quickly and decisively brought to a standstill.
Key Concepts
| Concept | Description |
|---|---|
| Automated Trigger | Conditions (e.g., rapid price drops, significant P&L deviation, high error rates) that automatically activate the kill switch. |
| Manual Override | A human-initiated command to activate or deactivate the kill switch, often used in emergencies or for system reset. |
| System Health Check | Monitoring of various system components (connectivity, API responses, latency, resource utilization) to detect operational failures. |
| Performance Monitor | Tracking of key trading metrics (P&L, drawdown, volume, order fill rates) to identify undesirable trading outcomes. |
| Trade Halting Mechanism | The actual method by which trading is stopped (e.g., cancelling all open orders, rejecting new orders, shutting down trading modules). |
| State Management | How the kill switch's status (active/inactive, triggers met) is tracked and communicated across the trading system. |
| Logging & Alerting | Comprehensive recording of kill switch events and immediate notifications to operators and stakeholders. |
2. Dependency Installation
This section installs all necessary Python packages. We'll use pandas for data handling, numpy for numerical operations, matplotlib and seaborn for visualizations, backoff for retry mechanisms, and the standard logging module.
[2]
pip install pandas numpy matplotlib seaborn backoffRequirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2) Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (2.0.2) Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0) Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2) Requirement already satisfied: backoff in /usr/local/lib/python3.12/dist-packages (2.2.1) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.2) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.63.0) Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.2) Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)
3. Library Imports
This section imports all required Python libraries. Standard libraries are imported first, followed by third-party libraries.
[3]
import logging
import time
import random
from collections import deque
from datetime import datetime, timedelta
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import backoff
# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
4. Core Functions
This section defines the core functions necessary for implementing a hard kill switch. Each function is presented with a detailed markdown header, comprehensive docstrings, type hints, and appropriate logging, as specified in the requirements.
Function Name: create_kill_switch_state
This function initializes the state dictionary for the kill switch. It sets up initial values for the kill switch status, trigger conditions, manual override, and a history of activation events. This state will be passed to and modified by other functions.
Parameters: None
Returns: (dict): An initialized dictionary representing the kill switch's current state.
[4]
def create_kill_switch_state() -> dict:
"""
Initializes the state dictionary for the hard kill switch.
The state includes:
- `is_active`: Boolean indicating if the kill switch is currently active (trading halted).
- `manual_override_active`: Boolean indicating if a manual override has activated the switch.
- `auto_triggers_met`: A list of automatic trigger conditions that have been met.
- `activation_history`: A list of dictionaries, each detailing an activation event.
- `last_status_update`: Timestamp of the last status update.
- `config`: Configuration parameters for the kill switch.
Returns
-------
dict
An initialized dictionary representing the kill switch's current state.
"""
logger.info("Initializing kill switch state.")
state = {
'is_active': False,
'manual_override_active': False,
'auto_triggers_met': [],
'activation_history': [],
'last_status_update': datetime.now(),
'config': {
'auto_trigger_thresholds': {
'pnl_drawdown_percent': -5.0, # e.g., -5% drawdown
'max_open_orders': 100, # e.g., if > 100 open orders
'error_rate_threshold': 0.1, # e.g., 10% error rate in API calls
'max_latency_ms': 500 # e.g., > 500ms API latency
},
'backoff_factor': 0.5,
'max_retries': 5
}
}
logger.debug(f"Kill switch state initialized: {state}")
return state
Function Name: activate_kill_switch
This function is responsible for activating the kill switch. When called, it updates the kill switch's state to is_active = True, logs the activation event with details about the trigger (manual or automatic), and records the timestamp. It also adds a new entry to the activation_history list.
Parameters: state (dict): The current state dictionary of the kill switch. trigger_type (str): The type of trigger that activated the kill switch (e.g., 'manual', 'auto_pnl_drawdown', 'auto_error_rate'). trigger_details (dict, optional): Additional details about the trigger, such as specific values that exceeded thresholds. Defaults to an empty dictionary.
Returns: (dict): The updated state dictionary with the kill switch activated.
[5]
def activate_kill_switch(state: dict, trigger_type: str, trigger_details: dict = None) -> dict:
"""
Activates the hard kill switch.
Updates the `is_active` status to True and logs the activation event.
If the kill switch is already active, it logs a warning.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
trigger_type : str
The type of trigger that activated the kill switch (e.g., 'manual', 'auto_pnl_drawdown').
trigger_details : dict, optional
Additional details about the trigger, such as specific values that exceeded thresholds.
Defaults to an empty dictionary.
Returns
-------
dict
The updated state dictionary with the kill switch activated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state = activate_kill_switch(state, 'manual_override')
>>> state['is_active']
True
>>> len(state['activation_history'])
1
"""
if trigger_details is None:
trigger_details = {}
if state['is_active']:
logger.warning("Kill switch is already active. Ignoring activation request.")
return state
state['is_active'] = True
state['last_status_update'] = datetime.now()
activation_event = {
'timestamp': state['last_status_update'].isoformat(),
'trigger_type': trigger_type,
'details': trigger_details
}
state['activation_history'].append(activation_event)
logger.critical(f"KILL SWITCH ACTIVATED by {trigger_type}. Details: {trigger_details}")
if trigger_type == 'manual_override':
state['manual_override_active'] = True
else:
state['auto_triggers_met'].append(trigger_type)
return state
Function Name: deactivate_kill_switch
This function deactivates the kill switch, setting its is_active status to False. It clears any automatic triggers that were met and optionally allows for a reset of the manual override status. This function is crucial for safely resuming trading operations after an incident has been resolved.
Parameters:
state (dict): The current state dictionary of the kill switch.
clear_manual_override (bool, optional): If True, also clears the manual_override_active flag. Defaults to True.
Returns: (dict): The updated state dictionary with the kill switch deactivated.
[6]
def deactivate_kill_switch(state: dict, clear_manual_override: bool = True) -> dict:
"""
Deactivates the hard kill switch, allowing trading to resume.
Resets the `is_active` status to False and clears any met auto triggers.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
clear_manual_override : bool, optional
If True, also clears the `manual_override_active` flag. Defaults to True.
Returns
-------
dict
The updated state dictionary with the kill switch deactivated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state = activate_kill_switch(state, 'manual_override')
>>> state['is_active']
True
>>> state = deactivate_kill_switch(state)
>>> state['is_active']
False
"""
if not state['is_active']:
logger.info("Kill switch is already inactive. No action needed.")
return state
state['is_active'] = False
state['auto_triggers_met'] = [] # Clear auto triggers upon deactivation
if clear_manual_override:
state['manual_override_active'] = False
state['last_status_update'] = datetime.now()
logger.info("KILL SWITCH DEACTIVATED. Trading can resume.")
return state
Function Name: check_pnl_drawdown
This function checks if the current profit and loss (P&L) drawdown percentage exceeds a predefined threshold. If the drawdown is worse (more negative) than the pnl_drawdown_percent configured in the state, it triggers the kill switch. This is a critical automated trigger to prevent significant capital loss.
Parameters: state (dict): The current state dictionary of the kill switch. current_pnl_percent (float): The current P&L as a percentage (e.g., -3.5 for a 3.5% loss).
Returns: (dict): The updated state dictionary, potentially with the kill switch activated.
[7]
def check_pnl_drawdown(state: dict, current_pnl_percent: float) -> dict:
"""
Checks if the P&L drawdown exceeds the configured threshold and activates the kill switch if it does.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
current_pnl_percent : float
The current P&L as a percentage (e.g., -3.5 for a 3.5% loss).
Returns
-------
dict
The updated state dictionary, potentially with the kill switch activated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state['config']['auto_trigger_thresholds']['pnl_drawdown_percent'] = -2.0
>>> state = check_pnl_drawdown(state, -2.5)
>>> state['is_active']
True
"""
threshold = state['config']['auto_trigger_thresholds']['pnl_drawdown_percent']
logger.debug(f"Checking P&L drawdown: current={current_pnl_percent}%, threshold={threshold}%")
if current_pnl_percent < threshold:
logger.warning(f"P&L drawdown threshold exceeded: current={current_pnl_percent}% < threshold={threshold}%")
trigger_details = {'pnl_percent': current_pnl_percent, 'threshold': threshold}
state = activate_kill_switch(state, 'auto_pnl_drawdown', trigger_details)
else:
logger.info(f"P&L drawdown within limits: current={current_pnl_percent}% >= threshold={threshold}%")
return state
Function Name: check_open_orders_count
This function monitors the current number of open orders. If this count exceeds the max_open_orders threshold defined in the kill switch configuration, it triggers the kill switch. This is crucial for preventing a system from accumulating an excessive number of unmanaged orders, which could lead to significant risk.
Parameters: state (dict): The current state dictionary of the kill switch. current_open_orders (int): The current number of open trading orders.
Returns: (dict): The updated state dictionary, potentially with the kill switch activated.
[8]
def check_open_orders_count(state: dict, current_open_orders: int) -> dict:
"""
Checks if the number of open orders exceeds the configured maximum and activates the kill switch if it does.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
current_open_orders : int
The current number of open trading orders.
Returns
-------
dict
The updated state dictionary, potentially with the kill switch activated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state['config']['auto_trigger_thresholds']['max_open_orders'] = 50
>>> state = check_open_orders_count(state, 55)
>>> state['is_active']
True
"""
threshold = state['config']['auto_trigger_thresholds']['max_open_orders']
logger.debug(f"Checking open orders count: current={current_open_orders}, threshold={threshold}")
if current_open_orders > threshold:
logger.warning(f"Max open orders threshold exceeded: current={current_open_orders} > threshold={threshold}")
trigger_details = {'open_orders': current_open_orders, 'threshold': threshold}
state = activate_kill_switch(state, 'auto_max_open_orders', trigger_details)
else:
logger.info(f"Open orders count within limits: current={current_open_orders} <= threshold={threshold}")
return state
Function Name: check_error_rate
This function assesses the current error rate (e.g., failed API calls, rejected orders) against a predefined error_rate_threshold. If the actual error rate surpasses this threshold, it indicates potential system instability or external issues, prompting the activation of the kill switch to halt trading operations.
Parameters: state (dict): The current state dictionary of the kill switch. current_error_rate (float): The current proportion of errors, as a value between 0.0 and 1.0 (e.g., 0.05 for 5% error rate).
Returns: (dict): The updated state dictionary, potentially with the kill switch activated.
[9]
def check_error_rate(state: dict, current_error_rate: float) -> dict:
"""
Checks if the current error rate exceeds the configured threshold and activates the kill switch if it does.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
current_error_rate : float
The current proportion of errors, as a value between 0.0 and 1.0 (e.g., 0.05 for 5% error rate).
Returns
-------
dict
The updated state dictionary, potentially with the kill switch activated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state['config']['auto_trigger_thresholds']['error_rate_threshold'] = 0.05
>>> state = check_error_rate(state, 0.06)
>>> state['is_active']
True
"""
threshold = state['config']['auto_trigger_thresholds']['error_rate_threshold']
logger.debug(f"Checking error rate: current={current_error_rate:.2f}, threshold={threshold:.2f}")
if current_error_rate > threshold:
logger.warning(f"Error rate threshold exceeded: current={current_error_rate:.2f} > threshold={threshold:.2f}")
trigger_details = {'error_rate': current_error_rate, 'threshold': threshold}
state = activate_kill_switch(state, 'auto_error_rate', trigger_details)
else:
logger.info(f"Error rate within limits: current={current_error_rate:.2f} <= threshold={threshold:.2f}")
return state
Function Name: check_latency
This function monitors the system's operational latency, such as API response times. If the current_latency_ms exceeds the max_latency_ms configured threshold, it indicates potential connectivity issues or system overload, leading to the activation of the kill switch. High latency can lead to stale data and mispriced orders, making it a critical trigger.
Parameters: state (dict): The current state dictionary of the kill switch. current_latency_ms (float): The current latency in milliseconds.
Returns: (dict): The updated state dictionary, potentially with the kill switch activated.
[10]
def check_latency(state: dict, current_latency_ms: float) -> dict:
"""
Checks if the current latency exceeds the configured maximum and activates the kill switch if it does.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
current_latency_ms : float
The current latency in milliseconds.
Returns
-------
dict
The updated state dictionary, potentially with the kill switch activated.
Examples
--------
>>> state = create_kill_switch_state()
>>> state['config']['auto_trigger_thresholds']['max_latency_ms'] = 200
>>> state = check_latency(state, 250)
>>> state['is_active']
True
"""
threshold = state['config']['auto_trigger_thresholds']['max_latency_ms']
logger.debug(f"Checking latency: current={current_latency_ms:.2f}ms, threshold={threshold:.2f}ms")
if current_latency_ms > threshold:
logger.warning(f"Latency threshold exceeded: current={current_latency_ms:.2f}ms > threshold={threshold:.2f}ms")
trigger_details = {'latency_ms': current_latency_ms, 'threshold': threshold}
state = activate_kill_switch(state, 'auto_max_latency', trigger_details)
else:
logger.info(f"Latency within limits: current={current_latency_ms:.2f}ms <= threshold={threshold:.2f}ms")
return state
Function Name: run_auto_trigger_checks
This function acts as an orchestrator for all automated trigger checks. It takes the current kill switch state and various real-time metrics (P&L, open orders, error rate, latency) as input. It then calls each individual check_ function to evaluate if any critical threshold has been crossed. If any check activates the kill switch, this function ensures that the updated state reflects the activation.
Parameters: state (dict): The current state dictionary of the kill switch. current_pnl_percent (float): Current profit and loss percentage. current_open_orders (int): Current count of open orders. current_error_rate (float): Current error rate (0.0 to 1.0). current_latency_ms (float): Current system latency in milliseconds.
Returns: (dict): The updated state dictionary, which might have the kill switch activated if any trigger condition was met.
[11]
def run_auto_trigger_checks(
state: dict,
current_pnl_percent: float,
current_open_orders: int,
current_error_rate: float,
current_latency_ms: float
) -> dict:
"""
Runs all automated trigger checks and updates the kill switch state.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
current_pnl_percent : float
Current profit and loss percentage.
current_open_orders : int
Current count of open orders.
current_error_rate : float
Current error rate (0.0 to 1.0).
current_latency_ms : float
Current system latency in milliseconds.
Returns
-------
dict
The updated state dictionary, which might have the kill switch activated
if any trigger condition was met.
"""
logger.info("Running automated kill switch trigger checks.")
# Check P&L drawdown
state = check_pnl_drawdown(state, current_pnl_percent)
if state['is_active']: return state # Stop if activated
# Check open orders count
state = check_open_orders_count(state, current_open_orders)
if state['is_active']: return state # Stop if activated
# Check error rate
state = check_error_rate(state, current_error_rate)
if state['is_active']: return state # Stop if activated
# Check latency
state = check_latency(state, current_latency_ms)
if state['is_active']: return state # Stop if activated
logger.info("All automated checks passed. Kill switch remains inactive.")
return state
Function Name: execute_trading_command_safely
This function wraps any critical trading operation, ensuring it only executes if the kill switch is inactive. It also incorporates a retry mechanism with exponential backoff and random jitter using the backoff library, adhering to robust engineering practices. If the kill switch is active, it will prevent the command from executing.
Parameters:
state (dict): The current state dictionary of the kill switch.
command_func (callable): The function representing the trading command to execute (e.g., place_order, cancel_all_orders).
*args: Variable length argument list for command_func.
**kwargs: Arbitrary keyword arguments for command_func.
Returns:
(any): The result of command_func if executed successfully, otherwise None.
[12]
def execute_trading_command_safely(state: dict, command_func: callable, *args, **kwargs) -> any:
"""
Executes a trading command only if the kill switch is not active.
Includes retry logic with exponential backoff and random jitter.
Parameters
----------
state : dict
The current state dictionary of the kill switch.
command_func : callable
The function representing the trading command to execute.
*args
Variable length argument list for `command_func`.
**kwargs
Arbitrary keyword arguments for `command_func`.
Returns
-------
any
The result of `command_func` if executed successfully and kill switch is inactive,
otherwise None.
Examples
--------
>>> state = create_kill_switch_state()
>>> def mock_place_order(symbol, quantity): print(f"Placing {quantity} of {symbol}"); return True
>>> execute_trading_command_safely(state, mock_place_order, 'AAPL', 10)
Placing 10 of AAPL
True
>>> state = activate_kill_switch(state, 'manual_override')
>>> execute_trading_command_safely(state, mock_place_order, 'GOOG', 5)
"""
if state['is_active']:
logger.warning(f"Kill switch is active. Blocking command: {command_func.__name__}")
return None
logger.info(f"Executing trading command: {command_func.__name__} safely.")
# Define a local helper function to be decorated, so it can access 'state' from the outer scope
@backoff.on_exception(
backoff.expo,
(IOError, TimeoutError), # Simulate network/API errors
max_tries=state['config']['max_retries'] + 1, # Pass the value directly
factor=state['config']['backoff_factor'], # Pass the value directly
jitter=backoff.full_jitter
)
def _execute_with_retry_local(func_to_retry, *f_args, **f_kwargs):
"""
Internal helper to execute a command with retry logic, defined locally
to capture the `state` configuration.
"""
return func_to_retry(*f_args, **f_kwargs)
try:
result = _execute_with_retry_local(command_func, *args, **kwargs)
logger.info(f"Command {command_func.__name__} executed successfully.")
return result
except Exception as e:
logger.error(f"Command {command_func.__name__} failed after retries: {e}")
return None5. Demonstration/Visualization
This section provides practical demonstrations of the hard kill switch functions in action, including simulations of various trading scenarios and visual representations of the kill switch's behavior. We will use mock data and functions to simulate P&L, open orders, error rates, and latency to observe how the kill switch activates and deactivates.
Simulate Trading Environment and Metrics
To demonstrate the kill switch, we need to simulate a dynamic trading environment that generates various metrics like P&L, open orders, error rates, and latency. These mock functions will provide the input for our check_ functions.
[13]
def create_mock_trading_metrics(initial_pnl: float = 0.0, initial_orders: int = 0, initial_errors: float = 0.0, initial_latency: float = 100.0) -> dict:
"""
Initializes a dictionary for mock trading metrics.
Parameters
----------
initial_pnl : float, optional
Starting P&L percentage, defaults to 0.0.
initial_orders : int, optional
Starting number of open orders, defaults to 0.
initial_errors : float, optional
Starting error rate, defaults to 0.0.
initial_latency : float, optional
Starting latency in ms, defaults to 100.0.
Returns
-------
dict
A dictionary containing the initial mock trading metrics.
"""
return {
'pnl_percent': initial_pnl,
'open_orders': initial_orders,
'error_rate': initial_errors,
'latency_ms': initial_latency
}
def simulate_metrics_step(metrics: dict, step: int) -> dict:
"""
Simulates a step in trading metrics, introducing some variability and potential trigger conditions.
Parameters
----------
metrics : dict
Current mock trading metrics.
step : int
The current simulation step, used to influence metric changes.
Returns
-------
dict
Updated mock trading metrics.
"""
new_metrics = metrics.copy()
# Simulate P&L with a general trend and some volatility
new_metrics['pnl_percent'] += np.random.normal(0.05, 0.5) # Slight positive drift, but high volatility
if step > 20 and step < 30: # Simulate a sharp drawdown
new_metrics['pnl_percent'] -= np.random.uniform(0.5, 2.0)
# Simulate open orders, increasing over time then potentially spiking
new_metrics['open_orders'] = int(metrics['open_orders'] + np.random.randint(-2, 5))
if step > 40 and step < 50: # Simulate a sudden surge in orders
new_metrics['open_orders'] += np.random.randint(10, 30)
new_metrics['open_orders'] = max(0, new_metrics['open_orders']) # Orders can't be negative
# Simulate error rate, generally low but with occasional spikes
new_metrics['error_rate'] = max(0.0, min(1.0, metrics['error_rate'] + np.random.normal(0, 0.02)))
if step > 60 and step < 70: # Simulate a period of high error rates
new_metrics['error_rate'] += np.random.uniform(0.05, 0.15)
# Simulate latency, generally stable but with occasional spikes
new_metrics['latency_ms'] = max(50.0, metrics['latency_ms'] + np.random.normal(0, 20))
if step > 80 and step < 90: # Simulate high latency
new_metrics['latency_ms'] += np.random.uniform(100, 300)
logger.debug(f"Simulated metrics at step {step}: {new_metrics}")
return new_metrics
def mock_place_order(symbol: str, quantity: int) -> bool:
"""
A mock function for placing an order.
"""
logger.info(f"Mock: Placing {quantity} of {symbol}.")
# Simulate occasional failure for retry mechanism
if random.random() < 0.1: # 10% chance of failure
raise IOError("Mock API error: Failed to place order.")
return True
def mock_cancel_all_orders() -> bool:
"""
A mock function for canceling all open orders.
"""
logger.info("Mock: Cancelling all open orders.")
return True
def mock_get_market_data(symbol: str) -> dict:
"""
A mock function for getting market data.
"""
logger.info(f"Mock: Getting market data for {symbol}.")
# Simulate occasional high latency for retry mechanism
if random.random() < 0.05: # 5% chance of delay
time.sleep(random.uniform(0.1, 0.5))
return {'price': round(random.uniform(100, 200), 2), 'timestamp': datetime.now().isoformat()}
Function Name: run_kill_switch_simulation
This function orchestrates a full simulation of the trading environment interacting with the hard kill switch. It simulates trading metrics over time, periodically runs automated trigger checks, and attempts to execute a mock trading command. All relevant metrics and the kill switch state are recorded for later analysis and visualization.
Parameters: num_steps (int): The number of simulation steps to run.
Returns: (tuple): A tuple containing:
kill_switch_state(dict): The final state of the kill switch after the simulation.simulation_data(pd.DataFrame): A DataFrame containing recorded metrics and kill switch status for each step.
[14]
def run_kill_switch_simulation(num_steps: int = 100) -> tuple[dict, pd.DataFrame]:
"""
Runs a simulation of the kill switch's behavior over multiple steps.
Parameters
----------
num_steps : int, optional
The number of simulation steps to run, defaults to 100.
Returns
-------
tuple[dict, pd.DataFrame]
A tuple containing the final kill switch state and a DataFrame of simulation data.
"""
logger.info(f"Starting kill switch simulation for {num_steps} steps.")
kill_switch_state = create_kill_switch_state()
mock_metrics = create_mock_trading_metrics(
initial_pnl=random.uniform(-1, 1),
initial_orders=random.randint(0, 50),
initial_errors=random.uniform(0, 0.01),
initial_latency=random.uniform(50, 150)
)
simulation_records = []
for step in range(num_steps):
logger.debug(f"--- Simulation Step {step + 1}/{num_steps} ---")
# 1. Simulate new trading metrics
mock_metrics = simulate_metrics_step(mock_metrics, step)
# 2. Run automated trigger checks
kill_switch_state = run_auto_trigger_checks(
kill_switch_state,
mock_metrics['pnl_percent'],
mock_metrics['open_orders'],
mock_metrics['error_rate'],
mock_metrics['latency_ms']
)
# 3. Attempt to execute a mock trading command safely
# We'll use a placeholder command here
if not kill_switch_state['is_active']:
execute_trading_command_safely(kill_switch_state, mock_place_order, 'AAPL', 10)
else:
logger.info("Trading commands blocked due to active kill switch.")
# If kill switch is active, simulate trying to cancel all orders
execute_trading_command_safely(kill_switch_state, mock_cancel_all_orders)
# 4. Occasionally simulate manual intervention
if step == 25 and not kill_switch_state['is_active']:
logger.info("Simulating manual activation of kill switch at step 25.")
kill_switch_state = activate_kill_switch(kill_switch_state, 'manual_intervention', {'reason': 'Emergency!'})
if step == 75 and kill_switch_state['is_active']:
logger.info("Simulating manual deactivation of kill switch at step 75.")
kill_switch_state = deactivate_kill_switch(kill_switch_state)
# 5. Record current state for visualization
record = {
'step': step,
'is_active': kill_switch_state['is_active'],
'pnl_percent': mock_metrics['pnl_percent'],
'open_orders': mock_metrics['open_orders'],
'error_rate': mock_metrics['error_rate'],
'latency_ms': mock_metrics['latency_ms'],
'triggers_met': ', '.join(kill_switch_state['auto_triggers_met']),
'manual_override': kill_switch_state['manual_override_active']
}
simulation_records.append(record)
# Introduce small random delay to simulate real-world processing
time.sleep(random.uniform(0.01, 0.05))
simulation_df = pd.DataFrame(simulation_records)
logger.info("Kill switch simulation completed.")
return kill_switch_state, simulation_df
Function Name: plot_kill_switch_simulation_results
This function visualizes the results of the kill switch simulation. It generates a multi-panel plot showing the trends of P&L, open orders, error rate, and latency over time, along with an overlay indicating when the kill switch was active. This helps in understanding the conditions that led to activation and the system's overall response.
Parameters: simulation_df (pd.DataFrame): A DataFrame containing the simulation records.
Returns: None: Displays the plot directly.
[15]
def plot_kill_switch_simulation_results(simulation_df: pd.DataFrame):
"""
Plots the results of the kill switch simulation.
Visualizes P&L, open orders, error rate, and latency over time,
with kill switch activation periods highlighted.
Parameters
----------
simulation_df : pd.DataFrame
A DataFrame containing the simulation records generated by `run_kill_switch_simulation`.
"""
logger.info("Generating visualization of simulation results.")
# Ensure 'is_active' is boolean for proper plotting
simulation_df['is_active_int'] = simulation_df['is_active'].astype(int)
fig, axes = plt.subplots(nrows=4, ncols=1, figsize=(14, 16), sharex=True)
fig.suptitle('Hard Kill Switch Simulation Results', fontsize=18, y=0.92)
# P&L Percentage
axes[0].plot(simulation_df['step'], simulation_df['pnl_percent'], label='P&L (%)', color='skyblue')
axes[0].axhline(y=simulation_df['pnl_percent'].mean(), color='gray', linestyle='--', linewidth=0.8, label='Avg P&L')
# Highlight kill switch active periods
for start, end in simulation_df[simulation_df['is_active']].groupby((simulation_df['is_active'] != simulation_df['is_active'].shift()).cumsum()).apply(lambda x: (x.index.min(), x.index.max())):
axes[0].axvspan(start, end, color='red', alpha=0.1, label='_Kill Switch Active' if start == simulation_df[simulation_df['is_active']].index.min() else "")
axes[0].set_ylabel('P&L (%)')
axes[0].set_title('P&L Percentage Over Time')
axes[0].grid(True, linestyle=':', alpha=0.6)
axes[0].legend()
# Open Orders Count
axes[1].plot(simulation_df['step'], simulation_df['open_orders'], label='Open Orders', color='lightgreen')
axes[1].axhline(y=simulation_df['open_orders'].mean(), color='gray', linestyle='--', linewidth=0.8, label='Avg Orders')
for start, end in simulation_df[simulation_df['is_active']].groupby((simulation_df['is_active'] != simulation_df['is_active'].shift()).cumsum()).apply(lambda x: (x.index.min(), x.index.max())):
axes[1].axvspan(start, end, color='red', alpha=0.1)
axes[1].set_ylabel('Open Orders Count')
axes[1].set_title('Open Orders Count Over Time')
axes[1].grid(True, linestyle=':', alpha=0.6)
axes[1].legend()
# Error Rate
axes[2].plot(simulation_df['step'], simulation_df['error_rate'], label='Error Rate', color='orange')
axes[2].axhline(y=simulation_df['error_rate'].mean(), color='gray', linestyle='--', linewidth=0.8, label='Avg Error Rate')
for start, end in simulation_df[simulation_df['is_active']].groupby((simulation_df['is_active'] != simulation_df['is_active'].shift()).cumsum()).apply(lambda x: (x.index.min(), x.index.max())):
axes[2].axvspan(start, end, color='red', alpha=0.1)
axes[2].set_ylabel('Error Rate')
axes[2].set_title('Error Rate Over Time')
axes[2].grid(True, linestyle=':', alpha=0.6)
axes[2].legend()
# Latency (ms)
axes[3].plot(simulation_df['step'], simulation_df['latency_ms'], label='Latency (ms)', color='lightcoral')
axes[3].axhline(y=simulation_df['latency_ms'].mean(), color='gray', linestyle='--', linewidth=0.8, label='Avg Latency')
for start, end in simulation_df[simulation_df['is_active']].groupby((simulation_df['is_active'] != simulation_df['is_active'].shift()).cumsum()).apply(lambda x: (x.index.min(), x.index.max())):
axes[3].axvspan(start, end, color='red', alpha=0.1)
axes[3].set_xlabel('Simulation Step')
axes[3].set_ylabel('Latency (ms)')
axes[3].set_title('Latency Over Time')
axes[3].grid(True, linestyle=':', alpha=0.6)
axes[3].legend()
plt.tight_layout(rect=[0, 0.03, 1, 0.9])
plt.show()
logger.info("Plotting complete.")
def display_summary_statistics(simulation_df: pd.DataFrame):
"""
Displays summary statistics of the simulation data.
Parameters
----------
simulation_df : pd.DataFrame
A DataFrame containing the simulation records.
"""
logger.info("Displaying summary statistics.")
print("\n--- Simulation Summary Statistics ---")
print("Kill Switch Active % of time: ", simulation_df['is_active'].mean() * 100, "%")
print(simulation_df[['pnl_percent', 'open_orders', 'error_rate', 'latency_ms']].describe().to_markdown(numalign="left", stralign="left"))
print("\n--- Kill Switch Activation Events ---")
for i, row in simulation_df[simulation_df['is_active'] & ~simulation_df['is_active'].shift(1).fillna(False)].iterrows():
print(f"Step {row['step']}: Activated by {row['triggers_met'] if row['triggers_met'] else 'Manual'}")
print("\n--- Kill Switch Deactivation Events ---")
for i, row in simulation_df[~simulation_df['is_active'] & simulation_df['is_active'].shift(1).fillna(False)].iterrows():
print(f"Step {row['step']}: Deactivated.")
Run Simulation and Visualize Results
Here, we execute the run_kill_switch_simulation function and then use the plot_kill_switch_simulation_results and display_summary_statistics functions to visualize and summarize the outcomes.
[16]
# Run the simulation
final_state, sim_df = run_kill_switch_simulation(num_steps=100)
# Plot the results
plot_kill_switch_simulation_results(sim_df)
# Display summary statistics
display_summary_statistics(sim_df)
INFO:__main__:Starting kill switch simulation for 100 steps.
INFO:__main__:Initializing kill switch state.
DEBUG:__main__:Kill switch state initialized: {'is_active': False, 'manual_override_active': False, 'auto_triggers_met': [], 'activation_history': [], 'last_status_update': datetime.datetime(2026, 6, 18, 6, 33, 4, 929581), 'config': {'auto_trigger_thresholds': {'pnl_drawdown_percent': -5.0, 'max_open_orders': 100, 'error_rate_threshold': 0.1, 'max_latency_ms': 500}, 'backoff_factor': 0.5, 'max_retries': 5}}
DEBUG:__main__:--- Simulation Step 1/100 ---
DEBUG:__main__:Simulated metrics at step 0: {'pnl_percent': -1.3033694102009121, 'open_orders': 22, 'error_rate': 0.03878178389809538, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.3033694102009121%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.3033694102009121% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=22, threshold=100
INFO:__main__:Open orders count within limits: current=22 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.04, threshold=0.10
INFO:__main__:Error rate within limits: current=0.04 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 2/100 ---
DEBUG:__main__:Simulated metrics at step 1: {'pnl_percent': -1.0484969411419032, 'open_orders': 23, 'error_rate': 0.044126288454599046, 'latency_ms': 61.97658090332634}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.0484969411419032%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.0484969411419032% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=23, threshold=100
INFO:__main__:Open orders count within limits: current=23 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.04, threshold=0.10
INFO:__main__:Error rate within limits: current=0.04 <= threshold=0.10
DEBUG:__main__:Checking latency: current=61.98ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=61.98ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 3/100 ---
DEBUG:__main__:Simulated metrics at step 2: {'pnl_percent': -0.6876269181082821, 'open_orders': 25, 'error_rate': 0.01702951618760889, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.6876269181082821%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.6876269181082821% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=25, threshold=100
INFO:__main__:Open orders count within limits: current=25 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.02, threshold=0.10
INFO:__main__:Error rate within limits: current=0.02 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 4/100 ---
DEBUG:__main__:Simulated metrics at step 3: {'pnl_percent': -0.9831341447173501, 'open_orders': 27, 'error_rate': 0.0, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.9831341447173501%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.9831341447173501% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=27, threshold=100
INFO:__main__:Open orders count within limits: current=27 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.00, threshold=0.10
INFO:__main__:Error rate within limits: current=0.00 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 5/100 ---
DEBUG:__main__:Simulated metrics at step 4: {'pnl_percent': -0.7420058363042676, 'open_orders': 31, 'error_rate': 0.0, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.7420058363042676%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.7420058363042676% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=31, threshold=100
INFO:__main__:Open orders count within limits: current=31 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.00, threshold=0.10
INFO:__main__:Error rate within limits: current=0.00 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 6/100 ---
DEBUG:__main__:Simulated metrics at step 5: {'pnl_percent': -1.1467840525693604, 'open_orders': 33, 'error_rate': 0.0, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.1467840525693604%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.1467840525693604% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=33, threshold=100
INFO:__main__:Open orders count within limits: current=33 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.00, threshold=0.10
INFO:__main__:Error rate within limits: current=0.00 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:backoff:Backing off _execute_with_retry_local(...) for 0.4s (OSError: Mock API error: Failed to place order.)
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:backoff:Backing off _execute_with_retry_local(...) for 0.0s (OSError: Mock API error: Failed to place order.)
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 7/100 ---
DEBUG:__main__:Simulated metrics at step 6: {'pnl_percent': -1.2922686016816987, 'open_orders': 33, 'error_rate': 0.0, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.2922686016816987%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.2922686016816987% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=33, threshold=100
INFO:__main__:Open orders count within limits: current=33 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.00, threshold=0.10
INFO:__main__:Error rate within limits: current=0.00 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 8/100 ---
DEBUG:__main__:Simulated metrics at step 7: {'pnl_percent': -1.262901822130897, 'open_orders': 33, 'error_rate': 0.01749593334794381, 'latency_ms': 55.129412288119944}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.262901822130897%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.262901822130897% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=33, threshold=100
INFO:__main__:Open orders count within limits: current=33 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.02, threshold=0.10
INFO:__main__:Error rate within limits: current=0.02 <= threshold=0.10
DEBUG:__main__:Checking latency: current=55.13ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=55.13ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 9/100 ---
DEBUG:__main__:Simulated metrics at step 8: {'pnl_percent': -1.2757394161648734, 'open_orders': 31, 'error_rate': 0.0, 'latency_ms': 63.636217458333576}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-1.2757394161648734%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-1.2757394161648734% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=31, threshold=100
INFO:__main__:Open orders count within limits: current=31 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.00, threshold=0.10
INFO:__main__:Error rate within limits: current=0.00 <= threshold=0.10
DEBUG:__main__:Checking latency: current=63.64ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=63.64ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 10/100 ---
DEBUG:__main__:Simulated metrics at step 9: {'pnl_percent': -0.820623159167494, 'open_orders': 33, 'error_rate': 0.02072423805907569, 'latency_ms': 50.0}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.820623159167494%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.820623159167494% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=33, threshold=100
INFO:__main__:Open orders count within limits: current=33 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.02, threshold=0.10
INFO:__main__:Error rate within limits: current=0.02 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.00ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.00ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 11/100 ---
DEBUG:__main__:Simulated metrics at step 10: {'pnl_percent': -0.6508723362238029, 'open_orders': 34, 'error_rate': 0.04498482654032944, 'latency_ms': 52.42589410084061}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.6508723362238029%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.6508723362238029% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=34, threshold=100
INFO:__main__:Open orders count within limits: current=34 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.04, threshold=0.10
INFO:__main__:Error rate within limits: current=0.04 <= threshold=0.10
DEBUG:__main__:Checking latency: current=52.43ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=52.43ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:backoff:Backing off _execute_with_retry_local(...) for 0.3s (OSError: Mock API error: Failed to place order.)
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 12/100 ---
DEBUG:__main__:Simulated metrics at step 11: {'pnl_percent': -0.12210374571773608, 'open_orders': 37, 'error_rate': 0.05834942831096397, 'latency_ms': 67.15119176269835}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.12210374571773608%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.12210374571773608% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=37, threshold=100
INFO:__main__:Open orders count within limits: current=37 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.06, threshold=0.10
INFO:__main__:Error rate within limits: current=0.06 <= threshold=0.10
DEBUG:__main__:Checking latency: current=67.15ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=67.15ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 13/100 ---
DEBUG:__main__:Simulated metrics at step 12: {'pnl_percent': -0.29443172670003853, 'open_orders': 38, 'error_rate': 0.05286810962873054, 'latency_ms': 50.20625547787508}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.29443172670003853%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.29443172670003853% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=38, threshold=100
INFO:__main__:Open orders count within limits: current=38 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.05, threshold=0.10
INFO:__main__:Error rate within limits: current=0.05 <= threshold=0.10
DEBUG:__main__:Checking latency: current=50.21ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=50.21ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 14/100 ---
DEBUG:__main__:Simulated metrics at step 13: {'pnl_percent': -0.2573048300390972, 'open_orders': 40, 'error_rate': 0.027307393549489255, 'latency_ms': 74.75609504862962}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.2573048300390972%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.2573048300390972% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=40, threshold=100
INFO:__main__:Open orders count within limits: current=40 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.03, threshold=0.10
INFO:__main__:Error rate within limits: current=0.03 <= threshold=0.10
DEBUG:__main__:Checking latency: current=74.76ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=74.76ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 15/100 ---
DEBUG:__main__:Simulated metrics at step 14: {'pnl_percent': -0.33594045015774354, 'open_orders': 40, 'error_rate': 0.035218319425281065, 'latency_ms': 85.85626037814514}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.33594045015774354%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.33594045015774354% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=40, threshold=100
INFO:__main__:Open orders count within limits: current=40 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.04, threshold=0.10
INFO:__main__:Error rate within limits: current=0.04 <= threshold=0.10
DEBUG:__main__:Checking latency: current=85.86ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=85.86ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 16/100 ---
DEBUG:__main__:Simulated metrics at step 15: {'pnl_percent': 0.7417137790514469, 'open_orders': 41, 'error_rate': 0.03291015842113993, 'latency_ms': 81.19051449838351}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=0.7417137790514469%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=0.7417137790514469% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=41, threshold=100
INFO:__main__:Open orders count within limits: current=41 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.03, threshold=0.10
INFO:__main__:Error rate within limits: current=0.03 <= threshold=0.10
DEBUG:__main__:Checking latency: current=81.19ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=81.19ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 17/100 ---
DEBUG:__main__:Simulated metrics at step 16: {'pnl_percent': 0.6006175941817773, 'open_orders': 39, 'error_rate': 0.0657281849795637, 'latency_ms': 104.95259629178199}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=0.6006175941817773%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=0.6006175941817773% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=39, threshold=100
INFO:__main__:Open orders count within limits: current=39 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.07, threshold=0.10
INFO:__main__:Error rate within limits: current=0.07 <= threshold=0.10
DEBUG:__main__:Checking latency: current=104.95ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=104.95ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 18/100 ---
DEBUG:__main__:Simulated metrics at step 17: {'pnl_percent': 0.3940860171828388, 'open_orders': 41, 'error_rate': 0.05065441366603539, 'latency_ms': 111.22854268099583}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=0.3940860171828388%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=0.3940860171828388% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=41, threshold=100
INFO:__main__:Open orders count within limits: current=41 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.05, threshold=0.10
INFO:__main__:Error rate within limits: current=0.05 <= threshold=0.10
DEBUG:__main__:Checking latency: current=111.23ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=111.23ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 19/100 ---
DEBUG:__main__:Simulated metrics at step 18: {'pnl_percent': -0.3515791209542192, 'open_orders': 39, 'error_rate': 0.060470828951428045, 'latency_ms': 169.4975013844579}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.3515791209542192%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.3515791209542192% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=39, threshold=100
INFO:__main__:Open orders count within limits: current=39 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.06, threshold=0.10
INFO:__main__:Error rate within limits: current=0.06 <= threshold=0.10
DEBUG:__main__:Checking latency: current=169.50ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=169.50ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 20/100 ---
DEBUG:__main__:Simulated metrics at step 19: {'pnl_percent': -0.2761080327831082, 'open_orders': 40, 'error_rate': 0.06666476095568578, 'latency_ms': 171.63184762418052}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.2761080327831082%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.2761080327831082% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=40, threshold=100
INFO:__main__:Open orders count within limits: current=40 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.07, threshold=0.10
INFO:__main__:Error rate within limits: current=0.07 <= threshold=0.10
DEBUG:__main__:Checking latency: current=171.63ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=171.63ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 21/100 ---
DEBUG:__main__:Simulated metrics at step 20: {'pnl_percent': -0.5889555074546973, 'open_orders': 44, 'error_rate': 0.04761599340288561, 'latency_ms': 138.08374217180932}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-0.5889555074546973%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-0.5889555074546973% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=44, threshold=100
INFO:__main__:Open orders count within limits: current=44 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.05, threshold=0.10
INFO:__main__:Error rate within limits: current=0.05 <= threshold=0.10
DEBUG:__main__:Checking latency: current=138.08ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=138.08ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 22/100 ---
DEBUG:__main__:Simulated metrics at step 21: {'pnl_percent': -2.2439079795927213, 'open_orders': 44, 'error_rate': 0.046994301853563104, 'latency_ms': 141.82225054341444}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-2.2439079795927213%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-2.2439079795927213% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=44, threshold=100
INFO:__main__:Open orders count within limits: current=44 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.05, threshold=0.10
INFO:__main__:Error rate within limits: current=0.05 <= threshold=0.10
DEBUG:__main__:Checking latency: current=141.82ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=141.82ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 23/100 ---
DEBUG:__main__:Simulated metrics at step 22: {'pnl_percent': -4.650132893701487, 'open_orders': 44, 'error_rate': 0.029081709149489862, 'latency_ms': 176.8723544156758}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-4.650132893701487%, threshold=-5.0%
INFO:__main__:P&L drawdown within limits: current=-4.650132893701487% >= threshold=-5.0%
DEBUG:__main__:Checking open orders count: current=44, threshold=100
INFO:__main__:Open orders count within limits: current=44 <= threshold=100
DEBUG:__main__:Checking error rate: current=0.03, threshold=0.10
INFO:__main__:Error rate within limits: current=0.03 <= threshold=0.10
DEBUG:__main__:Checking latency: current=176.87ms, threshold=500.00ms
INFO:__main__:Latency within limits: current=176.87ms <= threshold=500.00ms
INFO:__main__:All automated checks passed. Kill switch remains inactive.
INFO:__main__:Executing trading command: mock_place_order safely.
INFO:__main__:Mock: Placing 10 of AAPL.
INFO:__main__:Command mock_place_order executed successfully.
DEBUG:__main__:--- Simulation Step 24/100 ---
DEBUG:__main__:Simulated metrics at step 23: {'pnl_percent': -5.779991123775235, 'open_orders': 43, 'error_rate': 0.03676066029927269, 'latency_ms': 173.85028598623188}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-5.779991123775235%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-5.779991123775235% < threshold=-5.0%
CRITICAL:__main__:KILL SWITCH ACTIVATED by auto_pnl_drawdown. Details: {'pnl_percent': -5.779991123775235, 'threshold': -5.0}
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 25/100 ---
DEBUG:__main__:Simulated metrics at step 24: {'pnl_percent': -6.578531057463371, 'open_orders': 44, 'error_rate': 0.037394041830768936, 'latency_ms': 146.24287141698295}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-6.578531057463371%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-6.578531057463371% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 26/100 ---
DEBUG:__main__:Simulated metrics at step 25: {'pnl_percent': -8.765334912132017, 'open_orders': 48, 'error_rate': 0.050824548465243265, 'latency_ms': 129.77460636545254}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-8.765334912132017%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-8.765334912132017% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 27/100 ---
DEBUG:__main__:Simulated metrics at step 26: {'pnl_percent': -9.489759302716648, 'open_orders': 50, 'error_rate': 0.051974247979240495, 'latency_ms': 137.01704399555138}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-9.489759302716648%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-9.489759302716648% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 28/100 ---
DEBUG:__main__:Simulated metrics at step 27: {'pnl_percent': -10.781922181815471, 'open_orders': 53, 'error_rate': 0.03734460833270187, 'latency_ms': 105.68534214639129}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.781922181815471%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.781922181815471% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 29/100 ---
DEBUG:__main__:Simulated metrics at step 28: {'pnl_percent': -11.595548027619984, 'open_orders': 56, 'error_rate': 0.013389496559373243, 'latency_ms': 87.02814915790213}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.595548027619984%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.595548027619984% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 30/100 ---
DEBUG:__main__:Simulated metrics at step 29: {'pnl_percent': -12.431196652630108, 'open_orders': 60, 'error_rate': 0.0, 'latency_ms': 76.46777726904561}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.431196652630108%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.431196652630108% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 31/100 ---
DEBUG:__main__:Simulated metrics at step 30: {'pnl_percent': -12.550572369834317, 'open_orders': 58, 'error_rate': 0.0, 'latency_ms': 79.48130464086088}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.550572369834317%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.550572369834317% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 32/100 ---
DEBUG:__main__:Simulated metrics at step 31: {'pnl_percent': -12.076430669029573, 'open_orders': 56, 'error_rate': 0.008732575814753844, 'latency_ms': 72.78963383766101}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.076430669029573%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.076430669029573% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 33/100 ---
DEBUG:__main__:Simulated metrics at step 32: {'pnl_percent': -11.561627019773606, 'open_orders': 58, 'error_rate': 0.015381049918342351, 'latency_ms': 107.21003578284203}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.561627019773606%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.561627019773606% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 34/100 ---
DEBUG:__main__:Simulated metrics at step 33: {'pnl_percent': -10.823146550146308, 'open_orders': 59, 'error_rate': 0.02977475025013132, 'latency_ms': 119.41634096798788}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.823146550146308%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.823146550146308% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 35/100 ---
DEBUG:__main__:Simulated metrics at step 34: {'pnl_percent': -11.005349061082319, 'open_orders': 57, 'error_rate': 0.013248487467945474, 'latency_ms': 130.9304521743026}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.005349061082319%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.005349061082319% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 36/100 ---
DEBUG:__main__:Simulated metrics at step 35: {'pnl_percent': -10.66435885052012, 'open_orders': 56, 'error_rate': 0.003941491685357212, 'latency_ms': 118.1934407522736}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.66435885052012%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.66435885052012% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 37/100 ---
DEBUG:__main__:Simulated metrics at step 36: {'pnl_percent': -10.72453593887659, 'open_orders': 56, 'error_rate': 0.0, 'latency_ms': 86.8769044525886}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.72453593887659%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.72453593887659% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 38/100 ---
DEBUG:__main__:Simulated metrics at step 37: {'pnl_percent': -11.842037377799814, 'open_orders': 55, 'error_rate': 0.016572605670721777, 'latency_ms': 74.29597939468788}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.842037377799814%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.842037377799814% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 39/100 ---
DEBUG:__main__:Simulated metrics at step 38: {'pnl_percent': -12.738061086679014, 'open_orders': 58, 'error_rate': 0.0, 'latency_ms': 86.01142628379259}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.738061086679014%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.738061086679014% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 40/100 ---
DEBUG:__main__:Simulated metrics at step 39: {'pnl_percent': -12.448782493453749, 'open_orders': 56, 'error_rate': 0.0, 'latency_ms': 92.62831825618862}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.448782493453749%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.448782493453749% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 41/100 ---
DEBUG:__main__:Simulated metrics at step 40: {'pnl_percent': -12.780379067805677, 'open_orders': 59, 'error_rate': 0.015834137397178493, 'latency_ms': 84.05722097433011}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.780379067805677%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.780379067805677% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 42/100 ---
DEBUG:__main__:Simulated metrics at step 41: {'pnl_percent': -12.632466443259617, 'open_orders': 83, 'error_rate': 0.0, 'latency_ms': 82.71915994576929}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.632466443259617%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.632466443259617% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 43/100 ---
DEBUG:__main__:Simulated metrics at step 42: {'pnl_percent': -12.046581395474844, 'open_orders': 112, 'error_rate': 0.0, 'latency_ms': 108.21961686800714}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.046581395474844%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.046581395474844% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 44/100 ---
DEBUG:__main__:Simulated metrics at step 43: {'pnl_percent': -12.11788160243302, 'open_orders': 126, 'error_rate': 0.0, 'latency_ms': 116.66960624823224}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.11788160243302%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.11788160243302% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 45/100 ---
DEBUG:__main__:Simulated metrics at step 44: {'pnl_percent': -12.036065404734748, 'open_orders': 139, 'error_rate': 0.0, 'latency_ms': 88.69863039156678}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.036065404734748%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.036065404734748% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 46/100 ---
DEBUG:__main__:Simulated metrics at step 45: {'pnl_percent': -12.175313769437011, 'open_orders': 153, 'error_rate': 0.00885562971453122, 'latency_ms': 66.40597241559567}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.175313769437011%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.175313769437011% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 47/100 ---
DEBUG:__main__:Simulated metrics at step 46: {'pnl_percent': -12.353142634462856, 'open_orders': 179, 'error_rate': 0.006077599439553336, 'latency_ms': 96.05071349728095}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.353142634462856%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.353142634462856% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 48/100 ---
DEBUG:__main__:Simulated metrics at step 47: {'pnl_percent': -12.370662159036748, 'open_orders': 202, 'error_rate': 0.008018455713327847, 'latency_ms': 141.5170660857011}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.370662159036748%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.370662159036748% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 49/100 ---
DEBUG:__main__:Simulated metrics at step 48: {'pnl_percent': -12.073586024400521, 'open_orders': 213, 'error_rate': 0.03261979853801608, 'latency_ms': 168.17014560427552}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.073586024400521%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.073586024400521% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 50/100 ---
DEBUG:__main__:Simulated metrics at step 49: {'pnl_percent': -11.927868810962483, 'open_orders': 223, 'error_rate': 0.050210484820262476, 'latency_ms': 155.55022213868327}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.927868810962483%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.927868810962483% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 51/100 ---
DEBUG:__main__:Simulated metrics at step 50: {'pnl_percent': -12.106778316309772, 'open_orders': 224, 'error_rate': 0.027665001749806153, 'latency_ms': 138.68597009654135}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.106778316309772%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.106778316309772% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 52/100 ---
DEBUG:__main__:Simulated metrics at step 51: {'pnl_percent': -12.077406501500255, 'open_orders': 224, 'error_rate': 0.025041441770155572, 'latency_ms': 146.8728190329861}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.077406501500255%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.077406501500255% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 53/100 ---
DEBUG:__main__:Simulated metrics at step 52: {'pnl_percent': -11.565990285328866, 'open_orders': 224, 'error_rate': 0.012393044657160363, 'latency_ms': 145.16201376457695}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.565990285328866%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.565990285328866% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 54/100 ---
DEBUG:__main__:Simulated metrics at step 53: {'pnl_percent': -11.420528070189329, 'open_orders': 228, 'error_rate': 0.02458705297797901, 'latency_ms': 115.03689888538435}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.420528070189329%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.420528070189329% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 55/100 ---
DEBUG:__main__:Simulated metrics at step 54: {'pnl_percent': -10.890307259540911, 'open_orders': 232, 'error_rate': 0.013107522042880116, 'latency_ms': 116.66460247255337}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.890307259540911%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.890307259540911% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 56/100 ---
DEBUG:__main__:Simulated metrics at step 55: {'pnl_percent': -11.542887332833223, 'open_orders': 231, 'error_rate': 0.007499495739790367, 'latency_ms': 90.91012674895735}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.542887332833223%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.542887332833223% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 57/100 ---
DEBUG:__main__:Simulated metrics at step 56: {'pnl_percent': -11.413350749199992, 'open_orders': 233, 'error_rate': 0.04093632815435699, 'latency_ms': 94.64050099169171}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.413350749199992%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.413350749199992% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 58/100 ---
DEBUG:__main__:Simulated metrics at step 57: {'pnl_percent': -11.828084907323507, 'open_orders': 237, 'error_rate': 0.03330660789297707, 'latency_ms': 77.28915357728577}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.828084907323507%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.828084907323507% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 59/100 ---
DEBUG:__main__:Simulated metrics at step 58: {'pnl_percent': -11.328939470392704, 'open_orders': 235, 'error_rate': 0.042507897232309355, 'latency_ms': 72.86409619764083}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.328939470392704%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.328939470392704% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 60/100 ---
DEBUG:__main__:Simulated metrics at step 59: {'pnl_percent': -11.39706911914247, 'open_orders': 235, 'error_rate': 0.06210933288139653, 'latency_ms': 64.77108770745872}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.39706911914247%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.39706911914247% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 61/100 ---
DEBUG:__main__:Simulated metrics at step 60: {'pnl_percent': -12.357227982788338, 'open_orders': 238, 'error_rate': 0.07281636794094354, 'latency_ms': 102.08419728520592}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.357227982788338%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.357227982788338% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 62/100 ---
DEBUG:__main__:Simulated metrics at step 61: {'pnl_percent': -12.520407305597631, 'open_orders': 240, 'error_rate': 0.13366922999827818, 'latency_ms': 125.23263397015063}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.520407305597631%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.520407305597631% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 63/100 ---
DEBUG:__main__:Simulated metrics at step 62: {'pnl_percent': -13.06004605347857, 'open_orders': 243, 'error_rate': 0.21285881941013554, 'latency_ms': 109.24166900706038}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.06004605347857%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.06004605347857% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 64/100 ---
DEBUG:__main__:Simulated metrics at step 63: {'pnl_percent': -13.559271999535095, 'open_orders': 246, 'error_rate': 0.2830120179704698, 'latency_ms': 109.5716959314291}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.559271999535095%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.559271999535095% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 65/100 ---
DEBUG:__main__:Simulated metrics at step 64: {'pnl_percent': -14.761054274677699, 'open_orders': 248, 'error_rate': 0.3665204639381281, 'latency_ms': 98.4010814960337}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.761054274677699%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.761054274677699% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 66/100 ---
DEBUG:__main__:Simulated metrics at step 65: {'pnl_percent': -14.7107648673755, 'open_orders': 246, 'error_rate': 0.48358006217229993, 'latency_ms': 111.71565390705544}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.7107648673755%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.7107648673755% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 67/100 ---
DEBUG:__main__:Simulated metrics at step 66: {'pnl_percent': -14.470942211632073, 'open_orders': 244, 'error_rate': 0.6143285701012724, 'latency_ms': 109.42852900591603}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.470942211632073%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.470942211632073% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 68/100 ---
DEBUG:__main__:Simulated metrics at step 67: {'pnl_percent': -13.83263672072284, 'open_orders': 244, 'error_rate': 0.7321237125081965, 'latency_ms': 111.72700137180757}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.83263672072284%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.83263672072284% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 69/100 ---
DEBUG:__main__:Simulated metrics at step 68: {'pnl_percent': -13.754110674556127, 'open_orders': 248, 'error_rate': 0.8110378926129259, 'latency_ms': 128.1192769746527}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.754110674556127%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.754110674556127% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 70/100 ---
DEBUG:__main__:Simulated metrics at step 69: {'pnl_percent': -14.257669823378729, 'open_orders': 249, 'error_rate': 0.9029514782735216, 'latency_ms': 143.57870995374498}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.257669823378729%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.257669823378729% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 71/100 ---
DEBUG:__main__:Simulated metrics at step 70: {'pnl_percent': -14.636164082317615, 'open_orders': 253, 'error_rate': 0.8882698295356818, 'latency_ms': 158.7672346857768}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.636164082317615%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.636164082317615% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 72/100 ---
DEBUG:__main__:Simulated metrics at step 71: {'pnl_percent': -15.42696254572929, 'open_orders': 256, 'error_rate': 0.8542222356423164, 'latency_ms': 151.49992983400492}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-15.42696254572929%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-15.42696254572929% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 73/100 ---
DEBUG:__main__:Simulated metrics at step 72: {'pnl_percent': -14.484863183554793, 'open_orders': 254, 'error_rate': 0.8930454279547214, 'latency_ms': 161.97103386238342}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.484863183554793%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.484863183554793% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 74/100 ---
DEBUG:__main__:Simulated metrics at step 73: {'pnl_percent': -14.116146732569888, 'open_orders': 253, 'error_rate': 0.8686043250973724, 'latency_ms': 163.64204540269367}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-14.116146732569888%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-14.116146732569888% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 75/100 ---
DEBUG:__main__:Simulated metrics at step 74: {'pnl_percent': -13.42266093969524, 'open_orders': 255, 'error_rate': 0.8601396688676572, 'latency_ms': 157.46060506910143}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.42266093969524%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.42266093969524% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 76/100 ---
DEBUG:__main__:Simulated metrics at step 75: {'pnl_percent': -13.576399383709544, 'open_orders': 259, 'error_rate': 0.8679211558336104, 'latency_ms': 149.7934012416467}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-13.576399383709544%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-13.576399383709544% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
INFO:__main__:Simulating manual deactivation of kill switch at step 75.
INFO:__main__:KILL SWITCH DEACTIVATED. Trading can resume.
DEBUG:__main__:--- Simulation Step 77/100 ---
DEBUG:__main__:Simulated metrics at step 76: {'pnl_percent': -12.657152076219681, 'open_orders': 258, 'error_rate': 0.8640173725140977, 'latency_ms': 178.89340357420852}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.657152076219681%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.657152076219681% < threshold=-5.0%
CRITICAL:__main__:KILL SWITCH ACTIVATED by auto_pnl_drawdown. Details: {'pnl_percent': -12.657152076219681, 'threshold': -5.0}
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 78/100 ---
DEBUG:__main__:Simulated metrics at step 77: {'pnl_percent': -12.193995100962285, 'open_orders': 262, 'error_rate': 0.8593878069231815, 'latency_ms': 191.45643825488312}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.193995100962285%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.193995100962285% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 79/100 ---
DEBUG:__main__:Simulated metrics at step 78: {'pnl_percent': -12.203555880068228, 'open_orders': 261, 'error_rate': 0.8767071310916136, 'latency_ms': 183.03901999836145}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.203555880068228%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.203555880068228% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 80/100 ---
DEBUG:__main__:Simulated metrics at step 79: {'pnl_percent': -12.079248026219522, 'open_orders': 259, 'error_rate': 0.9111511830949537, 'latency_ms': 115.74358740589825}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.079248026219522%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.079248026219522% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 81/100 ---
DEBUG:__main__:Simulated metrics at step 80: {'pnl_percent': -12.282783604464736, 'open_orders': 263, 'error_rate': 0.9117982602756683, 'latency_ms': 85.6311159266752}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.282783604464736%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.282783604464736% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 82/100 ---
DEBUG:__main__:Simulated metrics at step 81: {'pnl_percent': -12.33324556264192, 'open_orders': 266, 'error_rate': 0.9130564881893207, 'latency_ms': 217.1940417982431}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.33324556264192%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.33324556264192% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 83/100 ---
DEBUG:__main__:Simulated metrics at step 82: {'pnl_percent': -12.452361290314878, 'open_orders': 266, 'error_rate': 0.9061344458258617, 'latency_ms': 424.4069386604538}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.452361290314878%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.452361290314878% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 84/100 ---
DEBUG:__main__:Simulated metrics at step 83: {'pnl_percent': -12.355115318722817, 'open_orders': 264, 'error_rate': 0.9141891352015934, 'latency_ms': 605.1067189663972}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.355115318722817%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.355115318722817% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 85/100 ---
DEBUG:__main__:Simulated metrics at step 84: {'pnl_percent': -12.170287198290847, 'open_orders': 266, 'error_rate': 0.901223306180569, 'latency_ms': 738.0021166919305}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.170287198290847%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.170287198290847% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 86/100 ---
DEBUG:__main__:Simulated metrics at step 85: {'pnl_percent': -12.665851201267294, 'open_orders': 270, 'error_rate': 0.8658557596141978, 'latency_ms': 1037.9278639022968}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.665851201267294%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.665851201267294% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 87/100 ---
DEBUG:__main__:Simulated metrics at step 86: {'pnl_percent': -11.901819540632085, 'open_orders': 268, 'error_rate': 0.8537974457659133, 'latency_ms': 1298.9370166903905}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.901819540632085%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.901819540632085% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 88/100 ---
DEBUG:__main__:Simulated metrics at step 87: {'pnl_percent': -11.478786428311633, 'open_orders': 266, 'error_rate': 0.8243767980976073, 'latency_ms': 1471.0868486355562}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.478786428311633%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.478786428311633% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 89/100 ---
DEBUG:__main__:Simulated metrics at step 88: {'pnl_percent': -11.80169344189095, 'open_orders': 264, 'error_rate': 0.7873655716091932, 'latency_ms': 1666.1722998187315}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.80169344189095%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.80169344189095% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 90/100 ---
DEBUG:__main__:Simulated metrics at step 89: {'pnl_percent': -12.153668547669286, 'open_orders': 265, 'error_rate': 0.7906688792262826, 'latency_ms': 1839.842889912726}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.153668547669286%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.153668547669286% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 91/100 ---
DEBUG:__main__:Simulated metrics at step 90: {'pnl_percent': -11.639031297197283, 'open_orders': 264, 'error_rate': 0.7822597245301182, 'latency_ms': 1867.5146012518446}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.639031297197283%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.639031297197283% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 92/100 ---
DEBUG:__main__:Simulated metrics at step 91: {'pnl_percent': -12.036969917690572, 'open_orders': 263, 'error_rate': 0.7877264059053919, 'latency_ms': 1842.54270399724}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-12.036969917690572%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-12.036969917690572% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 93/100 ---
DEBUG:__main__:Simulated metrics at step 92: {'pnl_percent': -11.331632990387806, 'open_orders': 261, 'error_rate': 0.7904972820462777, 'latency_ms': 1858.0884091342398}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.331632990387806%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.331632990387806% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 94/100 ---
DEBUG:__main__:Simulated metrics at step 93: {'pnl_percent': -11.45928454566367, 'open_orders': 265, 'error_rate': 0.7827412252490626, 'latency_ms': 1854.54000067189}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.45928454566367%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.45928454566367% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 95/100 ---
DEBUG:__main__:Simulated metrics at step 94: {'pnl_percent': -11.267261409792965, 'open_orders': 263, 'error_rate': 0.7791050180873887, 'latency_ms': 1858.2255117551772}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.267261409792965%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.267261409792965% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 96/100 ---
DEBUG:__main__:Simulated metrics at step 95: {'pnl_percent': -10.724884654803905, 'open_orders': 267, 'error_rate': 0.7789107296342834, 'latency_ms': 1817.8408974104575}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.724884654803905%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.724884654803905% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 97/100 ---
DEBUG:__main__:Simulated metrics at step 96: {'pnl_percent': -10.872993921908648, 'open_orders': 271, 'error_rate': 0.7940697818825201, 'latency_ms': 1791.898077805596}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.872993921908648%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.872993921908648% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 98/100 ---
DEBUG:__main__:Simulated metrics at step 97: {'pnl_percent': -10.917463354732591, 'open_orders': 274, 'error_rate': 0.8052590399798877, 'latency_ms': 1809.7557209171207}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.917463354732591%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.917463354732591% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 99/100 ---
DEBUG:__main__:Simulated metrics at step 98: {'pnl_percent': -10.85116592154796, 'open_orders': 277, 'error_rate': 0.7767784901190126, 'latency_ms': 1809.352413762929}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-10.85116592154796%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-10.85116592154796% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
DEBUG:__main__:--- Simulation Step 100/100 ---
DEBUG:__main__:Simulated metrics at step 99: {'pnl_percent': -11.296556987246186, 'open_orders': 279, 'error_rate': 0.7774211152838618, 'latency_ms': 1797.5195943838519}
INFO:__main__:Running automated kill switch trigger checks.
DEBUG:__main__:Checking P&L drawdown: current=-11.296556987246186%, threshold=-5.0%
WARNING:__main__:P&L drawdown threshold exceeded: current=-11.296556987246186% < threshold=-5.0%
WARNING:__main__:Kill switch is already active. Ignoring activation request.
INFO:__main__:Trading commands blocked due to active kill switch.
WARNING:__main__:Kill switch is active. Blocking command: mock_cancel_all_orders
INFO:__main__:Kill switch simulation completed.
INFO:__main__:Generating visualization of simulation results.
INFO:__main__:Plotting complete.
INFO:__main__:Displaying summary statistics.
/tmp/ipykernel_3140/750244018.py:83: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
for i, row in simulation_df[simulation_df['is_active'] & ~simulation_df['is_active'].shift(1).fillna(False)].iterrows():
/tmp/ipykernel_3140/750244018.py:87: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
for i, row in simulation_df[~simulation_df['is_active'] & simulation_df['is_active'].shift(1).fillna(False)].iterrows():
--- Simulation Summary Statistics --- Kill Switch Active % of time: 76.0 % | | pnl_percent | open_orders | error_rate | latency_ms | |:------|:--------------|:--------------|:-------------|:-------------| | count | 100 | 100 | 100 | 100 | | mean | -9.44614 | 158.85 | 0.313627 | 364.042 | | std | 4.95625 | 101.661 | 0.382508 | 582.375 | | min | -15.427 | 22 | 0 | 50 | | 25% | -12.3606 | 47 | 0.0157209 | 85.2376 | | 50% | -11.6173 | 223.5 | 0.0489132 | 116.667 | | 75% | -8.21863 | 256.5 | 0.787456 | 170.031 | | max | 0.741714 | 279 | 0.914189 | 1867.51 | --- Kill Switch Activation Events --- Step 23: Activated by auto_pnl_drawdown Step 76: Activated by auto_pnl_drawdown --- Kill Switch Deactivation Events --- Step 75: Deactivated.
6. Production Considerations
Implementing a hard kill switch in a production trading environment requires careful consideration of several best practices to ensure its effectiveness, reliability, and safety. This table outlines key considerations.
| Aspect | Best Practice |
|---|---|
| Redundancy | Implement the kill switch across multiple independent systems and networks. A single point of failure could compromise its effectiveness. |
| Isolation | The kill switch mechanism should be as isolated as possible from the trading system itself to prevent common mode failures. Ideally, it should run on separate infrastructure. |
| Fail-Safe Design | Design the kill switch to fail-safe, meaning if there's any ambiguity or failure in the kill switch itself, it defaults to the "active" (halted trading) state. |
| Alerting | Implement immediate, multi-channel alerting (SMS, email, PagerDuty, etc.) for kill switch activation/deactivation and any failures within the kill switch system. Alerts should be actionable and reach responsible personnel 24/7. |
| Logging & Audit | Comprehensive, immutable logging of all kill switch events (activation, deactivation, trigger details, manual overrides) is crucial for post-incident analysis and regulatory compliance. |
| Testing | Regular, rigorous testing of the kill switch in a simulated production environment is essential. This includes both automated and manual tests to verify its functionality under various failure scenarios without impacting live trading. |
| Manual Override | Provide a clear, simple, and physically accessible manual override mechanism for human intervention when automated triggers may not suffice or in cases of unforeseen circumstances. |
| Permissions | Strictly control access to kill switch activation/deactivation. Implement role-based access control (RBAC) and require multi-factor authentication for any manual operations. |
| State Persistence | Ensure the kill switch state is persistent across restarts or failures. If the system reboots, the kill switch should retain its active state until explicitly deactivated. |
| Graceful Shutdown | The kill switch should ideally initiate a graceful shutdown process for trading components, ensuring open positions are managed (e.g., cancelling all open orders) before completely halting. |
| Threshold Tuning | Continuously monitor and fine-tune trigger thresholds to balance responsiveness to genuine issues with minimizing false positives. Use historical data and backtesting to inform threshold adjustments. |
| Dependency Review | Regularly review all external dependencies of the kill switch (e.g., monitoring services, messaging queues) to ensure their reliability and minimize the risk of external failures impacting the kill switch. |
7. Conclusion
This notebook has provided a comprehensive framework for developing a Hard Kill Switch for Automated Trading Systems. We've covered the essential components, from state management and automatic trigger definitions to safe command execution and a full simulation demonstrating its behavior.
Key components implemented include:
- State Management: A central
statedictionary to track the kill switch status, activation history, and configuration. - Activation/Deactivation: Functions (
activate_kill_switch,deactivate_kill_switch) for programmatic control, supporting both manual and automated triggers. - Automated Triggers: Specific
check_functions (check_pnl_drawdown,check_open_orders_count,check_error_rate,check_latency) to monitor critical trading metrics and system health indicators. - Orchestrated Checks: A
run_auto_trigger_checksfunction to consolidate all automated monitoring, ensuring rapid response to any breach. - Safe Command Execution: The
execute_trading_command_safelyfunction, which integrates kill switch status with robust retry mechanisms (exponential backoff and jitter) to prevent trading operations when the switch is active. - Simulation & Visualization: A full
run_kill_switch_simulationto demonstrate the system's dynamic response to simulated market conditions, along withplot_kill_switch_simulation_resultsanddisplay_summary_statisticsfor analysis.
The hard kill switch is a critical safety component, acting as the last line of defense against catastrophic trading system failures. Its design emphasizes isolation, redundancy, and rigorous testing to ensure it functions reliably when needed most, safeguarding capital and maintaining system integrity.