Live Trading·Trading Infrastructure·Intermediate

Graceful Shutdown Handler

Implement a production-grade graceful shutdown handler that safely exits all open trading positions, cancels all pending exchange orders, flushes pending log buffers, and atomically persists all system state when the trading system receives a SIGTERM termination signal for clean restarts.

live-tradingsafety

Graceful System Shutdown

This notebook explores the critical concept of graceful system shutdown in applications. A graceful shutdown ensures that an application can terminate cleanly, releasing resources, saving state, and completing in-progress operations without data loss or corruption. It's a fundamental aspect of building robust and reliable software systems.

Why Graceful Shutdown?

  • Data Integrity: Prevent data loss by ensuring all pending writes are completed.
  • Resource Release: Close file handles, database connections, network sockets, and other system resources.
  • State Persistence: Save application state so it can resume correctly upon restart.
  • User Experience: Provide a smooth termination, avoiding abrupt disconnections for clients.
  • Operational Stability: Facilitate planned maintenance, deployments, and restarts without service disruption.

Key Concepts:

ConceptDescription
Signal HandlingIntercepting operating system signals (e.g., SIGINT for Ctrl+C, SIGTERM for kill command) to trigger shutdown procedures.
Resource CleanupExplicitly closing open resources like files, network connections, or database sessions.
State SavingPersisting in-memory data or ongoing computations to durable storage before termination.
TimeoutsSetting limits on how long cleanup operations can take to prevent indefinite hangs, ensuring eventual termination even if some tasks cannot complete.
IdempotencyDesigning cleanup and state-saving operations to be safely repeatable, meaning they can be executed multiple times without adverse effects.
Worker ManagementStopping background tasks, threads, or processes cleanly, allowing them to finish current work or be interrupted safely.
Liveness/ReadinessUpdating health checks to indicate a shutting down state, preventing new requests from being routed to the terminating instance.

Dependency Installation

We'll install loguru for robust logging and tenacity for implementing retry logic with exponential backoff, which can be useful for resilient cleanup operations.

[1]
pip install loguru tenacity pandas matplotlib seaborn
Collecting loguru
  Downloading loguru-0.7.3-py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: tenacity in /usr/local/lib/python3.12/dist-packages (9.1.4)
Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0)
Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2)
Requirement already satisfied: numpy>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.2)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.2)
Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)
Downloading loguru-0.7.3-py3-none-any.whl (61 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 783.8 kB/s eta 0:00:00
[?25hInstalling collected packages: loguru
Successfully installed loguru-0.7.3

Library Imports

This section imports all necessary libraries. Standard Python libraries are imported first, followed by third-party libraries.

[2]
import os
import signal
import time
import sys
import random
from collections import deque
import threading

from loguru import logger
from tenacity import retry, wait_exponential, stop_after_attempt, after_log
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Core Functions

This section defines the core functions required for managing and executing a graceful system shutdown. Each function is presented in its own dedicated code block, adhering to the specified format including markdown headers, complete docstrings, type hints, and logger statements.

Function Name: create_shutdown_state

This function initializes the application's shutdown state dictionary. It sets up flags and data structures to manage the shutdown process, including the global shutdown flag, active tasks, and records of cleanup activities.

Parameters: None

Returns: dict: An initialized dictionary representing the application's shutdown state.

[3]
def create_shutdown_state() -> dict:
    """
    Initializes a dictionary to manage the application's shutdown state.

    The state includes a flag to signal shutdown, a list of active tasks,
    and a record of cleanup operations.

    Parameters
    ----------
    None

    Returns
    -------
    dict
        An initialized dictionary with shutdown-related keys:
        - 'shutdown_flag': bool, indicates if shutdown has been requested.
        - 'active_tasks': list, tracks currently running tasks.
        - 'cleanup_log': list, records of cleanup operations.
        - 'start_time': float, timestamp when the state was created.
    """
    logger.info("Initializing shutdown state.")
    state = {
        'shutdown_flag': False,
        'active_tasks': [],
        'cleanup_log': [],
        'start_time': time.monotonic(),
        'shutdown_requested_time': None
    }
    logger.debug(f"Shutdown state initialized: {state}")
    return state

Function Name: _handle_signal

This is an internal helper function designed to be used as a signal handler. When an OS signal (like SIGINT or SIGTERM) is received, this function updates the global shutdown_flag in the application state, triggering the graceful shutdown sequence. It logs the received signal.

Parameters: state (dict): The current application state dictionary. signum (int): The number of the signal received. frame (object): The current stack frame (unused but required by signal handler signature).

Returns: None

[4]
def _handle_signal(state: dict, signum: int, frame: object) -> None:
    """
    Internal signal handler to set the shutdown flag in the application state.

    Parameters
    ----------
    state : dict
        The current application state dictionary, which includes the 'shutdown_flag'.
    signum : int
        The number of the signal received (e.g., signal.SIGINT, signal.SIGTERM).
    frame : object
        The current stack frame (required by signal handler signature, but unused).

    Returns
    -------
    None

    Examples
    --------
    >>> app_state = create_shutdown_state()
    >>> # In a real scenario, this would be called by the OS when a signal is received.
    >>> _handle_signal(app_state, signal.SIGINT, None)
    >>> app_state['shutdown_flag']
    True
    """
    signal_name = signal.Signals(signum).name
    logger.warning(f"Received signal {signal_name} ({signum}). Initiating graceful shutdown.")
    state['shutdown_flag'] = True
    state['shutdown_requested_time'] = time.monotonic()
    logger.debug("Shutdown flag set to True.")

Function Name: register_signal_handlers

This function configures the application to listen for specific operating system signals (SIGINT and SIGTERM) that typically request process termination. It registers the _handle_signal function to be called when these signals are received, enabling the application to respond gracefully.

Parameters: state (dict): The current application state dictionary.

Returns: dict: The updated application state dictionary after registering handlers.

[5]
def register_signal_handlers(state: dict) -> dict:
    """
    Registers signal handlers for SIGINT (Ctrl+C) and SIGTERM (kill command).

    When these signals are received, the `_handle_signal` function will be called
    to set the shutdown flag in the provided state dictionary.

    Parameters
    ----------
    state : dict
        The application state dictionary where the shutdown flag will be set.

    Returns
    -------
    dict
        The updated state dictionary with signal handlers registered.

    Examples
    --------
    >>> app_state = create_shutdown_state()
    >>> updated_state = register_signal_handlers(app_state)
    >>> # Now, if SIGINT or SIGTERM are sent, updated_state['shutdown_flag'] will become True.
    """
    logger.info("Registering signal handlers for SIGINT and SIGTERM.")
    # Use a lambda to pass the state dictionary to the signal handler
    signal.signal(signal.SIGINT, lambda s, f: _handle_signal(state, s, f))
    signal.signal(signal.SIGTERM, lambda s, f: _handle_signal(state, s, f))
    logger.debug("Signal handlers registered successfully.")
    return state

Function Name: perform_cleanup_task

This function simulates a critical cleanup operation, such as closing a database connection or flushing logs. It includes retry logic with exponential backoff using tenacity to handle transient failures, along with random jitter to prevent thundering herd problems. The function also incorporates a cleanup_timeout to ensure cleanup completes within a reasonable timeframe.

Parameters: state (dict): The current application state dictionary. task_name (str): A descriptive name for the cleanup task. duration (float): The simulated time (in seconds) the task takes. fail_chance (float, optional): Probability (0.0 to 1.0) of the task failing once. Defaults to 0.2.

Returns: dict: The updated application state dictionary, with the cleanup task logged.

[6]
@retry(
    wait=wait_exponential(multiplier=1, min=0.5, max=5), # Exponential backoff with min 0.5s, max 5s
    stop=stop_after_attempt(3), # Retry up to 3 times
    after=after_log(logger, 'warning') # Log retries as warnings
)
def perform_cleanup_task(state: dict, task_name: str, duration: float, fail_chance: float = 0.2) -> dict:
    """
    Simulates a cleanup task that might fail and be retried, with random jitter.

    Parameters
    ----------
    state : dict
        The current application state dictionary.
    task_name : str
        A descriptive name for the cleanup task.
    duration : float
        The simulated time (in seconds) the task takes.
    fail_chance : float, optional
        Probability (0.0 to 1.0) of the task failing once. Defaults to 0.2.

    Returns
    -------
    dict
        The updated application state dictionary, with the cleanup task logged.

    Raises
    ------
    RuntimeError
        If the cleanup task fails after all retries.

    Examples
    --------
    >>> app_state = create_shutdown_state()
    >>> # Example of a successful cleanup task
    >>> updated_state = perform_cleanup_task(app_state, "Close DB Connection", 1.0, fail_chance=0.0)
    >>> # Example of a potentially failing cleanup task (requires mocking or careful testing)
    >>> # updated_state = perform_cleanup_task(app_state, "Flush Logs", 0.5, fail_chance=0.8)
    """
    jitter = random.uniform(0, 0.1) # Add random jitter to simulate variability
    actual_duration = duration + jitter

    logger.info(f"Attempting cleanup task: '{task_name}' (simulated duration: {actual_duration:.2f}s).")

    if random.random() < fail_chance: # Simulate occasional failure
        logger.warning(f"Cleanup task '{task_name}' failed temporarily. Retrying...")
        raise RuntimeError(f"Simulated failure for task '{task_name}'")

    time.sleep(actual_duration)
    log_entry = {
        'task': task_name,
        'status': 'completed',
        'time_taken': actual_duration,
        'timestamp': time.monotonic() - state['start_time']
    }
    state['cleanup_log'].append(log_entry)
    logger.success(f"Cleanup task '{task_name}' completed successfully.")
    return state

Function Name: simulate_workload

This function simulates an application's ongoing work. It continuously performs small units of work until the shutdown_flag in the provided state dictionary is set to True. This demonstrates how an application's main loop can gracefully stop processing new work upon a shutdown request.

Parameters: state (dict): The current application state dictionary. task_duration (float): The simulated time (in seconds) each unit of work takes.

Returns: dict: The updated application state dictionary after the workload has ceased.

[7]
def simulate_workload(state: dict, task_duration: float = 0.1) -> dict:
    """
    Simulates an ongoing application workload that checks for a shutdown signal.

    It adds tasks to `state['active_tasks']` until `state['shutdown_flag']` is set.

    Parameters
    ----------
    state : dict
        The current application state dictionary, containing 'shutdown_flag'
        and 'active_tasks'.
    task_duration : float, optional
        The simulated time (in seconds) each small unit of work takes. Defaults to 0.1.

    Returns
    -------
    dict
        The updated state dictionary after the workload has finished.

    Examples
    --------
    >>> app_state = create_shutdown_state()
    >>> # In a separate thread or process, you'd call this:
    >>> # t = threading.Thread(target=simulate_workload, args=(app_state, 0.05))
    >>> # t.start()
    >>> # time.sleep(1) # Let it run for a bit
    >>> # app_state['shutdown_flag'] = True # Signal shutdown
    >>> # t.join() # Wait for it to finish
    """
    task_count = 0
    logger.info("Starting simulated workload. Will run until shutdown is requested.")
    while not state['shutdown_flag']:
        jitter = random.uniform(0, 0.05) # Add small random jitter
        time.sleep(task_duration + jitter)
        task_count += 1
        state['active_tasks'].append(f"task_{task_count}")
        if task_count % 10 == 0:
            logger.debug(f"Workload active: processed {task_count} tasks.")

    logger.info(f"Simulated workload received shutdown signal. Processed {task_count} tasks.")
    return state

Function Name: run_graceful_shutdown

This function orchestrates the entire graceful shutdown process. It first registers signal handlers, then starts a simulated workload in a separate thread. After a specified duration (or upon receiving an external signal), it initiates the cleanup phase, calling various perform_cleanup_task functions. It ensures that all cleanup tasks are attempted within a given cleanup_timeout.

Parameters: state (dict): The application state dictionary. workload_duration (float): The simulated time (in seconds) the main workload runs before a shutdown is internally triggered (if no external signal is sent). cleanup_timeout (float): The maximum time (in seconds) allowed for all cleanup tasks to complete.

Returns: dict: The final application state dictionary after the shutdown process.

[8]
def run_graceful_shutdown(state: dict, workload_duration: float = 5, cleanup_timeout: float = 10) -> dict:
    """
    Orchestrates the graceful shutdown process for the application.

    This involves registering signal handlers, running a simulated workload,
    and then performing cleanup tasks once a shutdown is requested or triggered.

    Parameters
    ----------
    state : dict
        The application state dictionary.
    workload_duration : float, optional
        The simulated time (in seconds) the main workload runs before
        a shutdown is internally triggered (if no external signal is sent).
        Defaults to 5.
    cleanup_timeout : float, optional
        The maximum time (in seconds) allowed for all cleanup tasks to complete.
        Defaults to 10.

    Returns
    -------
    dict
        The final application state dictionary after the shutdown process.

    Examples
    --------
    >>> app_state = create_shutdown_state()
    >>> final_state = run_graceful_shutdown(app_state, workload_duration=3, cleanup_timeout=5)
    >>> # The 'cleanup_log' in final_state will show executed cleanup tasks.
    """
    logger.info("Starting graceful shutdown orchestration.")

    # 1. Register signal handlers
    state = register_signal_handlers(state)

    # 2. Start a simulated workload in a separate thread
    workload_thread = threading.Thread(target=simulate_workload, args=(state, 0.05))
    workload_thread.start()
    logger.info(f"Workload thread started. Will run for ~{workload_duration}s or until signal.")

    # 3. Wait for the shutdown flag or a simulated duration
    start_waiting = time.monotonic()
    while not state['shutdown_flag'] and (time.monotonic() - start_waiting < workload_duration):
        time.sleep(0.1) # Polling for shutdown flag

    if not state['shutdown_flag']:
        logger.info("Simulated workload duration elapsed. Internally triggering shutdown.")
        state['shutdown_flag'] = True
        state['shutdown_requested_time'] = time.monotonic()
    else:
        logger.info("Shutdown signal received or duration elapsed, proceeding to cleanup.")

    # Wait for workload to acknowledge shutdown flag and finish its current cycle
    logger.info("Waiting for workload thread to finish acknowledging shutdown flag...")
    workload_thread.join(timeout=2) # Give it a little time to finish its loop
    if workload_thread.is_alive():
        logger.warning("Workload thread did not finish cleanly within timeout. Forcing continue.")

    # 4. Perform cleanup tasks with a global timeout
    logger.info(f"Starting cleanup phase with a timeout of {cleanup_timeout} seconds.")
    cleanup_start_time = time.monotonic()

    cleanup_tasks = [
        ("Close Database Connection", 1.5, 0.3), # Name, Duration, Fail Chance
        ("Flush Application Logs", 0.8, 0.1),
        ("Save In-progress Work", 2.0, 0.05),
        ("Release Network Socket", 1.0, 0.2)
    ]

    for task_name, task_duration, fail_chance in cleanup_tasks:
        remaining_time = cleanup_timeout - (time.monotonic() - cleanup_start_time)
        if remaining_time <= 0:
            logger.error(f"Cleanup timeout reached. Aborting remaining cleanup tasks. Failed to complete: {task_name}")
            break
        try:
            # Simulate potential task delays or failures
            state = perform_cleanup_task(state, task_name, task_duration, fail_chance)
        except RuntimeError as e:
            logger.error(f"Cleanup task '{task_name}' failed critically after retries: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred during cleanup task '{task_name}': {e}")

    cleanup_end_time = time.monotonic()
    total_cleanup_time = cleanup_end_time - cleanup_start_time
    logger.info(f"Cleanup phase completed in {total_cleanup_time:.2f} seconds.")

    if state['shutdown_requested_time']:
        total_shutdown_duration = cleanup_end_time - state['shutdown_requested_time']
        logger.info(f"Total graceful shutdown duration from signal to completion: {total_shutdown_duration:.2f} seconds.")

    logger.info("Graceful shutdown orchestration finished.")
    return state

Demonstration/Visualization

This section demonstrates the graceful shutdown process by executing the run_graceful_shutdown function and visualizing its effects. We will simulate a workload being interrupted and cleanup tasks being performed. The output will show logs of the process and a summary of cleanup activities.

[9]
# Configure Loguru to write to stderr and capture it for display
logger.remove()
logger.add(sys.stderr, format="<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>")

logger.info("### Starting Graceful Shutdown Demonstration ###")

# Initialize the application state
app_state = create_shutdown_state()

# Run the graceful shutdown process
# You can try interrupting this cell (Ctrl+C) while it's running to see the signal handler in action.
final_state = run_graceful_shutdown(app_state, workload_duration=3, cleanup_timeout=7)

logger.info("### Graceful Shutdown Demonstration Complete ###")

# Display summary of cleanup activities
print("\n--- Cleanup Summary ---")
if final_state['cleanup_log']:
    cleanup_df = pd.DataFrame(final_state['cleanup_log'])
    display(cleanup_df)
else:
    print("No cleanup tasks were logged.")

print(f"\nTotal active tasks before shutdown: {len(app_state['active_tasks'])}")
print(f"Shutdown flag final state: {final_state['shutdown_flag']}")
2026-06-10 10:06:22.106 | INFO     | __main__:<cell line: 0>:5 - ### Starting Graceful Shutdown Demonstration ###
2026-06-10 10:06:22.109 | INFO     | __main__:create_shutdown_state:21 - Initializing shutdown state.
2026-06-10 10:06:22.110 | DEBUG    | __main__:create_shutdown_state:29 - Shutdown state initialized: {'shutdown_flag': False, 'active_tasks': [], 'cleanup_log': [], 'start_time': 796.053242071, 'shutdown_requested_time': None}
2026-06-10 10:06:22.111 | INFO     | __main__:run_graceful_shutdown:31 - Starting graceful shutdown orchestration.
2026-06-10 10:06:22.112 | INFO     | __main__:register_signal_handlers:24 - Registering signal handlers for SIGINT and SIGTERM.
2026-06-10 10:06:22.113 | DEBUG    | __main__:register_signal_handlers:28 - Signal handlers registered successfully.
2026-06-10 10:06:22.125 | INFO     | __main__:simulate_workload:31 - Starting simulated workload. Will run until shutdown is requested.
2026-06-10 10:06:22.125 | INFO     | __main__:run_graceful_shutdown:39 - Workload thread started. Will run for ~3s or until signal.
2026-06-10 10:06:22.864 | DEBUG    | __main__:simulate_workload:38 - Workload active: processed 10 tasks.
2026-06-10 10:06:23.668 | DEBUG    | __main__:simulate_workload:38 - Workload active: processed 20 tasks.
2026-06-10 10:06:24.435 | DEBUG    | __main__:simulate_workload:38 - Workload active: processed 30 tasks.
2026-06-10 10:06:25.136 | INFO     | __main__:run_graceful_shutdown:47 - Simulated workload duration elapsed. Internally triggering shutdown.
2026-06-10 10:06:25.137 | INFO     | __main__:run_graceful_shutdown:54 - Waiting for workload thread to finish acknowledging shutdown flag...
2026-06-10 10:06:25.174 | DEBUG    | __main__:simulate_workload:38 - Workload active: processed 40 tasks.
2026-06-10 10:06:25.176 | INFO     | __main__:simulate_workload:40 - Simulated workload received shutdown signal. Processed 40 tasks.
2026-06-10 10:06:25.180 | INFO     | __main__:run_graceful_shutdown:60 - Starting cleanup phase with a timeout of 7 seconds.
2026-06-10 10:06:25.182 | INFO     | __main__:perform_cleanup_task:42 - Attempting cleanup task: 'Close Database Connection' (simulated duration: 1.55s).
2026-06-10 10:06:25.183 | WARNING  | __main__:perform_cleanup_task:45 - Cleanup task 'Close Database Connection' failed temporarily. Retrying...
2026-06-10 10:06:25.185 | ERROR    | __main__:run_graceful_shutdown:81 - An unexpected error occurred during cleanup task 'Close Database Connection': Level 'warning' does not exist
2026-06-10 10:06:25.187 | INFO     | __main__:perform_cleanup_task:42 - Attempting cleanup task: 'Flush Application Logs' (simulated duration: 0.89s).
2026-06-10 10:06:26.079 | SUCCESS  | __main__:perform_cleanup_task:56 - Cleanup task 'Flush Application Logs' completed successfully.
2026-06-10 10:06:26.080 | INFO     | __main__:perform_cleanup_task:42 - Attempting cleanup task: 'Save In-progress Work' (simulated duration: 2.07s).
2026-06-10 10:06:28.154 | SUCCESS  | __main__:perform_cleanup_task:56 - Cleanup task 'Save In-progress Work' completed successfully.
2026-06-10 10:06:28.156 | INFO     | __main__:perform_cleanup_task:42 - Attempting cleanup task: 'Release Network Socket' (simulated duration: 1.05s).
2026-06-10 10:06:29.206 | SUCCESS  | __main__:perform_cleanup_task:56 - Cleanup task 'Release Network Socket' completed successfully.
2026-06-10 10:06:29.207 | INFO     | __main__:run_graceful_shutdown:85 - Cleanup phase completed in 4.03 seconds.
2026-06-10 10:06:29.208 | INFO     | __main__:run_graceful_shutdown:89 - Total graceful shutdown duration from signal to completion: 4.07 seconds.
2026-06-10 10:06:29.210 | INFO     | __main__:run_graceful_shutdown:91 - Graceful shutdown orchestration finished.
2026-06-10 10:06:29.212 | INFO     | __main__:<cell line: 0>:14 - ### Graceful Shutdown Demonstration Complete ###

--- Cleanup Summary ---
task status time_taken timestamp
0 Flush Application Logs completed 0.890009 3.967998
1 Save In-progress Work completed 2.072272 6.044170
2 Release Network Socket completed 1.046861 7.095284

Total active tasks before shutdown: 40
Shutdown flag final state: True
[10]
if final_state['cleanup_log']:
    cleanup_df = pd.DataFrame(final_state['cleanup_log'])

    # Calculate time relative to shutdown request for visualization
    # If shutdown was internal, use state creation time as reference
    reference_time = final_state['shutdown_requested_time'] if final_state['shutdown_requested_time'] else final_state['start_time']
    cleanup_df['time_relative_to_shutdown'] = cleanup_df['timestamp'] - (reference_time - final_state['start_time'])

    plt.figure(figsize=(10, 6))
    sns.barplot(x='task', y='time_taken', data=cleanup_df, palette='viridis')
    plt.axvline(x=-0.5, color='red', linestyle='--', label='Shutdown Signal/Trigger')
    plt.title('Cleanup Task Durations During Graceful Shutdown')
    plt.xlabel('Cleanup Task')
    plt.ylabel('Time Taken (seconds)')
    plt.xticks(rotation=45, ha='right')
    plt.legend()
    plt.tight_layout()
    plt.show()

    # Visualize cleanup progress over time
    cleanup_df = cleanup_df.sort_values('time_relative_to_shutdown')
    cleanup_df['cumulative_time'] = cleanup_df['time_taken'].cumsum()

    plt.figure(figsize=(12, 6))
    plt.plot(cleanup_df['time_relative_to_shutdown'], cleanup_df['cumulative_time'], marker='o', linestyle='-', color='skyblue')
    plt.axvline(x=0, color='red', linestyle='--', label='Shutdown Initiated')
    plt.axhline(y=7, color='green', linestyle=':', label='Cleanup Timeout (7s)')
    plt.title('Cumulative Cleanup Time After Shutdown Initiation')
    plt.xlabel('Time Relative to Shutdown Initiation (seconds)')
    plt.ylabel('Cumulative Time Spent on Cleanup (seconds)')
    plt.grid(True, linestyle='--', alpha=0.7)
    plt.legend()
    plt.tight_layout()
    plt.show()
else:
    print("No data to visualize cleanup activities.")
/tmp/ipykernel_3102/2508657904.py:10: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x='task', y='time_taken', data=cleanup_df, palette='viridis')
cell output
cell output

Production Considerations

Implementing graceful shutdown in a production environment involves more than just signal handling. It requires careful consideration of the entire application ecosystem. Here's a table outlining key best practices and considerations:

ConsiderationDescription
Process ManagersUse tools like systemd, Supervisor, Kubernetes (with terminationGracePeriodSeconds) to manage application lifecycles, ensuring SIGTERM is sent and providing configurable grace periods before forced termination (SIGKILL).
Health ChecksImplement liveness and readiness probes. During shutdown, transition readiness probes to 'unready' status to stop receiving new traffic, allowing in-flight requests to complete.
Distributed TracingIn microservice architectures, use distributed tracing (e.g., OpenTelemetry, Zipkin) to understand how shutdown signals propagate and how long dependent services take to respond and clean up.
Idempotent OperationsEnsure that cleanup and state-saving operations are idempotent. This means they can be safely repeated without causing adverse effects, which is crucial in systems with retries or partial failures.
Monitoring & AlertingMonitor shutdown events and durations. Set up alerts for processes that consistently fail to shut down gracefully within their allocated time, indicating potential resource leaks or stuck operations.
Configuration ManagementExternalize shutdown timeouts, retry policies, and cleanup task configurations. This allows operators to fine-tune shutdown behavior without code changes, adapting to different environments or workloads.
Connection DrainingFor network services, actively close listening sockets and stop accepting new connections while allowing existing connections to finish their current transactions or gracefully disconnect.
Circuit Breakers/BackoffWhen interacting with external services during cleanup, use circuit breakers or exponential backoff to prevent cascading failures if a dependency is also shutting down or experiencing issues.
Resource QuotasEnsure cleanup routines respect resource quotas (CPU, memory, I/O) to prevent the shutdown process itself from consuming too many resources and impacting other critical services during a system-wide event.

Conclusion

This notebook has demonstrated the fundamental principles and practical implementation of graceful system shutdown. We've covered:

  • Signal Handling: How to intercept OS signals (SIGINT, SIGTERM) to trigger controlled application termination.
  • State Management: Using a central state dictionary to coordinate the shutdown process across different components.
  • Simulated Workload: Illustrating how an application's main loop can cooperatively stop processing new tasks.
  • Resilient Cleanup: Implementing cleanup operations with retry logic and timeouts to ensure resources are released reliably.
  • Visualization: Providing insights into the timing and completion of cleanup tasks.

Graceful shutdown is not merely a best practice; it's a necessity for building resilient, data-safe, and operationally sound applications. By thoughtfully designing your systems to handle termination with care, you contribute significantly to the stability and reliability of your software in production environments.