Execution·Advanced·Advanced

Retry Failed Orders

Build a robust order retry system with exponential backoff, partial fill detection, and error classification for production trading.

retryerror handlingrobustness

Order Submission Retry Framework


1. Overview

This notebook presents a production-grade retry framework designed for robust handling of transient failures during order submission across various supported exchanges.

Failure Classification

Order submission failures are categorized to determine appropriate error handling and retry strategies:

Failure TypePrimary CauseRetry ActionOperational Response
Network timeoutConnectivity issues✅ YesImplement exponential backoff.
Rate limit (HTTP 429)Excessive requests✅ YesIntroduce a wait period before retrying.
Insufficient balanceAccount state anomaly⚠️ ConditionalReduce order quantity and re-attempt.
Invalid parameterApplication logic error❌ NoLog the error and terminate the process.
Exchange outage (HTTP 503)Service unavailability✅ YesApply a prolonged wait before re-attempting.
Authentication failure (HTTP 401)Invalid credentials❌ NoTerminate immediately; no retry attempted.

Strategic Implementation

Key strategies employed within this framework include:

  • Exponential Backoff with Jitter: This mechanism prevents system overload by progressively increasing delay between retries while adding a random component to mitigate synchronized retry storms.
  • Maximum Retry Cap: A defined limit on retry attempts prevents indefinite loops and resource exhaustion.
  • Quantity Reduction: For 'insufficient balance' errors, the order quantity is automatically reduced, allowing for a retry attempt with a viable size.
  • Circuit Breaker Pattern: This component monitors consecutive failures and, upon reaching a specified threshold, temporarily halts order submission to prevent cascading failures and allow the system to recover.

2. Dependencies and Initialization

This section outlines the required Python packages and configures the logging system for monitoring framework operations.

[23]
import time       # For managing delays between retries
import math       # Potentially for advanced backoff calculations (though not directly used in simple exponential)
import random     # For introducing jitter into backoff delays
import logging    # For structured logging of retry events
import functools  # For creating decorators that preserve function metadata
from typing import Callable, Any, Optional # For type hinting to improve code readability and maintainability
from dataclasses import dataclass, field # For creating data classes for configuration
from enum import Enum # For defining enumerated types (not directly used here, but common in such frameworks)

# Configure structured logging to record events, warnings, and errors
logging.basicConfig(
    level=logging.INFO, # Set the minimum logging level to INFO
    format="%(asctime)s | %(levelname)-8s | %(message)s", # Define log message format
    datefmt="%Y-%m-%d %H:%M:%S", # Define timestamp format
)
logger = logging.getLogger("retry_engine") # Obtain a logger instance for the retry engine
logger.info("Module dependencies loaded and logger initialized.")

3. Error Classification System

This section defines custom exception types and a classification utility to map raw exchange errors into standardized categories. This mapping is crucial for determining the appropriate retry strategy.

[24]
class OrderError(Exception):
    """Base exception for all order-related failures within the system."""
    pass

class RetryableError(OrderError):
    """Indicates a transient error that warrants a retry after a backoff period."""
    pass

class FatalError(OrderError):
    """Represents a permanent error, necessitating immediate abortion without retry."""
    pass

class BalanceError(OrderError):
    """Signifies an insufficient balance condition, triggering a quantity reduction and retry attempt."""
    pass

def classify_exchange_error(exception: Exception, exchange: str = "generic") -> str:
    """
    Maps a raw exception from an exchange API to a predefined failure category.

    This function inspects the exception message to identify keywords indicative of specific error types,
    thereby guiding the retry mechanism.

    Parameters
    ----------
    exception : The caught exception object.
    exchange  : An optional string indicating the exchange context, allowing for exchange-specific error mapping.

    Returns
    -------
    str
        One of the following classification strings:
        'fatal'     : Non-retryable error, immediate termination.
        'balance'   : Balance-related error, reduce quantity and retry.
        'rate_limit': Rate limit exceeded, apply specific delay and retry.
        'retryable' : General transient error, apply exponential backoff and retry.
    """
    msg = str(exception).lower()

    # Classification for non-retryable errors (e.g., authentication, invalid parameters)
    if any(kw in msg for kw in ["invalid api", "signature", "unauthorized", "401"]):
        return "fatal"
    if any(kw in msg for kw in ["invalid symbol", "invalid parameter", "bad request"]):
        return "fatal"

    # Classification for balance-related errors (triggering quantity reduction)
    if any(kw in msg for kw in ["insufficient", "not enough", "balance"]):
        return "balance"

    # Classification for rate limit errors (triggering specific delay)
    if any(kw in msg for kw in ["rate limit", "too many", "429", "10018"]):
        return "rate_limit"

    # Default classification for transient or network-related errors
    if any(kw in msg for kw in ["timeout", "connection", "503", "502", "reset"]):
        return "retryable"

    return "retryable"   # Default assumption: any unclassified error is retryable


logger.info("Error classification system and custom exception types defined.")

4. Core Retry Engine Components

This section defines the foundational elements of the retry engine: RetryConfig for parameterizing retry behavior, CircuitBreaker for system stability, and compute_backoff_delay for calculating dynamic retry intervals.

[25]
@dataclass
class RetryConfig:
    """
    Configuration parameters for controlling the behavior of the retry mechanism.

    Attributes
    ----------
    max_attempts      : int
        The maximum number of total attempts (including the initial attempt) before declaring failure.
    base_delay_s      : float
        The initial delay in seconds for exponential backoff.
    max_delay_s       : float
        The maximum allowable delay in seconds between retry attempts.
    backoff_multiplier: float
        The factor by which the delay increases with each subsequent attempt.
    jitter_fraction   : float
        A fractional value (0–1) representing the amplitude of random jitter added to the delay.
    qty_reduction_pct : float
        The percentage by which the order quantity is reduced upon encountering a `BalanceError`.
    circuit_breaker_n : int
        The number of consecutive failures that trigger the circuit breaker to open.
    """
    max_attempts:       int   = 5
    base_delay_s:       float = 1.0
    max_delay_s:        float = 60.0
    backoff_multiplier: float = 2.0
    jitter_fraction:    float = 0.2
    qty_reduction_pct:  float = 10.0
    circuit_breaker_n:  int   = 10


class CircuitBreaker:
    """
    Implements the Circuit Breaker pattern to prevent repeated operations that are likely to fail.

    This component manages three states: CLOSED (normal operation), OPEN (requests halted), and
    HALF-OPEN (allows a limited number of test requests to determine recovery).

    Attributes
    ----------
    threshold   : int
        The number of consecutive failures required to trip the circuit to the OPEN state.
    recovery_s  : float
        The duration in seconds for which the circuit remains OPEN before transitioning to HALF-OPEN.
    _failures   : int
        Internal counter for consecutive failures.
    _opened_at  : Optional[float]
        Timestamp when the circuit transitioned to the OPEN state.
    """
    def __init__(self, threshold: int = 10, recovery_s: float = 300):
        self.threshold   = threshold
        self.recovery_s  = recovery_s
        self._failures   = 0
        self._opened_at  = None

    @property
    def is_open(self) -> bool:
        """
        Evaluates the current state of the circuit breaker.

        Returns
        -------
        bool
            True if the circuit is OPEN and requests should be halted; False otherwise.
        """
        if self._opened_at is None:
            # Circuit is CLOSED
            return False
        if time.time() - self._opened_at >= self.recovery_s:
            # Circuit has been OPEN for longer than recovery_s, transition to HALF-OPEN
            self._opened_at = None
            return False
        # Circuit is OPEN
        return True

    def record_failure(self) -> None:
        """
        Registers a single failure event. If the failure threshold is met, the circuit transitions to OPEN.
        """
        self._failures += 1
        if self._failures >= self.threshold:
            self._opened_at = time.time()
            logger.error(f"[CIRCUIT OPEN] {self._failures} consecutive failures. "
                         f"Trading halted for {self.recovery_s}s.")

    def record_success(self) -> None:
        """
        Registers a success event. Resets the failure counter and closes the circuit if it was OPEN or HALF-OPEN.
        """
        self._failures  = 0
        self._opened_at = None


# Instantiate a global circuit breaker to manage system-wide order submission stability.
# This instance operates with a threshold of 10 consecutive failures and a 300-second recovery period.
circuit_breaker = CircuitBreaker(threshold=10, recovery_s=300)

def compute_backoff_delay(attempt: int, config: RetryConfig) -> float:
    """
    Calculates the exponential backoff delay, incorporating a jitter component.

    The formula used is: `min(base * multiplier^attempt, max_delay) * (1 ± jitter)`

    Parameters
    ----------
    attempt : int
        The zero-indexed current retry attempt number.
    config  : RetryConfig
        The configuration object containing backoff parameters.

    Returns
    -------
    float
        The computed delay in seconds, which is always non-negative.
    """
    # Calculate the exponential backoff component
    base_delay = config.base_delay_s * (config.backoff_multiplier ** attempt)
    # Apply the maximum delay cap
    capped     = min(base_delay, config.max_delay_s)
    # Introduce random jitter to prevent thundering herd problem
    jitter     = capped * config.jitter_fraction * (random.random() * 2 - 1)
    # Ensure the calculated delay is not negative
    return max(0, capped + jitter)


logger.info("Retry engine core components (RetryConfig, CircuitBreaker, compute_backoff_delay) defined.")

5. Order Retry Wrapper (retry_order)

This section introduces the retry_order function, a direct execution wrapper that encapsulates the retry logic. It processes order submission attempts, classifies failures, applies backoff, manages quantity reduction, and interacts with the circuit breaker.

[26]
def retry_order(
    order_fn: Callable[[float], Any],
    quantity: float,
    config: Optional[RetryConfig] = None,
    on_reduce_qty: Optional[Callable[[float], None]] = None
) -> Any:
    """
    Executes an order placement function with built-in automatic retry, failure classification,
    exponential backoff, and quantity reduction capabilities.

    This function iterates through retry attempts, handling different error types based on the
    `classify_exchange_error` mechanism and managing the circuit breaker state.

    Parameters
    ----------
    order_fn      : Callable[[float], Any]
        A callable function that accepts the order quantity as its sole argument and attempts
        to place an order. It is expected to return the order result on success or raise an
        exception on failure.
    quantity      : float
        The initial order quantity to be submitted.
    config        : Optional[RetryConfig]
        An instance of `RetryConfig` to customize retry parameters. If `None`, default configuration
        values are utilized.
    on_reduce_qty : Optional[Callable[[float], None]]
        An optional callback function that is invoked when the order quantity is reduced due to
        a `BalanceError`. It receives the new, reduced quantity as an argument.

    Returns
    -------
    Any
        The result returned by `order_fn` upon a successful order placement.

    Raises
    ------
    RuntimeError
        If the circuit breaker is open, indicating a system-wide halt in trading operations.
        Also raised if all retry attempts are exhausted without a successful order.
    FatalError
        If a non-retryable error (e.g., authentication failure, invalid parameter) is encountered,
        resulting in immediate termination.

    Example
    -------
    ```python
    # Assume 'exchange' is an initialized exchange API client.
    # Define a mock function for placing an order.
    def place_buy_order(qty: float):
        # Simulate an exchange call
        print(f"Attempting to place buy order for {qty} units...")
        if random.random() < 0.7: # Simulate failure 70% of the time
            if random.random() < 0.3: # Simulate fatal error 30% of failed attempts
                raise ConnectionError("API-KEY-INVALID")
            elif random.random() < 0.5: # Simulate balance error 20% of failed attempts
                raise ValueError("Not enough balance")
            else:
                raise ConnectionError("Network timeout")
        return {"status": "FILLED", "price": 100, "qty": qty}

    try:
        order_result = retry_order(
            order_fn=lambda qty: place_buy_order(qty=qty),
            quantity=10.0,
            config=RetryConfig(max_attempts=4, base_delay_s=0.5, qty_reduction_pct=20.0),
            on_reduce_qty=lambda new_qty: print(f"Quantity reduced to {new_qty}")
        )
        print(f"Order successful: {order_result}")
    except (RuntimeError, FatalError) as e:
        print(f"Order failed after all retries: {e}")
    ```
    """
    config = config or RetryConfig() # Use provided config or default instance
    current_qty = quantity           # Initialize current order quantity

    # Check if the circuit breaker is in an OPEN state. If so, halt operations immediately.
    if circuit_breaker.is_open:
        logger.error("Attempted order submission while circuit breaker is OPEN. Trading halted.")
        raise RuntimeError("Circuit breaker OPEN — trading halted.")

    # Iterate through the defined number of retry attempts
    for attempt in range(config.max_attempts):
        try:
            # Attempt to execute the order function with the current quantity
            result = order_fn(current_qty)
            # If successful, record success with the circuit breaker and return the result
            circuit_breaker.record_success()
            logger.info(f"[SUCCESS] Order filled on attempt {attempt + 1}. Quantity: {current_qty}")
            return result

        except Exception as e:
            # Classify the caught exception to determine the appropriate handling strategy
            error_class = classify_exchange_error(e)
            logger.warning(f"[ATTEMPT {attempt+1}/{config.max_attempts}] Error classified as: {error_class} | Exception: {e}")

            if error_class == "fatal":
                # For fatal errors, record a failure and raise a FatalError immediately
                circuit_breaker.record_failure()
                logger.error(f"Fatal error encountered: {e}. Aborting retry process.")
                raise FatalError(f"Non-retryable error: {e}") from e

            if error_class == "balance":
                # For balance errors, reduce the order quantity and retry without delay
                current_qty *= (1 - config.qty_reduction_pct / 100)
                logger.warning(f"[QTY REDUCE] Insufficient balance. New quantity: {current_qty:.6f}")
                if on_reduce_qty:
                    on_reduce_qty(current_qty) # Invoke callback if provided
                # Continue to the next attempt immediately (no backoff for balance errors)
                continue

            if error_class == "rate_limit":
                # For rate limit errors, compute a delay and pause before the next attempt
                delay = compute_backoff_delay(attempt, config)
                logger.info(f"[RATE LIMIT] Waiting {delay:.1f}s before next retry (attempt {attempt+2}).")
                time.sleep(delay)
                continue

            # For general retryable errors, apply exponential backoff delay
            delay = compute_backoff_delay(attempt, config)
            logger.info(f"[BACKOFF] Waiting {delay:.1f}s before next retry (attempt {attempt+2}).")
            time.sleep(delay)

    # If the loop completes, all attempts have been exhausted without success
    circuit_breaker.record_failure()
    logger.critical(f"Order submission failed after exhausting {config.max_attempts} attempts.")
    raise RuntimeError(f"Order failed after {config.max_attempts} attempts.")


logger.info("`retry_order` function, the primary order submission wrapper, has been defined.")

6. Decorator-Based Retry Mechanism (with_retry)

This section introduces with_retry, a Python decorator factory that enables the application of retry logic to any function declaratively. This provides a clean and reusable way to add retry capabilities without modifying the original function's implementation.

[27]
def with_retry(max_attempts: int = 5, base_delay: float = 1.0):
    """
    A decorator factory for applying retry logic to any synchronous function.

    This decorator enables functions to automatically re-attempt execution upon encountering
    retryable exceptions, utilizing exponential backoff with jitter.

    Parameters
    ----------
    max_attempts : int
        The maximum number of times the decorated function will be attempted, including the initial call.
    base_delay   : float
        The initial delay in seconds before the first retry, used in exponential backoff calculation.

    Returns
    -------
    Callable
        A decorator function that wraps the target function with retry logic.

    Usage Example
    -------------
    ```python
    @with_retry(max_attempts=3, base_delay=2.0)
    def place_my_order_decorated(qty):
        # Simulated function that might fail
        if random.random() < 0.8: # 80% chance of failure
            raise ConnectionError("Failed to connect to exchange")
        return {"status": "Success", "qty": qty}

    try:
        result = place_my_order_decorated(1.0)
        print(f"Decorated order successful: {result}")
    except RuntimeError as e:
        print(f"Decorated order failed: {e}")
    ```
    """
    def decorator(fn: Callable) -> Callable:
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            # Initialize RetryConfig with parameters provided to the decorator
            config = RetryConfig(max_attempts=max_attempts, base_delay_s=base_delay)
            # Iterate through the defined number of retry attempts
            for attempt in range(config.max_attempts):
                try:
                    # Attempt to execute the wrapped function
                    return fn(*args, **kwargs)
                except Exception as e:
                    # Classify the exception to determine if it's retryable
                    error_class = classify_exchange_error(e)
                    if error_class == "fatal":
                        # If fatal, re-raise immediately without retry
                        logger.error(f"[{fn.__name__}] Fatal error encountered: {e}. Aborting.")
                        raise
                    # If it's the last attempt, and still an error, raise RuntimeError
                    if attempt < config.max_attempts - 1:
                        # Compute backoff delay for the next attempt
                        delay = compute_backoff_delay(attempt, config)
                        logger.warning(f"[{fn.__name__}] Attempt {attempt+1} failed: {e}. "
                                       f"Retrying in {delay:.1f}s.")
                        time.sleep(delay)
                    else:
                        # All attempts exhausted, raise a RuntimeError
                        logger.critical(f"[{fn.__name__}] Failed after {config.max_attempts} attempts.")
                        raise RuntimeError(f"{fn.__name__} failed after {config.max_attempts} attempts.") from e
        return wrapper
    return decorator


logger.info("Initiating test of decorator-based retry mechanism.")
# ── Example usage of the decorator for demonstration ───────────────────────
@with_retry(max_attempts=3, base_delay=1.0)
def example_order_function(quantity: float) -> dict:
    """Simulated order function that raises an error on the first two invocations."""
    # Use a persistent attribute to track call count across invocations
    if not hasattr(example_order_function, "_call_count"):
        example_order_function._call_count = 0
    example_order_function._call_count += 1
    logger.info(f"Simulated order function call count: {example_order_function._call_count}")
    if example_order_function._call_count < 3:
        # Simulate a transient network error for the first two calls
        raise ConnectionError("Simulated network timeout")
    # On the third call, simulate success
    return {"status": "filled", "price": 50_000, "qty": quantity}


try:
    result = example_order_function(1.0)
    logger.info(f"Result from decorated function: {result}")
except RuntimeError as e:
    logger.error(f"Decorated function failed as expected: {e}")


logger.info("Decorator-based retry mechanism (`with_retry`) defined and demonstrated.")
WARNING:retry_engine:[example_order_function] Attempt 1 failed: Simulated network timeout. Retrying in 0.9s.
WARNING:retry_engine:[example_order_function] Attempt 2 failed: Simulated network timeout. Retrying in 2.3s.

7. Backoff Delay Visualization

This section provides a visual representation of the exponential backoff delay, including the effect of jitter and the maximum delay cap. This visualization aids in understanding the non-deterministic nature and bounds of retry intervals.

[28]
import matplotlib.pyplot as plt # For plotting and visualization
import numpy as np              # For numerical operations, especially mean and percentiles

# Instantiate RetryConfig with specific parameters for visualization purposes
config = RetryConfig(base_delay_s=1.0, max_delay_s=30.0, backoff_multiplier=2.0, jitter_fraction=0.2)
# Define the range of attempt numbers to visualize
attempts = list(range(10))

# Generate multiple delay samples for each attempt to demonstrate the effect of jitter.
# For each attempt, 100 delay values are computed to capture the random variation.
samples = [[compute_backoff_delay(a, config) for _ in range(100)] for a in attempts]
# Calculate the mean delay for each attempt
means   = [np.mean(s) for s in samples]
# Calculate the 10th percentile of delays for each attempt to show lower bound of jitter
lower   = [np.percentile(s, 10) for s in samples]
# Calculate the 90th percentile of delays for each attempt to show upper bound of jitter
upper   = [np.percentile(s, 90) for s in samples]

# Create a figure and an axes object for the plot
fig, ax = plt.subplots(figsize=(10, 5))

# Plot the mean delay across attempts
ax.plot(attempts, means, marker="o", color="#e74c3c", linewidth=2, label="Mean delay")
# Fill the area between the 10th and 90th percentiles to visualize jitter range
ax.fill_between(attempts, lower, upper, alpha=0.3, color="#e74c3c", label="10–90th percentile")
# Draw a horizontal line to indicate the maximum delay cap
ax.axhline(config.max_delay_s, linestyle="--", color="#333", label=f"Cap ({config.max_delay_s}s)")

# Set plot labels and title for clarity
ax.set_xlabel("Attempt Number")
ax.set_ylabel("Delay (seconds)")
ax.set_title("Exponential Backoff Delay with Jitter and Max Cap")
ax.legend() # Display the legend
ax.grid(True, alpha=0.3) # Add a subtle grid for readability

plt.tight_layout() # Adjust plot to ensure all elements fit without overlap
# Save the figure to a file (optional)
# plt.savefig("/home/claude/retry_backoff.png", dpi=150, bbox_inches="tight")
plt.show() # Display the generated plot
logger.info("Backoff delay visualization generated and displayed.")
cell output

8. Integration Pattern

This section illustrates a practical integration of the retry_order function into an exchange client's order placement method. This example demonstrates how to adapt an existing function (self.client.futures_create_order) to leverage the robust retry capabilities provided by the framework.

# ── Example Integration into a Hypothetical BinanceUtils.market_order Method ──

from binance.client import Client # Assuming this import is present in the actual BinanceUtils module
from binance.enums import *        # Assuming necessary Binance enums like ORDER_TYPE_MARKET

class BinanceUtils:
    def __init__(self, api_key, api_secret, ticker):
        self.client = Client(api_key, api_secret)
        self.ticker = ticker

    def market_order_with_retry(self, quantity, direction, order_for):
        """
        Places a market order with integrated retry logic.

        Parameters
        ----------
        quantity  : float
            The quantity of the asset to trade.
        direction : str
            The order side (e.g., 'BUY', 'SELL').
        order_for : str
            Contextual information for the order.

        Returns
        -------
        str
            The order ID upon successful placement.
        """
        # Configure specific retry parameters for this market order context
        config = RetryConfig(
            max_attempts      = 5,    # Allow up to 5 attempts
            base_delay_s      = 2.0,  # Start with a 2-second delay
            qty_reduction_pct = 10.0, # Reduce quantity by 10% on balance errors
        )

        # Define the core order placement function to be wrapped by retry_order.
        # This function encapsulates the actual exchange API call.
        def order_fn(qty_to_place: float):
            response = self.client.futures_create_order(
                symbol=self.ticker,
                side=direction,
                type=ORDER_TYPE_MARKET,
                quantity=qty_to_place,
            )
            # The order_fn should return a meaningful success indicator, e.g., order ID.
            return response["orderId"]

        try:
            # Execute the order placement with retry logic
            order_id = retry_order(order_fn, quantity, config)
            logger.info(f"Market order placed successfully: Order ID {order_id}")
            return order_id
        except (RuntimeError, FatalError) as e:
            logger.error(f"Market order failed after all retries: {e}")
            raise # Re-raise the exception to indicate overall failure

# Example of how to use this within a client context (conceptual)
# api_key = "YOUR_BINANCE_API_KEY"
# api_secret = "YOUR_BINANCE_API_SECRET"
# binance_client = BinanceUtils(api_key, api_secret, ticker="BTCUSDT")
# try:
#     new_order_id = binance_client.market_order_with_retry(quantity=0.001, direction='BUY', order_for='test_trade')
#     print(f"Successfully placed order with ID: {new_order_id}")
# except Exception as e:
#     print(f"Failed to place order: {e}")

logger.info("Integration pattern for retry_order demonstrated.")

9. Limit Order Price Adjustment Strategy

This section introduces a strategy to dynamically adjust the price of a limit order, or convert it to a market order, if the current market price deviates significantly from the initial intended limit price. This addresses scenarios where a limit order might not fill due to market movement and provides a mechanism to ensure order execution by adapting to current market conditions.

[29]
def place_adjusted_order(
    exchange_order_fn: Callable[[float, float], Any],
    intended_price: float,
    current_market_price: float,
    quantity: float,
    order_side: str, # e.g., 'BUY', 'SELL'
    price_tolerance_pct: float = 0.05, # 0.05% tolerance
    price_adjustment_factor: float = 0.01, # 0.01% adjustment for limit orders
    fallback_to_market: bool = True,
    config: Optional[RetryConfig] = None
) -> Any:
    """
    Places an order with dynamic price adjustment or fallback to a market order
    if the market price deviates from the intended limit price.

    This function assesses the difference between the intended limit price and the current market price.
    If the deviation exceeds a specified tolerance, the order price is adjusted, or the order type
    is converted to a market order to facilitate execution.

    Parameters
    ----------
    exchange_order_fn       : Callable[[float, float], Any]
        A function that accepts `quantity` and `price` (or `None` for market orders)
        and attempts to place an order. Expected to return order details or raise an exception.
    intended_price          : float
        The initial price at which the order was intended to be placed (limit price).
    current_market_price    : float
        The current relevant market price (e.g., best ask for buy, best bid for sell).
    quantity                : float
        The quantity of the asset to trade.
    order_side              : str
        The side of the order, typically 'BUY' or 'SELL'.
    price_tolerance_pct     : float, optional
        The percentage deviation allowed between `intended_price` and `current_market_price`
        before triggering an adjustment. Default is 0.05 (0.05%).
    price_adjustment_factor : float, optional
        The percentage by which the `intended_price` is adjusted if it's out of tolerance
        but still to be placed as a limit order. Default is 0.01 (0.01%).
    fallback_to_market      : bool, optional
        If True, the order will be placed as a market order if the price deviation
        is too large and `price_adjustment_factor` is not applied. Default is True.
    config                  : Optional[RetryConfig], optional
        An instance of `RetryConfig` for the `retry_order` mechanism. Uses default if None.

    Returns
    -------
    Any
        The result returned by the `exchange_order_fn` upon successful order placement.

    Raises
    ------
    RuntimeError
        If the order fails after all retries or if the circuit breaker is open.
    FatalError
        If a non-retryable error occurs during order submission.
    """
    config = config or RetryConfig()
    adjusted_price = intended_price
    order_type_str = "LIMIT"

    # Calculate the percentage deviation of the market price from the intended limit price.
    price_deviation_pct = abs((current_market_price - intended_price) / intended_price) * 100

    if price_deviation_pct > price_tolerance_pct:
        logger.warning(f"Market price ({current_market_price}) deviates by {price_deviation_pct:.4f}% "
                       f"from intended limit price ({intended_price}).")

        if fallback_to_market:
            # If deviation is too large and fallback is enabled, switch to a market order.
            adjusted_price = None  # Indicate a market order
            order_type_str = "MARKET"
            logger.info("Falling back to market order for guaranteed execution.")
        else:
            # Adjust limit price based on order side.
            adjustment = intended_price * (price_adjustment_factor / 100)
            if order_side == 'BUY':
                # For buy orders, increase the limit price slightly to improve fill probability.
                adjusted_price = current_market_price + adjustment
                logger.info(f"Adjusting BUY limit price to {adjusted_price:.6f}.")
            elif order_side == 'SELL':
                # For sell orders, decrease the limit price slightly to improve fill probability.
                adjusted_price = current_market_price - adjustment
                logger.info(f"Adjusting SELL limit price to {adjusted_price:.6f}.")
            else:
                logger.error("Invalid order_side specified.")
                raise ValueError("Invalid order_side. Must be 'BUY' or 'SELL'.")

    # Define the order function for retry_order, encapsulating the exchange call with potentially adjusted price.
    def order_placement_func(qty: float):
        # For market orders (adjusted_price is None), the exchange_order_fn should handle it accordingly.
        # This assumes exchange_order_fn can accept None for price for market orders,
        # or the caller needs to provide a wrapper that correctly handles `adjusted_price`.
        if adjusted_price is None:
            # Call the exchange function without a price for market orders
            return exchange_order_fn(qty, None)
        else:
            # Call the exchange function with the adjusted limit price
            return exchange_order_fn(qty, adjusted_price)

    try:
        # Execute the order placement function with retry logic
        order_result = retry_order(order_placement_func, quantity, config=config)
        # Conditionally format the price for logging, handling None for market orders
        price_log_str = 'N/A' if adjusted_price is None else f"{adjusted_price:.6f}"
        logger.info(f"Order placed successfully: Type={order_type_str}, Price={price_log_str}, Qty={quantity}")
        return order_result
    except (RuntimeError, FatalError) as e:
        logger.error(f"Adjusted order placement failed: {e}")
        raise

logger.info("Limit order price adjustment strategy (`place_adjusted_order`) defined.")

10. Dummy Data Generation and Strategy Demonstration

This section simulates an exchange environment and demonstrates the place_adjusted_order function under various market conditions. It includes:

  • A Simulated Exchange Order Placement Function: This function mimics the behavior of a real exchange API, including potential failures (network, balance, fatal) and successful order placement.
  • Demonstration Scenarios: Practical examples showcasing how place_adjusted_order handles different price deviations, adjusts limit prices, or falls back to market orders, leveraging the retry framework.
[30]
class MockExchange:
    """
    A mock exchange client to simulate order placement and market data retrieval.

    This class is now enhanced to allow for deterministic error sequences for testing
    the retry and price adjustment mechanisms without relying on pure randomness.
    """
    def __init__(self, ticker: str = "BTCUSDT", error_sequence: Optional[list] = None):
        self.ticker = ticker
        self.successful_orders = []
        self._call_count = 0
        # Predefined sequence of errors to simulate. Each element is an error type string.
        # 'none': no error, 'fatal': FatalError, 'balance': BalanceError, 'retryable': RetryableError (ConnectionError)
        self._error_sequence = error_sequence if error_sequence is not None else []
        logger.info("MockExchange initialized.")

    def get_current_market_price(self) -> float:
        """
        Simulates fetching the current market price for the configured ticker.
        Returns a slightly randomized price around a base value.
        """
        base_price = 50000.0
        fluctuation = random.uniform(-0.0005, 0.0005) * base_price # +/- 0.05%
        return base_price * (1 + fluctuation)

    def simulate_exchange_order_placement(self, quantity: float, price: Optional[float] = None) -> dict:
        """
        Simulates an order placement on the exchange, using a predefined error sequence
        or falling back to a default probabilistic model if the sequence is exhausted.

        Parameters
        ----------
        quantity : float
            The quantity of the asset to trade.
        price    : Optional[float]
            The limit price for the order. If None, it's considered a market order.

        Returns
        -------
        dict
            A dictionary representing the order confirmation if successful.

        Raises
        ------
        ConnectionError
            Simulates a network-related or transient error.
        ValueError
            Simulates an insufficient balance or invalid parameter error.
        """
        self._call_count += 1
        log_prefix = f"Mock Exchange (call {self._call_count}):"
        logger.info(f"{log_prefix} Attempting to place order - Qty: {quantity}, Price: {price if price else 'MARKET'}")

        # Use predefined error sequence if available
        if self._call_count - 1 < len(self._error_sequence):
            sim_error = self._error_sequence[self._call_count - 1]
            if sim_error == 'fatal':
                logger.warning(f"{log_prefix} Simulating FATAL error.")
                raise ValueError("Invalid SYMBOL detected") # Example fatal error
            elif sim_error == 'balance':
                logger.warning(f"{log_prefix} Simulating BALANCE error.")
                raise ValueError("Insufficient balance for this order")
            elif sim_error == 'retryable':
                logger.warning(f"{log_prefix} Simulating RETRYABLE error (Network timeout).")
                raise ConnectionError("Network timeout during order submission")
            # 'none' will proceed to successful placement
        else:
            # Fallback to probabilistic model if sequence exhausted (or empty)
            rand_val = random.random()
            if rand_val < 0.01: # 1% chance of a FATAL error
                logger.warning(f"{log_prefix} Probabilistically Simulating FATAL error.")
                if random.random() < 0.5: raise ValueError("Invalid API-KEY")
                else: raise ValueError("Invalid SYMBOL detected")
            elif rand_val < 0.06: # 5% chance of a BALANCE error
                logger.warning(f"{log_prefix} Probabilistically Simulating BALANCE error.")
                raise ValueError("Insufficient balance for this order")
            elif rand_val < 0.40: # 34% chance of a RETRYABLE error
                logger.warning(f"{log_prefix} Probabilistically Simulating RETRYABLE error.")
                if random.random() < 0.5: raise ConnectionError("Exchange rate limit exceeded (HTTP 429)")
                else: raise ConnectionError("Network timeout during order submission")

        # Successful order placement
        order_id = f"MOCK_ORDER_{len(self.successful_orders) + 1}_{int(time.time() * 1000)}"
        order_result = {
            "orderId": order_id,
            "status": "FILLED",
            "price": price if price is not None else self.get_current_market_price(),
            "executedQty": quantity
        }
        self.successful_orders.append(order_result)
        logger.info(f"{log_prefix} Order {order_id} placed successfully.")
        return order_result

# NOTE: Initializing a default mock_exchange for general use if not explicitly defined in scenarios.
# However, for demonstrations, each scenario will create its own instance with a specific error_sequence.
mock_exchange = MockExchange()

logger.info("Mock exchange client for dummy data generation has been initialized.")

Scenario 1: Order with No Significant Price Deviation (Successful Limit Order)

This scenario demonstrates the placement of a limit order where the market price is within the acceptable price_tolerance_pct of the intended_price. The order is expected to be placed as a limit order at the intended_price.

[31]
# Define parameters for the first scenario
intended_price_scenario1 = 50000.0
current_market_price_scenario1 = intended_price_scenario1 * 1.00002 # 0.002% deviation
quantity_scenario1 = 0.01
order_side_scenario1 = 'BUY'

logger.info("--- SCENARIO 1: No Significant Price Deviation (Limit Order, Retryable/Balance errors) ---")
# Simulate 2 balance errors, then 1 retryable error, then success
mock_exchange_s1 = MockExchange(error_sequence=['balance', 'balance', 'retryable', 'none'])
try:
    result1 = place_adjusted_order(
        exchange_order_fn=mock_exchange_s1.simulate_exchange_order_placement,
        intended_price=intended_price_scenario1,
        current_market_price=current_market_price_scenario1,
        quantity=quantity_scenario1,
        order_side=order_side_scenario1,
        price_tolerance_pct=0.05,
        fallback_to_market=True
    )
    logger.info(f"Scenario 1 Result: {result1}")
except (RuntimeError, FatalError) as e:
    logger.error(f"Scenario 1 Failed: {e}")
logger.info("Scenario 1 demonstration completed.")
WARNING:retry_engine:Mock Exchange (call 1): Simulating BALANCE error.
WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: balance | Exception: Insufficient balance for this order
WARNING:retry_engine:[QTY REDUCE] Insufficient balance. New quantity: 0.009000
WARNING:retry_engine:Mock Exchange (call 2): Simulating BALANCE error.
WARNING:retry_engine:[ATTEMPT 2/5] Error classified as: balance | Exception: Insufficient balance for this order
WARNING:retry_engine:[QTY REDUCE] Insufficient balance. New quantity: 0.008100
WARNING:retry_engine:Mock Exchange (call 3): Simulating RETRYABLE error (Network timeout).
WARNING:retry_engine:[ATTEMPT 3/5] Error classified as: retryable | Exception: Network timeout during order submission

Scenario 2: Order with Slight Price Deviation (Adjusted Limit Order, No Fallback)

This scenario simulates a situation where the market price deviates slightly beyond the price_tolerance_pct from the intended_price. With fallback_to_market set to False, the system will adjust the limit order's price slightly to improve its chances of execution, rather than converting it to a market order.

[32]
# Define parameters for the second scenario
intended_price_scenario2 = 50000.0
current_market_price_scenario2 = intended_price_scenario2 * 1.001 # 0.1% deviation
quantity_scenario2 = 0.02
order_side_scenario2 = 'BUY'

logger.info("--- SCENARIO 2: Slight Price Deviation (Adjusted Limit Order, No Fallback) ---")
# Simulate 1 retryable error, then success
mock_exchange_s2 = MockExchange(error_sequence=['retryable', 'none'])
try:
    result2 = place_adjusted_order(
        exchange_order_fn=mock_exchange_s2.simulate_exchange_order_placement,
        intended_price=intended_price_scenario2,
        current_market_price=current_market_price_scenario2,
        quantity=quantity_scenario2,
        order_side=order_side_scenario2,
        price_tolerance_pct=0.05, # Tolerance exceeded (0.1% > 0.05%)
        price_adjustment_factor=0.02,
        fallback_to_market=False # Crucially, do NOT fall back to market order
    )
    logger.info(f"Scenario 2 Result: {result2}")
except (RuntimeError, FatalError) as e:
    logger.error(f"Scenario 2 Failed: {e}")
logger.info("Scenario 2 demonstration completed.")
WARNING:retry_engine:Market price (50049.99999999999) deviates by 0.1000% from intended limit price (50000.0).
WARNING:retry_engine:Mock Exchange (call 1): Simulating RETRYABLE error (Network timeout).
WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Network timeout during order submission

Scenario 3: Order with Significant Price Deviation (Fallback to Market Order)

This scenario demonstrates a significant divergence between the intended_price and the current_market_price, exceeding the price_tolerance_pct. With fallback_to_market enabled, the place_adjusted_order function converts the order into a market order to ensure execution, prioritizing fill over a specific limit price.

[33]
# Define parameters for the third scenario
intended_price_scenario3 = 50000.0
current_market_price_scenario3 = intended_price_scenario3 * 1.005 # 0.5% deviation
quantity_scenario3 = 0.03
order_side_scenario3 = 'SELL'

logger.info("--- SCENARIO 3: Significant Price Deviation (Fallback to Market Order) ---")
# Simulate 1 retryable error, then success
mock_exchange_s3 = MockExchange(error_sequence=['retryable', 'none'])
try:
    result3 = place_adjusted_order(
        exchange_order_fn=mock_exchange_s3.simulate_exchange_order_placement,
        intended_price=intended_price_scenario3,
        current_market_price=current_market_price_scenario3,
        quantity=quantity_scenario3,
        order_side=order_side_scenario3,
        price_tolerance_pct=0.1, # Tolerance exceeded (0.5% > 0.1%)
        fallback_to_market=True # Crucially, enable fallback to market order
    )
    logger.info(f"Scenario 3 Result: {result3}")
except (RuntimeError, FatalError) as e:
    logger.error(f"Scenario 3 Failed: {e}")
logger.info("Scenario 3 demonstration completed.")
WARNING:retry_engine:Market price (50249.99999999999) deviates by 0.5000% from intended limit price (50000.0).
WARNING:retry_engine:Mock Exchange (call 1): Simulating RETRYABLE error (Network timeout).
WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Network timeout during order submission

Scenario 4: Error Handling within place_adjusted_order

This scenario demonstrates how the place_adjusted_order function, leveraging the underlying retry_order mechanism, handles transient failures during the actual order submission attempt. It will simulate a series of failures before a successful order or exhaustion of retries.

[34]
class MockExchangeReliable:
    """
    A mock exchange client that guarantees success after a few attempts.
    Used to specifically test the retry logic within `place_adjusted_order`.
    """
    def __init__(self, fail_count: int = 2):
        self.call_count = 0
        self.fail_count = fail_count
        logger.info(f"MockExchangeReliable initialized to fail {self.fail_count} times.")

    def get_current_market_price(self) -> float:
        return 50000.0

    def simulate_order_placement_with_transient_failure(self, quantity: float, price: Optional[float] = None) -> dict:
        self.call_count += 1
        if self.call_count <= self.fail_count:
            logger.warning(f"Mock Exchange Reliable: Simulating transient error (call {self.call_count}).")
            raise ConnectionError("Simulated network hiccup")
        else:
            order_id = f"RELIABLE_ORDER_{int(time.time() * 1000)}"
            order_result = {
                "orderId": order_id,
                "status": "FILLED",
                "price": price if price is not None else self.get_current_market_price(),
                "executedQty": quantity
            }
            logger.info(f"Mock Exchange Reliable: Order {order_id} placed successfully after {self.call_count} attempts.")
            return order_result


# Instantiate a reliable mock exchange that fails a few times initially
mock_exchange_reliable = MockExchangeReliable(fail_count=2)

# Define parameters for the fourth scenario
intended_price_scenario4 = 49990.0
current_market_price_scenario4 = 49990.0 # No price deviation for simplicity in this test
quantity_scenario4 = 0.05
order_side_scenario4 = 'BUY'

logger.info("--- SCENARIO 4: Error Handling within place_adjusted_order (Always Succeeds) ---")
try:
    # Use a custom RetryConfig to limit attempts for a quicker test
    custom_config_scenario4 = RetryConfig(max_attempts=5, base_delay_s=0.1, jitter_fraction=0.1)

    result4 = place_adjusted_order(
        exchange_order_fn=mock_exchange_reliable.simulate_order_placement_with_transient_failure,
        intended_price=intended_price_scenario4,
        current_market_price=current_market_price_scenario4,
        quantity=quantity_scenario4,
        order_side=order_side_scenario4,
        price_tolerance_pct=0.01,
        fallback_to_market=False,
        config=custom_config_scenario4
    )
    logger.info(f"Scenario 4 Result: {result4}")
except (RuntimeError, FatalError) as e:
    logger.error(f"Scenario 4 Failed: {e}")
logger.info("Scenario 4 demonstration completed.")
WARNING:retry_engine:Mock Exchange Reliable: Simulating transient error (call 1).
WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Simulated network hiccup
WARNING:retry_engine:Mock Exchange Reliable: Simulating transient error (call 2).
WARNING:retry_engine:[ATTEMPT 2/5] Error classified as: retryable | Exception: Simulated network hiccup