Retry Failed Orders
Build a robust order retry system with exponential backoff, partial fill detection, and error classification for production trading.
Order Submission Retry Framework
1. Overview
This notebook presents a production-grade retry framework designed for robust handling of transient failures during order submission across various supported exchanges.
Failure Classification
Order submission failures are categorized to determine appropriate error handling and retry strategies:
| Failure Type | Primary Cause | Retry Action | Operational Response |
|---|---|---|---|
| Network timeout | Connectivity issues | ✅ Yes | Implement exponential backoff. |
| Rate limit (HTTP 429) | Excessive requests | ✅ Yes | Introduce a wait period before retrying. |
| Insufficient balance | Account state anomaly | ⚠️ Conditional | Reduce order quantity and re-attempt. |
| Invalid parameter | Application logic error | ❌ No | Log the error and terminate the process. |
| Exchange outage (HTTP 503) | Service unavailability | ✅ Yes | Apply a prolonged wait before re-attempting. |
| Authentication failure (HTTP 401) | Invalid credentials | ❌ No | Terminate immediately; no retry attempted. |
Strategic Implementation
Key strategies employed within this framework include:
- Exponential Backoff with Jitter: This mechanism prevents system overload by progressively increasing delay between retries while adding a random component to mitigate synchronized retry storms.
- Maximum Retry Cap: A defined limit on retry attempts prevents indefinite loops and resource exhaustion.
- Quantity Reduction: For 'insufficient balance' errors, the order quantity is automatically reduced, allowing for a retry attempt with a viable size.
- Circuit Breaker Pattern: This component monitors consecutive failures and, upon reaching a specified threshold, temporarily halts order submission to prevent cascading failures and allow the system to recover.
2. Dependencies and Initialization
This section outlines the required Python packages and configures the logging system for monitoring framework operations.
import time # For managing delays between retries
import math # Potentially for advanced backoff calculations (though not directly used in simple exponential)
import random # For introducing jitter into backoff delays
import logging # For structured logging of retry events
import functools # For creating decorators that preserve function metadata
from typing import Callable, Any, Optional # For type hinting to improve code readability and maintainability
from dataclasses import dataclass, field # For creating data classes for configuration
from enum import Enum # For defining enumerated types (not directly used here, but common in such frameworks)
# Configure structured logging to record events, warnings, and errors
logging.basicConfig(
level=logging.INFO, # Set the minimum logging level to INFO
format="%(asctime)s | %(levelname)-8s | %(message)s", # Define log message format
datefmt="%Y-%m-%d %H:%M:%S", # Define timestamp format
)
logger = logging.getLogger("retry_engine") # Obtain a logger instance for the retry engine
logger.info("Module dependencies loaded and logger initialized.")3. Error Classification System
This section defines custom exception types and a classification utility to map raw exchange errors into standardized categories. This mapping is crucial for determining the appropriate retry strategy.
class OrderError(Exception):
"""Base exception for all order-related failures within the system."""
pass
class RetryableError(OrderError):
"""Indicates a transient error that warrants a retry after a backoff period."""
pass
class FatalError(OrderError):
"""Represents a permanent error, necessitating immediate abortion without retry."""
pass
class BalanceError(OrderError):
"""Signifies an insufficient balance condition, triggering a quantity reduction and retry attempt."""
pass
def classify_exchange_error(exception: Exception, exchange: str = "generic") -> str:
"""
Maps a raw exception from an exchange API to a predefined failure category.
This function inspects the exception message to identify keywords indicative of specific error types,
thereby guiding the retry mechanism.
Parameters
----------
exception : The caught exception object.
exchange : An optional string indicating the exchange context, allowing for exchange-specific error mapping.
Returns
-------
str
One of the following classification strings:
'fatal' : Non-retryable error, immediate termination.
'balance' : Balance-related error, reduce quantity and retry.
'rate_limit': Rate limit exceeded, apply specific delay and retry.
'retryable' : General transient error, apply exponential backoff and retry.
"""
msg = str(exception).lower()
# Classification for non-retryable errors (e.g., authentication, invalid parameters)
if any(kw in msg for kw in ["invalid api", "signature", "unauthorized", "401"]):
return "fatal"
if any(kw in msg for kw in ["invalid symbol", "invalid parameter", "bad request"]):
return "fatal"
# Classification for balance-related errors (triggering quantity reduction)
if any(kw in msg for kw in ["insufficient", "not enough", "balance"]):
return "balance"
# Classification for rate limit errors (triggering specific delay)
if any(kw in msg for kw in ["rate limit", "too many", "429", "10018"]):
return "rate_limit"
# Default classification for transient or network-related errors
if any(kw in msg for kw in ["timeout", "connection", "503", "502", "reset"]):
return "retryable"
return "retryable" # Default assumption: any unclassified error is retryable
logger.info("Error classification system and custom exception types defined.")4. Core Retry Engine Components
This section defines the foundational elements of the retry engine: RetryConfig for parameterizing retry behavior, CircuitBreaker for system stability, and compute_backoff_delay for calculating dynamic retry intervals.
@dataclass
class RetryConfig:
"""
Configuration parameters for controlling the behavior of the retry mechanism.
Attributes
----------
max_attempts : int
The maximum number of total attempts (including the initial attempt) before declaring failure.
base_delay_s : float
The initial delay in seconds for exponential backoff.
max_delay_s : float
The maximum allowable delay in seconds between retry attempts.
backoff_multiplier: float
The factor by which the delay increases with each subsequent attempt.
jitter_fraction : float
A fractional value (0–1) representing the amplitude of random jitter added to the delay.
qty_reduction_pct : float
The percentage by which the order quantity is reduced upon encountering a `BalanceError`.
circuit_breaker_n : int
The number of consecutive failures that trigger the circuit breaker to open.
"""
max_attempts: int = 5
base_delay_s: float = 1.0
max_delay_s: float = 60.0
backoff_multiplier: float = 2.0
jitter_fraction: float = 0.2
qty_reduction_pct: float = 10.0
circuit_breaker_n: int = 10
class CircuitBreaker:
"""
Implements the Circuit Breaker pattern to prevent repeated operations that are likely to fail.
This component manages three states: CLOSED (normal operation), OPEN (requests halted), and
HALF-OPEN (allows a limited number of test requests to determine recovery).
Attributes
----------
threshold : int
The number of consecutive failures required to trip the circuit to the OPEN state.
recovery_s : float
The duration in seconds for which the circuit remains OPEN before transitioning to HALF-OPEN.
_failures : int
Internal counter for consecutive failures.
_opened_at : Optional[float]
Timestamp when the circuit transitioned to the OPEN state.
"""
def __init__(self, threshold: int = 10, recovery_s: float = 300):
self.threshold = threshold
self.recovery_s = recovery_s
self._failures = 0
self._opened_at = None
@property
def is_open(self) -> bool:
"""
Evaluates the current state of the circuit breaker.
Returns
-------
bool
True if the circuit is OPEN and requests should be halted; False otherwise.
"""
if self._opened_at is None:
# Circuit is CLOSED
return False
if time.time() - self._opened_at >= self.recovery_s:
# Circuit has been OPEN for longer than recovery_s, transition to HALF-OPEN
self._opened_at = None
return False
# Circuit is OPEN
return True
def record_failure(self) -> None:
"""
Registers a single failure event. If the failure threshold is met, the circuit transitions to OPEN.
"""
self._failures += 1
if self._failures >= self.threshold:
self._opened_at = time.time()
logger.error(f"[CIRCUIT OPEN] {self._failures} consecutive failures. "
f"Trading halted for {self.recovery_s}s.")
def record_success(self) -> None:
"""
Registers a success event. Resets the failure counter and closes the circuit if it was OPEN or HALF-OPEN.
"""
self._failures = 0
self._opened_at = None
# Instantiate a global circuit breaker to manage system-wide order submission stability.
# This instance operates with a threshold of 10 consecutive failures and a 300-second recovery period.
circuit_breaker = CircuitBreaker(threshold=10, recovery_s=300)
def compute_backoff_delay(attempt: int, config: RetryConfig) -> float:
"""
Calculates the exponential backoff delay, incorporating a jitter component.
The formula used is: `min(base * multiplier^attempt, max_delay) * (1 ± jitter)`
Parameters
----------
attempt : int
The zero-indexed current retry attempt number.
config : RetryConfig
The configuration object containing backoff parameters.
Returns
-------
float
The computed delay in seconds, which is always non-negative.
"""
# Calculate the exponential backoff component
base_delay = config.base_delay_s * (config.backoff_multiplier ** attempt)
# Apply the maximum delay cap
capped = min(base_delay, config.max_delay_s)
# Introduce random jitter to prevent thundering herd problem
jitter = capped * config.jitter_fraction * (random.random() * 2 - 1)
# Ensure the calculated delay is not negative
return max(0, capped + jitter)
logger.info("Retry engine core components (RetryConfig, CircuitBreaker, compute_backoff_delay) defined.")5. Order Retry Wrapper (retry_order)
This section introduces the retry_order function, a direct execution wrapper that encapsulates the retry logic. It processes order submission attempts, classifies failures, applies backoff, manages quantity reduction, and interacts with the circuit breaker.
def retry_order(
order_fn: Callable[[float], Any],
quantity: float,
config: Optional[RetryConfig] = None,
on_reduce_qty: Optional[Callable[[float], None]] = None
) -> Any:
"""
Executes an order placement function with built-in automatic retry, failure classification,
exponential backoff, and quantity reduction capabilities.
This function iterates through retry attempts, handling different error types based on the
`classify_exchange_error` mechanism and managing the circuit breaker state.
Parameters
----------
order_fn : Callable[[float], Any]
A callable function that accepts the order quantity as its sole argument and attempts
to place an order. It is expected to return the order result on success or raise an
exception on failure.
quantity : float
The initial order quantity to be submitted.
config : Optional[RetryConfig]
An instance of `RetryConfig` to customize retry parameters. If `None`, default configuration
values are utilized.
on_reduce_qty : Optional[Callable[[float], None]]
An optional callback function that is invoked when the order quantity is reduced due to
a `BalanceError`. It receives the new, reduced quantity as an argument.
Returns
-------
Any
The result returned by `order_fn` upon a successful order placement.
Raises
------
RuntimeError
If the circuit breaker is open, indicating a system-wide halt in trading operations.
Also raised if all retry attempts are exhausted without a successful order.
FatalError
If a non-retryable error (e.g., authentication failure, invalid parameter) is encountered,
resulting in immediate termination.
Example
-------
```python
# Assume 'exchange' is an initialized exchange API client.
# Define a mock function for placing an order.
def place_buy_order(qty: float):
# Simulate an exchange call
print(f"Attempting to place buy order for {qty} units...")
if random.random() < 0.7: # Simulate failure 70% of the time
if random.random() < 0.3: # Simulate fatal error 30% of failed attempts
raise ConnectionError("API-KEY-INVALID")
elif random.random() < 0.5: # Simulate balance error 20% of failed attempts
raise ValueError("Not enough balance")
else:
raise ConnectionError("Network timeout")
return {"status": "FILLED", "price": 100, "qty": qty}
try:
order_result = retry_order(
order_fn=lambda qty: place_buy_order(qty=qty),
quantity=10.0,
config=RetryConfig(max_attempts=4, base_delay_s=0.5, qty_reduction_pct=20.0),
on_reduce_qty=lambda new_qty: print(f"Quantity reduced to {new_qty}")
)
print(f"Order successful: {order_result}")
except (RuntimeError, FatalError) as e:
print(f"Order failed after all retries: {e}")
```
"""
config = config or RetryConfig() # Use provided config or default instance
current_qty = quantity # Initialize current order quantity
# Check if the circuit breaker is in an OPEN state. If so, halt operations immediately.
if circuit_breaker.is_open:
logger.error("Attempted order submission while circuit breaker is OPEN. Trading halted.")
raise RuntimeError("Circuit breaker OPEN — trading halted.")
# Iterate through the defined number of retry attempts
for attempt in range(config.max_attempts):
try:
# Attempt to execute the order function with the current quantity
result = order_fn(current_qty)
# If successful, record success with the circuit breaker and return the result
circuit_breaker.record_success()
logger.info(f"[SUCCESS] Order filled on attempt {attempt + 1}. Quantity: {current_qty}")
return result
except Exception as e:
# Classify the caught exception to determine the appropriate handling strategy
error_class = classify_exchange_error(e)
logger.warning(f"[ATTEMPT {attempt+1}/{config.max_attempts}] Error classified as: {error_class} | Exception: {e}")
if error_class == "fatal":
# For fatal errors, record a failure and raise a FatalError immediately
circuit_breaker.record_failure()
logger.error(f"Fatal error encountered: {e}. Aborting retry process.")
raise FatalError(f"Non-retryable error: {e}") from e
if error_class == "balance":
# For balance errors, reduce the order quantity and retry without delay
current_qty *= (1 - config.qty_reduction_pct / 100)
logger.warning(f"[QTY REDUCE] Insufficient balance. New quantity: {current_qty:.6f}")
if on_reduce_qty:
on_reduce_qty(current_qty) # Invoke callback if provided
# Continue to the next attempt immediately (no backoff for balance errors)
continue
if error_class == "rate_limit":
# For rate limit errors, compute a delay and pause before the next attempt
delay = compute_backoff_delay(attempt, config)
logger.info(f"[RATE LIMIT] Waiting {delay:.1f}s before next retry (attempt {attempt+2}).")
time.sleep(delay)
continue
# For general retryable errors, apply exponential backoff delay
delay = compute_backoff_delay(attempt, config)
logger.info(f"[BACKOFF] Waiting {delay:.1f}s before next retry (attempt {attempt+2}).")
time.sleep(delay)
# If the loop completes, all attempts have been exhausted without success
circuit_breaker.record_failure()
logger.critical(f"Order submission failed after exhausting {config.max_attempts} attempts.")
raise RuntimeError(f"Order failed after {config.max_attempts} attempts.")
logger.info("`retry_order` function, the primary order submission wrapper, has been defined.")6. Decorator-Based Retry Mechanism (with_retry)
This section introduces with_retry, a Python decorator factory that enables the application of retry logic to any function declaratively. This provides a clean and reusable way to add retry capabilities without modifying the original function's implementation.
def with_retry(max_attempts: int = 5, base_delay: float = 1.0):
"""
A decorator factory for applying retry logic to any synchronous function.
This decorator enables functions to automatically re-attempt execution upon encountering
retryable exceptions, utilizing exponential backoff with jitter.
Parameters
----------
max_attempts : int
The maximum number of times the decorated function will be attempted, including the initial call.
base_delay : float
The initial delay in seconds before the first retry, used in exponential backoff calculation.
Returns
-------
Callable
A decorator function that wraps the target function with retry logic.
Usage Example
-------------
```python
@with_retry(max_attempts=3, base_delay=2.0)
def place_my_order_decorated(qty):
# Simulated function that might fail
if random.random() < 0.8: # 80% chance of failure
raise ConnectionError("Failed to connect to exchange")
return {"status": "Success", "qty": qty}
try:
result = place_my_order_decorated(1.0)
print(f"Decorated order successful: {result}")
except RuntimeError as e:
print(f"Decorated order failed: {e}")
```
"""
def decorator(fn: Callable) -> Callable:
@functools.wraps(fn)
def wrapper(*args, **kwargs):
# Initialize RetryConfig with parameters provided to the decorator
config = RetryConfig(max_attempts=max_attempts, base_delay_s=base_delay)
# Iterate through the defined number of retry attempts
for attempt in range(config.max_attempts):
try:
# Attempt to execute the wrapped function
return fn(*args, **kwargs)
except Exception as e:
# Classify the exception to determine if it's retryable
error_class = classify_exchange_error(e)
if error_class == "fatal":
# If fatal, re-raise immediately without retry
logger.error(f"[{fn.__name__}] Fatal error encountered: {e}. Aborting.")
raise
# If it's the last attempt, and still an error, raise RuntimeError
if attempt < config.max_attempts - 1:
# Compute backoff delay for the next attempt
delay = compute_backoff_delay(attempt, config)
logger.warning(f"[{fn.__name__}] Attempt {attempt+1} failed: {e}. "
f"Retrying in {delay:.1f}s.")
time.sleep(delay)
else:
# All attempts exhausted, raise a RuntimeError
logger.critical(f"[{fn.__name__}] Failed after {config.max_attempts} attempts.")
raise RuntimeError(f"{fn.__name__} failed after {config.max_attempts} attempts.") from e
return wrapper
return decorator
logger.info("Initiating test of decorator-based retry mechanism.")
# ── Example usage of the decorator for demonstration ───────────────────────
@with_retry(max_attempts=3, base_delay=1.0)
def example_order_function(quantity: float) -> dict:
"""Simulated order function that raises an error on the first two invocations."""
# Use a persistent attribute to track call count across invocations
if not hasattr(example_order_function, "_call_count"):
example_order_function._call_count = 0
example_order_function._call_count += 1
logger.info(f"Simulated order function call count: {example_order_function._call_count}")
if example_order_function._call_count < 3:
# Simulate a transient network error for the first two calls
raise ConnectionError("Simulated network timeout")
# On the third call, simulate success
return {"status": "filled", "price": 50_000, "qty": quantity}
try:
result = example_order_function(1.0)
logger.info(f"Result from decorated function: {result}")
except RuntimeError as e:
logger.error(f"Decorated function failed as expected: {e}")
logger.info("Decorator-based retry mechanism (`with_retry`) defined and demonstrated.")WARNING:retry_engine:[example_order_function] Attempt 1 failed: Simulated network timeout. Retrying in 0.9s. WARNING:retry_engine:[example_order_function] Attempt 2 failed: Simulated network timeout. Retrying in 2.3s.
7. Backoff Delay Visualization
This section provides a visual representation of the exponential backoff delay, including the effect of jitter and the maximum delay cap. This visualization aids in understanding the non-deterministic nature and bounds of retry intervals.
import matplotlib.pyplot as plt # For plotting and visualization
import numpy as np # For numerical operations, especially mean and percentiles
# Instantiate RetryConfig with specific parameters for visualization purposes
config = RetryConfig(base_delay_s=1.0, max_delay_s=30.0, backoff_multiplier=2.0, jitter_fraction=0.2)
# Define the range of attempt numbers to visualize
attempts = list(range(10))
# Generate multiple delay samples for each attempt to demonstrate the effect of jitter.
# For each attempt, 100 delay values are computed to capture the random variation.
samples = [[compute_backoff_delay(a, config) for _ in range(100)] for a in attempts]
# Calculate the mean delay for each attempt
means = [np.mean(s) for s in samples]
# Calculate the 10th percentile of delays for each attempt to show lower bound of jitter
lower = [np.percentile(s, 10) for s in samples]
# Calculate the 90th percentile of delays for each attempt to show upper bound of jitter
upper = [np.percentile(s, 90) for s in samples]
# Create a figure and an axes object for the plot
fig, ax = plt.subplots(figsize=(10, 5))
# Plot the mean delay across attempts
ax.plot(attempts, means, marker="o", color="#e74c3c", linewidth=2, label="Mean delay")
# Fill the area between the 10th and 90th percentiles to visualize jitter range
ax.fill_between(attempts, lower, upper, alpha=0.3, color="#e74c3c", label="10–90th percentile")
# Draw a horizontal line to indicate the maximum delay cap
ax.axhline(config.max_delay_s, linestyle="--", color="#333", label=f"Cap ({config.max_delay_s}s)")
# Set plot labels and title for clarity
ax.set_xlabel("Attempt Number")
ax.set_ylabel("Delay (seconds)")
ax.set_title("Exponential Backoff Delay with Jitter and Max Cap")
ax.legend() # Display the legend
ax.grid(True, alpha=0.3) # Add a subtle grid for readability
plt.tight_layout() # Adjust plot to ensure all elements fit without overlap
# Save the figure to a file (optional)
# plt.savefig("/home/claude/retry_backoff.png", dpi=150, bbox_inches="tight")
plt.show() # Display the generated plot
logger.info("Backoff delay visualization generated and displayed.")8. Integration Pattern
This section illustrates a practical integration of the retry_order function into an exchange client's order placement method. This example demonstrates how to adapt an existing function (self.client.futures_create_order) to leverage the robust retry capabilities provided by the framework.
# ── Example Integration into a Hypothetical BinanceUtils.market_order Method ──
from binance.client import Client # Assuming this import is present in the actual BinanceUtils module
from binance.enums import * # Assuming necessary Binance enums like ORDER_TYPE_MARKET
class BinanceUtils:
def __init__(self, api_key, api_secret, ticker):
self.client = Client(api_key, api_secret)
self.ticker = ticker
def market_order_with_retry(self, quantity, direction, order_for):
"""
Places a market order with integrated retry logic.
Parameters
----------
quantity : float
The quantity of the asset to trade.
direction : str
The order side (e.g., 'BUY', 'SELL').
order_for : str
Contextual information for the order.
Returns
-------
str
The order ID upon successful placement.
"""
# Configure specific retry parameters for this market order context
config = RetryConfig(
max_attempts = 5, # Allow up to 5 attempts
base_delay_s = 2.0, # Start with a 2-second delay
qty_reduction_pct = 10.0, # Reduce quantity by 10% on balance errors
)
# Define the core order placement function to be wrapped by retry_order.
# This function encapsulates the actual exchange API call.
def order_fn(qty_to_place: float):
response = self.client.futures_create_order(
symbol=self.ticker,
side=direction,
type=ORDER_TYPE_MARKET,
quantity=qty_to_place,
)
# The order_fn should return a meaningful success indicator, e.g., order ID.
return response["orderId"]
try:
# Execute the order placement with retry logic
order_id = retry_order(order_fn, quantity, config)
logger.info(f"Market order placed successfully: Order ID {order_id}")
return order_id
except (RuntimeError, FatalError) as e:
logger.error(f"Market order failed after all retries: {e}")
raise # Re-raise the exception to indicate overall failure
# Example of how to use this within a client context (conceptual)
# api_key = "YOUR_BINANCE_API_KEY"
# api_secret = "YOUR_BINANCE_API_SECRET"
# binance_client = BinanceUtils(api_key, api_secret, ticker="BTCUSDT")
# try:
# new_order_id = binance_client.market_order_with_retry(quantity=0.001, direction='BUY', order_for='test_trade')
# print(f"Successfully placed order with ID: {new_order_id}")
# except Exception as e:
# print(f"Failed to place order: {e}")
logger.info("Integration pattern for retry_order demonstrated.")
9. Limit Order Price Adjustment Strategy
This section introduces a strategy to dynamically adjust the price of a limit order, or convert it to a market order, if the current market price deviates significantly from the initial intended limit price. This addresses scenarios where a limit order might not fill due to market movement and provides a mechanism to ensure order execution by adapting to current market conditions.
def place_adjusted_order(
exchange_order_fn: Callable[[float, float], Any],
intended_price: float,
current_market_price: float,
quantity: float,
order_side: str, # e.g., 'BUY', 'SELL'
price_tolerance_pct: float = 0.05, # 0.05% tolerance
price_adjustment_factor: float = 0.01, # 0.01% adjustment for limit orders
fallback_to_market: bool = True,
config: Optional[RetryConfig] = None
) -> Any:
"""
Places an order with dynamic price adjustment or fallback to a market order
if the market price deviates from the intended limit price.
This function assesses the difference between the intended limit price and the current market price.
If the deviation exceeds a specified tolerance, the order price is adjusted, or the order type
is converted to a market order to facilitate execution.
Parameters
----------
exchange_order_fn : Callable[[float, float], Any]
A function that accepts `quantity` and `price` (or `None` for market orders)
and attempts to place an order. Expected to return order details or raise an exception.
intended_price : float
The initial price at which the order was intended to be placed (limit price).
current_market_price : float
The current relevant market price (e.g., best ask for buy, best bid for sell).
quantity : float
The quantity of the asset to trade.
order_side : str
The side of the order, typically 'BUY' or 'SELL'.
price_tolerance_pct : float, optional
The percentage deviation allowed between `intended_price` and `current_market_price`
before triggering an adjustment. Default is 0.05 (0.05%).
price_adjustment_factor : float, optional
The percentage by which the `intended_price` is adjusted if it's out of tolerance
but still to be placed as a limit order. Default is 0.01 (0.01%).
fallback_to_market : bool, optional
If True, the order will be placed as a market order if the price deviation
is too large and `price_adjustment_factor` is not applied. Default is True.
config : Optional[RetryConfig], optional
An instance of `RetryConfig` for the `retry_order` mechanism. Uses default if None.
Returns
-------
Any
The result returned by the `exchange_order_fn` upon successful order placement.
Raises
------
RuntimeError
If the order fails after all retries or if the circuit breaker is open.
FatalError
If a non-retryable error occurs during order submission.
"""
config = config or RetryConfig()
adjusted_price = intended_price
order_type_str = "LIMIT"
# Calculate the percentage deviation of the market price from the intended limit price.
price_deviation_pct = abs((current_market_price - intended_price) / intended_price) * 100
if price_deviation_pct > price_tolerance_pct:
logger.warning(f"Market price ({current_market_price}) deviates by {price_deviation_pct:.4f}% "
f"from intended limit price ({intended_price}).")
if fallback_to_market:
# If deviation is too large and fallback is enabled, switch to a market order.
adjusted_price = None # Indicate a market order
order_type_str = "MARKET"
logger.info("Falling back to market order for guaranteed execution.")
else:
# Adjust limit price based on order side.
adjustment = intended_price * (price_adjustment_factor / 100)
if order_side == 'BUY':
# For buy orders, increase the limit price slightly to improve fill probability.
adjusted_price = current_market_price + adjustment
logger.info(f"Adjusting BUY limit price to {adjusted_price:.6f}.")
elif order_side == 'SELL':
# For sell orders, decrease the limit price slightly to improve fill probability.
adjusted_price = current_market_price - adjustment
logger.info(f"Adjusting SELL limit price to {adjusted_price:.6f}.")
else:
logger.error("Invalid order_side specified.")
raise ValueError("Invalid order_side. Must be 'BUY' or 'SELL'.")
# Define the order function for retry_order, encapsulating the exchange call with potentially adjusted price.
def order_placement_func(qty: float):
# For market orders (adjusted_price is None), the exchange_order_fn should handle it accordingly.
# This assumes exchange_order_fn can accept None for price for market orders,
# or the caller needs to provide a wrapper that correctly handles `adjusted_price`.
if adjusted_price is None:
# Call the exchange function without a price for market orders
return exchange_order_fn(qty, None)
else:
# Call the exchange function with the adjusted limit price
return exchange_order_fn(qty, adjusted_price)
try:
# Execute the order placement function with retry logic
order_result = retry_order(order_placement_func, quantity, config=config)
# Conditionally format the price for logging, handling None for market orders
price_log_str = 'N/A' if adjusted_price is None else f"{adjusted_price:.6f}"
logger.info(f"Order placed successfully: Type={order_type_str}, Price={price_log_str}, Qty={quantity}")
return order_result
except (RuntimeError, FatalError) as e:
logger.error(f"Adjusted order placement failed: {e}")
raise
logger.info("Limit order price adjustment strategy (`place_adjusted_order`) defined.")10. Dummy Data Generation and Strategy Demonstration
This section simulates an exchange environment and demonstrates the place_adjusted_order function under various market conditions. It includes:
- A Simulated Exchange Order Placement Function: This function mimics the behavior of a real exchange API, including potential failures (network, balance, fatal) and successful order placement.
- Demonstration Scenarios: Practical examples showcasing how
place_adjusted_orderhandles different price deviations, adjusts limit prices, or falls back to market orders, leveraging the retry framework.
class MockExchange:
"""
A mock exchange client to simulate order placement and market data retrieval.
This class is now enhanced to allow for deterministic error sequences for testing
the retry and price adjustment mechanisms without relying on pure randomness.
"""
def __init__(self, ticker: str = "BTCUSDT", error_sequence: Optional[list] = None):
self.ticker = ticker
self.successful_orders = []
self._call_count = 0
# Predefined sequence of errors to simulate. Each element is an error type string.
# 'none': no error, 'fatal': FatalError, 'balance': BalanceError, 'retryable': RetryableError (ConnectionError)
self._error_sequence = error_sequence if error_sequence is not None else []
logger.info("MockExchange initialized.")
def get_current_market_price(self) -> float:
"""
Simulates fetching the current market price for the configured ticker.
Returns a slightly randomized price around a base value.
"""
base_price = 50000.0
fluctuation = random.uniform(-0.0005, 0.0005) * base_price # +/- 0.05%
return base_price * (1 + fluctuation)
def simulate_exchange_order_placement(self, quantity: float, price: Optional[float] = None) -> dict:
"""
Simulates an order placement on the exchange, using a predefined error sequence
or falling back to a default probabilistic model if the sequence is exhausted.
Parameters
----------
quantity : float
The quantity of the asset to trade.
price : Optional[float]
The limit price for the order. If None, it's considered a market order.
Returns
-------
dict
A dictionary representing the order confirmation if successful.
Raises
------
ConnectionError
Simulates a network-related or transient error.
ValueError
Simulates an insufficient balance or invalid parameter error.
"""
self._call_count += 1
log_prefix = f"Mock Exchange (call {self._call_count}):"
logger.info(f"{log_prefix} Attempting to place order - Qty: {quantity}, Price: {price if price else 'MARKET'}")
# Use predefined error sequence if available
if self._call_count - 1 < len(self._error_sequence):
sim_error = self._error_sequence[self._call_count - 1]
if sim_error == 'fatal':
logger.warning(f"{log_prefix} Simulating FATAL error.")
raise ValueError("Invalid SYMBOL detected") # Example fatal error
elif sim_error == 'balance':
logger.warning(f"{log_prefix} Simulating BALANCE error.")
raise ValueError("Insufficient balance for this order")
elif sim_error == 'retryable':
logger.warning(f"{log_prefix} Simulating RETRYABLE error (Network timeout).")
raise ConnectionError("Network timeout during order submission")
# 'none' will proceed to successful placement
else:
# Fallback to probabilistic model if sequence exhausted (or empty)
rand_val = random.random()
if rand_val < 0.01: # 1% chance of a FATAL error
logger.warning(f"{log_prefix} Probabilistically Simulating FATAL error.")
if random.random() < 0.5: raise ValueError("Invalid API-KEY")
else: raise ValueError("Invalid SYMBOL detected")
elif rand_val < 0.06: # 5% chance of a BALANCE error
logger.warning(f"{log_prefix} Probabilistically Simulating BALANCE error.")
raise ValueError("Insufficient balance for this order")
elif rand_val < 0.40: # 34% chance of a RETRYABLE error
logger.warning(f"{log_prefix} Probabilistically Simulating RETRYABLE error.")
if random.random() < 0.5: raise ConnectionError("Exchange rate limit exceeded (HTTP 429)")
else: raise ConnectionError("Network timeout during order submission")
# Successful order placement
order_id = f"MOCK_ORDER_{len(self.successful_orders) + 1}_{int(time.time() * 1000)}"
order_result = {
"orderId": order_id,
"status": "FILLED",
"price": price if price is not None else self.get_current_market_price(),
"executedQty": quantity
}
self.successful_orders.append(order_result)
logger.info(f"{log_prefix} Order {order_id} placed successfully.")
return order_result
# NOTE: Initializing a default mock_exchange for general use if not explicitly defined in scenarios.
# However, for demonstrations, each scenario will create its own instance with a specific error_sequence.
mock_exchange = MockExchange()
logger.info("Mock exchange client for dummy data generation has been initialized.")Scenario 1: Order with No Significant Price Deviation (Successful Limit Order)
This scenario demonstrates the placement of a limit order where the market price is within the acceptable price_tolerance_pct of the intended_price. The order is expected to be placed as a limit order at the intended_price.
# Define parameters for the first scenario
intended_price_scenario1 = 50000.0
current_market_price_scenario1 = intended_price_scenario1 * 1.00002 # 0.002% deviation
quantity_scenario1 = 0.01
order_side_scenario1 = 'BUY'
logger.info("--- SCENARIO 1: No Significant Price Deviation (Limit Order, Retryable/Balance errors) ---")
# Simulate 2 balance errors, then 1 retryable error, then success
mock_exchange_s1 = MockExchange(error_sequence=['balance', 'balance', 'retryable', 'none'])
try:
result1 = place_adjusted_order(
exchange_order_fn=mock_exchange_s1.simulate_exchange_order_placement,
intended_price=intended_price_scenario1,
current_market_price=current_market_price_scenario1,
quantity=quantity_scenario1,
order_side=order_side_scenario1,
price_tolerance_pct=0.05,
fallback_to_market=True
)
logger.info(f"Scenario 1 Result: {result1}")
except (RuntimeError, FatalError) as e:
logger.error(f"Scenario 1 Failed: {e}")
logger.info("Scenario 1 demonstration completed.")WARNING:retry_engine:Mock Exchange (call 1): Simulating BALANCE error. WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: balance | Exception: Insufficient balance for this order WARNING:retry_engine:[QTY REDUCE] Insufficient balance. New quantity: 0.009000 WARNING:retry_engine:Mock Exchange (call 2): Simulating BALANCE error. WARNING:retry_engine:[ATTEMPT 2/5] Error classified as: balance | Exception: Insufficient balance for this order WARNING:retry_engine:[QTY REDUCE] Insufficient balance. New quantity: 0.008100 WARNING:retry_engine:Mock Exchange (call 3): Simulating RETRYABLE error (Network timeout). WARNING:retry_engine:[ATTEMPT 3/5] Error classified as: retryable | Exception: Network timeout during order submission
Scenario 2: Order with Slight Price Deviation (Adjusted Limit Order, No Fallback)
This scenario simulates a situation where the market price deviates slightly beyond the price_tolerance_pct from the intended_price. With fallback_to_market set to False, the system will adjust the limit order's price slightly to improve its chances of execution, rather than converting it to a market order.
# Define parameters for the second scenario
intended_price_scenario2 = 50000.0
current_market_price_scenario2 = intended_price_scenario2 * 1.001 # 0.1% deviation
quantity_scenario2 = 0.02
order_side_scenario2 = 'BUY'
logger.info("--- SCENARIO 2: Slight Price Deviation (Adjusted Limit Order, No Fallback) ---")
# Simulate 1 retryable error, then success
mock_exchange_s2 = MockExchange(error_sequence=['retryable', 'none'])
try:
result2 = place_adjusted_order(
exchange_order_fn=mock_exchange_s2.simulate_exchange_order_placement,
intended_price=intended_price_scenario2,
current_market_price=current_market_price_scenario2,
quantity=quantity_scenario2,
order_side=order_side_scenario2,
price_tolerance_pct=0.05, # Tolerance exceeded (0.1% > 0.05%)
price_adjustment_factor=0.02,
fallback_to_market=False # Crucially, do NOT fall back to market order
)
logger.info(f"Scenario 2 Result: {result2}")
except (RuntimeError, FatalError) as e:
logger.error(f"Scenario 2 Failed: {e}")
logger.info("Scenario 2 demonstration completed.")WARNING:retry_engine:Market price (50049.99999999999) deviates by 0.1000% from intended limit price (50000.0). WARNING:retry_engine:Mock Exchange (call 1): Simulating RETRYABLE error (Network timeout). WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Network timeout during order submission
Scenario 3: Order with Significant Price Deviation (Fallback to Market Order)
This scenario demonstrates a significant divergence between the intended_price and the current_market_price, exceeding the price_tolerance_pct. With fallback_to_market enabled, the place_adjusted_order function converts the order into a market order to ensure execution, prioritizing fill over a specific limit price.
# Define parameters for the third scenario
intended_price_scenario3 = 50000.0
current_market_price_scenario3 = intended_price_scenario3 * 1.005 # 0.5% deviation
quantity_scenario3 = 0.03
order_side_scenario3 = 'SELL'
logger.info("--- SCENARIO 3: Significant Price Deviation (Fallback to Market Order) ---")
# Simulate 1 retryable error, then success
mock_exchange_s3 = MockExchange(error_sequence=['retryable', 'none'])
try:
result3 = place_adjusted_order(
exchange_order_fn=mock_exchange_s3.simulate_exchange_order_placement,
intended_price=intended_price_scenario3,
current_market_price=current_market_price_scenario3,
quantity=quantity_scenario3,
order_side=order_side_scenario3,
price_tolerance_pct=0.1, # Tolerance exceeded (0.5% > 0.1%)
fallback_to_market=True # Crucially, enable fallback to market order
)
logger.info(f"Scenario 3 Result: {result3}")
except (RuntimeError, FatalError) as e:
logger.error(f"Scenario 3 Failed: {e}")
logger.info("Scenario 3 demonstration completed.")WARNING:retry_engine:Market price (50249.99999999999) deviates by 0.5000% from intended limit price (50000.0). WARNING:retry_engine:Mock Exchange (call 1): Simulating RETRYABLE error (Network timeout). WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Network timeout during order submission
Scenario 4: Error Handling within place_adjusted_order
This scenario demonstrates how the place_adjusted_order function, leveraging the underlying retry_order mechanism, handles transient failures during the actual order submission attempt. It will simulate a series of failures before a successful order or exhaustion of retries.
class MockExchangeReliable:
"""
A mock exchange client that guarantees success after a few attempts.
Used to specifically test the retry logic within `place_adjusted_order`.
"""
def __init__(self, fail_count: int = 2):
self.call_count = 0
self.fail_count = fail_count
logger.info(f"MockExchangeReliable initialized to fail {self.fail_count} times.")
def get_current_market_price(self) -> float:
return 50000.0
def simulate_order_placement_with_transient_failure(self, quantity: float, price: Optional[float] = None) -> dict:
self.call_count += 1
if self.call_count <= self.fail_count:
logger.warning(f"Mock Exchange Reliable: Simulating transient error (call {self.call_count}).")
raise ConnectionError("Simulated network hiccup")
else:
order_id = f"RELIABLE_ORDER_{int(time.time() * 1000)}"
order_result = {
"orderId": order_id,
"status": "FILLED",
"price": price if price is not None else self.get_current_market_price(),
"executedQty": quantity
}
logger.info(f"Mock Exchange Reliable: Order {order_id} placed successfully after {self.call_count} attempts.")
return order_result
# Instantiate a reliable mock exchange that fails a few times initially
mock_exchange_reliable = MockExchangeReliable(fail_count=2)
# Define parameters for the fourth scenario
intended_price_scenario4 = 49990.0
current_market_price_scenario4 = 49990.0 # No price deviation for simplicity in this test
quantity_scenario4 = 0.05
order_side_scenario4 = 'BUY'
logger.info("--- SCENARIO 4: Error Handling within place_adjusted_order (Always Succeeds) ---")
try:
# Use a custom RetryConfig to limit attempts for a quicker test
custom_config_scenario4 = RetryConfig(max_attempts=5, base_delay_s=0.1, jitter_fraction=0.1)
result4 = place_adjusted_order(
exchange_order_fn=mock_exchange_reliable.simulate_order_placement_with_transient_failure,
intended_price=intended_price_scenario4,
current_market_price=current_market_price_scenario4,
quantity=quantity_scenario4,
order_side=order_side_scenario4,
price_tolerance_pct=0.01,
fallback_to_market=False,
config=custom_config_scenario4
)
logger.info(f"Scenario 4 Result: {result4}")
except (RuntimeError, FatalError) as e:
logger.error(f"Scenario 4 Failed: {e}")
logger.info("Scenario 4 demonstration completed.")WARNING:retry_engine:Mock Exchange Reliable: Simulating transient error (call 1). WARNING:retry_engine:[ATTEMPT 1/5] Error classified as: retryable | Exception: Simulated network hiccup WARNING:retry_engine:Mock Exchange Reliable: Simulating transient error (call 2). WARNING:retry_engine:[ATTEMPT 2/5] Error classified as: retryable | Exception: Simulated network hiccup