Backtesting·Backtest Realism·Intermediate

Partial Fill Simulation

Simulate partial order fills in backtests by probabilistically modeling the extent of execution based on limit order queue position, available opposing liquidity, and the statistical distribution of partial fill outcomes observed in real markets.

backtest-realismbacktestingsimulation

Understanding Partial Fill Simulation

Introduction: What is a Partial Fill Simulation?

In financial markets, an order placed to buy or sell an asset might not be executed in its entirety at a single price or even at all. This phenomenon is known as a partial fill. A partial fill occurs when only a portion of the total order quantity is matched with available liquidity in the market.

Partial Fill Simulation is the process of modeling and predicting how a large order would be executed against a given order book, accounting for the possibility of only a fraction of the order being filled at various price levels. It helps traders, quantitative analysts, and system designers understand the potential impact of their orders on the market, estimate execution costs (like slippage), and optimize their trading strategies.

Purpose and Importance

  1. Slippage Estimation: Predict the average price at which a large order will be filled, considering the available liquidity at different price levels. This helps in estimating the slippage, which is the difference between the expected price and the actual execution price.
  2. Order Strategy Optimization: Inform decisions on how to break down large orders into smaller ones (e.g., using Limit Orders, Market Orders, or more complex algorithms like VWAP/TWAP) to minimize market impact and achieve better execution prices.
  3. Risk Management: Understand the risk of not getting a full fill, especially in illiquid markets, and how this might affect overall portfolio exposure.
  4. Backtesting: Evaluate the performance of historical trading strategies under realistic execution conditions.
  5. Market Impact Analysis: Assess how a specific order size might consume liquidity and move the market price.

This notebook will guide you through creating a simple partial fill simulation model, generating mock order book data, and visualizing the simulation results.

Core Concepts for Partial Fill Simulation

To simulate partial fills, we need to understand the fundamental components of an order book and how orders interact with it.

1. Order Book

The order book is a real-time list of outstanding buy and sell orders for a particular asset, organized by price level. It typically consists of two sides:

  • Bid Side (Buy Orders): Orders from buyers indicating the maximum price they are willing to pay for a certain quantity.
  • Ask Side (Sell Orders): Orders from sellers indicating the minimum price they are willing to accept for a certain quantity.

Each entry in the order book contains a price and a quantity at that price.

2. Market Order vs. Limit Order

While limit orders specify a price, partial fill simulations are typically most relevant for Market Orders (or large limit orders that sweep through multiple price levels) as they consume available liquidity from the best available prices upwards/downwards until filled or until liquidity runs out.

3. Key Simulation Parameters

  • Total Order Quantity: The total amount of the asset the trader wishes to buy or sell.
  • Order Book Snapshot: The current state of the order book (prices and quantities available).
  • Side of Order: Whether the order is a buy (consumes ask-side liquidity) or a sell (consumes bid-side liquidity).

4. Output Metrics

  • Filled Quantity: The total quantity of the order that was successfully executed.
  • Remaining Quantity: The portion of the order that could not be filled with the available liquidity.
  • Average Execution Price: The volume-weighted average price (VWAP) at which the filled quantity was executed.
  • Slippage: The difference between the initial best available price and the average execution price.

Simulation Function: partial_fill_simulator

Let's implement a Python function to simulate the partial filling of a market order against a given order book. This function will iterate through the relevant side of the order book, consuming liquidity until the order is fully filled or the order book runs out of relevant liquidity.

[1]
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker

def partial_fill_simulator(
    order_quantity: float,
    order_book_side: pd.DataFrame,
    order_type: str
) -> dict:
    """
    Simulates the partial filling of a market order against a given order book side.

    This function processes an order against the provided order book, calculating
    how much is filled, what remains, and the average execution price.

    Args:
        order_quantity (float): The total quantity of the asset to buy or sell.
        order_book_side (pd.DataFrame): A DataFrame representing one side of the
                                         order book (e.g., 'asks' for a buy order).
                                         It must have 'price' and 'quantity' columns.
                                         Prices should be sorted appropriately (ascending for asks,
                                         descending for bids).
        order_type (str): The type of order, 'buy' or 'sell'. This determines
                          which side of the order book is consumed.

    Returns:
        dict: A dictionary containing:
            'filled_quantity': The total quantity filled.
            'remaining_quantity': The quantity of the order that could not be filled.
            'average_execution_price': The volume-weighted average price of the filled quantity.
            'filled_details': A list of dictionaries, each containing 'price', 'quantity_filled'
                              for each level where the order was partially filled.
            'initial_best_price': The best price available at the start of the simulation.
    """
    if order_book_side.empty:
        return {
            'filled_quantity': 0.0,
            'remaining_quantity': order_quantity,
            'average_execution_price': None,
            'filled_details': [],
            'initial_best_price': None
        }

    filled_quantity = 0.0
    total_cost_or_revenue = 0.0
    filled_details = []
    initial_best_price = order_book_side['price'].iloc[0]

    remaining_order_quantity = order_quantity

    for _, row in order_book_side.iterrows():
        price = row['price']
        available_quantity = row['quantity']

        if remaining_order_quantity <= 0:
            break

        quantity_to_fill = min(remaining_order_quantity, available_quantity)

        filled_quantity += quantity_to_fill
        total_cost_or_revenue += quantity_to_fill * price
        remaining_order_quantity -= quantity_to_fill

        filled_details.append({
            'price': price,
            'quantity_filled': quantity_to_fill
        })

    average_execution_price = total_cost_or_revenue / filled_quantity if filled_quantity > 0 else None

    return {
        'filled_quantity': filled_quantity,
        'remaining_quantity': remaining_order_quantity,
        'average_execution_price': average_execution_price,
        'filled_details': filled_details,
        'initial_best_price': initial_best_price
    }

print("Partial fill simulator function defined.")
Partial fill simulator function defined.

Generating Mock Order Book Data

To demonstrate the simulator, we need a realistic-looking order book. We'll generate a mock 'ask' side of an order book, which a 'buy' market order would consume. The prices will increase as we move away from the best ask, and quantities will vary.

[2]
# Seed for reproducibility
np.random.seed(42)

# Define parameters for the mock order book
best_ask_price = 100.00
num_price_levels = 20
price_increment = 0.01 # Smallest tick size

# Generate ask prices (increasing from best ask)
ask_prices = np.linspace(best_ask_price, best_ask_price + (num_price_levels - 1) * price_increment, num_price_levels)

# Generate quantities for each ask price level
# Quantities can be somewhat random, but often decrease further from the best price
ask_quantities = np.random.randint(50, 500, size=num_price_levels)
# Let's make the quantities generally larger closer to the best price, then drop off
ask_quantities = ask_quantities * (1 + np.exp(-(np.arange(num_price_levels) / 5)))
ask_quantities = np.round(ask_quantities, 0)

mock_asks_df = pd.DataFrame({
    'price': ask_prices,
    'quantity': ask_quantities
})

print("Mock Ask Order Book (first 5 levels):")
display(mock_asks_df.head())

print("\nMock Ask Order Book (last 5 levels):")
display(mock_asks_df.tail())
Mock Ask Order Book (first 5 levels):
price quantity
0 100.00 304.0
1 100.01 882.0
2 100.02 665.0
3 100.03 496.0
4 100.04 226.0

Mock Ask Order Book (last 5 levels):
price quantity
15 100.15 429.0
16 100.16 209.0
17 100.17 186.0
18 100.18 204.0
19 100.19 366.0

Demonstration of Partial Fill Simulation

Now, let's use our simulator with a sample buy order and interpret the results.

[3]
buy_order_quantity = 2500  # A relatively large buy order

print(f"Simulating a BUY order of {buy_order_quantity} units.\n")

simulation_results = partial_fill_simulator(
    order_quantity=buy_order_quantity,
    order_book_side=mock_asks_df,
    order_type='buy'
)

print("--- Simulation Results ---")
print(f"Total Order Quantity: {buy_order_quantity}")
print(f"Filled Quantity: {simulation_results['filled_quantity']:.2f}")
print(f"Remaining Quantity: {simulation_results['remaining_quantity']:.2f}")
if simulation_results['average_execution_price'] is not None:
    print(f"Average Execution Price: {simulation_results['average_execution_price']:.4f}")
    print(f"Initial Best Ask Price: {simulation_results['initial_best_price']:.4f}")
    slippage = simulation_results['average_execution_price'] - simulation_results['initial_best_price']
    print(f"Slippage: {slippage:.4f}")
else:
    print("Could not calculate average execution price (no fill).")

print("\n--- Filled Details ---")
filled_df = pd.DataFrame(simulation_results['filled_details'])
display(filled_df)
Simulating a BUY order of 2500 units.

--- Simulation Results ---
Total Order Quantity: 2500
Filled Quantity: 2500.00
Remaining Quantity: 0.00
Average Execution Price: 100.0172
Initial Best Ask Price: 100.0000
Slippage: 0.0172

--- Filled Details ---
price quantity_filled
0 100.00 304.0
1 100.01 882.0
2 100.02 665.0
3 100.03 496.0
4 100.04 153.0

Interpretation of Results

From the simulation above, we can observe:

  • The Filled Quantity indicates how much of our Total Order Quantity was successfully executed against the available ask-side liquidity.
  • The Remaining Quantity shows the portion of the order that could not be filled, suggesting that the order was larger than the available liquidity in our mock order book.
  • The Average Execution Price is the volume-weighted average of all the prices at which parts of the order were filled. Since we're consuming liquidity upwards through the ask side, this price is typically higher than the Initial Best Ask Price.
  • Slippage quantifies this difference, representing the additional cost incurred due to the order's size pushing the execution price away from the initial best price.
  • The Filled Details table provides a granular view, showing exactly how much quantity was filled at each price level as the order swept through the order book.

Visualizations

Visualizations help in understanding the dynamics of order execution and the impact of large orders.

Visualization 1: Order Book Liquidity vs. Filled Quantity

This chart visualizes the total available quantity at each price level in the order book and highlights how much of that liquidity was consumed by our simulated order. This helps to clearly see which price levels were partially or fully consumed.

[4]
fig, ax = plt.subplots(figsize=(12, 7))

# Prepare data for plotting
plot_df = mock_asks_df.copy()
plot_df['quantity_filled'] = 0.0

for detail in simulation_results['filled_details']:
    price = detail['price']
    qty_filled = detail['quantity_filled']
    plot_df.loc[plot_df['price'] == price, 'quantity_filled'] = qty_filled

plot_df['quantity_remaining_at_level'] = plot_df['quantity'] - plot_df['quantity_filled']

# Create stacked bar chart
bar_width = price_increment * 0.8 # Make bars slightly narrower than price increment

ax.bar(plot_df['price'], plot_df['quantity_filled'], label='Quantity Filled', color='skyblue', width=bar_width)
ax.bar(plot_df['price'], plot_df['quantity_remaining_at_level'], bottom=plot_df['quantity_filled'], label='Remaining Liquidity at Level', color='lightgray', width=bar_width)

# Add a line for the average execution price
if simulation_results['average_execution_price'] is not None:
    ax.axvline(x=simulation_results['average_execution_price'], color='red', linestyle='--',
               label=f"Avg. Exec. Price: {simulation_results['average_execution_price']:.2f}")
    ax.text(simulation_results['average_execution_price'] + 0.005, ax.get_ylim()[1] * 0.9,
            'Avg. Exec. Price',
            color='red', rotation=90, va='top')

# Add a line for the initial best price
ax.axvline(x=simulation_results['initial_best_price'], color='green', linestyle='--',
           label=f"Initial Best Ask: {simulation_results['initial_best_price']:.2f}")
ax.text(simulation_results['initial_best_price'] - 0.005, ax.get_ylim()[1] * 0.9,
        'Initial Best Ask',
        color='green', rotation=90, va='top', ha='right')

ax.set_title(f'Order Book Liquidity Consumption for Buy Order of {buy_order_quantity} units')
ax.set_xlabel('Price')
ax.set_ylabel('Quantity')
ax.legend()
ax.grid(axis='y', linestyle='--', alpha=0.7)

# Format x-axis to show more price precision
formatter = mticker.FormatStrFormatter('$%.2f')
ax.xaxis.set_major_formatter(formatter)
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
cell output

Interpretation of Visualization 1

The stacked bar chart visually represents the interaction of our buy_order_quantity with the mock ask-side order book:

  • Light Blue Bars: Show the quantity filled at each price level. You can see how the order progressively consumes liquidity at increasing prices.
  • Gray Bars (on top of blue): Represent the liquidity that remained at a given price level after our order was executed (or if the order didn't reach that level).
  • Red Dashed Line: Indicates the Average Execution Price, which is higher than the initial best ask price due to consuming liquidity at higher price levels.
  • Green Dashed Line: Marks the Initial Best Ask Price, the price at which the first portion of the order would have been filled.

This visualization clearly demonstrates how larger orders 'walk up' or 'walk down' the order book, consuming liquidity and leading to slippage as the average execution price deviates from the initial best available price.

Visualization 2: Slippage vs. Order Quantity

This plot shows how the average execution price (and thus slippage) changes as the order quantity increases. This is crucial for understanding the market impact of different order sizes.

[5]
order_quantities = np.arange(100, 5000, 200) # Test various order quantities
slippages = []
avg_prices = []

for qty in order_quantities:
    results = partial_fill_simulator(
        order_quantity=qty,
        order_book_side=mock_asks_df,
        order_type='buy'
    )
    if results['average_execution_price'] is not None:
        slippage = results['average_execution_price'] - results['initial_best_price']
        slippages.append(slippage)
        avg_prices.append(results['average_execution_price'])
    else:
        slippages.append(0.0) # Or NaN if preferred for unfilled orders
        avg_prices.append(mock_asks_df['price'].iloc[0]) # Initial best price if no fill

fig, ax1 = plt.subplots(figsize=(12, 7))

color = 'tab:blue'
ax1.set_xlabel('Order Quantity')
ax1.set_ylabel('Slippage (Price difference)', color=color)
ax1.plot(order_quantities, slippages, color=color, marker='o', linestyle='-')
ax1.tick_params(axis='y', labelcolor=color)
ax1.set_title('Slippage and Average Execution Price vs. Order Quantity')
ax1.grid(True, linestyle='--', alpha=0.6)

# Create a second y-axis for average execution price
ax2 = ax1.twinx()
color = 'tab:red'
ax2.set_ylabel('Average Execution Price', color=color)
ax2.plot(order_quantities, avg_prices, color=color, marker='x', linestyle='--')
ax2.tick_params(axis='y', labelcolor=color)

# Format y-axes to show currency
formatter = mticker.FormatStrFormatter('$%.4f')
ax1.yaxis.set_major_formatter(formatter)
ax2.yaxis.set_major_formatter(formatter)

# Add horizontal line for the initial best price
ax1.axhline(0, color='gray', linestyle=':', linewidth=0.8, label='No Slippage (Initial Best Price)')
ax2.axhline(mock_asks_df['price'].iloc[0], color='gray', linestyle=':', linewidth=0.8)

fig.tight_layout()  # otherwise the right y-label is slightly clipped
plt.show()
cell output

Interpretation of Visualization 2

This dual-axis plot illustrates the relationship between the Order Quantity and its execution characteristics:

  • Blue Line (Slippage): As the order quantity increases, the slippage (the difference between the average execution price and the initial best ask) generally increases. This is because larger orders consume more liquidity and must reach higher (for a buy order) or lower (for a sell order) price levels, leading to a worse average execution price.
  • Red Line (Average Execution Price): Correspondingly, the average execution price for a buy order tends to increase with larger order quantities, reflecting the consumption of more expensive liquidity further up the order book.
  • Gray Dashed Lines: Represent the baseline of no slippage (0 for the slippage axis) and the initial best price (for the average execution price axis). The divergence from these lines quantifies the market impact.

This visualization is critical for understanding the market depth and impact costs associated with different order sizes. It helps traders determine an optimal order size to balance execution speed and minimize slippage.

Conclusion

Partial fill simulation is a powerful tool for understanding and quantifying the execution risks and costs associated with trading larger order quantities in financial markets. By modeling how an order interacts with the available liquidity in an order book, we can:

  • Estimate Slippage: Accurately predict the average execution price and the resulting price deviation from the initial best price.
  • Optimize Order Sizing: Inform decisions on appropriate order sizes to mitigate market impact.
  • Develop Advanced Strategies: Lay the groundwork for more sophisticated order execution algorithms that dynamically adapt to market conditions.

This notebook provided a foundational understanding and a practical Python implementation of a partial fill simulator, along with visualizations to interpret its behavior. While this model is simplified, it captures the essential mechanics that underpin more complex real-world execution models.