Infrastructure·System Monitoring·Intermediate

Prometheus Metrics Setup

Instrument trading system code to expose critical operational metrics including real-time PnL, open position counts, order execution latency histograms, and error rate counters as Prometheus metric endpoints for centralized collection, alerting rule evaluation, and Grafana dashboard visualization.

infrastructuremonitoringperformance-metrics

Expose Metrics with Prometheus

This notebook provides a comprehensive guide on how to expose custom application metrics for Prometheus scraping using the prometheus_client library in Python. Prometheus is an open-source monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Exposing metrics is crucial for understanding the behavior and performance of your applications in real-time. By providing detailed insights into various aspects like request rates, error counts, and resource utilization, you can effectively monitor, troubleshoot, and optimize your systems.

Key Concepts:

Concept	Description
Prometheus	An open-source monitoring system that collects and stores time-series data.
Metrics	Numerical measurements representing some aspect of an application or system at a given time (e.g., request count, error rate).
Exporters	Applications or services that expose metrics in a format that Prometheus can scrape (pull).
Scraping	The process by which Prometheus pulls metrics from configured targets.
`prometheus_client`	A Python client library for Prometheus to instrument applications and expose metrics.
Metric Types	Different types of metrics available in Prometheus, such as Counters, Gauges, Histograms, and Summaries, each suited for specific use cases.
HTTP Server	A lightweight HTTP server is often used to serve the `/metrics` endpoint where Prometheus can scrape the exposed metrics.

Dependency Installation

First, we need to install the prometheus_client library, which allows us to expose Prometheus metrics from Python applications.

[1]

!pip install prometheus_client

Requirement already satisfied: prometheus_client in /usr/local/lib/python3.12/dist-packages (0.25.0)

Library Imports

This section imports all necessary libraries for the notebook. Standard Python libraries are imported first, followed by third-party libraries like prometheus_client.

[2]

import time
import random
import logging
from collections import deque
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
import os

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from prometheus_client import start_http_server, Gauge, Counter, Histogram, Summary, generate_latest

Core Functions

This section defines the core functions for initializing and managing Prometheus metrics. Each function is presented in its own block with a detailed markdown header, docstrings, type hints, and logging.

Function Name: `create_metrics_state`

This function initializes various Prometheus metrics (Counter, Gauge, Histogram, Summary) and sets up a logger. It returns a dictionary containing these initialized metrics and the logger, which will serve as the application's state for metric management.

Algorithm:

Initialize a logging.Logger instance.
Define a Counter for total requests.
Define a Gauge for current active requests.
Define a Histogram for request durations.
Define a Summary for response sizes.
Store these metrics and the logger in a dictionary.

Parameters: None

Returns: (dict): A dictionary containing initialized Prometheus metrics and the logger.

[3]

def create_metrics_state() -> dict:
    """
    Initializes Prometheus metrics and a logger for the application.

    This function sets up a Counter for total requests, a Gauge for current active requests,
    a Histogram for request durations, and a Summary for response sizes. It also configures
    a basic logger.

    Parameters
    ----------
    None

    Returns
    -------
    dict
        A dictionary containing the initialized metrics and the logger:
        - 'logger': Configured logging.Logger instance.
        - 'requests_total': Counter metric for total requests.
        - 'active_requests': Gauge metric for current active requests.
        - 'request_duration_seconds': Histogram metric for request durations.
        - 'response_size_bytes': Summary metric for response sizes.
    """
    # Configure logger
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.INFO)
    if not logger.handlers:
        handler = logging.StreamHandler()
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        handler.setFormatter(formatter)
        logger.addHandler(handler)

    logger.info("Initializing Prometheus metrics...")

    # Initialize Prometheus metrics
    requests_total = Counter(
        'app_requests_total',
        'Total number of requests received by the application',
        ['method', 'endpoint', 'status']
    )
    active_requests = Gauge(
        'app_active_requests',
        'Current number of active requests'
    )
    request_duration_seconds = Histogram(
        'app_request_duration_seconds',
        'Histogram of request durations',
        buckets=[0.1, 0.5, 1.0, 2.5, 5.0, 10.0, float('inf')]
    )
    response_size_bytes = Summary(
        'app_response_size_bytes',
        'Summary of response sizes in bytes'
    )

    state = {
        'logger': logger,
        'requests_total': requests_total,
        'active_requests': active_requests,
        'request_duration_seconds': request_duration_seconds,
        'response_size_bytes': response_size_bytes
    }

    logger.info("Prometheus metrics initialized successfully.")
    return state

Function Name: `update_metrics`

This function simulates a request processing cycle and updates the Prometheus metrics accordingly. It demonstrates how to use the Counter, Gauge, Histogram, and Summary metrics.

Algorithm:

Increment the total request counter with specified labels.
Increment the active requests gauge at the start of the 'request'.
Simulate work being done for a random duration.
Observe the request duration in the histogram.
Decrement the active requests gauge at the end of the 'request'.
Observe the response size in the summary.

Parameters: state (dict): The current application state containing the initialized Prometheus metrics and logger. method (str): The HTTP method of the simulated request (e.g., 'GET', 'POST'). endpoint (str): The endpoint of the simulated request (e.g., '/home', '/api/data'). status (str): The HTTP status code of the simulated request (e.g., '200', '404', '500'). response_size (int): The size of the simulated response in bytes.

Returns: dict: The updated state dictionary.

[4]

def update_metrics(state: dict, method: str, endpoint: str, status: str, response_size: int) -> dict:
    """
    Updates various Prometheus metrics based on a simulated request.

    This function simulates a request by incrementing a counter, managing a gauge
    for active requests, observing a request duration with a histogram, and
    recording a response size with a summary.

    Parameters
    ----------
    state : dict
        The current application state dictionary containing initialized Prometheus metrics.
    method : str
        The HTTP method of the simulated request (e.g., 'GET', 'POST').
    endpoint : str
        The endpoint of the simulated request (e.g., '/home', '/api/data').
    status : str
        The HTTP status code of the simulated request (e.g., '200', '404', '500').
    response_size : int
        The size of the simulated response in bytes.

    Returns
    -------
    dict
        The updated state dictionary.
    """
    logger = state['logger']
    requests_total = state['requests_total']
    active_requests = state['active_requests']
    request_duration_seconds = state['request_duration_seconds']
    response_size_bytes = state['response_size_bytes']

    logger.debug(f"Updating metrics for request: {method} {endpoint} -> {status}")

    # Increment total requests counter
    requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
    logger.debug(f"Counter 'requests_total' incremented for labels: method={method}, endpoint={endpoint}, status={status}")

    # Increment active requests gauge (simulate start of request)
    active_requests.inc()
    logger.debug("Gauge 'active_requests' incremented.")

    # Simulate request processing time
    start_time = time.time()
    # Add random jitter to simulate varying processing times
    processing_time = random.uniform(0.05, 0.5) + random.gauss(0, 0.05)
    time.sleep(max(0, processing_time)) # Ensure processing_time is not negative
    duration = time.time() - start_time

    # Observe request duration in histogram
    request_duration_seconds.observe(duration)
    logger.debug(f"Histogram 'request_duration_seconds' observed duration: {duration:.4f}s")

    # Decrement active requests gauge (simulate end of request)
    active_requests.dec()
    logger.debug("Gauge 'active_requests' decremented.")

    # Observe response size in summary
    response_size_bytes.observe(response_size)
    logger.debug(f"Summary 'response_size_bytes' observed size: {response_size} bytes")

    logger.info(f"Metrics updated for request: {method} {endpoint} (Duration: {duration:.4f}s, Size: {response_size} bytes)")
    return state

Function Name: `start_metrics_server`

This function starts a lightweight HTTP server in a separate thread to expose Prometheus metrics on a specified port. This server will respond to /metrics requests from Prometheus.

Algorithm:

Define a custom MetricsHandler to serve Prometheus metrics.
Start an HTTPServer with the handler on the given port.
Run the server in a separate thread to avoid blocking the main execution flow.
Include a retry mechanism with exponential backoff if the port is already in use.

Parameters: state (dict): The current application state dictionary containing the logger. port (int): The port number on which the metrics server will listen.

Returns: dict: The updated state dictionary, potentially with a reference to the server thread.

[5]

class MetricsHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/metrics':
            self.send_response(200)
            self.send_header('Content-Type', 'text/plain; version=0.0.4; charset=utf-8')
            self.end_headers()
            self.wfile.write(generate_latest())
        else:
            self.send_response(404)
            self.end_headers()
            self.wfile.write(b'Not Found')

    def log_message(self, format, *args):
        # Suppress HTTP server logging to console to avoid clutter
        pass

def start_metrics_server(state: dict, port: int = 8000) -> dict:
    """
    Starts an HTTP server in a separate thread to expose Prometheus metrics.

    The server listens on the specified port and serves metrics at the /metrics endpoint.
    It includes a retry mechanism for port binding issues.

    Parameters
    ----------
    state : dict
        The current application state dictionary containing the logger.
    port : int, optional
        The port number to expose metrics on, defaults to 8000.

    Returns
    -------
    dict
        The updated state dictionary.
    """
    logger = state['logger']
    max_retries = 5
    base_delay = 1 # seconds

    for attempt in range(max_retries):
        try:
            # Prometheus client's start_http_server is simpler but sometimes we need more control
            # For this example, we will use a custom handler to demonstrate how to expose metrics
            # without relying solely on the client library's built-in server.

            # However, for simplicity and Colab's environment, we will use prometheus_client's built-in server
            # which is easier to manage in a non-daemonized environment.
            # The class MetricsHandler is shown for educational purposes if a custom server is needed.

            logger.info(f"Attempting to start Prometheus metrics server on port {port} (Attempt {attempt + 1}/{max_retries})...")
            start_http_server(port)
            state['metrics_server_port'] = port
            logger.info(f"Prometheus metrics server started successfully on port {port}.")
            return state
        except OSError as e:
            if "Address already in use" in str(e) and attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt) + random.uniform(0, 0.1) # Exponential backoff with jitter
                logger.warning(f"Port {port} is already in use. Retrying in {delay:.2f} seconds...")
                time.sleep(delay)
            else:
                logger.error(f"Failed to start Prometheus metrics server on port {port}: {e}")
                raise

    return state

Function Name: `stop_metrics_server`

This function attempts to gracefully stop the Prometheus metrics HTTP server if it was started. This is important for freeing up resources and ports, especially in interactive environments or when re-running code.

Algorithm:

Check if the server thread is stored in the state.
If it exists, attempt to shut down the server and join its thread.

Parameters: state (dict): The current application state dictionary, potentially containing the server thread and logger.

Returns: dict: The updated state dictionary.

[6]

def stop_metrics_server(state: dict) -> dict:
    """
    Stops the Prometheus metrics HTTP server if it is running.

    Parameters
    ----------
    state : dict
        The current application state dictionary.

    Returns
    -------
    dict
        The updated state dictionary.
    """
    logger = state['logger']
    if 'metrics_server_port' in state:
        logger.info(f"Attempting to stop Prometheus metrics server on port {state['metrics_server_port']}...")
        # Unfortunately, prometheus_client.start_http_server doesn't return a server object to stop.
        # In a real application, you would manage the server instance directly.
        # For Colab, simply notifying the user that a new server will bind to a new port if run again.
        logger.warning("Due to the nature of `prometheus_client.start_http_server`, a running server cannot be programmatically stopped easily within the same process.")
        logger.warning("If you re-run `start_metrics_server`, it will attempt to bind to a new port if the previous one is still in use.")
        del state['metrics_server_port'] # Remove the port to indicate no server is actively managed
    else:
        logger.info("No Prometheus metrics server found to stop.")
    return state

Demonstration/Visualization

This section demonstrates the usage of the defined functions to expose Prometheus metrics. We will initialize the metrics, start a local HTTP server to expose them, simulate application activity to update the metrics, and then visualize the collected data.

Step 1: Initialize Metrics State and Start Server

[7]

# Initialize the metrics state
app_state = create_metrics_state()

# Start the metrics server on an available port
# In Colab, we need to pick a high port to avoid conflicts, and ensure it's accessible.
# The start_http_server function usually picks a free port automatically if none is specified, or if the specified one is in use.
# For demonstration purposes, we will try port 8000.
# If it fails, the retry mechanism will try other ports.
app_state = start_metrics_server(app_state, port=8000)

2026-06-11 06:10:19,574 - __main__ - INFO - Initializing Prometheus metrics...
INFO:__main__:Initializing Prometheus metrics...
2026-06-11 06:10:19,576 - __main__ - INFO - Prometheus metrics initialized successfully.
INFO:__main__:Prometheus metrics initialized successfully.
2026-06-11 06:10:19,577 - __main__ - INFO - Attempting to start Prometheus metrics server on port 8000 (Attempt 1/5)...
INFO:__main__:Attempting to start Prometheus metrics server on port 8000 (Attempt 1/5)...
2026-06-11 06:10:19,580 - __main__ - INFO - Prometheus metrics server started successfully on port 8000.
INFO:__main__:Prometheus metrics server started successfully on port 8000.

Step 2: Simulate Application Activity

Now, let's simulate some requests to our application to update the Prometheus metrics. We will call the update_metrics function with various parameters to mimic different types of requests (GET, POST), endpoints, statuses, and response sizes.

[8]

num_simulated_requests = 50

logger = app_state['logger']
logger.info(f"Simulating {num_simulated_requests} application requests...")

for i in range(num_simulated_requests):
    method = random.choice(['GET', 'POST', 'PUT', 'DELETE'])
    endpoint = random.choice(['/users', '/products', '/orders', '/status', '/admin'])
    status = random.choice(['200', '201', '204', '400', '401', '404', '500', '503'])
    response_size = random.randint(100, 5000)

    # Simulate some requests that might take longer or fail
    if endpoint == '/admin' and random.random() < 0.2: # 20% chance of 500 on /admin
        status = '500'
    if method == 'POST' and random.random() < 0.1: # 10% chance of 400 on POST
        status = '400'

    app_state = update_metrics(app_state, method, endpoint, status, response_size)
    time.sleep(random.uniform(0.01, 0.1)) # Small delay between requests with jitter

logger.info("Simulated application requests complete.")

# You can verify the metrics by trying to access the following URL in a new browser tab:
# http://localhost:8000/metrics (if running locally)
# In Google Colab, this server is running within the Colab environment.
# To access it, you would typically need to expose the port using ngrok or similar, but for demonstration
# purposes, we will just show the raw metrics in the next step.

2026-06-11 06:10:34,414 - __main__ - INFO - Simulating 50 application requests...
INFO:__main__:Simulating 50 application requests...
2026-06-11 06:10:34,680 - __main__ - INFO - Metrics updated for request: GET /products (Duration: 0.2634s, Size: 808 bytes)
INFO:__main__:Metrics updated for request: GET /products (Duration: 0.2634s, Size: 808 bytes)
2026-06-11 06:10:34,947 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.2461s, Size: 331 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.2461s, Size: 331 bytes)
2026-06-11 06:10:35,248 - __main__ - INFO - Metrics updated for request: GET /products (Duration: 0.2216s, Size: 3330 bytes)
INFO:__main__:Metrics updated for request: GET /products (Duration: 0.2216s, Size: 3330 bytes)
2026-06-11 06:10:35,511 - __main__ - INFO - Metrics updated for request: GET /admin (Duration: 0.1924s, Size: 1955 bytes)
INFO:__main__:Metrics updated for request: GET /admin (Duration: 0.1924s, Size: 1955 bytes)
2026-06-11 06:10:35,968 - __main__ - INFO - Metrics updated for request: DELETE /orders (Duration: 0.4317s, Size: 2399 bytes)
INFO:__main__:Metrics updated for request: DELETE /orders (Duration: 0.4317s, Size: 2399 bytes)
2026-06-11 06:10:36,222 - __main__ - INFO - Metrics updated for request: DELETE /users (Duration: 0.1730s, Size: 1148 bytes)
INFO:__main__:Metrics updated for request: DELETE /users (Duration: 0.1730s, Size: 1148 bytes)
2026-06-11 06:10:36,584 - __main__ - INFO - Metrics updated for request: POST /orders (Duration: 0.3404s, Size: 2729 bytes)
INFO:__main__:Metrics updated for request: POST /orders (Duration: 0.3404s, Size: 2729 bytes)
2026-06-11 06:10:37,103 - __main__ - INFO - Metrics updated for request: GET /products (Duration: 0.4494s, Size: 2033 bytes)
INFO:__main__:Metrics updated for request: GET /products (Duration: 0.4494s, Size: 2033 bytes)
2026-06-11 06:10:37,322 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.1797s, Size: 3398 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.1797s, Size: 3398 bytes)
2026-06-11 06:10:37,812 - __main__ - INFO - Metrics updated for request: DELETE /orders (Duration: 0.4711s, Size: 2092 bytes)
INFO:__main__:Metrics updated for request: DELETE /orders (Duration: 0.4711s, Size: 2092 bytes)
2026-06-11 06:10:38,305 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.4788s, Size: 4067 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.4788s, Size: 4067 bytes)
2026-06-11 06:10:38,400 - __main__ - INFO - Metrics updated for request: DELETE /status (Duration: 0.0301s, Size: 3901 bytes)
INFO:__main__:Metrics updated for request: DELETE /status (Duration: 0.0301s, Size: 3901 bytes)
2026-06-11 06:10:38,992 - __main__ - INFO - Metrics updated for request: PUT /admin (Duration: 0.5172s, Size: 4970 bytes)
INFO:__main__:Metrics updated for request: PUT /admin (Duration: 0.5172s, Size: 4970 bytes)
2026-06-11 06:10:39,572 - __main__ - INFO - Metrics updated for request: PUT /users (Duration: 0.5052s, Size: 1770 bytes)
INFO:__main__:Metrics updated for request: PUT /users (Duration: 0.5052s, Size: 1770 bytes)
2026-06-11 06:10:40,166 - __main__ - INFO - Metrics updated for request: POST /orders (Duration: 0.4977s, Size: 3468 bytes)
INFO:__main__:Metrics updated for request: POST /orders (Duration: 0.4977s, Size: 3468 bytes)
2026-06-11 06:10:40,573 - __main__ - INFO - Metrics updated for request: POST /users (Duration: 0.3167s, Size: 3422 bytes)
INFO:__main__:Metrics updated for request: POST /users (Duration: 0.3167s, Size: 3422 bytes)
2026-06-11 06:10:40,719 - __main__ - INFO - Metrics updated for request: GET /admin (Duration: 0.1162s, Size: 2035 bytes)
INFO:__main__:Metrics updated for request: GET /admin (Duration: 0.1162s, Size: 2035 bytes)
2026-06-11 06:10:40,945 - __main__ - INFO - Metrics updated for request: DELETE /users (Duration: 0.2069s, Size: 3463 bytes)
INFO:__main__:Metrics updated for request: DELETE /users (Duration: 0.2069s, Size: 3463 bytes)
2026-06-11 06:10:41,456 - __main__ - INFO - Metrics updated for request: POST /products (Duration: 0.4257s, Size: 4453 bytes)
INFO:__main__:Metrics updated for request: POST /products (Duration: 0.4257s, Size: 4453 bytes)
2026-06-11 06:10:41,859 - __main__ - INFO - Metrics updated for request: PUT /products (Duration: 0.3383s, Size: 3726 bytes)
INFO:__main__:Metrics updated for request: PUT /products (Duration: 0.3383s, Size: 3726 bytes)
2026-06-11 06:10:42,387 - __main__ - INFO - Metrics updated for request: POST /users (Duration: 0.4474s, Size: 636 bytes)
INFO:__main__:Metrics updated for request: POST /users (Duration: 0.4474s, Size: 636 bytes)
2026-06-11 06:10:42,700 - __main__ - INFO - Metrics updated for request: DELETE /products (Duration: 0.2551s, Size: 972 bytes)
INFO:__main__:Metrics updated for request: DELETE /products (Duration: 0.2551s, Size: 972 bytes)
2026-06-11 06:10:42,922 - __main__ - INFO - Metrics updated for request: GET /status (Duration: 0.1555s, Size: 2241 bytes)
INFO:__main__:Metrics updated for request: GET /status (Duration: 0.1555s, Size: 2241 bytes)
2026-06-11 06:10:43,091 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.1164s, Size: 4552 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.1164s, Size: 4552 bytes)
2026-06-11 06:10:43,416 - __main__ - INFO - Metrics updated for request: PUT /users (Duration: 0.2661s, Size: 2061 bytes)
INFO:__main__:Metrics updated for request: PUT /users (Duration: 0.2661s, Size: 2061 bytes)
2026-06-11 06:10:43,719 - __main__ - INFO - Metrics updated for request: DELETE /products (Duration: 0.2513s, Size: 769 bytes)
INFO:__main__:Metrics updated for request: DELETE /products (Duration: 0.2513s, Size: 769 bytes)
2026-06-11 06:10:44,191 - __main__ - INFO - Metrics updated for request: POST /orders (Duration: 0.3794s, Size: 438 bytes)
INFO:__main__:Metrics updated for request: POST /orders (Duration: 0.3794s, Size: 438 bytes)
2026-06-11 06:10:44,625 - __main__ - INFO - Metrics updated for request: GET /users (Duration: 0.4184s, Size: 2607 bytes)
INFO:__main__:Metrics updated for request: GET /users (Duration: 0.4184s, Size: 2607 bytes)
2026-06-11 06:10:45,261 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.5689s, Size: 1159 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.5689s, Size: 1159 bytes)
2026-06-11 06:10:45,546 - __main__ - INFO - Metrics updated for request: GET /orders (Duration: 0.2479s, Size: 365 bytes)
INFO:__main__:Metrics updated for request: GET /orders (Duration: 0.2479s, Size: 365 bytes)
2026-06-11 06:10:45,628 - __main__ - INFO - Metrics updated for request: GET /status (Duration: 0.0612s, Size: 2413 bytes)
INFO:__main__:Metrics updated for request: GET /status (Duration: 0.0612s, Size: 2413 bytes)
2026-06-11 06:10:45,778 - __main__ - INFO - Metrics updated for request: POST /users (Duration: 0.0640s, Size: 954 bytes)
INFO:__main__:Metrics updated for request: POST /users (Duration: 0.0640s, Size: 954 bytes)
2026-06-11 06:10:46,053 - __main__ - INFO - Metrics updated for request: PUT /products (Duration: 0.2063s, Size: 4276 bytes)
INFO:__main__:Metrics updated for request: PUT /products (Duration: 0.2063s, Size: 4276 bytes)
2026-06-11 06:10:46,344 - __main__ - INFO - Metrics updated for request: DELETE /status (Duration: 0.2237s, Size: 402 bytes)
INFO:__main__:Metrics updated for request: DELETE /status (Duration: 0.2237s, Size: 402 bytes)
2026-06-11 06:10:46,639 - __main__ - INFO - Metrics updated for request: DELETE /orders (Duration: 0.2431s, Size: 446 bytes)
INFO:__main__:Metrics updated for request: DELETE /orders (Duration: 0.2431s, Size: 446 bytes)
2026-06-11 06:10:47,071 - __main__ - INFO - Metrics updated for request: DELETE /users (Duration: 0.3821s, Size: 3808 bytes)
INFO:__main__:Metrics updated for request: DELETE /users (Duration: 0.3821s, Size: 3808 bytes)
2026-06-11 06:10:47,296 - __main__ - INFO - Metrics updated for request: PUT /orders (Duration: 0.1552s, Size: 1908 bytes)
INFO:__main__:Metrics updated for request: PUT /orders (Duration: 0.1552s, Size: 1908 bytes)
2026-06-11 06:10:47,542 - __main__ - INFO - Metrics updated for request: POST /orders (Duration: 0.1589s, Size: 4604 bytes)
INFO:__main__:Metrics updated for request: POST /orders (Duration: 0.1589s, Size: 4604 bytes)
2026-06-11 06:10:48,077 - __main__ - INFO - Metrics updated for request: DELETE /orders (Duration: 0.4499s, Size: 4310 bytes)
INFO:__main__:Metrics updated for request: DELETE /orders (Duration: 0.4499s, Size: 4310 bytes)
2026-06-11 06:10:48,643 - __main__ - INFO - Metrics updated for request: PUT /admin (Duration: 0.5232s, Size: 956 bytes)
INFO:__main__:Metrics updated for request: PUT /admin (Duration: 0.5232s, Size: 956 bytes)
2026-06-11 06:10:49,102 - __main__ - INFO - Metrics updated for request: GET /products (Duration: 0.4449s, Size: 4583 bytes)
INFO:__main__:Metrics updated for request: GET /products (Duration: 0.4449s, Size: 4583 bytes)
2026-06-11 06:10:49,505 - __main__ - INFO - Metrics updated for request: GET /orders (Duration: 0.3695s, Size: 3994 bytes)
INFO:__main__:Metrics updated for request: GET /orders (Duration: 0.3695s, Size: 3994 bytes)
2026-06-11 06:10:49,810 - __main__ - INFO - Metrics updated for request: DELETE /admin (Duration: 0.2447s, Size: 2895 bytes)
INFO:__main__:Metrics updated for request: DELETE /admin (Duration: 0.2447s, Size: 2895 bytes)
2026-06-11 06:10:50,327 - __main__ - INFO - Metrics updated for request: POST /orders (Duration: 0.4223s, Size: 547 bytes)
INFO:__main__:Metrics updated for request: POST /orders (Duration: 0.4223s, Size: 547 bytes)
2026-06-11 06:10:50,795 - __main__ - INFO - Metrics updated for request: GET /users (Duration: 0.4168s, Size: 4198 bytes)
INFO:__main__:Metrics updated for request: GET /users (Duration: 0.4168s, Size: 4198 bytes)
2026-06-11 06:10:51,157 - __main__ - INFO - Metrics updated for request: PUT /users (Duration: 0.2601s, Size: 2958 bytes)
INFO:__main__:Metrics updated for request: PUT /users (Duration: 0.2601s, Size: 2958 bytes)
2026-06-11 06:10:51,520 - __main__ - INFO - Metrics updated for request: DELETE /admin (Duration: 0.3481s, Size: 1618 bytes)
INFO:__main__:Metrics updated for request: DELETE /admin (Duration: 0.3481s, Size: 1618 bytes)
2026-06-11 06:10:52,100 - __main__ - INFO - Metrics updated for request: PUT /admin (Duration: 0.5207s, Size: 770 bytes)
INFO:__main__:Metrics updated for request: PUT /admin (Duration: 0.5207s, Size: 770 bytes)
2026-06-11 06:10:52,686 - __main__ - INFO - Metrics updated for request: PUT /admin (Duration: 0.5481s, Size: 4423 bytes)
INFO:__main__:Metrics updated for request: PUT /admin (Duration: 0.5481s, Size: 4423 bytes)
2026-06-11 06:10:52,930 - __main__ - INFO - Metrics updated for request: POST /users (Duration: 0.1741s, Size: 3819 bytes)
INFO:__main__:Metrics updated for request: POST /users (Duration: 0.1741s, Size: 3819 bytes)
2026-06-11 06:10:53,028 - __main__ - INFO - Simulated application requests complete.
INFO:__main__:Simulated application requests complete.

Step 3: Retrieve and Display Raw Metrics

Although Prometheus would typically scrape the /metrics endpoint, we can directly retrieve the raw metrics exposed by our server within this Colab environment. This allows us to inspect the data generated by our simulated activity.

[9]

import requests

logger = app_state['logger']
metrics_port = app_state.get('metrics_server_port', 8000)
metrics_url = f"http://localhost:{metrics_port}/metrics"

try:
    logger.info(f"Attempting to fetch metrics from {metrics_url}...")
    response = requests.get(metrics_url)
    response.raise_for_status() # Raise an exception for HTTP errors (4xx or 5xx)
    print("--- Raw Prometheus Metrics ---")
    print(response.text)
    print("------------------------------")
    logger.info("Raw metrics fetched successfully.")
except requests.exceptions.ConnectionError as e:
    logger.error(f"Could not connect to the metrics server at {metrics_url}. Is it running? Error: {e}")
    print(f"Error: Could not connect to the metrics server at {metrics_url}. Please ensure the server is running.")
except requests.exceptions.RequestException as e:
    logger.error(f"An error occurred while fetching metrics from {metrics_url}. Error: {e}")
    print(f"Error fetching metrics: {e}")

2026-06-11 06:11:11,990 - __main__ - INFO - Attempting to fetch metrics from http://localhost:8000/metrics...
INFO:__main__:Attempting to fetch metrics from http://localhost:8000/metrics...
2026-06-11 06:11:11,998 - __main__ - INFO - Raw metrics fetched successfully.
INFO:__main__:Raw metrics fetched successfully.

--- Raw Prometheus Metrics ---
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 2478.0
python_gc_objects_collected_total{generation="1"} 330.0
python_gc_objects_collected_total{generation="2"} 106.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 623.0
python_gc_collections_total{generation="1"} 56.0
python_gc_collections_total{generation="2"} 5.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="12",patchlevel="13",version="3.12.13"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.267990528e+09
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.50699776e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.78115811943e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 4.11
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 53.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP app_requests_total Total number of requests received by the application
# TYPE app_requests_total counter
app_requests_total{endpoint="/products",method="GET",status="400"} 1.0
app_requests_total{endpoint="/orders",method="PUT",status="500"} 1.0
app_requests_total{endpoint="/products",method="GET",status="201"} 2.0
app_requests_total{endpoint="/admin",method="GET",status="500"} 2.0
app_requests_total{endpoint="/orders",method="DELETE",status="201"} 1.0
app_requests_total{endpoint="/users",method="DELETE",status="503"} 1.0
app_requests_total{endpoint="/orders",method="POST",status="404"} 1.0
app_requests_total{endpoint="/orders",method="PUT",status="401"} 1.0
app_requests_total{endpoint="/orders",method="DELETE",status="404"} 1.0
app_requests_total{endpoint="/orders",method="PUT",status="204"} 3.0
app_requests_total{endpoint="/status",method="DELETE",status="400"} 1.0
app_requests_total{endpoint="/admin",method="PUT",status="201"} 2.0
app_requests_total{endpoint="/users",method="PUT",status="500"} 1.0
app_requests_total{endpoint="/orders",method="POST",status="400"} 2.0
app_requests_total{endpoint="/users",method="POST",status="400"} 2.0
app_requests_total{endpoint="/users",method="DELETE",status="401"} 2.0
app_requests_total{endpoint="/products",method="POST",status="401"} 1.0
app_requests_total{endpoint="/products",method="PUT",status="500"} 1.0
app_requests_total{endpoint="/users",method="POST",status="503"} 1.0
app_requests_total{endpoint="/products",method="DELETE",status="201"} 1.0
app_requests_total{endpoint="/status",method="GET",status="201"} 1.0
app_requests_total{endpoint="/users",method="PUT",status="400"} 1.0
app_requests_total{endpoint="/products",method="DELETE",status="503"} 1.0
app_requests_total{endpoint="/users",method="GET",status="404"} 1.0
app_requests_total{endpoint="/orders",method="GET",status="204"} 1.0
app_requests_total{endpoint="/status",method="GET",status="204"} 1.0
app_requests_total{endpoint="/products",method="PUT",status="401"} 1.0
app_requests_total{endpoint="/status",method="DELETE",status="200"} 1.0
app_requests_total{endpoint="/orders",method="DELETE",status="200"} 1.0
app_requests_total{endpoint="/orders",method="PUT",status="200"} 1.0
app_requests_total{endpoint="/orders",method="POST",status="200"} 1.0
app_requests_total{endpoint="/orders",method="DELETE",status="204"} 1.0
app_requests_total{endpoint="/admin",method="PUT",status="500"} 1.0
app_requests_total{endpoint="/products",method="GET",status="200"} 1.0
app_requests_total{endpoint="/orders",method="GET",status="201"} 1.0
app_requests_total{endpoint="/admin",method="DELETE",status="401"} 1.0
app_requests_total{endpoint="/orders",method="POST",status="201"} 1.0
app_requests_total{endpoint="/users",method="GET",status="500"} 1.0
app_requests_total{endpoint="/users",method="PUT",status="204"} 1.0
app_requests_total{endpoint="/admin",method="DELETE",status="400"} 1.0
app_requests_total{endpoint="/admin",method="PUT",status="401"} 1.0
app_requests_total{endpoint="/users",method="POST",status="401"} 1.0
# HELP app_requests_created Total number of requests received by the application
# TYPE app_requests_created gauge
app_requests_created{endpoint="/products",method="GET",status="400"} 1.7811582344173706e+09
app_requests_created{endpoint="/orders",method="PUT",status="500"} 1.7811582347015238e+09
app_requests_created{endpoint="/products",method="GET",status="201"} 1.7811582350271525e+09
app_requests_created{endpoint="/admin",method="GET",status="500"} 1.7811582353191845e+09
app_requests_created{endpoint="/orders",method="DELETE",status="201"} 1.7811582355362773e+09
app_requests_created{endpoint="/users",method="DELETE",status="503"} 1.7811582360490382e+09
app_requests_created{endpoint="/orders",method="POST",status="404"} 1.7811582362435997e+09
app_requests_created{endpoint="/orders",method="PUT",status="401"} 1.7811582371430728e+09
app_requests_created{endpoint="/orders",method="DELETE",status="404"} 1.7811582373410661e+09
app_requests_created{endpoint="/orders",method="PUT",status="204"} 1.7811582378264725e+09
app_requests_created{endpoint="/status",method="DELETE",status="400"} 1.7811582383698685e+09
app_requests_created{endpoint="/admin",method="PUT",status="201"} 1.7811582384756112e+09
app_requests_created{endpoint="/users",method="PUT",status="500"} 1.7811582390675468e+09
app_requests_created{endpoint="/orders",method="POST",status="400"} 1.7811582396687527e+09
app_requests_created{endpoint="/users",method="POST",status="400"} 1.7811582402564476e+09
app_requests_created{endpoint="/users",method="DELETE",status="401"} 1.781158240738131e+09
app_requests_created{endpoint="/products",method="POST",status="401"} 1.7811582410306656e+09
app_requests_created{endpoint="/products",method="PUT",status="500"} 1.7811582415213804e+09
app_requests_created{endpoint="/users",method="POST",status="503"} 1.781158241940299e+09
app_requests_created{endpoint="/products",method="DELETE",status="201"} 1.7811582424448297e+09
app_requests_created{endpoint="/status",method="GET",status="201"} 1.7811582427667048e+09
app_requests_created{endpoint="/users",method="PUT",status="400"} 1.7811582431505191e+09
app_requests_created{endpoint="/products",method="DELETE",status="503"} 1.7811582434681017e+09
app_requests_created{endpoint="/users",method="GET",status="404"} 1.7811582442070928e+09
app_requests_created{endpoint="/orders",method="GET",status="204"} 1.7811582452986217e+09
app_requests_created{endpoint="/status",method="GET",status="204"} 1.7811582455671494e+09
app_requests_created{endpoint="/products",method="PUT",status="401"} 1.7811582458471649e+09
app_requests_created{endpoint="/status",method="DELETE",status="200"} 1.781158246120475e+09
app_requests_created{endpoint="/orders",method="DELETE",status="200"} 1.7811582463963249e+09
app_requests_created{endpoint="/orders",method="PUT",status="200"} 1.781158247141268e+09
app_requests_created{endpoint="/orders",method="POST",status="200"} 1.781158247383052e+09
app_requests_created{endpoint="/orders",method="DELETE",status="204"} 1.7811582476278608e+09
app_requests_created{endpoint="/admin",method="PUT",status="500"} 1.7811582481200776e+09
app_requests_created{endpoint="/products",method="GET",status="200"} 1.7811582486572316e+09
app_requests_created{endpoint="/orders",method="GET",status="201"} 1.7811582491356757e+09
app_requests_created{endpoint="/admin",method="DELETE",status="401"} 1.7811582495660214e+09
app_requests_created{endpoint="/orders",method="POST",status="201"} 1.781158249904738e+09
app_requests_created{endpoint="/users",method="GET",status="500"} 1.781158250378696e+09
app_requests_created{endpoint="/users",method="PUT",status="204"} 1.7811582508968785e+09
app_requests_created{endpoint="/admin",method="DELETE",status="400"} 1.7811582511725404e+09
app_requests_created{endpoint="/admin",method="PUT",status="401"} 1.781158252138123e+09
app_requests_created{endpoint="/users",method="POST",status="401"} 1.7811582527563267e+09
# HELP app_active_requests Current number of active requests
# TYPE app_active_requests gauge
app_active_requests 0.0
# HELP app_request_duration_seconds Histogram of request durations
# TYPE app_request_duration_seconds histogram
app_request_duration_seconds_bucket{le="0.1"} 3.0
app_request_duration_seconds_bucket{le="0.5"} 44.0
app_request_duration_seconds_bucket{le="1.0"} 50.0
app_request_duration_seconds_bucket{le="2.5"} 50.0
app_request_duration_seconds_bucket{le="5.0"} 50.0
app_request_duration_seconds_bucket{le="10.0"} 50.0
app_request_duration_seconds_bucket{le="+Inf"} 50.0
app_request_duration_seconds_count 50.0
app_request_duration_seconds_sum 15.724711179733276
# HELP app_request_duration_seconds_created Histogram of request durations
# TYPE app_request_duration_seconds_created gauge
app_request_duration_seconds_created 1.7811582195763412e+09
# HELP app_response_size_bytes Summary of response sizes in bytes
# TYPE app_response_size_bytes summary
app_response_size_bytes_count 50.0
app_response_size_bytes_sum 125180.0
# HELP app_response_size_bytes_created Summary of response sizes in bytes
# TYPE app_response_size_bytes_created gauge
app_response_size_bytes_created 1.7811582195764172e+09

------------------------------

Step 4: Parse Raw Metrics into a DataFrame

To effectively analyze and visualize the collected metrics, it's beneficial to parse the raw Prometheus text format into a structured data format, such as a Pandas DataFrame. This step extracts key information from the metrics and prepares it for plotting.

[10]

from io import StringIO
import re
import pandas as pd

def parse_prometheus_metrics(metrics_text: str) -> pd.DataFrame:
    """
    Parses raw Prometheus metrics text into a Pandas DataFrame.

    Parameters
    ----------
    metrics_text : str
        The raw text output from a Prometheus /metrics endpoint.

    Returns
    -------
    pd.DataFrame
        A DataFrame containing the parsed metrics, including metric name, labels, and value.
    """
    parsed_data = []
    current_metric_name = None
    current_metric_type = None
    current_metric_help = None

    # Split the text into lines and process each one
    for line in metrics_text.strip().split('\n'):
        line = line.strip()
        if not line or line.startswith('#'):
            # This is a comment or blank line, parse it for metadata
            if line.startswith('# HELP'):
                match = re.match(r'# HELP (\w+) (.*)', line)
                if match: # Add if for mypy compatibility
                    current_metric_name = match.group(1)
                    current_metric_help = match.group(2)
            elif line.startswith('# TYPE'):
                match = re.match(r'# TYPE (\w+) (\w+)', line)
                if match: # Add if for mypy compatibility
                    current_metric_name = match.group(1)
                    current_metric_type = match.group(2)
            continue

        # Process metric data lines
        # Regex to capture metric name, labels (optional), and value
        match = re.match(r'(\w+){(.*)} (\S+)', line)
        if match:
            metric_name = match.group(1)
            labels_str = match.group(2)
            value = match.group(3)
            labels = {}
            if labels_str:
                # Split labels_str by comma, but handle quoted strings
                for label_pair in re.findall(r'(\w+)="([^"]*)"(?:,|$)', labels_str):
                    labels[label_pair[0]] = label_pair[1]
        else:
            # Handle metrics without labels (e.g., simple gauges, counters)
            match = re.match(r'(\w+) (\S+)', line)
            if match:
                metric_name = match.group(1)
                labels = {}
                value = match.group(2)
            else:
                logger.warning(f"Could not parse line: {line}")
                continue

        parsed_data.append({
            'metric_name': metric_name,
            'value': float(value),
            **labels
        })

    return pd.DataFrame(parsed_data)

# Get the raw metrics text from the previous execution
raw_metrics_text = app_state.get('raw_metrics_output', None)
if 'response' in locals() and response.status_code == 200:
    raw_metrics_text = response.text

if raw_metrics_text:
    logger.info("Parsing raw Prometheus metrics...")
    metrics_df = parse_prometheus_metrics(raw_metrics_text)
    app_state['metrics_dataframe'] = metrics_df
    print("--- Parsed Metrics DataFrame (Head) ---")
    display(metrics_df.head(10))
    print("--------------------------------------")
    logger.info("Prometheus metrics parsed into DataFrame.")
else:
    logger.error("No raw metrics text available to parse. Please ensure the metrics fetching step ran successfully.")
    print("Error: No raw metrics text available to parse.")

2026-06-11 06:11:28,404 - __main__ - INFO - Parsing raw Prometheus metrics...
INFO:__main__:Parsing raw Prometheus metrics...

--- Parsed Metrics DataFrame (Head) ---

	metric_name	value	generation	implementation	major	minor	patchlevel	version	endpoint	method	status	le
0	python_gc_objects_collected_total	2478.0	0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	python_gc_objects_collected_total	330.0	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	python_gc_objects_collected_total	106.0	2	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	python_gc_objects_uncollectable_total	0.0	0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	python_gc_objects_uncollectable_total	0.0	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
5	python_gc_objects_uncollectable_total	0.0	2	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
6	python_gc_collections_total	623.0	0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
7	python_gc_collections_total	56.0	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
8	python_gc_collections_total	5.0	2	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
9	python_info	1.0	NaN	CPython	3	12	13	3.12.13	NaN	NaN	NaN	NaN

2026-06-11 06:11:28,451 - __main__ - INFO - Prometheus metrics parsed into DataFrame.
INFO:__main__:Prometheus metrics parsed into DataFrame.

--------------------------------------

Step 5: Visualize Key Metrics

With the metrics now in a structured DataFrame, we can create various visualizations to analyze our application's performance. We'll focus on the custom metrics we defined: app_requests_total, app_request_duration_seconds, and app_response_size_bytes.

Visualization 5.1: Total Requests by Method, Endpoint, and Status

This visualization shows the breakdown of the app_requests_total counter, illustrating which HTTP methods were used, on which endpoints, and what status codes were returned. This helps to quickly identify common request patterns or error hotspots.

[11]

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

logger = app_state['logger']
metrics_df = app_state.get('metrics_dataframe')

if metrics_df is not None:
    # Filter for the 'app_requests_total' metric
    requests_total_df = metrics_df[metrics_df['metric_name'] == 'app_requests_total']

    if not requests_total_df.empty:
        logger.info("Generating visualization for Total Requests by Method, Endpoint, and Status...")

        # Create a combined category for plotting
        requests_total_df['category'] = requests_total_df['method'] + ' ' + requests_total_df['endpoint'] + ' (' + requests_total_df['status'] + ')'

        plt.figure(figsize=(14, 8))
        sns.barplot(data=requests_total_df, x='value', y='category', orient='h', palette='viridis')
        plt.title('Total Requests by Method, Endpoint, and Status')
        plt.xlabel('Total Requests')
        plt.ylabel('Request Category (Method Endpoint Status)')
        plt.grid(axis='x', linestyle='--', alpha=0.7)
        plt.tight_layout()
        plt.show()
        logger.info("Visualization for Total Requests generated successfully.")
    else:
        logger.warning("No 'app_requests_total' metrics found in the DataFrame to visualize.")
        print("No 'app_requests_total' metrics found to visualize.")
else:
    logger.error("Metrics DataFrame not found in app_state. Please ensure parsing step ran successfully.")
    print("Error: Metrics DataFrame not found. Cannot visualize.")

2026-06-11 06:11:43,052 - __main__ - INFO - Generating visualization for Total Requests by Method, Endpoint, and Status...
INFO:__main__:Generating visualization for Total Requests by Method, Endpoint, and Status...
/tmp/ipykernel_880/3226007623.py:16: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  requests_total_df['category'] = requests_total_df['method'] + ' ' + requests_total_df['endpoint'] + ' (' + requests_total_df['status'] + ')'
/tmp/ipykernel_880/3226007623.py:19: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(data=requests_total_df, x='value', y='category', orient='h', palette='viridis')

2026-06-11 06:11:43,620 - __main__ - INFO - Visualization for Total Requests generated successfully.
INFO:__main__:Visualization for Total Requests generated successfully.

Visualization 5.2: Request Duration Histogram

This visualization analyzes the app_request_duration_seconds histogram metric, which provides insights into the distribution of request processing times. By plotting the cumulative count against the defined buckets, we can see how many requests fall within certain duration ranges.

[13]

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

logger = app_state['logger']
metrics_df = app_state.get('metrics_dataframe')

if metrics_df is not None:
    # Filter for histogram buckets of 'app_request_duration_seconds'
    duration_buckets_df = metrics_df[
        (metrics_df['metric_name'] == 'app_request_duration_seconds_bucket')
    ].copy() # Use .copy() to avoid SettingWithCopyWarning

    if not duration_buckets_df.empty:
        logger.info("Generating visualization for Request Duration Histogram...")

        # Convert 'le' (less than or equal to) label to numeric for plotting
        duration_buckets_df['le_numeric'] = pd.to_numeric(duration_buckets_df['le'], errors='coerce')
        # Drop rows where 'le_numeric' is NaN (e.g., if there were non-numeric strings other than '+Inf')
        duration_buckets_df = duration_buckets_df.dropna(subset=['le_numeric'])
        duration_buckets_df = duration_buckets_df.sort_values(by='le_numeric')

        plt.figure(figsize=(12, 7))
        sns.lineplot(data=duration_buckets_df, x='le_numeric', y='value', drawstyle='steps-pre', marker='o')
        plt.title('Request Duration Cumulative Histogram')
        plt.xlabel('Duration (seconds)')
        plt.ylabel('Cumulative Count of Requests')
        plt.xscale('log') # Log scale for better visualization of durations
        plt.grid(True, which="both", ls="--", c='0.7')

        # Filter out '+Inf' for xticks to avoid OverflowError with log scale
        finite_buckets_df = duration_buckets_df[duration_buckets_df['le_numeric'] != float('inf')]
        plt.xticks(finite_buckets_df['le_numeric'], labels=[f'<= {l}' for l in finite_buckets_df['le']], rotation=45, ha='right')
        plt.tight_layout()
        plt.show()
        logger.info("Visualization for Request Duration Histogram generated successfully.")
    else:
        logger.warning("No 'app_request_duration_seconds_bucket' metrics found in the DataFrame to visualize.")
        print("No 'app_request_duration_seconds' histogram metrics found to visualize.")
else:
    logger.error("Metrics DataFrame not found in app_state. Please ensure parsing step ran successfully.")
    print("Error: Metrics DataFrame not found. Cannot visualize.")

2026-06-11 06:12:30,334 - __main__ - INFO - Generating visualization for Request Duration Histogram...
INFO:__main__:Generating visualization for Request Duration Histogram...

2026-06-11 06:12:30,553 - __main__ - INFO - Visualization for Request Duration Histogram generated successfully.
INFO:__main__:Visualization for Request Duration Histogram generated successfully.

Visualization 5.3: Response Size Summary

This visualization examines the app_response_size_bytes summary metric, which provides statistical quantiles (like median, 90th percentile, etc.) for response sizes. While Prometheus summaries don't expose buckets like histograms, we can extract the count and sum, and if available, the quantiles to visualize the distribution.

[14]

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

logger = app_state['logger']
metrics_df = app_state.get('metrics_dataframe')

if metrics_df is not None:
    # Filter for the 'app_response_size_bytes' summary metrics (count, sum, and quantiles if available)
    response_size_df = metrics_df[
        metrics_df['metric_name'].str.startswith('app_response_size_bytes')
    ].copy() # Use .copy() to avoid SettingWithCopyWarning

    if not response_size_df.empty:
        logger.info("Generating visualization for Response Size Summary...")

        # Extract 'app_response_size_bytes_count' and 'app_response_size_bytes_sum'
        count = response_size_df[response_size_df['metric_name'] == 'app_response_size_bytes_count']['value'].sum()
        total_sum = response_size_df[response_size_df['metric_name'] == 'app_response_size_bytes_sum']['value'].sum()

        print(f"Total Response Count: {int(count)}")
        print(f"Total Response Size (Sum): {int(total_sum)} bytes")
        if count > 0:
            print(f"Average Response Size: {total_sum / count:.2f} bytes")
        else:
            print("No response size data to calculate average.")

        # Attempt to plot quantiles if they exist (Summary metrics usually provide them as separate entries if configured)
        # Prometheus Client library's Summary metrics only expose _count and _sum by default in the text format.
        # To get quantiles, you'd typically need to configure them explicitly, or use a Histogram.
        # For this demonstration, we'll assume the client does not expose quantiles by default in the raw /metrics output.
        # If we had quantiles like 'app_response_size_bytes{quantile="0.5"}', we could plot them.

        # Plotting a simple bar chart of the total sum of bytes for demonstration if quantiles are not directly available
        if total_sum > 0 and count > 0:
            plt.figure(figsize=(8, 6))
            sns.barplot(x=['Total Bytes', 'Average Bytes'], y=[total_sum, total_sum / count], palette='rocket')
            plt.title('Summary of Application Response Sizes')
            plt.ylabel('Bytes')
            plt.grid(axis='y', linestyle='--', alpha=0.7)
            plt.tight_layout()
            plt.show()
            logger.info("Visualization for Response Size Summary generated successfully.")
        else:
            logger.warning("No sufficient data to visualize Response Size Summary.")
            print("No sufficient data to visualize Response Size Summary (count or sum is zero).")

    else:
        logger.warning("No 'app_response_size_bytes' metrics found in the DataFrame to visualize.")
        print("No 'app_response_size_bytes' summary metrics found to visualize.")
else:
    logger.error("Metrics DataFrame not found in app_state. Please ensure parsing step ran successfully.")
    print("Error: Metrics DataFrame not found. Cannot visualize.")

2026-06-11 06:12:49,897 - __main__ - INFO - Generating visualization for Response Size Summary...
INFO:__main__:Generating visualization for Response Size Summary...

Total Response Count: 50
Total Response Size (Sum): 125180 bytes
Average Response Size: 2503.60 bytes

/tmp/ipykernel_880/1878120491.py:37: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x=['Total Bytes', 'Average Bytes'], y=[total_sum, total_sum / count], palette='rocket')

2026-06-11 06:12:50,010 - __main__ - INFO - Visualization for Response Size Summary generated successfully.
INFO:__main__:Visualization for Response Size Summary generated successfully.

Production Considerations

When exposing Prometheus metrics in a production environment, several factors need to be considered to ensure reliability, security, and efficiency.

1. Security

Exposing metrics often means exposing internal application state. In many environments, especially those with sensitive data, this endpoint (/metrics) should be secured.

Network Segmentation: Restrict access to the /metrics endpoint to internal networks or specific IP ranges where your Prometheus server resides.
Authentication/Authorization: For more stringent security, implement authentication (e.g., basic auth, mTLS) for the /metrics endpoint. The prometheus_client library's start_http_server does not offer built-in authentication, so you might need to run it behind a reverse proxy (like Nginx) or integrate it with a web framework that handles authentication.
HTTPS/TLS: Ensure communication to the /metrics endpoint is encrypted, especially if metrics contain sensitive information or are exposed over untrusted networks.

2. Metric Cardinality

Prometheus performs best when the number of unique label combinations (cardinality) for each metric is kept under control. High cardinality can lead to:

Increased Storage: More time-series data to store.
Increased Memory Usage: Higher memory consumption by Prometheus and the instrumented application.
Slower Queries: Queries over high-cardinality metrics can be slow.

Best Practices:

Avoid labels that are unique per request (e.g., request ID, user ID).
Aggregate data before labeling if possible.
Use service names, endpoint names, and status codes, but be mindful of their uniqueness.

3. Resource Usage

Running a metrics exporter, especially one serving a high volume of metrics, consumes resources (CPU, memory, network).

Scraping Interval: Configure Prometheus to scrape at appropriate intervals. Very frequent scraping of many targets can overload both Prometheus and the scraped targets.
Efficient Metric Generation: Ensure that your application generates metrics efficiently, without causing significant overhead to the primary application logic.
Dedicated Server/Sidecar: For critical applications, consider running the exporter as a separate sidecar process or on a dedicated metrics server to isolate resource usage.

4. Alerting and Dashboards

Metrics are most useful when they are integrated into a monitoring stack for alerting and visualization.

PromQL: Learn Prometheus Query Language (PromQL) to effectively query and aggregate your metrics for dashboards (e.g., Grafana) and alerting rules.
Thresholds: Define meaningful thresholds for your alerts based on observed application behavior and SLOs (Service Level Objectives).

5. Managing the Metrics Server in Production

In a real-world Python application, the start_http_server function from prometheus_client is often sufficient for simple cases. However, for more complex applications, you might integrate metric exposition directly into your web framework (e.g., Flask, Django) or use a more robust HTTP server (like Gunicorn, uWSGI) to serve the /metrics endpoint alongside your main application routes.

Example for a Flask application (Conceptual):

from flask import Flask, Response
from prometheus_client import generate_latest

app = Flask(__name__)

@app.route('/metrics')
def metrics():
    return Response(generate_latest(), mimetype='text/plain; version=0.0.4; charset=utf-8')

# ... other Flask routes and application logic ...

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

This approach gives you full control over the server environment and allows for consistent deployment practices.

Conclusion

This notebook has provided a comprehensive guide and demonstration on how to expose custom application metrics using the Prometheus Python client library. We covered the following key aspects:

Introduction to Prometheus Metrics: Understanding what Prometheus is, the concept of metrics, and different metric types (Counter, Gauge, Histogram, Summary).
Setting up prometheus_client: Installation of the library and importing necessary components.
Core Functions for Metric Management:
- create_metrics_state(): Initializing a logger and defining various Prometheus metric types.
- update_metrics(): Simulating application activity and updating the metrics with realistic data for different request methods, endpoints, statuses, durations, and response sizes.
- start_metrics_server(): Launching a lightweight HTTP server to expose the collected metrics on a dedicated /metrics endpoint, ready for Prometheus scraping.
- stop_metrics_server(): Acknowledging the challenges of programmatically stopping the prometheus_client's default server in a Colab environment.
Demonstration and Visualization:
- Simulating application requests to populate metrics.
- Retrieving and displaying the raw Prometheus metrics text as it would be scraped by a Prometheus server.
- Parsing the raw metrics into a Pandas DataFrame for structured analysis.
- Visualizing key metrics, including total requests by category (method, endpoint, status), the cumulative histogram of request durations, and a summary of response sizes.
Production Considerations: Discussing important aspects like security, metric cardinality, resource usage, and integration with alerting/dashboarding tools for real-world deployments.

By following these steps, you can effectively instrument your Python applications to provide valuable insights into their performance and behavior, enabling robust monitoring and more efficient troubleshooting in production environments.

Prometheus Metrics Setup

Expose Metrics with Prometheus

Key Concepts:

Dependency Installation

Library Imports

Core Functions

Function Name: create_metrics_state

Function Name: update_metrics

Function Name: start_metrics_server

Function Name: stop_metrics_server

Demonstration/Visualization

Step 1: Initialize Metrics State and Start Server

Step 2: Simulate Application Activity

Step 3: Retrieve and Display Raw Metrics

Step 4: Parse Raw Metrics into a DataFrame

Step 5: Visualize Key Metrics

Visualization 5.1: Total Requests by Method, Endpoint, and Status

Visualization 5.2: Request Duration Histogram

Visualization 5.3: Response Size Summary

Production Considerations

1. Security

2. Metric Cardinality

3. Resource Usage

4. Alerting and Dashboards

5. Managing the Metrics Server in Production

Conclusion

Function Name: `create_metrics_state`

Function Name: `update_metrics`

Function Name: `start_metrics_server`

Function Name: `stop_metrics_server`