Infrastructure·Security & Secrets·Advanced

Secrets Vault Setup

Securely store exchange API credentials and application secrets in a dedicated secrets management vault using HashiCorp Vault or cloud provider KMS with fine-grained access control policies, comprehensive audit logging of all secret access, and encryption both at rest and in transit for defense-in-depth security architecture.

infrastructuresecurity

Infrastructure Security: Storing Secrets in Vault/KMS

This notebook explores best practices for securely managing sensitive information within an application's infrastructure. We will focus on using dedicated secrets management systems like HashiCorp Vault or cloud-based Key Management Services (KMS) to store, access, and rotate secrets, thereby reducing the risk of exposure.

Key Concepts in Secrets Management

ConceptDescriptionWhy it's Important
SecretAny piece of sensitive information (API keys, database credentials, certificates, etc.)Protects critical system access and data.
Secrets EngineA component within a secrets management system that handles secrets for specific types (e.g., KV, database)Provides structured storage and retrieval for different secret types.
LeasingSecrets are granted for a limited time and must be renewed or re-requestedLimits exposure time, automatically revokes access if not renewed.
RevocationAbility to invalidate a secret or access token immediatelyCritical for incident response and mitigating breaches.
AuditingLogging all secret access and modification eventsEnsures accountability and aids in forensic analysis.
Dynamic SecretsSecrets generated on demand for a short lifespan, never storedDrastically reduces secret exposure, as secrets are never at rest for long periods.
EncryptionProtecting data at rest and in transit using cryptographic techniquesEnsures confidentiality and integrity of secrets even if storage is compromised.
Access ControlPolicies defining who can access which secrets under what conditionsEnforces least privilege, preventing unauthorized access.
RotationPeriodically changing secretsReduces the impact of a compromised secret over time.

Resources

[23]
#@title Dependency Installation
# Install necessary libraries
%pip install hvac google-cloud-kms loguru pandas matplotlib seaborn
Requirement already satisfied: hvac in /usr/local/lib/python3.12/dist-packages (2.4.0)
Requirement already satisfied: google-cloud-kms in /usr/local/lib/python3.12/dist-packages (3.13.0)
Requirement already satisfied: loguru in /usr/local/lib/python3.12/dist-packages (0.7.3)
Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0)
Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2)
Requirement already satisfied: requests<3.0.0,>=2.27.1 in /usr/local/lib/python3.12/dist-packages (from hvac) (2.32.4)
Requirement already satisfied: google-api-core<3.0.0,>=2.11.0 in /usr/local/lib/python3.12/dist-packages (from google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (2.30.3)
Requirement already satisfied: google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (2.47.0)
Requirement already satisfied: grpcio<2.0.0,>=1.33.2 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (1.81.0)
Requirement already satisfied: proto-plus<2.0.0,>=1.22.3 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (1.28.0)
Requirement already satisfied: protobuf<8.0.0,>=4.25.8 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (5.29.6)
Requirement already satisfied: grpc-google-iam-v1<1.0.0,>=0.14.0 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (0.14.4)
Requirement already satisfied: numpy>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.2)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.2)
Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2)
Requirement already satisfied: googleapis-common-protos<2.0.0,>=1.63.2 in /usr/local/lib/python3.12/dist-packages (from google-api-core<3.0.0,>=2.11.0->google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (1.75.0)
Requirement already satisfied: grpcio-status<2.0.0,>=1.33.2 in /usr/local/lib/python3.12/dist-packages (from google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (1.71.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.12/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (0.4.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.12/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (4.9.1)
Requirement already satisfied: typing-extensions~=4.12 in /usr/local/lib/python3.12/dist-packages (from grpcio<2.0.0,>=1.33.2->google-cloud-kms) (4.15.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (3.18)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (2026.5.20)
Requirement already satisfied: pyasn1<0.7.0,>=0.6.1 in /usr/local/lib/python3.12/dist-packages (from pyasn1-modules>=0.2.1->google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (0.6.3)
[42]
import os
import time
import subprocess

# 1. Download standalone Vault binary if not present
if not os.path.exists('./vault'):
    print("Downloading Vault binary...")
    !wget https://releases.hashicorp.com/vault/1.15.4/vault_1.15.4_linux_amd64.zip
    !unzip -o vault_1.15.4_linux_amd64.zip
    !chmod +x vault

# 2. Start Vault in dev mode in the background
print("Starting Vault dev server...")
!pkill vault || true
time.sleep(2)
get_ipython().system_raw('./vault server -dev -dev-root-token-id="myroottoken" > vault.log 2>&1 &')

# Wait for Vault to be ready
print("Waiting for Vault to be ready...")
for i in range(10):
    try:
        env = os.environ.copy()
        env['VAULT_ADDR'] = 'http://127.0.0.1:8200'
        result = subprocess.run(['./vault', 'status'], env=env, capture_output=True)
        if result.returncode in [0, 2]:
            print("Vault is ready!")
            break
    except:
        pass
    time.sleep(1)

# 3. Configure environment for hvac and CLI
os.environ['VAULT_ADDR'] = 'http://127.0.0.1:8200'
os.environ['VAULT_TOKEN'] = 'myroottoken'

# 4. Enable engines
print("Enabling KV and Transit engines...")
!./vault secrets enable -path=kv kv-v2 || true
!./vault secrets enable transit || true
!./vault write -f transit/keys/my-key || true

print("Vault is configured and running at http://127.0.0.1:8200")
Starting Vault dev server...
Waiting for Vault to be ready...
Vault is ready!
Enabling KV and Transit engines...
Success! Enabled the kv-v2 secrets engine at: kv/
Success! Enabled the transit secrets engine at: transit/
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1781084272]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      my-key
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96
Vault is configured and running at http://127.0.0.1:8200
[38]
# Enable the Transit secrets engine for local encryption/decryption
print("Enabling Transit secrets engine...")
!./vault secrets enable transit || echo "Transit engine might already be enabled"

# Create a named encryption key
print("Creating local encryption key 'my-key'...")
!./vault write -f transit/keys/my-key
Enabling Transit secrets engine...
Error enabling: Post "http://127.0.0.1:8200/v1/sys/mounts/transit": dial tcp 127.0.0.1:8200: connect: connection refused
Transit engine might already be enabled
Creating local encryption key 'my-key'...
Error writing data to transit/keys/my-key: Put "http://127.0.0.1:8200/v1/transit/keys/my-key": dial tcp 127.0.0.1:8200: connect: connection refused
[25]
#@title Library Imports
# Standard library imports
import os
import time
import logging
import random
from collections import deque
from typing import Dict, Any, Optional

# Third-party library imports
import hvac # HashiCorp Vault client
from google.cloud import kms_v1 # Google Cloud KMS client
from loguru import logger # Enhanced logging
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Core Functions

This section defines the essential functions for initializing our application's state, interacting with HashiCorp Vault for secrets management, and utilizing Google Cloud KMS for cryptographic operations.

Function Name: create_app_state

This function initializes the application's state as a dictionary. It sets up default configurations, including logging, and provides a structured way to manage parameters and connections throughout the notebook. The state dictionary will be passed to subsequent functions to maintain context and allow for modification.

Parameters:

  • vault_addr (Optional[str]): The address for the HashiCorp Vault server, defaults to os.getenv('VAULT_ADDR').
  • kms_project_id (Optional[str]): The Google Cloud Project ID for KMS, defaults to os.getenv('GCP_PROJECT_ID').
  • log_level (str): The logging level for the application, defaults to 'INFO'.

Returns:

  • dict: An initialized state dictionary containing configuration and client placeholders.
[26]
import sys
from loguru import logger
from typing import Dict, Any, Optional

def create_app_state(vault_addr: Optional[str] = None, kms_project_id: Optional[str] = None, log_level: str = "INFO") -> Dict[str, Any]:
    """
    Initializes the application's state dictionary.

    Parameters
    ----------
    vault_addr : Optional[str]
        The address for the HashiCorp Vault server. Defaults to VAULT_ADDR environment variable.
    kms_project_id : Optional[str]
        The Google Cloud Project ID for KMS. Defaults to GCP_PROJECT_ID environment variable.
    log_level : str
        The logging level for the application (e.g., 'INFO', 'DEBUG', 'WARNING').

    Returns
    -------
    Dict[str, Any]
        An initialized state dictionary containing configuration and client placeholders.

    Examples
    --------
    >>> state = create_app_state(vault_addr='http://localhost:8200', log_level='DEBUG')
    >>> assert 'vault_addr' in state
    >>> assert state['log_level'] == 'DEBUG'
    """
    logger.info("Initializing application state.")
    state = {
        "vault_addr": vault_addr if vault_addr else os.getenv("VAULT_ADDR"),
        "vault_token": os.getenv("VAULT_TOKEN"), # Assuming token is passed via env for simplicity in setup
        "kms_project_id": kms_project_id if kms_project_id else os.getenv("GCP_PROJECT_ID"),
        "kms_client": None,
        "vault_client": None,
        "log_level": log_level.upper(),
        "metrics": deque(maxlen=100), # For tracking performance/errors
        "retry_attempts": 0
    }
    logger.remove() # Remove default handler
    logger.add(sys.stderr, level=state["log_level"])
    logger.debug(f"Initial state created: {state}")
    return state

Function Name: connect_to_vault

This function establishes a connection to a HashiCorp Vault server using the hvac client library. It takes the current application state, which should contain the Vault address and token, and attempts to authenticate. Robust error handling with exponential backoff is included to manage transient network issues or temporary service unavailability. Upon successful connection, the Vault client is stored in the application state.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary.

Returns:

  • Dict[str, Any]: The updated state dictionary with the Vault client, or None if connection fails after retries.
[27]
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling

def connect_to_vault(state: Dict[str, Any]) -> Dict[str, Any]:
    """
    Establishes a connection to HashiCorp Vault with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'vault_addr' and 'vault_token'.

    Returns
    -------
    Dict[str, Any]
        The updated state dictionary with the 'vault_client' initialized upon successful connection.
        Returns the state with 'vault_client' as None if connection fails after retries.

    Examples
    --------
    >>> state = create_app_state(vault_addr='http://localhost:8200', vault_token='myroottoken')
    >>> updated_state = connect_to_vault(state)
    >>> assert updated_state['vault_client'] is not None
    """
    vault_addr = state.get("vault_addr")
    vault_token = state.get("vault_token")
    if not vault_addr:
        logger.error("Vault address not found in state or environment variables.")
        return state

    max_retries = 5
    base_delay = 1 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to connect to Vault at {vault_addr} (Attempt {i+1}/{max_retries})")
            client = hvac.Client(url=vault_addr, token=vault_token)
            if client.is_authenticated():
                state["vault_client"] = client
                logger.success("Successfully connected and authenticated to Vault.")
                state["retry_attempts"] = 0
                return state
            else:
                logger.warning("Vault client not authenticated. Check token or authentication method.")
        except hvac.exceptions.VaultError as e:
            logger.error(f"Vault connection error: {e}")
        except requests.exceptions.ConnectionError as e:
            logger.error(f"Network connection error to Vault: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred during Vault connection: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.5) # Exponential backoff with jitter
        logger.info(f"Retrying in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to connect to Vault after {max_retries} attempts.")
    state["vault_client"] = None # Ensure client is None if connection fails
    return state

Function Name: connect_to_kms

This function initializes a client for Google Cloud Key Management Service (KMS). It requires the Google Cloud Project ID from the application state to establish the connection. Similar to the Vault connection, it incorporates exponential backoff with jitter for retries to handle potential API rate limits or temporary service unavailability. The initialized KMS client is then stored in the application state.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary.

Returns:

  • Dict[str, Any]: The updated state dictionary with the KMS client, or None if connection fails after retries.
[28]
from typing import Dict, Any, Optional

def connect_to_kms(state: Dict[str, Any]) -> Dict[str, Any]:
    """
    Initializes a Google Cloud KMS client with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'kms_project_id'.

    Returns
    -------
    Dict[str, Any]
        The updated state dictionary with the 'kms_client' initialized upon successful connection.
        Returns the state with 'kms_client' as None if connection fails after retries.

    Examples
    --------
    >>> state = create_app_state(kms_project_id='my-gcp-project')
    >>> updated_state = connect_to_kms(state)
    >>> assert updated_state['kms_client'] is not None
    """
    kms_project_id = state.get("kms_project_id")
    if not kms_project_id:
        logger.error("GCP Project ID for KMS not found in state or environment variables.")
        return state

    max_retries = 5
    base_delay = 1 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to initialize KMS client for project {kms_project_id} (Attempt {i+1}/{max_retries})")
            client = kms_v1.KeyManagementServiceClient()
            # A simple way to verify client functionality without an actual KMS call
            # is to ensure the client object is created. More robust would be a dummy API call.
            state["kms_client"] = client
            logger.success("Successfully initialized Google Cloud KMS client.")
            state["retry_attempts"] = 0
            return state
        except Exception as e:
            logger.error(f"Error initializing Google Cloud KMS client: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.5) # Exponential backoff with jitter
        logger.info(f"Retrying in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to initialize Google Cloud KMS client after {max_retries} attempts.")
    state["kms_client"] = None # Ensure client is None if connection fails
    return state

Function Name: store_secret_in_vault

This function is responsible for securely storing a secret within HashiCorp Vault. It leverages Vault's Key-Value (KV) secrets engine. The function takes the application state (which includes the authenticated Vault client), the path where the secret should be stored, and the secret data itself. It incorporates robust error handling and retries with exponential backoff to ensure reliability even in the face of temporary network issues or Vault service fluctuations.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.
  • path (str): The path within the Vault KV secrets engine where the secret will be stored (e.g., 'kv/data/my-app/db-creds').
  • secret_data (Dict[str, Any]): A dictionary containing the secret key-value pairs to store.

Returns:

  • Dict[str, Any]: The updated state dictionary. Returns True if the secret was stored successfully, False otherwise.
[29]
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling

def store_secret_in_vault(state: Dict[str, Any], path: str, secret_data: Dict[str, Any]) -> bool:
    """
    Stores a secret in HashiCorp Vault's KV secrets engine with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'vault_client'.
    path : str
        The path within the Vault KV secrets engine (e.g., 'kv/data/my-app/db-creds').
    secret_data : Dict[str, Any]
        A dictionary containing the secret key-value pairs to store.

    Returns
    -------
    bool
        True if the secret was stored successfully, False otherwise.

    Examples
    --------
    >>> # Assume 'state' has an authenticated vault_client
    >>> # success = store_secret_in_vault(state, 'kv/data/my-app/test-secret', {'api_key': '12345', 'user': 'testuser'})
    >>> # assert success is True
    """
    vault_client = state.get("vault_client")
    if not vault_client:
        logger.error("Vault client not initialized. Cannot store secret.")
        return False

    max_retries = 3
    base_delay = 0.5 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to store secret at '{path}' in Vault (Attempt {i+1}/{max_retries})")
            vault_client.secrets.kv.v2.create_or_update_secret(path=path, secret=secret_data)
            logger.success(f"Secret successfully stored at '{path}' in Vault.")
            state["retry_attempts"] = 0
            return True
        except hvac.exceptions.VaultError as e:
            logger.error(f"Vault error while storing secret at '{path}': {e}")
        except requests.exceptions.ConnectionError as e:
            logger.error(f"Network error while connecting to Vault: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred while storing secret: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
        logger.warning(f"Retrying secret storage in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to store secret at '{path}' after {max_retries} attempts.")
    return False

Function Name: get_secret_from_vault

This function retrieves a secret from HashiCorp Vault's KV secrets engine. Given the application state and the path to the secret, it attempts to fetch the latest version of the secret data. Like other network-dependent functions, it includes an exponential backoff retry mechanism to enhance resilience against transient failures. If successful, it returns the secret data; otherwise, it returns None.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.
  • path (str): The path within the Vault KV secrets engine where the secret is stored (e.g., 'kv/data/my-app/db-creds').

Returns:

  • Optional[Dict[str, Any]]: The secret data if retrieved successfully, otherwise None.
[30]
from typing import Dict, Any, Optional

def get_secret_from_vault(state: Dict[str, Any], path: str) -> Optional[Dict[str, Any]]:
    """
    Retrieves a secret from HashiCorp Vault's KV secrets engine with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'vault_client'.
    path : str
        The path within the Vault KV secrets engine where the secret is stored.

    Returns
    -------
    Optional[Dict[str, Any]]
        The secret data if retrieved successfully, otherwise None.

    Examples
    --------
    >>> # Assume 'state' has an authenticated vault_client and a secret at 'kv/data/my-app/test-secret'
    >>> # secret = get_secret_from_vault(state, 'kv/data/my-app/test-secret')
    >>> # if secret: print(secret)
    """
    vault_client = state.get("vault_client")
    if not vault_client:
        logger.error("Vault client not initialized. Cannot retrieve secret.")
        return None

    max_retries = 3
    base_delay = 0.5 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to retrieve secret from '{path}' in Vault (Attempt {i+1}/{max_retries})")
            read_response = vault_client.secrets.kv.v2.read_secret_version(path=path)
            secret_data = read_response['data']['data']
            logger.success(f"Secret successfully retrieved from '{path}' in Vault.")
            state["retry_attempts"] = 0
            return secret_data
        except hvac.exceptions.VaultError as e:
            if "No key exists at" in str(e): # Specific check for secret not found
                logger.warning(f"Secret not found at path '{path}' in Vault: {e}")
                return None
            logger.error(f"Vault error while retrieving secret from '{path}': {e}")
        except requests.exceptions.ConnectionError as e:
            logger.error(f"Network error while connecting to Vault: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred while retrieving secret: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
        logger.warning(f"Retrying secret retrieval in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to retrieve secret from '{path}' after {max_retries} attempts.")
    return None

Function Name: encrypt_with_kms

This function uses Google Cloud KMS to encrypt a plaintext string. It requires the application state (containing the KMS client) and the full resource name of the KMS key to be used for encryption. The plaintext is first encoded to bytes, then sent to KMS for encryption. The resulting ciphertext is returned as a base64-encoded string, suitable for storage or transmission. Includes exponential backoff for retries.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, expected to contain a 'kms_client'.
  • key_name (str): The full resource name of the KMS key (e.g., 'projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID').
  • plaintext (str): The data to be encrypted.

Returns:

  • Optional[str]: The base64-encoded ciphertext if encryption is successful, otherwise None.
[31]
from typing import Dict, Any, Optional
import base64
from google.api_core.exceptions import GoogleAPIError

def encrypt_with_kms(state: Dict[str, Any], key_name: str, plaintext: str) -> Optional[str]:
    """
    Encrypts plaintext data using Google Cloud KMS with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'kms_client'.
    key_name : str
        The full resource name of the KMS key.
    plaintext : str
        The data to be encrypted.

    Returns
    -------
    Optional[str]
        The base64-encoded ciphertext if encryption is successful, otherwise None.

    Examples
    --------
    >>> # Assume 'state' has an initialized kms_client
    >>> # key_name = 'projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key'
    >>> # encrypted_text = encrypt_with_kms(state, key_name, 'sensitive data')
    >>> # if encrypted_text: print(f"Encrypted: {encrypted_text}")
    """
    kms_client = state.get("kms_client")
    if not kms_client:
        logger.error("KMS client not initialized. Cannot encrypt data.")
        return None

    plaintext_bytes = plaintext.encode('utf-8')

    max_retries = 3
    base_delay = 0.5 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to encrypt data with KMS key '{key_name}' (Attempt {i+1}/{max_retries})")
            encrypt_response = kms_client.encrypt(request={'name': key_name, 'plaintext': plaintext_bytes})
            ciphertext = base64.b64encode(encrypt_response.ciphertext).decode('utf-8')
            logger.success("Data successfully encrypted with KMS.")
            state["retry_attempts"] = 0
            return ciphertext
        except GoogleAPIError as e:
            logger.error(f"Google Cloud KMS API error during encryption: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred during KMS encryption: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
        logger.warning(f"Retrying KMS encryption in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to encrypt data with KMS key '{key_name}' after {max_retries} attempts.")
    return None

Function Name: decrypt_with_kms

This function utilizes Google Cloud KMS to decrypt a base64-encoded ciphertext. It requires the application state (with an initialized KMS client), the full resource name of the KMS key used for encryption, and the ciphertext. The ciphertext is first base64-decoded, then sent to KMS for decryption. The decrypted plaintext is returned as a string. It also includes an exponential backoff mechanism for retries to ensure robustness.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, expected to contain a 'kms_client'.
  • key_name (str): The full resource name of the KMS key used for decryption.
  • ciphertext (str): The base64-encoded ciphertext to be decrypted.

Returns:

  • Optional[str]: The decrypted plaintext string if decryption is successful, otherwise None.
[32]
from typing import Dict, Any, Optional

def decrypt_with_kms(state: Dict[str, Any], key_name: str, ciphertext: str) -> Optional[str]:
    """
    Decrypts ciphertext data using Google Cloud KMS with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'kms_client'.
    key_name : str
        The full resource name of the KMS key used for decryption.
    ciphertext : str
        The base64-encoded ciphertext to be decrypted.

    Returns
    -------
    Optional[str]
        The decrypted plaintext string if decryption is successful, otherwise None.

    Examples
    --------
    >>> # Assume 'state' has an initialized kms_client and 'encrypted_text'
    >>> # key_name = 'projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key'
    >>> # decrypted_text = decrypt_with_kms(state, key_name, encrypted_text)
    >>> # if decrypted_text: print(f"Decrypted: {decrypted_text}")
    """
    kms_client = state.get("kms_client")
    if not kms_client:
        logger.error("KMS client not initialized. Cannot decrypt data.")
        return None

    try:
        ciphertext_bytes = base64.b64decode(ciphertext)
    except Exception as e:
        logger.error(f"Error base64 decoding ciphertext: {e}")
        return None

    max_retries = 3
    base_delay = 0.5 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to decrypt data with KMS key '{key_name}' (Attempt {i+1}/{max_retries})")
            decrypt_response = kms_client.decrypt(request={'name': key_name, 'ciphertext': ciphertext_bytes})
            plaintext = decrypt_response.plaintext.decode('utf-8')
            logger.success("Data successfully decrypted with KMS.")
            state["retry_attempts"] = 0
            return plaintext
        except GoogleAPIError as e:
            logger.error(f"Google Cloud KMS API error during decryption: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred during KMS decryption: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
        logger.warning(f"Retrying KMS decryption in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to decrypt data with KMS key '{key_name}' after {max_retries} attempts.")
    return None

Local Alternative: Vault Transit Encryption

Since we are working locally, we can use Vault's Transit engine to perform cryptographic operations without external cloud providers.

[39]
def encrypt_with_vault_transit(state: Dict[str, Any], key_name: str, plaintext: str) -> Optional[str]:
    """Encrypts data using Vault's Transit engine."""
    client = state.get("vault_client")
    if not client:
        return None

    try:
        # Vault expects base64 encoded plaintext for Transit
        import base64
        encoded_plaintext = base64.b64encode(plaintext.encode('utf-8')).decode('utf-8')
        response = client.secrets.transit.encrypt_data(name=key_name, plaintext=encoded_plaintext)
        return response['data']['ciphertext']
    except Exception as e:
        logger.error(f"Vault Transit encryption error: {e}")
        return None

def decrypt_with_vault_transit(state: Dict[str, Any], key_name: str, ciphertext: str) -> Optional[str]:
    """Decrypts data using Vault's Transit engine."""
    client = state.get("vault_client")
    if not client:
        return None

    try:
        response = client.secrets.transit.decrypt_data(name=key_name, ciphertext=ciphertext)
        import base64
        return base64.b64decode(response['data']['plaintext']).decode('utf-8')
    except Exception as e:
        logger.error(f"Vault Transit decryption error: {e}")
        return None

Function Name: track_metric

This utility function is designed to record various operational metrics throughout the application's lifecycle. It stores metrics in a deque (double-ended queue) within the application state, allowing for a rolling window of recent activities. This is crucial for monitoring performance, error rates, and retry attempts, which can later be visualized to understand system behavior.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, containing the metrics deque.
  • metric_name (str): The name of the metric to track (e.g., 'vault_read_success', 'kms_encrypt_failure').
  • value (Any): The value associated with the metric.

Returns:

  • Dict[str, Any]: The updated state dictionary.
[33]
from typing import Dict, Any
import time

def track_metric(state: Dict[str, Any], metric_name: str, value: Any) -> Dict[str, Any]:
    """
    Tracks an operational metric in the application state's metrics deque.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain a 'metrics' deque.
    metric_name : str
        The name of the metric to track.
    value : Any
        The value associated with the metric.

    Returns
    -------
    Dict[str, Any]
        The updated state dictionary.

    Examples
    --------
    >>> state = {'metrics': deque(maxlen=10)}
    >>> state = track_metric(state, 'vault_read_latency', 0.05)
    >>> assert len(state['metrics']) == 1
    >>> assert state['metrics'][0]['name'] == 'vault_read_latency'
    """
    timestamp = time.time()
    state['metrics'].append({'timestamp': timestamp, 'name': metric_name, 'value': value})
    logger.debug(f"Metric tracked: {metric_name} = {value}")
    return state

Function Name: delete_secret_from_vault

This function removes a secret from HashiCorp Vault's KV secrets engine at a specified path. It's important for secret lifecycle management, allowing for cleanup of old or deprecated secrets. The function includes retry logic with exponential backoff to handle potential transient issues during the deletion process. It logs the outcome of the deletion attempt.

Parameters:

  • state (Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.
  • path (str): The path within the Vault KV secrets engine where the secret is to be deleted (e.g., 'kv/data/my-app/old-creds').

Returns:

  • bool: True if the secret was successfully deleted, False otherwise.
[34]
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling

def delete_secret_from_vault(state: Dict[str, Any], path: str) -> bool:
    """
    Deletes a secret from HashiCorp Vault's KV secrets engine with retry logic.

    Parameters
    ----------
    state : Dict[str, Any]
        The current application state dictionary, expected to contain 'vault_client'.
    path : str
        The path within the Vault KV secrets engine where the secret is to be deleted.

    Returns
    -------
    bool
        True if the secret was successfully deleted, False otherwise.

    Examples
    --------
    >>> # Assume 'state' has an authenticated vault_client and a secret at 'kv/data/my-app/temp-secret'
    >>> # success = delete_secret_from_vault(state, 'kv/data/my-app/temp-secret')
    >>> # assert success is True
    """
    vault_client = state.get("vault_client")
    if not vault_client:
        logger.error("Vault client not initialized. Cannot delete secret.")
        return False

    max_retries = 3
    base_delay = 0.5 # seconds
    for i in range(max_retries):
        try:
            logger.info(f"Attempting to delete secret from '{path}' in Vault (Attempt {i+1}/{max_retries})")
            vault_client.secrets.kv.v2.delete_latest_version_of_secret(path=path)
            logger.success(f"Secret successfully deleted from '{path}' in Vault.")
            state["retry_attempts"] = 0
            return True
        except hvac.exceptions.VaultError as e:
            if "Code: 404" in str(e): # Secret not found
                logger.warning(f"Secret at '{path}' not found or already deleted: {e}")
                return True # Consider it successful if it's already gone
            logger.error(f"Vault error while deleting secret from '{path}': {e}")
        except requests.exceptions.ConnectionError as e:
            logger.error(f"Network error while connecting to Vault: {e}")
        except Exception as e:
            logger.error(f"An unexpected error occurred while deleting secret: {e}")

        delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
        logger.warning(f"Retrying secret deletion in {delay:.2f} seconds...")
        time.sleep(delay)
        state["retry_attempts"] += 1

    logger.error(f"Failed to delete secret from '{path}' after {max_retries} attempts.")
    return False

Demonstration and Visualization

This section demonstrates the functionality of the secrets management functions. We will simulate an environment where secrets are stored in HashiCorp Vault and sensitive data is encrypted using Google Cloud KMS. We'll also visualize operational metrics like latency and retry attempts.

To run this section, ensure you have:

  1. A running HashiCorp Vault instance (e.g., vault server -dev).
  2. A Vault token with appropriate permissions (e.g., export VAULT_TOKEN='s.xxxxxx' and export VAULT_ADDR='http://127.0.0.1:8200' for dev server).
  3. Google Cloud project credentials configured (e.g., gcloud auth application-default login).
  4. A Google Cloud KMS Key Ring and CryptoKey created (e.g., in global or a specific region).

1. Initialize State and Connect to Services

First, we'll set up our simulated environment variables and initialize the application state, then attempt to connect to both Vault and KMS. The logger statements will provide real-time feedback on the connection status.

[35]
#@title Initialize App State and Connect to Vault/KMS
import os # Added import
from loguru import logger # Added import

# Simulate environment variables (replace with your actual values for a real setup)
os.environ['VAULT_ADDR'] = os.getenv('VAULT_ADDR', 'http://127.0.0.1:8200') # Default to dev server
os.environ['VAULT_TOKEN'] = os.getenv('VAULT_TOKEN', 'myroottoken') # Replace with your dev token or real token
os.environ['GCP_PROJECT_ID'] = os.getenv('GCP_PROJECT_ID', 'your-gcp-project-id') # Replace with your GCP Project ID

# Initialize application state
app_state = create_app_state(log_level='DEBUG')
logger.info(f"Application state initialized with log level: {app_state['log_level']}")

# Connect to Vault
app_state = connect_to_vault(app_state)

# Connect to KMS
app_state = connect_to_kms(app_state)

# Display connection status
print("\n--- Connection Status ---")
print(f"Vault Client Connected: {app_state['vault_client'] is not None}")
print(f"KMS Client Connected: {app_state['kms_client'] is not None}")
2026-06-10 09:33:53.299 | INFO     | __main__:create_app_state:29 - Initializing application state.
2026-06-10 09:33:53.326 | DEBUG    | __main__:create_app_state:42 - Initial state created: {'vault_addr': 'http://127.0.0.1:8200', 'vault_token': 'myroottoken', 'kms_project_id': 'your-gcp-project-id', 'kms_client': None, 'vault_client': None, 'log_level': 'DEBUG', 'metrics': deque([], maxlen=100), 'retry_attempts': 0}
2026-06-10 09:33:53.331 | INFO     | __main__:<cell line: 0>:12 - Application state initialized with log level: DEBUG
2026-06-10 09:33:53.336 | INFO     | __main__:connect_to_vault:35 - Attempting to connect to Vault at http://127.0.0.1:8200 (Attempt 1/5)
2026-06-10 09:33:53.348 | SUCCESS  | __main__:connect_to_vault:39 - Successfully connected and authenticated to Vault.
2026-06-10 09:33:53.355 | INFO     | __main__:connect_to_kms:33 - Attempting to initialize KMS client for project your-gcp-project-id (Attempt 1/5)
2026-06-10 09:33:53.384 | SUCCESS  | __main__:connect_to_kms:38 - Successfully initialized Google Cloud KMS client.

--- Connection Status ---
Vault Client Connected: True
KMS Client Connected: True

2. HashiCorp Vault: Store, Retrieve, and Delete Secrets

We'll demonstrate the full lifecycle of a secret in Vault: creating it, reading it back, and then deleting it. We'll also track metrics for these operations.

[36]
#@title Vault Secret Operations
import time # Added import
from loguru import logger # Added import

# Define a test secret
VAULT_TEST_PATH = 'kv/data/my-app/test-db-creds'
TEST_SECRET_DATA = {
    'username': 'app_user',
    'password': 'super_secret_password_123',
    'connection_string': 'jdbc:postgresql://db.example.com:5432/production'
}

# 1. Store a secret
logger.info(f"\n--- Storing secret at {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
store_success = store_secret_in_vault(app_state, VAULT_TEST_PATH, TEST_SECRET_DATA)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_store_latency_ms', latency)
app_state = track_metric(app_state, 'vault_store_success', int(store_success))

print(f"Secret store successful: {store_success} (Latency: {latency:.2f} ms)")

# 2. Retrieve the secret
logger.info(f"\n--- Retrieving secret from {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
retrieved_secret = get_secret_from_vault(app_state, VAULT_TEST_PATH)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_retrieve_latency_ms', latency)
app_state = track_metric(app_state, 'vault_retrieve_success', int(retrieved_secret is not None))

if retrieved_secret:
    print("Secret retrieved successfully:")
    for k, v in retrieved_secret.items():
        print(f"  {k}: {'*' * len(v) if k == 'password' else v}") # Mask password
else:
    print("Failed to retrieve secret.")

# 3. Attempt to retrieve a non-existent secret (edge case)
logger.info(f"\n--- Attempting to retrieve non-existent secret ---")
non_existent_secret = get_secret_from_vault(app_state, 'kv/data/my-app/non-existent-creds')
print(f"Non-existent secret retrieval result: {non_existent_secret}")
app_state = track_metric(app_state, 'vault_retrieve_non_existent_success', int(non_existent_secret is not None))

# 4. Delete the secret
logger.info(f"\n--- Deleting secret from {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
del_success = delete_secret_from_vault(app_state, VAULT_TEST_PATH)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_delete_latency_ms', latency)
app_state = track_metric(app_state, 'vault_delete_success', int(del_success))

print(f"Secret deletion successful: {del_success} (Latency: {latency:.2f} ms)")

# 5. Verify deletion
logger.info(f"\n--- Verifying secret deletion from {VAULT_TEST_PATH} ---")
verified_deleted = get_secret_from_vault(app_state, VAULT_TEST_PATH)
print(f"Verification after deletion: {verified_deleted is None}")
2026-06-10 09:33:53.418 | INFO     | __main__:<cell line: 0>:14 - 
--- Storing secret at kv/data/my-app/test-db-creds ---
2026-06-10 09:33:53.419 | INFO     | __main__:store_secret_in_vault:37 - Attempting to store secret at 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)
2026-06-10 09:33:53.427 | SUCCESS  | __main__:store_secret_in_vault:39 - Secret successfully stored at 'kv/data/my-app/test-db-creds' in Vault.
2026-06-10 09:33:53.429 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_store_latency_ms = 9.38200600012351
2026-06-10 09:33:53.431 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_store_success = 1
2026-06-10 09:33:53.432 | INFO     | __main__:<cell line: 0>:25 - 
--- Retrieving secret from kv/data/my-app/test-db-creds ---
2026-06-10 09:33:53.434 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)
/tmp/ipykernel_10995/394795625.py:35: DeprecationWarning: The raise_on_deleted_version parameter will change its default value to False in hvac v3.0.0. The current default of True will preserve previous behavior. To use the old behavior with no warning, explicitly set this value to True. See https://github.com/hvac/hvac/pull/907
  read_response = vault_client.secrets.kv.v2.read_secret_version(path=path)
2026-06-10 09:33:53.440 | SUCCESS  | __main__:get_secret_from_vault:37 - Secret successfully retrieved from 'kv/data/my-app/test-db-creds' in Vault.
2026-06-10 09:33:53.442 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_retrieve_latency_ms = 7.942938000269351
2026-06-10 09:33:53.444 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_retrieve_success = 1
2026-06-10 09:33:53.447 | INFO     | __main__:<cell line: 0>:41 - 
--- Attempting to retrieve non-existent secret ---
2026-06-10 09:33:53.450 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 1/3)
2026-06-10 09:33:53.457 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds
2026-06-10 09:33:53.458 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 0.53 seconds...
Secret store successful: True (Latency: 9.38 ms)
Secret retrieved successfully:
  connection_string: jdbc:postgresql://db.example.com:5432/production
  password: *************************
  username: app_user
2026-06-10 09:33:53.987 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 2/3)
2026-06-10 09:33:53.993 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds
2026-06-10 09:33:53.994 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 1.07 seconds...
2026-06-10 09:33:55.061 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 3/3)
2026-06-10 09:33:55.066 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds
2026-06-10 09:33:55.068 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 2.06 seconds...
2026-06-10 09:33:57.129 | ERROR    | __main__:get_secret_from_vault:55 - Failed to retrieve secret from 'kv/data/my-app/non-existent-creds' after 3 attempts.
2026-06-10 09:33:57.131 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_retrieve_non_existent_success = 0
2026-06-10 09:33:57.134 | INFO     | __main__:<cell line: 0>:47 - 
--- Deleting secret from kv/data/my-app/test-db-creds ---
2026-06-10 09:33:57.135 | INFO     | __main__:delete_secret_from_vault:35 - Attempting to delete secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)
2026-06-10 09:33:57.153 | SUCCESS  | __main__:delete_secret_from_vault:37 - Secret successfully deleted from 'kv/data/my-app/test-db-creds' in Vault.
2026-06-10 09:33:57.163 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_delete_latency_ms = 27.20103400042717
2026-06-10 09:33:57.171 | DEBUG    | __main__:track_metric:31 - Metric tracked: vault_delete_success = 1
2026-06-10 09:33:57.174 | INFO     | __main__:<cell line: 0>:58 - 
--- Verifying secret deletion from kv/data/my-app/test-db-creds ---
2026-06-10 09:33:57.179 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)
2026-06-10 09:33:57.198 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"d31fee39-c793-f8e3-c2a3-57ecee24ff70","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds
2026-06-10 09:33:57.203 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 0.68 seconds...
Non-existent secret retrieval result: None
Secret deletion successful: True (Latency: 27.20 ms)
2026-06-10 09:33:57.886 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 2/3)
2026-06-10 09:33:57.891 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"cbb73ae6-d538-a72b-7a37-d7a3900a1093","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds
2026-06-10 09:33:57.893 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 1.04 seconds...
2026-06-10 09:33:58.934 | INFO     | __main__:get_secret_from_vault:34 - Attempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 3/3)
2026-06-10 09:33:58.941 | ERROR    | __main__:get_secret_from_vault:44 - Vault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"65f06ed2-a529-f0e2-2908-326ef6cfe4e1","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds
2026-06-10 09:33:58.947 | WARNING  | __main__:get_secret_from_vault:51 - Retrying secret retrieval in 2.17 seconds...
2026-06-10 09:34:01.119 | ERROR    | __main__:get_secret_from_vault:55 - Failed to retrieve secret from 'kv/data/my-app/test-db-creds' after 3 attempts.
Verification after deletion: True

Local Encryption Demo (Vault Transit)

This replaces the Google KMS demo with a local version using the Vault dev server.

[43]
LOCAL_KEY_NAME = 'my-key'
PLAINTEXT = 'This is local sensitive data.'

# Perform local encryption
logger.info("--- Starting Local Transit Encryption ---")
ciphertext = encrypt_with_vault_transit(app_state, LOCAL_KEY_NAME, PLAINTEXT)

if ciphertext:
    print(f"Local Ciphertext: {ciphertext}")

    # Track metrics for the local operation
    app_state = track_metric(app_state, 'local_transit_encrypt_success', 1)

    # Perform local decryption
    decrypted = decrypt_with_vault_transit(app_state, LOCAL_KEY_NAME, ciphertext)

    if decrypted:
        print(f"Decrypted: {decrypted}")
        print(f"Original matches decrypted: {PLAINTEXT == decrypted}")
        app_state = track_metric(app_state, 'local_transit_decrypt_success', 1)
    else:
        print("Local decryption failed.")
else:
    print("Local encryption failed.")
2026-06-10 09:38:07.236 | INFO     | __main__:<cell line: 0>:5 - --- Starting Local Transit Encryption ---
2026-06-10 09:38:07.247 | DEBUG    | __main__:track_metric:31 - Metric tracked: local_transit_encrypt_success = 1
2026-06-10 09:38:07.253 | DEBUG    | __main__:track_metric:31 - Metric tracked: local_transit_decrypt_success = 1
Local Ciphertext: vault:v1:nTUL3XprrBsdpNWQNXyY9hrijxv+lVCcSKo5YgcjUTxGXky/St+rFINeBYQYd+gOCVlmbnYYP1O6
Decrypted: This is local sensitive data.
Original matches decrypted: True

4. Visualization of Operational Metrics

We'll use the tracked metrics to visualize the latency and success rates of our Vault and KMS operations. This helps in understanding the performance and reliability of the secrets management system.

[45]
#@title Plotting Metrics
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display

# Convert metrics deque to a Pandas DataFrame
metrics_df = pd.DataFrame(list(app_state['metrics']))

if not metrics_df.empty:
    metrics_df['timestamp'] = pd.to_datetime(metrics_df['timestamp'], unit='s')

    print("\n--- Summary of All Tracked Metrics ---")
    display(metrics_df)

    # Filter for success/failure metrics including the new local transit metrics
    success_metrics = metrics_df[metrics_df['name'].str.contains('_success')]
    if not success_metrics.empty:
        plt.figure(figsize=(10, 5))
        # Updated to resolve FutureWarning by assigning x to hue and setting legend=False
        sns.barplot(data=success_metrics, x='name', y='value', hue='name', legend=False, palette='viridis')
        plt.title('Vault Operation Success Indicators')
        plt.xlabel('Operation Type')
        plt.ylabel('Success (1=True, 0=False)')
        plt.xticks(rotation=45, ha='right')
        plt.ylim(0, 1.1)
        plt.grid(axis='y', linestyle='--', alpha=0.7)
        plt.tight_layout()
        plt.show()
    else:
        print("No success metrics to plot.")

    # Filter for latency metrics
    latency_metrics = metrics_df[metrics_df['name'].str.contains('_latency_ms')]
    if not latency_metrics.empty:
        plt.figure(figsize=(10, 5))
        sns.lineplot(data=latency_metrics, x='timestamp', y='value', hue='name', marker='o')
        plt.title('Vault Operation Latencies')
        plt.xlabel('Time')
        plt.ylabel('Latency (ms)')
        plt.grid(True)
        plt.tight_layout()
        plt.show()
else:
    print("No metrics have been tracked yet.")

--- Summary of All Tracked Metrics ---
timestamp name value
0 2026-06-10 09:33:53.429040194 vault_store_latency_ms 9.382006
1 2026-06-10 09:33:53.431044102 vault_store_success 1.000000
2 2026-06-10 09:33:53.442732334 vault_retrieve_latency_ms 7.942938
3 2026-06-10 09:33:53.444972515 vault_retrieve_success 1.000000
4 2026-06-10 09:33:57.131113291 vault_retrieve_non_existent_success 0.000000
5 2026-06-10 09:33:57.163095951 vault_delete_latency_ms 27.201034
6 2026-06-10 09:33:57.171158791 vault_delete_success 1.000000
7 2026-06-10 09:38:07.247468472 local_transit_encrypt_success 1.000000
8 2026-06-10 09:38:07.253608465 local_transit_decrypt_success 1.000000
cell output
cell output

Production Considerations

Deploying secrets management solutions like Vault and KMS in production requires careful planning and adherence to best practices to ensure security, reliability, and maintainability. This section outlines key considerations.

Best Practices for Secrets Management in Production

AspectBest PracticeRationale
AuthenticationUse strong, machine-oriented authentication methods (e.g., IAM roles for AWS/GCP, Kubernetes service accounts, AppRole for Vault, client certificates) instead of hardcoded tokens.Reduces the risk of static credential exposure and simplifies rotation.
Authorization (Least Privilege)Implement fine-grained access control policies (ACLs in Vault, IAM policies in GCP KMS) that grant only the minimum necessary permissions to secrets and keys.Minimizes the blast radius in case of compromise.
Auditing & MonitoringEnable comprehensive audit logging for all secret access and modification events. Integrate logs with SIEM (Security Information and Event Management) systems for real-time alerting and analysis.Provides a forensic trail, helps detect anomalous behavior, and ensures compliance.
Secret RotationAutomate secret rotation wherever possible. For Vault, utilize dynamic secrets; for static secrets, implement periodic rotation (e.g., every 90 days). KMS keys should also be rotated according to policy.Limits the window of exposure for compromised secrets and reduces operational overhead.
High AvailabilityDeploy secrets management systems in a highly available configuration (e.g., Vault in HA mode with integrated storage or Consul, KMS is inherently highly available by GCP).Ensures continuous access to critical secrets even during outages of individual components.
Disaster RecoveryImplement robust backup and disaster recovery procedures for your secrets management infrastructure and the secrets themselves. Test these procedures regularly.Guarantees business continuity and recovery from catastrophic data loss or system failure.
Network SecurityRestrict network access to secrets management systems using firewalls, VPC private access, and dedicated subnets. Communicate over TLS/SSL and ensure certificate validation.Prevents unauthorized network access and protects data in transit.
Key Management StrategyDefine a clear strategy for managing cryptographic keys, including key hierarchy, lifecycle, and revocation policies. Use hardware security modules (HSMs) where regulatory compliance or highest security is required.Establishes a secure foundation for all cryptographic operations and data protection.
Secure Application IntegrationUse official client libraries and SDKs. Avoid embedding secrets directly in application code or configuration files. Retrieve secrets at runtime from Vault/KMS.Prevents accidental exposure of secrets in source code repositories or build artifacts.
TestingRegularly test your secrets management setup, including access patterns, rotation mechanisms, and disaster recovery scenarios. Conduct penetration testing and security audits.Identifies vulnerabilities and ensures the system functions as expected under various conditions.

Conclusion

This notebook has provided a comprehensive overview and practical demonstration of infrastructure security best practices concerning the storage and management of secrets using HashiCorp Vault and Google Cloud Key Management Service (KMS). We've explored:

  • Core Functions: Implemented modular functions for initializing application state, connecting to Vault and KMS, and performing essential operations like storing, retrieving, encrypting, and decrypting sensitive data.
  • Robustness: Ensured reliability through the incorporation of exponential backoff and jitter for retry mechanisms in network-dependent operations, enhancing resilience against transient failures.
  • Demonstration: Illustrated the end-to-end lifecycle of a secret in Vault and a sensitive data encryption/decryption flow with KMS, complete with simulated environment variables and real-time logging.
  • Visualization: Utilized operational metrics to visualize the performance (latency) and outcomes (success/failure) of various secrets management actions, providing insights into system health and behavior.
  • Production Considerations: Outlined a set of critical best practices for deploying and managing these solutions in a production environment, emphasizing authentication, authorization, auditing, rotation, high availability, disaster recovery, and network security.

By following these principles and leveraging dedicated tools like Vault and KMS, organizations can significantly enhance their security posture, protect sensitive information from compromise, and comply with regulatory requirements. The modular and well-documented approach presented in this notebook serves as a foundation for building secure and scalable applications.