Secrets Vault Setup
Securely store exchange API credentials and application secrets in a dedicated secrets management vault using HashiCorp Vault or cloud provider KMS with fine-grained access control policies, comprehensive audit logging of all secret access, and encryption both at rest and in transit for defense-in-depth security architecture.
Infrastructure Security: Storing Secrets in Vault/KMS
This notebook explores best practices for securely managing sensitive information within an application's infrastructure. We will focus on using dedicated secrets management systems like HashiCorp Vault or cloud-based Key Management Services (KMS) to store, access, and rotate secrets, thereby reducing the risk of exposure.
Key Concepts in Secrets Management
| Concept | Description | Why it's Important |
|---|---|---|
| Secret | Any piece of sensitive information (API keys, database credentials, certificates, etc.) | Protects critical system access and data. |
| Secrets Engine | A component within a secrets management system that handles secrets for specific types (e.g., KV, database) | Provides structured storage and retrieval for different secret types. |
| Leasing | Secrets are granted for a limited time and must be renewed or re-requested | Limits exposure time, automatically revokes access if not renewed. |
| Revocation | Ability to invalidate a secret or access token immediately | Critical for incident response and mitigating breaches. |
| Auditing | Logging all secret access and modification events | Ensures accountability and aids in forensic analysis. |
| Dynamic Secrets | Secrets generated on demand for a short lifespan, never stored | Drastically reduces secret exposure, as secrets are never at rest for long periods. |
| Encryption | Protecting data at rest and in transit using cryptographic techniques | Ensures confidentiality and integrity of secrets even if storage is compromised. |
| Access Control | Policies defining who can access which secrets under what conditions | Enforces least privilege, preventing unauthorized access. |
| Rotation | Periodically changing secrets | Reduces the impact of a compromised secret over time. |
Resources
#@title Dependency Installation
# Install necessary libraries
%pip install hvac google-cloud-kms loguru pandas matplotlib seabornRequirement already satisfied: hvac in /usr/local/lib/python3.12/dist-packages (2.4.0) Requirement already satisfied: google-cloud-kms in /usr/local/lib/python3.12/dist-packages (3.13.0) Requirement already satisfied: loguru in /usr/local/lib/python3.12/dist-packages (0.7.3) Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2) Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0) Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2) Requirement already satisfied: requests<3.0.0,>=2.27.1 in /usr/local/lib/python3.12/dist-packages (from hvac) (2.32.4) Requirement already satisfied: google-api-core<3.0.0,>=2.11.0 in /usr/local/lib/python3.12/dist-packages (from google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (2.30.3) Requirement already satisfied: google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (2.47.0) Requirement already satisfied: grpcio<2.0.0,>=1.33.2 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (1.81.0) Requirement already satisfied: proto-plus<2.0.0,>=1.22.3 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (1.28.0) Requirement already satisfied: protobuf<8.0.0,>=4.25.8 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (5.29.6) Requirement already satisfied: grpc-google-iam-v1<1.0.0,>=0.14.0 in /usr/local/lib/python3.12/dist-packages (from google-cloud-kms) (0.14.4) Requirement already satisfied: numpy>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.0.2) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.2) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.63.0) Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.2) Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2) Requirement already satisfied: googleapis-common-protos<2.0.0,>=1.63.2 in /usr/local/lib/python3.12/dist-packages (from google-api-core<3.0.0,>=2.11.0->google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (1.75.0) Requirement already satisfied: grpcio-status<2.0.0,>=1.33.2 in /usr/local/lib/python3.12/dist-packages (from google-api-core[grpc]<3.0.0,>=2.11.0->google-cloud-kms) (1.71.2) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.12/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (0.4.2) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.12/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (4.9.1) Requirement already satisfied: typing-extensions~=4.12 in /usr/local/lib/python3.12/dist-packages (from grpcio<2.0.0,>=1.33.2->google-cloud-kms) (4.15.0) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0) Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (3.4.7) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (3.18) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (2.5.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.27.1->hvac) (2026.5.20) Requirement already satisfied: pyasn1<0.7.0,>=0.6.1 in /usr/local/lib/python3.12/dist-packages (from pyasn1-modules>=0.2.1->google-auth!=2.24.0,!=2.25.0,<3.0.0,>=2.14.1->google-cloud-kms) (0.6.3)
import os
import time
import subprocess
# 1. Download standalone Vault binary if not present
if not os.path.exists('./vault'):
print("Downloading Vault binary...")
!wget https://releases.hashicorp.com/vault/1.15.4/vault_1.15.4_linux_amd64.zip
!unzip -o vault_1.15.4_linux_amd64.zip
!chmod +x vault
# 2. Start Vault in dev mode in the background
print("Starting Vault dev server...")
!pkill vault || true
time.sleep(2)
get_ipython().system_raw('./vault server -dev -dev-root-token-id="myroottoken" > vault.log 2>&1 &')
# Wait for Vault to be ready
print("Waiting for Vault to be ready...")
for i in range(10):
try:
env = os.environ.copy()
env['VAULT_ADDR'] = 'http://127.0.0.1:8200'
result = subprocess.run(['./vault', 'status'], env=env, capture_output=True)
if result.returncode in [0, 2]:
print("Vault is ready!")
break
except:
pass
time.sleep(1)
# 3. Configure environment for hvac and CLI
os.environ['VAULT_ADDR'] = 'http://127.0.0.1:8200'
os.environ['VAULT_TOKEN'] = 'myroottoken'
# 4. Enable engines
print("Enabling KV and Transit engines...")
!./vault secrets enable -path=kv kv-v2 || true
!./vault secrets enable transit || true
!./vault write -f transit/keys/my-key || true
print("Vault is configured and running at http://127.0.0.1:8200")Starting Vault dev server... Waiting for Vault to be ready... Vault is ready! Enabling KV and Transit engines... [0mSuccess! Enabled the kv-v2 secrets engine at: kv/[0m [0mSuccess! Enabled the transit secrets engine at: transit/[0m [0mKey Value --- ----- allow_plaintext_backup false auto_rotate_period 0s deletion_allowed false derived false exportable false imported_key false keys map[1:1781084272] latest_version 1 min_available_version 0 min_decryption_version 1 min_encryption_version 0 name my-key supports_decryption true supports_derivation true supports_encryption true supports_signing false type aes256-gcm96[0m Vault is configured and running at http://127.0.0.1:8200
# Enable the Transit secrets engine for local encryption/decryption
print("Enabling Transit secrets engine...")
!./vault secrets enable transit || echo "Transit engine might already be enabled"
# Create a named encryption key
print("Creating local encryption key 'my-key'...")
!./vault write -f transit/keys/my-keyEnabling Transit secrets engine... [91mError enabling: Post "http://127.0.0.1:8200/v1/sys/mounts/transit": dial tcp 127.0.0.1:8200: connect: connection refused[0m Transit engine might already be enabled Creating local encryption key 'my-key'... [91mError writing data to transit/keys/my-key: Put "http://127.0.0.1:8200/v1/transit/keys/my-key": dial tcp 127.0.0.1:8200: connect: connection refused[0m
#@title Library Imports
# Standard library imports
import os
import time
import logging
import random
from collections import deque
from typing import Dict, Any, Optional
# Third-party library imports
import hvac # HashiCorp Vault client
from google.cloud import kms_v1 # Google Cloud KMS client
from loguru import logger # Enhanced logging
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
Core Functions
This section defines the essential functions for initializing our application's state, interacting with HashiCorp Vault for secrets management, and utilizing Google Cloud KMS for cryptographic operations.
Function Name: create_app_state
This function initializes the application's state as a dictionary. It sets up default configurations, including logging, and provides a structured way to manage parameters and connections throughout the notebook. The state dictionary will be passed to subsequent functions to maintain context and allow for modification.
Parameters:
vault_addr(Optional[str]): The address for the HashiCorp Vault server, defaults toos.getenv('VAULT_ADDR').kms_project_id(Optional[str]): The Google Cloud Project ID for KMS, defaults toos.getenv('GCP_PROJECT_ID').log_level(str): The logging level for the application, defaults to 'INFO'.
Returns:
dict: An initialized state dictionary containing configuration and client placeholders.
import sys
from loguru import logger
from typing import Dict, Any, Optional
def create_app_state(vault_addr: Optional[str] = None, kms_project_id: Optional[str] = None, log_level: str = "INFO") -> Dict[str, Any]:
"""
Initializes the application's state dictionary.
Parameters
----------
vault_addr : Optional[str]
The address for the HashiCorp Vault server. Defaults to VAULT_ADDR environment variable.
kms_project_id : Optional[str]
The Google Cloud Project ID for KMS. Defaults to GCP_PROJECT_ID environment variable.
log_level : str
The logging level for the application (e.g., 'INFO', 'DEBUG', 'WARNING').
Returns
-------
Dict[str, Any]
An initialized state dictionary containing configuration and client placeholders.
Examples
--------
>>> state = create_app_state(vault_addr='http://localhost:8200', log_level='DEBUG')
>>> assert 'vault_addr' in state
>>> assert state['log_level'] == 'DEBUG'
"""
logger.info("Initializing application state.")
state = {
"vault_addr": vault_addr if vault_addr else os.getenv("VAULT_ADDR"),
"vault_token": os.getenv("VAULT_TOKEN"), # Assuming token is passed via env for simplicity in setup
"kms_project_id": kms_project_id if kms_project_id else os.getenv("GCP_PROJECT_ID"),
"kms_client": None,
"vault_client": None,
"log_level": log_level.upper(),
"metrics": deque(maxlen=100), # For tracking performance/errors
"retry_attempts": 0
}
logger.remove() # Remove default handler
logger.add(sys.stderr, level=state["log_level"])
logger.debug(f"Initial state created: {state}")
return state
Function Name: connect_to_vault
This function establishes a connection to a HashiCorp Vault server using the hvac client library. It takes the current application state, which should contain the Vault address and token, and attempts to authenticate. Robust error handling with exponential backoff is included to manage transient network issues or temporary service unavailability. Upon successful connection, the Vault client is stored in the application state.
Parameters:
state(Dict[str, Any]): The current application state dictionary.
Returns:
Dict[str, Any]: The updated state dictionary with the Vault client, orNoneif connection fails after retries.
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling
def connect_to_vault(state: Dict[str, Any]) -> Dict[str, Any]:
"""
Establishes a connection to HashiCorp Vault with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'vault_addr' and 'vault_token'.
Returns
-------
Dict[str, Any]
The updated state dictionary with the 'vault_client' initialized upon successful connection.
Returns the state with 'vault_client' as None if connection fails after retries.
Examples
--------
>>> state = create_app_state(vault_addr='http://localhost:8200', vault_token='myroottoken')
>>> updated_state = connect_to_vault(state)
>>> assert updated_state['vault_client'] is not None
"""
vault_addr = state.get("vault_addr")
vault_token = state.get("vault_token")
if not vault_addr:
logger.error("Vault address not found in state or environment variables.")
return state
max_retries = 5
base_delay = 1 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to connect to Vault at {vault_addr} (Attempt {i+1}/{max_retries})")
client = hvac.Client(url=vault_addr, token=vault_token)
if client.is_authenticated():
state["vault_client"] = client
logger.success("Successfully connected and authenticated to Vault.")
state["retry_attempts"] = 0
return state
else:
logger.warning("Vault client not authenticated. Check token or authentication method.")
except hvac.exceptions.VaultError as e:
logger.error(f"Vault connection error: {e}")
except requests.exceptions.ConnectionError as e:
logger.error(f"Network connection error to Vault: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred during Vault connection: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.5) # Exponential backoff with jitter
logger.info(f"Retrying in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to connect to Vault after {max_retries} attempts.")
state["vault_client"] = None # Ensure client is None if connection fails
return state
Function Name: connect_to_kms
This function initializes a client for Google Cloud Key Management Service (KMS). It requires the Google Cloud Project ID from the application state to establish the connection. Similar to the Vault connection, it incorporates exponential backoff with jitter for retries to handle potential API rate limits or temporary service unavailability. The initialized KMS client is then stored in the application state.
Parameters:
state(Dict[str, Any]): The current application state dictionary.
Returns:
Dict[str, Any]: The updated state dictionary with the KMS client, orNoneif connection fails after retries.
from typing import Dict, Any, Optional
def connect_to_kms(state: Dict[str, Any]) -> Dict[str, Any]:
"""
Initializes a Google Cloud KMS client with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'kms_project_id'.
Returns
-------
Dict[str, Any]
The updated state dictionary with the 'kms_client' initialized upon successful connection.
Returns the state with 'kms_client' as None if connection fails after retries.
Examples
--------
>>> state = create_app_state(kms_project_id='my-gcp-project')
>>> updated_state = connect_to_kms(state)
>>> assert updated_state['kms_client'] is not None
"""
kms_project_id = state.get("kms_project_id")
if not kms_project_id:
logger.error("GCP Project ID for KMS not found in state or environment variables.")
return state
max_retries = 5
base_delay = 1 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to initialize KMS client for project {kms_project_id} (Attempt {i+1}/{max_retries})")
client = kms_v1.KeyManagementServiceClient()
# A simple way to verify client functionality without an actual KMS call
# is to ensure the client object is created. More robust would be a dummy API call.
state["kms_client"] = client
logger.success("Successfully initialized Google Cloud KMS client.")
state["retry_attempts"] = 0
return state
except Exception as e:
logger.error(f"Error initializing Google Cloud KMS client: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.5) # Exponential backoff with jitter
logger.info(f"Retrying in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to initialize Google Cloud KMS client after {max_retries} attempts.")
state["kms_client"] = None # Ensure client is None if connection fails
return state
Function Name: store_secret_in_vault
This function is responsible for securely storing a secret within HashiCorp Vault. It leverages Vault's Key-Value (KV) secrets engine. The function takes the application state (which includes the authenticated Vault client), the path where the secret should be stored, and the secret data itself. It incorporates robust error handling and retries with exponential backoff to ensure reliability even in the face of temporary network issues or Vault service fluctuations.
Parameters:
state(Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.path(str): The path within the Vault KV secrets engine where the secret will be stored (e.g., 'kv/data/my-app/db-creds').secret_data(Dict[str, Any]): A dictionary containing the secret key-value pairs to store.
Returns:
Dict[str, Any]: The updated state dictionary. ReturnsTrueif the secret was stored successfully,Falseotherwise.
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling
def store_secret_in_vault(state: Dict[str, Any], path: str, secret_data: Dict[str, Any]) -> bool:
"""
Stores a secret in HashiCorp Vault's KV secrets engine with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'vault_client'.
path : str
The path within the Vault KV secrets engine (e.g., 'kv/data/my-app/db-creds').
secret_data : Dict[str, Any]
A dictionary containing the secret key-value pairs to store.
Returns
-------
bool
True if the secret was stored successfully, False otherwise.
Examples
--------
>>> # Assume 'state' has an authenticated vault_client
>>> # success = store_secret_in_vault(state, 'kv/data/my-app/test-secret', {'api_key': '12345', 'user': 'testuser'})
>>> # assert success is True
"""
vault_client = state.get("vault_client")
if not vault_client:
logger.error("Vault client not initialized. Cannot store secret.")
return False
max_retries = 3
base_delay = 0.5 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to store secret at '{path}' in Vault (Attempt {i+1}/{max_retries})")
vault_client.secrets.kv.v2.create_or_update_secret(path=path, secret=secret_data)
logger.success(f"Secret successfully stored at '{path}' in Vault.")
state["retry_attempts"] = 0
return True
except hvac.exceptions.VaultError as e:
logger.error(f"Vault error while storing secret at '{path}': {e}")
except requests.exceptions.ConnectionError as e:
logger.error(f"Network error while connecting to Vault: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred while storing secret: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
logger.warning(f"Retrying secret storage in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to store secret at '{path}' after {max_retries} attempts.")
return False
Function Name: get_secret_from_vault
This function retrieves a secret from HashiCorp Vault's KV secrets engine. Given the application state and the path to the secret, it attempts to fetch the latest version of the secret data. Like other network-dependent functions, it includes an exponential backoff retry mechanism to enhance resilience against transient failures. If successful, it returns the secret data; otherwise, it returns None.
Parameters:
state(Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.path(str): The path within the Vault KV secrets engine where the secret is stored (e.g., 'kv/data/my-app/db-creds').
Returns:
Optional[Dict[str, Any]]: The secret data if retrieved successfully, otherwiseNone.
from typing import Dict, Any, Optional
def get_secret_from_vault(state: Dict[str, Any], path: str) -> Optional[Dict[str, Any]]:
"""
Retrieves a secret from HashiCorp Vault's KV secrets engine with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'vault_client'.
path : str
The path within the Vault KV secrets engine where the secret is stored.
Returns
-------
Optional[Dict[str, Any]]
The secret data if retrieved successfully, otherwise None.
Examples
--------
>>> # Assume 'state' has an authenticated vault_client and a secret at 'kv/data/my-app/test-secret'
>>> # secret = get_secret_from_vault(state, 'kv/data/my-app/test-secret')
>>> # if secret: print(secret)
"""
vault_client = state.get("vault_client")
if not vault_client:
logger.error("Vault client not initialized. Cannot retrieve secret.")
return None
max_retries = 3
base_delay = 0.5 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to retrieve secret from '{path}' in Vault (Attempt {i+1}/{max_retries})")
read_response = vault_client.secrets.kv.v2.read_secret_version(path=path)
secret_data = read_response['data']['data']
logger.success(f"Secret successfully retrieved from '{path}' in Vault.")
state["retry_attempts"] = 0
return secret_data
except hvac.exceptions.VaultError as e:
if "No key exists at" in str(e): # Specific check for secret not found
logger.warning(f"Secret not found at path '{path}' in Vault: {e}")
return None
logger.error(f"Vault error while retrieving secret from '{path}': {e}")
except requests.exceptions.ConnectionError as e:
logger.error(f"Network error while connecting to Vault: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred while retrieving secret: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
logger.warning(f"Retrying secret retrieval in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to retrieve secret from '{path}' after {max_retries} attempts.")
return None
Function Name: encrypt_with_kms
This function uses Google Cloud KMS to encrypt a plaintext string. It requires the application state (containing the KMS client) and the full resource name of the KMS key to be used for encryption. The plaintext is first encoded to bytes, then sent to KMS for encryption. The resulting ciphertext is returned as a base64-encoded string, suitable for storage or transmission. Includes exponential backoff for retries.
Parameters:
state(Dict[str, Any]): The current application state dictionary, expected to contain a 'kms_client'.key_name(str): The full resource name of the KMS key (e.g., 'projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID').plaintext(str): The data to be encrypted.
Returns:
Optional[str]: The base64-encoded ciphertext if encryption is successful, otherwiseNone.
from typing import Dict, Any, Optional
import base64
from google.api_core.exceptions import GoogleAPIError
def encrypt_with_kms(state: Dict[str, Any], key_name: str, plaintext: str) -> Optional[str]:
"""
Encrypts plaintext data using Google Cloud KMS with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'kms_client'.
key_name : str
The full resource name of the KMS key.
plaintext : str
The data to be encrypted.
Returns
-------
Optional[str]
The base64-encoded ciphertext if encryption is successful, otherwise None.
Examples
--------
>>> # Assume 'state' has an initialized kms_client
>>> # key_name = 'projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key'
>>> # encrypted_text = encrypt_with_kms(state, key_name, 'sensitive data')
>>> # if encrypted_text: print(f"Encrypted: {encrypted_text}")
"""
kms_client = state.get("kms_client")
if not kms_client:
logger.error("KMS client not initialized. Cannot encrypt data.")
return None
plaintext_bytes = plaintext.encode('utf-8')
max_retries = 3
base_delay = 0.5 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to encrypt data with KMS key '{key_name}' (Attempt {i+1}/{max_retries})")
encrypt_response = kms_client.encrypt(request={'name': key_name, 'plaintext': plaintext_bytes})
ciphertext = base64.b64encode(encrypt_response.ciphertext).decode('utf-8')
logger.success("Data successfully encrypted with KMS.")
state["retry_attempts"] = 0
return ciphertext
except GoogleAPIError as e:
logger.error(f"Google Cloud KMS API error during encryption: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred during KMS encryption: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
logger.warning(f"Retrying KMS encryption in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to encrypt data with KMS key '{key_name}' after {max_retries} attempts.")
return None
Function Name: decrypt_with_kms
This function utilizes Google Cloud KMS to decrypt a base64-encoded ciphertext. It requires the application state (with an initialized KMS client), the full resource name of the KMS key used for encryption, and the ciphertext. The ciphertext is first base64-decoded, then sent to KMS for decryption. The decrypted plaintext is returned as a string. It also includes an exponential backoff mechanism for retries to ensure robustness.
Parameters:
state(Dict[str, Any]): The current application state dictionary, expected to contain a 'kms_client'.key_name(str): The full resource name of the KMS key used for decryption.ciphertext(str): The base64-encoded ciphertext to be decrypted.
Returns:
Optional[str]: The decrypted plaintext string if decryption is successful, otherwiseNone.
from typing import Dict, Any, Optional
def decrypt_with_kms(state: Dict[str, Any], key_name: str, ciphertext: str) -> Optional[str]:
"""
Decrypts ciphertext data using Google Cloud KMS with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'kms_client'.
key_name : str
The full resource name of the KMS key used for decryption.
ciphertext : str
The base64-encoded ciphertext to be decrypted.
Returns
-------
Optional[str]
The decrypted plaintext string if decryption is successful, otherwise None.
Examples
--------
>>> # Assume 'state' has an initialized kms_client and 'encrypted_text'
>>> # key_name = 'projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key'
>>> # decrypted_text = decrypt_with_kms(state, key_name, encrypted_text)
>>> # if decrypted_text: print(f"Decrypted: {decrypted_text}")
"""
kms_client = state.get("kms_client")
if not kms_client:
logger.error("KMS client not initialized. Cannot decrypt data.")
return None
try:
ciphertext_bytes = base64.b64decode(ciphertext)
except Exception as e:
logger.error(f"Error base64 decoding ciphertext: {e}")
return None
max_retries = 3
base_delay = 0.5 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to decrypt data with KMS key '{key_name}' (Attempt {i+1}/{max_retries})")
decrypt_response = kms_client.decrypt(request={'name': key_name, 'ciphertext': ciphertext_bytes})
plaintext = decrypt_response.plaintext.decode('utf-8')
logger.success("Data successfully decrypted with KMS.")
state["retry_attempts"] = 0
return plaintext
except GoogleAPIError as e:
logger.error(f"Google Cloud KMS API error during decryption: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred during KMS decryption: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
logger.warning(f"Retrying KMS decryption in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to decrypt data with KMS key '{key_name}' after {max_retries} attempts.")
return None
Local Alternative: Vault Transit Encryption
Since we are working locally, we can use Vault's Transit engine to perform cryptographic operations without external cloud providers.
def encrypt_with_vault_transit(state: Dict[str, Any], key_name: str, plaintext: str) -> Optional[str]:
"""Encrypts data using Vault's Transit engine."""
client = state.get("vault_client")
if not client:
return None
try:
# Vault expects base64 encoded plaintext for Transit
import base64
encoded_plaintext = base64.b64encode(plaintext.encode('utf-8')).decode('utf-8')
response = client.secrets.transit.encrypt_data(name=key_name, plaintext=encoded_plaintext)
return response['data']['ciphertext']
except Exception as e:
logger.error(f"Vault Transit encryption error: {e}")
return None
def decrypt_with_vault_transit(state: Dict[str, Any], key_name: str, ciphertext: str) -> Optional[str]:
"""Decrypts data using Vault's Transit engine."""
client = state.get("vault_client")
if not client:
return None
try:
response = client.secrets.transit.decrypt_data(name=key_name, ciphertext=ciphertext)
import base64
return base64.b64decode(response['data']['plaintext']).decode('utf-8')
except Exception as e:
logger.error(f"Vault Transit decryption error: {e}")
return NoneFunction Name: track_metric
This utility function is designed to record various operational metrics throughout the application's lifecycle. It stores metrics in a deque (double-ended queue) within the application state, allowing for a rolling window of recent activities. This is crucial for monitoring performance, error rates, and retry attempts, which can later be visualized to understand system behavior.
Parameters:
state(Dict[str, Any]): The current application state dictionary, containing themetricsdeque.metric_name(str): The name of the metric to track (e.g., 'vault_read_success', 'kms_encrypt_failure').value(Any): The value associated with the metric.
Returns:
Dict[str, Any]: The updated state dictionary.
from typing import Dict, Any
import time
def track_metric(state: Dict[str, Any], metric_name: str, value: Any) -> Dict[str, Any]:
"""
Tracks an operational metric in the application state's metrics deque.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain a 'metrics' deque.
metric_name : str
The name of the metric to track.
value : Any
The value associated with the metric.
Returns
-------
Dict[str, Any]
The updated state dictionary.
Examples
--------
>>> state = {'metrics': deque(maxlen=10)}
>>> state = track_metric(state, 'vault_read_latency', 0.05)
>>> assert len(state['metrics']) == 1
>>> assert state['metrics'][0]['name'] == 'vault_read_latency'
"""
timestamp = time.time()
state['metrics'].append({'timestamp': timestamp, 'name': metric_name, 'value': value})
logger.debug(f"Metric tracked: {metric_name} = {value}")
return state
Function Name: delete_secret_from_vault
This function removes a secret from HashiCorp Vault's KV secrets engine at a specified path. It's important for secret lifecycle management, allowing for cleanup of old or deprecated secrets. The function includes retry logic with exponential backoff to handle potential transient issues during the deletion process. It logs the outcome of the deletion attempt.
Parameters:
state(Dict[str, Any]): The current application state dictionary, expected to contain an authenticated 'vault_client'.path(str): The path within the Vault KV secrets engine where the secret is to be deleted (e.g., 'kv/data/my-app/old-creds').
Returns:
bool:Trueif the secret was successfully deleted,Falseotherwise.
from typing import Dict, Any, Optional
import requests # Import requests for specific exceptions handling
def delete_secret_from_vault(state: Dict[str, Any], path: str) -> bool:
"""
Deletes a secret from HashiCorp Vault's KV secrets engine with retry logic.
Parameters
----------
state : Dict[str, Any]
The current application state dictionary, expected to contain 'vault_client'.
path : str
The path within the Vault KV secrets engine where the secret is to be deleted.
Returns
-------
bool
True if the secret was successfully deleted, False otherwise.
Examples
--------
>>> # Assume 'state' has an authenticated vault_client and a secret at 'kv/data/my-app/temp-secret'
>>> # success = delete_secret_from_vault(state, 'kv/data/my-app/temp-secret')
>>> # assert success is True
"""
vault_client = state.get("vault_client")
if not vault_client:
logger.error("Vault client not initialized. Cannot delete secret.")
return False
max_retries = 3
base_delay = 0.5 # seconds
for i in range(max_retries):
try:
logger.info(f"Attempting to delete secret from '{path}' in Vault (Attempt {i+1}/{max_retries})")
vault_client.secrets.kv.v2.delete_latest_version_of_secret(path=path)
logger.success(f"Secret successfully deleted from '{path}' in Vault.")
state["retry_attempts"] = 0
return True
except hvac.exceptions.VaultError as e:
if "Code: 404" in str(e): # Secret not found
logger.warning(f"Secret at '{path}' not found or already deleted: {e}")
return True # Consider it successful if it's already gone
logger.error(f"Vault error while deleting secret from '{path}': {e}")
except requests.exceptions.ConnectionError as e:
logger.error(f"Network error while connecting to Vault: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred while deleting secret: {e}")
delay = base_delay * (2 ** i) + random.uniform(0, 0.2) # Exponential backoff with jitter
logger.warning(f"Retrying secret deletion in {delay:.2f} seconds...")
time.sleep(delay)
state["retry_attempts"] += 1
logger.error(f"Failed to delete secret from '{path}' after {max_retries} attempts.")
return False
Demonstration and Visualization
This section demonstrates the functionality of the secrets management functions. We will simulate an environment where secrets are stored in HashiCorp Vault and sensitive data is encrypted using Google Cloud KMS. We'll also visualize operational metrics like latency and retry attempts.
To run this section, ensure you have:
- A running HashiCorp Vault instance (e.g.,
vault server -dev). - A Vault token with appropriate permissions (e.g.,
export VAULT_TOKEN='s.xxxxxx'andexport VAULT_ADDR='http://127.0.0.1:8200'for dev server). - Google Cloud project credentials configured (e.g.,
gcloud auth application-default login). - A Google Cloud KMS Key Ring and CryptoKey created (e.g., in
globalor a specific region).
1. Initialize State and Connect to Services
First, we'll set up our simulated environment variables and initialize the application state, then attempt to connect to both Vault and KMS. The logger statements will provide real-time feedback on the connection status.
#@title Initialize App State and Connect to Vault/KMS
import os # Added import
from loguru import logger # Added import
# Simulate environment variables (replace with your actual values for a real setup)
os.environ['VAULT_ADDR'] = os.getenv('VAULT_ADDR', 'http://127.0.0.1:8200') # Default to dev server
os.environ['VAULT_TOKEN'] = os.getenv('VAULT_TOKEN', 'myroottoken') # Replace with your dev token or real token
os.environ['GCP_PROJECT_ID'] = os.getenv('GCP_PROJECT_ID', 'your-gcp-project-id') # Replace with your GCP Project ID
# Initialize application state
app_state = create_app_state(log_level='DEBUG')
logger.info(f"Application state initialized with log level: {app_state['log_level']}")
# Connect to Vault
app_state = connect_to_vault(app_state)
# Connect to KMS
app_state = connect_to_kms(app_state)
# Display connection status
print("\n--- Connection Status ---")
print(f"Vault Client Connected: {app_state['vault_client'] is not None}")
print(f"KMS Client Connected: {app_state['kms_client'] is not None}")
[32m2026-06-10 09:33:53.299[0m | [1mINFO [0m | [36m__main__[0m:[36mcreate_app_state[0m:[36m29[0m - [1mInitializing application state.[0m
[32m2026-06-10 09:33:53.326[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mcreate_app_state[0m:[36m42[0m - [34m[1mInitial state created: {'vault_addr': 'http://127.0.0.1:8200', 'vault_token': 'myroottoken', 'kms_project_id': 'your-gcp-project-id', 'kms_client': None, 'vault_client': None, 'log_level': 'DEBUG', 'metrics': deque([], maxlen=100), 'retry_attempts': 0}[0m
[32m2026-06-10 09:33:53.331[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m12[0m - [1mApplication state initialized with log level: DEBUG[0m
[32m2026-06-10 09:33:53.336[0m | [1mINFO [0m | [36m__main__[0m:[36mconnect_to_vault[0m:[36m35[0m - [1mAttempting to connect to Vault at http://127.0.0.1:8200 (Attempt 1/5)[0m
[32m2026-06-10 09:33:53.348[0m | [32m[1mSUCCESS [0m | [36m__main__[0m:[36mconnect_to_vault[0m:[36m39[0m - [32m[1mSuccessfully connected and authenticated to Vault.[0m
[32m2026-06-10 09:33:53.355[0m | [1mINFO [0m | [36m__main__[0m:[36mconnect_to_kms[0m:[36m33[0m - [1mAttempting to initialize KMS client for project your-gcp-project-id (Attempt 1/5)[0m
[32m2026-06-10 09:33:53.384[0m | [32m[1mSUCCESS [0m | [36m__main__[0m:[36mconnect_to_kms[0m:[36m38[0m - [32m[1mSuccessfully initialized Google Cloud KMS client.[0m
--- Connection Status --- Vault Client Connected: True KMS Client Connected: True
2. HashiCorp Vault: Store, Retrieve, and Delete Secrets
We'll demonstrate the full lifecycle of a secret in Vault: creating it, reading it back, and then deleting it. We'll also track metrics for these operations.
#@title Vault Secret Operations
import time # Added import
from loguru import logger # Added import
# Define a test secret
VAULT_TEST_PATH = 'kv/data/my-app/test-db-creds'
TEST_SECRET_DATA = {
'username': 'app_user',
'password': 'super_secret_password_123',
'connection_string': 'jdbc:postgresql://db.example.com:5432/production'
}
# 1. Store a secret
logger.info(f"\n--- Storing secret at {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
store_success = store_secret_in_vault(app_state, VAULT_TEST_PATH, TEST_SECRET_DATA)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_store_latency_ms', latency)
app_state = track_metric(app_state, 'vault_store_success', int(store_success))
print(f"Secret store successful: {store_success} (Latency: {latency:.2f} ms)")
# 2. Retrieve the secret
logger.info(f"\n--- Retrieving secret from {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
retrieved_secret = get_secret_from_vault(app_state, VAULT_TEST_PATH)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_retrieve_latency_ms', latency)
app_state = track_metric(app_state, 'vault_retrieve_success', int(retrieved_secret is not None))
if retrieved_secret:
print("Secret retrieved successfully:")
for k, v in retrieved_secret.items():
print(f" {k}: {'*' * len(v) if k == 'password' else v}") # Mask password
else:
print("Failed to retrieve secret.")
# 3. Attempt to retrieve a non-existent secret (edge case)
logger.info(f"\n--- Attempting to retrieve non-existent secret ---")
non_existent_secret = get_secret_from_vault(app_state, 'kv/data/my-app/non-existent-creds')
print(f"Non-existent secret retrieval result: {non_existent_secret}")
app_state = track_metric(app_state, 'vault_retrieve_non_existent_success', int(non_existent_secret is not None))
# 4. Delete the secret
logger.info(f"\n--- Deleting secret from {VAULT_TEST_PATH} ---")
start_time = time.perf_counter()
del_success = delete_secret_from_vault(app_state, VAULT_TEST_PATH)
end_time = time.perf_counter()
latency = (end_time - start_time) * 1000 # milliseconds
app_state = track_metric(app_state, 'vault_delete_latency_ms', latency)
app_state = track_metric(app_state, 'vault_delete_success', int(del_success))
print(f"Secret deletion successful: {del_success} (Latency: {latency:.2f} ms)")
# 5. Verify deletion
logger.info(f"\n--- Verifying secret deletion from {VAULT_TEST_PATH} ---")
verified_deleted = get_secret_from_vault(app_state, VAULT_TEST_PATH)
print(f"Verification after deletion: {verified_deleted is None}")
[32m2026-06-10 09:33:53.418[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m14[0m - [1m --- Storing secret at kv/data/my-app/test-db-creds ---[0m [32m2026-06-10 09:33:53.419[0m | [1mINFO [0m | [36m__main__[0m:[36mstore_secret_in_vault[0m:[36m37[0m - [1mAttempting to store secret at 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)[0m [32m2026-06-10 09:33:53.427[0m | [32m[1mSUCCESS [0m | [36m__main__[0m:[36mstore_secret_in_vault[0m:[36m39[0m - [32m[1mSecret successfully stored at 'kv/data/my-app/test-db-creds' in Vault.[0m [32m2026-06-10 09:33:53.429[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_store_latency_ms = 9.38200600012351[0m [32m2026-06-10 09:33:53.431[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_store_success = 1[0m [32m2026-06-10 09:33:53.432[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m25[0m - [1m --- Retrieving secret from kv/data/my-app/test-db-creds ---[0m [32m2026-06-10 09:33:53.434[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)[0m /tmp/ipykernel_10995/394795625.py:35: DeprecationWarning: The raise_on_deleted_version parameter will change its default value to False in hvac v3.0.0. The current default of True will preserve previous behavior. To use the old behavior with no warning, explicitly set this value to True. See https://github.com/hvac/hvac/pull/907 read_response = vault_client.secrets.kv.v2.read_secret_version(path=path) [32m2026-06-10 09:33:53.440[0m | [32m[1mSUCCESS [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m37[0m - [32m[1mSecret successfully retrieved from 'kv/data/my-app/test-db-creds' in Vault.[0m [32m2026-06-10 09:33:53.442[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_retrieve_latency_ms = 7.942938000269351[0m [32m2026-06-10 09:33:53.444[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_retrieve_success = 1[0m [32m2026-06-10 09:33:53.447[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m41[0m - [1m --- Attempting to retrieve non-existent secret ---[0m [32m2026-06-10 09:33:53.450[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 1/3)[0m [32m2026-06-10 09:33:53.457[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds[0m [32m2026-06-10 09:33:53.458[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 0.53 seconds...[0m
Secret store successful: True (Latency: 9.38 ms) Secret retrieved successfully: connection_string: jdbc:postgresql://db.example.com:5432/production password: ************************* username: app_user
[32m2026-06-10 09:33:53.987[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 2/3)[0m
[32m2026-06-10 09:33:53.993[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds[0m
[32m2026-06-10 09:33:53.994[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 1.07 seconds...[0m
[32m2026-06-10 09:33:55.061[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/non-existent-creds' in Vault (Attempt 3/3)[0m
[32m2026-06-10 09:33:55.066[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/non-existent-creds': None, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/non-existent-creds[0m
[32m2026-06-10 09:33:55.068[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 2.06 seconds...[0m
[32m2026-06-10 09:33:57.129[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m55[0m - [31m[1mFailed to retrieve secret from 'kv/data/my-app/non-existent-creds' after 3 attempts.[0m
[32m2026-06-10 09:33:57.131[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_retrieve_non_existent_success = 0[0m
[32m2026-06-10 09:33:57.134[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m47[0m - [1m
--- Deleting secret from kv/data/my-app/test-db-creds ---[0m
[32m2026-06-10 09:33:57.135[0m | [1mINFO [0m | [36m__main__[0m:[36mdelete_secret_from_vault[0m:[36m35[0m - [1mAttempting to delete secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)[0m
[32m2026-06-10 09:33:57.153[0m | [32m[1mSUCCESS [0m | [36m__main__[0m:[36mdelete_secret_from_vault[0m:[36m37[0m - [32m[1mSecret successfully deleted from 'kv/data/my-app/test-db-creds' in Vault.[0m
[32m2026-06-10 09:33:57.163[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_delete_latency_ms = 27.20103400042717[0m
[32m2026-06-10 09:33:57.171[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: vault_delete_success = 1[0m
[32m2026-06-10 09:33:57.174[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m58[0m - [1m
--- Verifying secret deletion from kv/data/my-app/test-db-creds ---[0m
[32m2026-06-10 09:33:57.179[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 1/3)[0m
[32m2026-06-10 09:33:57.198[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"d31fee39-c793-f8e3-c2a3-57ecee24ff70","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds[0m
[32m2026-06-10 09:33:57.203[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 0.68 seconds...[0m
Non-existent secret retrieval result: None Secret deletion successful: True (Latency: 27.20 ms)
[32m2026-06-10 09:33:57.886[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 2/3)[0m
[32m2026-06-10 09:33:57.891[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"cbb73ae6-d538-a72b-7a37-d7a3900a1093","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds[0m
[32m2026-06-10 09:33:57.893[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 1.04 seconds...[0m
[32m2026-06-10 09:33:58.934[0m | [1mINFO [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m34[0m - [1mAttempting to retrieve secret from 'kv/data/my-app/test-db-creds' in Vault (Attempt 3/3)[0m
[32m2026-06-10 09:33:58.941[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m44[0m - [31m[1mVault error while retrieving secret from 'kv/data/my-app/test-db-creds': {"request_id":"65f06ed2-a529-f0e2-2908-326ef6cfe4e1","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":null,"metadata":{"created_time":"2026-06-10T09:33:53.426064208Z","custom_metadata":null,"deletion_time":"2026-06-10T09:33:57.146062189Z","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}, on get http://127.0.0.1:8200/v1/secret/data/kv/data/my-app/test-db-creds[0m
[32m2026-06-10 09:33:58.947[0m | [33m[1mWARNING [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m51[0m - [33m[1mRetrying secret retrieval in 2.17 seconds...[0m
[32m2026-06-10 09:34:01.119[0m | [31m[1mERROR [0m | [36m__main__[0m:[36mget_secret_from_vault[0m:[36m55[0m - [31m[1mFailed to retrieve secret from 'kv/data/my-app/test-db-creds' after 3 attempts.[0m
Verification after deletion: True
Local Encryption Demo (Vault Transit)
This replaces the Google KMS demo with a local version using the Vault dev server.
LOCAL_KEY_NAME = 'my-key'
PLAINTEXT = 'This is local sensitive data.'
# Perform local encryption
logger.info("--- Starting Local Transit Encryption ---")
ciphertext = encrypt_with_vault_transit(app_state, LOCAL_KEY_NAME, PLAINTEXT)
if ciphertext:
print(f"Local Ciphertext: {ciphertext}")
# Track metrics for the local operation
app_state = track_metric(app_state, 'local_transit_encrypt_success', 1)
# Perform local decryption
decrypted = decrypt_with_vault_transit(app_state, LOCAL_KEY_NAME, ciphertext)
if decrypted:
print(f"Decrypted: {decrypted}")
print(f"Original matches decrypted: {PLAINTEXT == decrypted}")
app_state = track_metric(app_state, 'local_transit_decrypt_success', 1)
else:
print("Local decryption failed.")
else:
print("Local encryption failed.")[32m2026-06-10 09:38:07.236[0m | [1mINFO [0m | [36m__main__[0m:[36m<cell line: 0>[0m:[36m5[0m - [1m--- Starting Local Transit Encryption ---[0m [32m2026-06-10 09:38:07.247[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: local_transit_encrypt_success = 1[0m [32m2026-06-10 09:38:07.253[0m | [34m[1mDEBUG [0m | [36m__main__[0m:[36mtrack_metric[0m:[36m31[0m - [34m[1mMetric tracked: local_transit_decrypt_success = 1[0m
Local Ciphertext: vault:v1:nTUL3XprrBsdpNWQNXyY9hrijxv+lVCcSKo5YgcjUTxGXky/St+rFINeBYQYd+gOCVlmbnYYP1O6 Decrypted: This is local sensitive data. Original matches decrypted: True
4. Visualization of Operational Metrics
We'll use the tracked metrics to visualize the latency and success rates of our Vault and KMS operations. This helps in understanding the performance and reliability of the secrets management system.
#@title Plotting Metrics
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display
# Convert metrics deque to a Pandas DataFrame
metrics_df = pd.DataFrame(list(app_state['metrics']))
if not metrics_df.empty:
metrics_df['timestamp'] = pd.to_datetime(metrics_df['timestamp'], unit='s')
print("\n--- Summary of All Tracked Metrics ---")
display(metrics_df)
# Filter for success/failure metrics including the new local transit metrics
success_metrics = metrics_df[metrics_df['name'].str.contains('_success')]
if not success_metrics.empty:
plt.figure(figsize=(10, 5))
# Updated to resolve FutureWarning by assigning x to hue and setting legend=False
sns.barplot(data=success_metrics, x='name', y='value', hue='name', legend=False, palette='viridis')
plt.title('Vault Operation Success Indicators')
plt.xlabel('Operation Type')
plt.ylabel('Success (1=True, 0=False)')
plt.xticks(rotation=45, ha='right')
plt.ylim(0, 1.1)
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
else:
print("No success metrics to plot.")
# Filter for latency metrics
latency_metrics = metrics_df[metrics_df['name'].str.contains('_latency_ms')]
if not latency_metrics.empty:
plt.figure(figsize=(10, 5))
sns.lineplot(data=latency_metrics, x='timestamp', y='value', hue='name', marker='o')
plt.title('Vault Operation Latencies')
plt.xlabel('Time')
plt.ylabel('Latency (ms)')
plt.grid(True)
plt.tight_layout()
plt.show()
else:
print("No metrics have been tracked yet.")--- Summary of All Tracked Metrics ---
| timestamp | name | value | |
|---|---|---|---|
| 0 | 2026-06-10 09:33:53.429040194 | vault_store_latency_ms | 9.382006 |
| 1 | 2026-06-10 09:33:53.431044102 | vault_store_success | 1.000000 |
| 2 | 2026-06-10 09:33:53.442732334 | vault_retrieve_latency_ms | 7.942938 |
| 3 | 2026-06-10 09:33:53.444972515 | vault_retrieve_success | 1.000000 |
| 4 | 2026-06-10 09:33:57.131113291 | vault_retrieve_non_existent_success | 0.000000 |
| 5 | 2026-06-10 09:33:57.163095951 | vault_delete_latency_ms | 27.201034 |
| 6 | 2026-06-10 09:33:57.171158791 | vault_delete_success | 1.000000 |
| 7 | 2026-06-10 09:38:07.247468472 | local_transit_encrypt_success | 1.000000 |
| 8 | 2026-06-10 09:38:07.253608465 | local_transit_decrypt_success | 1.000000 |
Production Considerations
Deploying secrets management solutions like Vault and KMS in production requires careful planning and adherence to best practices to ensure security, reliability, and maintainability. This section outlines key considerations.
Best Practices for Secrets Management in Production
| Aspect | Best Practice | Rationale |
|---|---|---|
| Authentication | Use strong, machine-oriented authentication methods (e.g., IAM roles for AWS/GCP, Kubernetes service accounts, AppRole for Vault, client certificates) instead of hardcoded tokens. | Reduces the risk of static credential exposure and simplifies rotation. |
| Authorization (Least Privilege) | Implement fine-grained access control policies (ACLs in Vault, IAM policies in GCP KMS) that grant only the minimum necessary permissions to secrets and keys. | Minimizes the blast radius in case of compromise. |
| Auditing & Monitoring | Enable comprehensive audit logging for all secret access and modification events. Integrate logs with SIEM (Security Information and Event Management) systems for real-time alerting and analysis. | Provides a forensic trail, helps detect anomalous behavior, and ensures compliance. |
| Secret Rotation | Automate secret rotation wherever possible. For Vault, utilize dynamic secrets; for static secrets, implement periodic rotation (e.g., every 90 days). KMS keys should also be rotated according to policy. | Limits the window of exposure for compromised secrets and reduces operational overhead. |
| High Availability | Deploy secrets management systems in a highly available configuration (e.g., Vault in HA mode with integrated storage or Consul, KMS is inherently highly available by GCP). | Ensures continuous access to critical secrets even during outages of individual components. |
| Disaster Recovery | Implement robust backup and disaster recovery procedures for your secrets management infrastructure and the secrets themselves. Test these procedures regularly. | Guarantees business continuity and recovery from catastrophic data loss or system failure. |
| Network Security | Restrict network access to secrets management systems using firewalls, VPC private access, and dedicated subnets. Communicate over TLS/SSL and ensure certificate validation. | Prevents unauthorized network access and protects data in transit. |
| Key Management Strategy | Define a clear strategy for managing cryptographic keys, including key hierarchy, lifecycle, and revocation policies. Use hardware security modules (HSMs) where regulatory compliance or highest security is required. | Establishes a secure foundation for all cryptographic operations and data protection. |
| Secure Application Integration | Use official client libraries and SDKs. Avoid embedding secrets directly in application code or configuration files. Retrieve secrets at runtime from Vault/KMS. | Prevents accidental exposure of secrets in source code repositories or build artifacts. |
| Testing | Regularly test your secrets management setup, including access patterns, rotation mechanisms, and disaster recovery scenarios. Conduct penetration testing and security audits. | Identifies vulnerabilities and ensures the system functions as expected under various conditions. |
Conclusion
This notebook has provided a comprehensive overview and practical demonstration of infrastructure security best practices concerning the storage and management of secrets using HashiCorp Vault and Google Cloud Key Management Service (KMS). We've explored:
- Core Functions: Implemented modular functions for initializing application state, connecting to Vault and KMS, and performing essential operations like storing, retrieving, encrypting, and decrypting sensitive data.
- Robustness: Ensured reliability through the incorporation of exponential backoff and jitter for retry mechanisms in network-dependent operations, enhancing resilience against transient failures.
- Demonstration: Illustrated the end-to-end lifecycle of a secret in Vault and a sensitive data encryption/decryption flow with KMS, complete with simulated environment variables and real-time logging.
- Visualization: Utilized operational metrics to visualize the performance (latency) and outcomes (success/failure) of various secrets management actions, providing insights into system health and behavior.
- Production Considerations: Outlined a set of critical best practices for deploying and managing these solutions in a production environment, emphasizing authentication, authorization, auditing, rotation, high availability, disaster recovery, and network security.
By following these principles and leveraging dedicated tools like Vault and KMS, organizations can significantly enhance their security posture, protect sensitive information from compromise, and comply with regulatory requirements. The modular and well-documented approach presented in this notebook serves as a foundation for building secure and scalable applications.