Data Storage Framework — CSV and Parquet

This notebook defines a standardized protocol for persisting cleaned OHLCV data to flat-file formats: CSV for human-readable interchange and Parquet for columnar, compressed analytical storage. All timestamps are represented as UTC datetime objects.

1. Dependency Installation

[ ]

!pip install pandas pyarrow

Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)
Requirement already satisfied: pyarrow in /usr/local/lib/python3.12/dist-packages (18.1.0)
Requirement already satisfied: numpy>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2026.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)

2. Library Imports

[ ]

import warnings
warnings.filterwarnings("ignore")

import pandas as pd
from pathlib import Path

Code Logic

pandas: Provides DataFrame.to_csv() and DataFrame.to_parquet() serialization methods.
pyarrow: Backend engine consumed by pandas for Parquet read/write. Installed separately, not imported directly.
pathlib.Path: Supplies OS-agnostic file path construction and directory creation.

3. Storage Format Definitions

CSV (Comma-Separated Values) CSV is a plain-text format where each row is a line and each column is separated by a comma. It is universally readable — any spreadsheet application, text editor, or programming language can open a CSV file without special tooling. For OHLCV data, CSV is appropriate for sharing data with external parties, manual inspection, or small datasets. Its disadvantages are size (no compression) and performance (every value is stored as a string and must be re-parsed on read).

Parquet Parquet is a binary columnar storage format designed for analytical workloads. Instead of storing data row by row, Parquet stores each column as a contiguous block. For OHLCV data — where close prices across thousands of rows are far more similar to each other than any given row's five fields are to each other — columnar storage achieves compression ratios of 5–20× compared to CSV. Parquet also embeds the schema (column names and types) inside the file, so datetime, float64, and int64 types are preserved on read without re-parsing. In a trading pipeline, Parquet is the standard format for storing historical OHLCV data on disk because it is compact, fast to read, and self-describing.

Snappy Compression Snappy is a compression algorithm developed by Google, optimized for speed rather than maximum compression ratio. For OHLCV Parquet files it typically achieves 4–8× size reduction at near-zero CPU cost. It is the default compression choice for financial time-series data because decompression speed matters more than achieving the smallest possible file size — backtests read data frequently.

4. Dummy Dataset

[ ]

raw_data = {
    "datetime": [
        "2024-01-01 00:00:00+00:00",
        "2024-01-01 00:01:00+00:00",
        "2024-01-01 00:02:00+00:00",
        "2024-01-01 00:03:00+00:00",
        "2024-01-01 00:04:00+00:00",
    ],
    "open":   [42100.0, 42200.0, 42150.0, 42300.0, 42250.0],
    "high":   [42300.0, 42400.0, 42350.0, 42500.0, 42450.0],
    "low":    [41900.0, 42000.0, 41950.0, 42100.0, 42050.0],
    "close":  [42200.0, 42150.0, 42300.0, 42250.0, 42400.0],
    "volume": [10.5,    8.2,     9.1,     11.3,    7.6    ],
}

df = pd.DataFrame(raw_data)
df["datetime"] = pd.to_datetime(df["datetime"], utc=True)

print("--- OHLCV DataFrame ---")
display(df)
df.info()

--- OHLCV DataFrame ---

	datetime	open	high	low	close	volume
0	2024-01-01 00:00:00+00:00	42100.0	42300.0	41900.0	42200.0	10.5
1	2024-01-01 00:01:00+00:00	42200.0	42400.0	42000.0	42150.0	8.2
2	2024-01-01 00:02:00+00:00	42150.0	42350.0	41950.0	42300.0	9.1
3	2024-01-01 00:03:00+00:00	42300.0	42500.0	42100.0	42250.0	11.3
4	2024-01-01 00:04:00+00:00	42250.0	42450.0	42050.0	42400.0	7.6

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype              
---  ------    --------------  -----              
 0   datetime  5 non-null      datetime64[ns, UTC]
 1   open      5 non-null      float64            
 2   high      5 non-null      float64            
 3   low       5 non-null      float64            
 4   close     5 non-null      float64            
 5   volume    5 non-null      float64            
dtypes: datetime64[ns, UTC](1), float64(5)
memory usage: 372.0 bytes

5. Storage Configuration

[ ]

SYMBOL     = "BTCUSDT"
EXCHANGE   = "bybit"
OUTPUT_DIR = Path("data") / EXCHANGE / SYMBOL

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

CSV_PATH     = OUTPUT_DIR / f"{SYMBOL}.csv"
PARQUET_PATH = OUTPUT_DIR / f"{SYMBOL}.parquet"

Code Logic

OUTPUT_DIR: Constructs a hierarchical directory data/{exchange}/{symbol}/ to namespace output files by exchange and instrument.
Path.mkdir(parents=True, exist_ok=True): Creates the full directory tree if absent; no-ops silently if it already exists.

6. Save Function

[ ]

def save_data(df: pd.DataFrame, csv_path: Path, parquet_path: Path) -> None:
    df = df.copy()

    # --- CSV ---
    df.to_csv(csv_path, index=False)
    print(f"CSV saved     → {csv_path}  ({csv_path.stat().st_size:,} bytes)")

    # --- Parquet ---
    df.to_parquet(
        parquet_path,
        engine      = "pyarrow",
        index       = False,
        compression = "snappy",
    )
    print(f"Parquet saved → {parquet_path}  ({parquet_path.stat().st_size:,} bytes)")

Code Logic

df.to_csv(index=False): Writes the DataFrame to a plain-text CSV without the integer RangeIndex. The datetime column is serialized as an ISO 8601 string including the UTC offset, making it human-readable.
df.to_parquet(engine="pyarrow", index=False, compression="snappy"): Writes a binary Parquet file using PyArrow as the serialization engine. See Section 3 for Parquet and Snappy definitions.
csv_path.stat().st_size: Reports on-disk file size in bytes immediately after write — a direct comparison between CSV and Parquet sizes demonstrates the compression benefit.

7. Execution

[ ]

save_data(df, CSV_PATH, PARQUET_PATH)

CSV saved     → data/bybit/BTCUSDT/BTCUSDT.csv  (348 bytes)
Parquet saved → data/bybit/BTCUSDT/BTCUSDT.parquet  (4,059 bytes)

8. Verification — Read Back and Inspect

[ ]

df_csv     = pd.read_csv(CSV_PATH, parse_dates=["datetime"])
df_parquet = pd.read_parquet(PARQUET_PATH, engine="pyarrow")

print("--- CSV Read-Back ---")
display(df_csv.head())
df_csv.info()

print("\n--- Parquet Read-Back ---")
display(df_parquet.head())
df_parquet.info()

--- CSV Read-Back ---

	datetime	open	high	low	close	volume
0	2024-01-01 00:00:00+00:00	42100.0	42300.0	41900.0	42200.0	10.5
1	2024-01-01 00:01:00+00:00	42200.0	42400.0	42000.0	42150.0	8.2
2	2024-01-01 00:02:00+00:00	42150.0	42350.0	41950.0	42300.0	9.1
3	2024-01-01 00:03:00+00:00	42300.0	42500.0	42100.0	42250.0	11.3
4	2024-01-01 00:04:00+00:00	42250.0	42450.0	42050.0	42400.0	7.6

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype              
---  ------    --------------  -----              
 0   datetime  5 non-null      datetime64[ns, UTC]
 1   open      5 non-null      float64            
 2   high      5 non-null      float64            
 3   low       5 non-null      float64            
 4   close     5 non-null      float64            
 5   volume    5 non-null      float64            
dtypes: datetime64[ns, UTC](1), float64(5)
memory usage: 372.0 bytes

--- Parquet Read-Back ---

	datetime	open	high	low	close	volume
0	2024-01-01 00:00:00+00:00	42100.0	42300.0	41900.0	42200.0	10.5
1	2024-01-01 00:01:00+00:00	42200.0	42400.0	42000.0	42150.0	8.2
2	2024-01-01 00:02:00+00:00	42150.0	42350.0	41950.0	42300.0	9.1
3	2024-01-01 00:03:00+00:00	42300.0	42500.0	42100.0	42250.0	11.3
4	2024-01-01 00:04:00+00:00	42250.0	42450.0	42050.0	42400.0	7.6

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype              
---  ------    --------------  -----              
 0   datetime  5 non-null      datetime64[ns, UTC]
 1   open      5 non-null      float64            
 2   high      5 non-null      float64            
 3   low       5 non-null      float64            
 4   close     5 non-null      float64            
 5   volume    5 non-null      float64            
dtypes: datetime64[ns, UTC](1), float64(5)
memory usage: 372.0 bytes

Code Logic

pd.read_csv(..., parse_dates=["datetime"]): Re-parses the datetime column from its serialized ISO 8601 string back into a pandas datetime type. This step is required for CSV because the format stores all values as plain text — type information is lost on write.
pd.read_parquet(..., engine="pyarrow"): Reads the binary Parquet file and restores the original schema — DatetimeTZDtype[ns, UTC], float64 — without any additional parsing. Type metadata is embedded in the Parquet file itself, which is one of its primary advantages over CSV.

Conclusion

This notebook demonstrates a robust and efficient workflow for persisting OHLCV data using both CSV and Parquet formats. CSV is suitable for human-readable interchange and small datasets, while Parquet, with Snappy compression, is ideal for analytical workloads due to its columnar storage, high compression ratios, and schema preservation. This approach ensures data integrity, efficiency, and flexibility for various downstream tasks in a trading pipeline.

Save to CSV Parquet

Data Storage Framework — CSV and Parquet

1. Dependency Installation

2. Library Imports

Code Logic

3. Storage Format Definitions

4. Dummy Dataset

5. Storage Configuration

Code Logic

6. Save Function

Code Logic

7. Execution

8. Verification — Read Back and Inspect

Code Logic

Conclusion