pandas rolling window running out of memory on tick data
Rolling statistics over tens of millions of ticks blow up when you chain them, call .apply, or roll over a DatetimeIndex with a large time window.
Why it happens
Each .rolling(...).mean() allocates a full-length float64 result, and chained
expressions keep the intermediates alive. .rolling().apply(f) also calls Python once per row,
which makes it slow. With 50M rows, each intermediate is 400 MB.
Fix it with plain pandas
import numpy as np
import pandas as pd
# Resample to the horizon you actually trade on before rolling: 1s bars
# turn ~50M ticks into ~86k rows per day.
px = ticks["price"].resample("1s").last().ffill()
ret = np.log(px).diff()
vol = ret.rolling(3600).std() * np.sqrt(365 * 24 * 3600) # 1h rolling vol, annualised
# Prefer built-in reducers (mean/std/sum/min/max) over .apply(lambda ...)Or with Deptho
Deptho Pro computes rolling mean and standard deviation in one streaming O(n) pass over a memory-mapped column, so full tick history never sits in RAM.
import deptho_pro as dpp
f = dpp.load("BTCUSDT-aggTrades-2026-09-30.zip") # converts once, then memory-maps
mean, std = f.rolling_mean_std("price", window=100_000)Working with files bigger than RAM? Deptho Pro memory-maps them.