h5i-dev/h5i-db: open-source project for self-hosters

English · Español · Français · 简体中文 · 日本語

*A fast, agent-native time-series database and backtesting engine for quant research. Embedded, written in Rust.*

• Fast for time-series shape: over 4.5× faster than DuckDB and Polars on OHLCV+VWAP rollups over 20M rows.
• Native time-series SQL: ASOF join, timezone-aware time_bucket, gapfill/resample, rolling windows, vwap, ewma.
• Point-in-time reads: pin a decision time and the frame that reaches pandas cannot contain rows from after it. No lookahead bias, by construction.
• Efficient event-driven backtester: 3.05M events/s through the replay kernel, 11.7× NautilusTrader and 31× LEAN on a shared top-of-book workload.
• Native venue support: Kalshi, Polymarket, Hyperliquid, Binance and more.
• Professional statistical analysis: factor and performance metrics at alphalens and empyrical parity, plus deflated Sharpe and overfitting-probability detection.
• Fork a database in milliseconds: forks share data instead of copying it. Agents can run wide trial-and-error loops (fork, mutate, evaluate, discard) at almost zero cost.
• Every write is an atomic, versioned commit: any past version reads in O(1), so a bad ingest (human or agent) is one restore away from undone.
• Safety policies for agent writes: previewable mutations, policy gates, fail-closed constraints that block destructive operations, and an audit trail of what changed and why.

📖 Documentation · Backtesting · Quant · Python API · Cookbook · Agent skill

Quickstart

CLI

cargo install h5i-db-cli

h5i-db init market.db
h5i-db create-table market.db trades --like ticks.parquet --time-column ts
h5i-db ingest market.db trades ticks.parquet --idempotency-key load-1
h5i-db context market.db                                           # orient in one call
h5i-db query market.db "SELECT symbol, vwap(price,size) FROM trades GROUP BY symbol"
h5i-db query market.db "SELECT count(*) FROM trades" \
  --decision-time 2026-07-01T00:00:00Z                             # the future is unreadable
h5i-db ui market.db                                                # review + experiments surface

Notebook in the terminal

h5i-db nb new research.ipynb --kernel python3 --db market.db   # the database SQL cells query
h5i-db nb view research.ipynb        # ⏎ edit · e run · a insert · ? all keys · q quit
h5i-db nb watch research.ipynb --split right   # follow along while an agent runs cells
h5i-db nb export research.ipynb --to html

It writes a real .ipynb, so JupyterLab opens the same file. A Python kernel needs pip install ipykernel. Agents drive the same session non-interactively with h5i-db nb exec.

Python Library for DataFrames and SQL

pip install h5i-db

import pyarrow as pa
import h5i_db

db = h5i_db.Database("market.db", create=True)

db.create_table(
    "trades",
    pa.schema([("ts", pa.timestamp("us")), ("symbol", pa.string()), ("price", pa.float64())]),
    time_column="ts",
)
db.append("trades", pa.table({
    "ts": pa.array([1_700_000_000_000_000, 1_700_000_060_000_000], pa.timestamp("us")),
    "symbol": ["AAPL", "MSFT"], "price": [187.4, 411.2],
}))

df = db.sql("SELECT symbol, avg(price) AS px FROM trades GROUP BY symbol").to_pandas()
# df = db.table("trades").group_by("symbol").agg(px=col("price").mean()).to_pandas()
old = db.read("trades", version=1)                # time travel: read any past version

plan = db.plan_delete_range("trades", 1_700_0_000_000)
print(plan.summary)                               # preview the mutation before it lands
plan.apply()

Python Library for Backtesting (same install, no server)

from h5i_db import backtest

config = backtest.BacktestConfig(
    run_id="momentum-001",
    data=backtest.DataConfig(signals="signals", snapshot="2024-q1"),   # the pin
    portfolio=backtest.PortfolioConfig(starting_cash=100_000.0),
    execution=backtest.ExecutionConfig(fee_kind="kalshi", fee_rate=0.07),
    risk=backtest.RiskConfig(max_order_quantity=500.0),
)

backtest.inspect(db, config).raise_for_errors()  # refuse claims the data can't support
result = backtest.execute(db, config)            # replays inside fork "bt-momentum-001"

result.summary()                  # fills, final cash, how far it actually simulated
result.explain()                  # why orders were rejected or never filled
result.fills                      # Arrow, or just SQL it: SELECT * FROM bt_fills
result.tearsheet("run.html")
result.verify()                   # re-execute the stored config and compare

A parameter grid becomes one fork per trial, and the winner is ranked without an export step. Give it explicit train and holdout windows and every trial runs both phases, so the leaderboard can be read out of sample:

board = backtest.study(
    db, study_id="fees", base=config,
    parameters={"execution.fee_rate": [0.0, 0.02, 0.07]},
    validation=backtest.ValidationWindows(
        train=("2024-01-01", "2024-04-01"), holdout=("2024-04-01", "2024-07-01")
    ),
).leaderboard("holdout_final_cash")

Agent skill (Claude Code, Codex, Cursor, …)

npx skills add h5i-dev/h5i-db        # installs the h5i-db skill from skills/h5i-db/

See it work

python examples/backtest_report_demo.py

Why it's fast

• Manifest pruning: every version's manifest carries per-segment time ranges and column min/max. Narrow queries prune whole segments before a single file is opened.
• Declared sort order: segments are stored time-sorted and the query layer tells DataFusion so. OHLCV rollups stream instead of sorting 20M rows first (every baseline pays that sort), and the ASOF join is sort-free.
• Immutable segments: footer metadata is cached unconditionally (sound because segments never change), cutting ~40% off warm scans.
• Version-aware aggregate states: OHLCV/VWAP rollups persist mergeable states per immutable segment; re-queries merge states in milliseconds instead of recomputing, scanning only newly appended segments.
• Lazy replay: the backtest kernel pulls records one at a time instead of materializing a window, so memory stays flat whether a run replays a day or a hundred million events.
• Indexed order matching: resting orders are keyed by market and price, so a new print wakes only the orders it actually crosses instead of rescanning every open one.

Why for agents

• Reproducible inputs: every read resolves to a version, so "which data did this run see" has an answer, and re-running against that version is O(1) rather than an archaeology project.
• Don't let a result destroy the context window. H5I_DB_PROFILE=agent caps every query and spills the rest to Parquet, reporting the true row count and where the withheld rows live.
• Errors that can be acted on: the stderr envelope carries next_actions (runnable commands), did_you_mean for typos, and a retryable flag.
• Branch without copying. fork opens a writable workspace over a pinned view of every table and duplicates no data, so an edit or an experiment costs one small file and is as cheap to discard as to keep.
• Privilege control. Mutations preview through plan/apply and policy can require that gate; --idempotency-key makes a retried ingest replay; an opt-in data-policy rejects malformed rows fail-closed.
• A backtest run is a branch. Each run executes inside its own fork and writes its orders, fills, positions and equity curve there as ordinary tables. So two runs diff at fill level with fork_diff, a whole sweep aggregates in one cross-fork query, the one worth keeping is promoted and the rest are dropped.
• The review surface routes attention rather than ranking. h5i-db ui orders trials by what needs a human next: decision required, then failed or warned, then finished and unseen, then running, then seen. Scanning a list does not mark work reviewed; a trial counts as seen only when its detail is opened.

Data sources

Loaders read files and payloads you already have. Nothing here fetches, so credentials, retries and rate limits stay in your script.

derived means bars are aggregated from that source's own prints rather than fetched, so gaps stay visible as missing bars. n/a means the venue has no such concept: Manifold is an automated market maker, so it has prints but no book. See venue guide for the detail.

When not to use h5i-db

• Distributed, multi-terabyte warehouses: single-node and embedded by design. Reach for ClickHouse, Snowflake or a lakehouse.
• OLTP or high-concurrency serving: one writer at a time, no row-level MVCC, no interactive transactions. Use Postgres.
• Sub-microsecond tick capture: the write cadence this is built for is minute bars, end-of-day, and vendor files, not the capture layer itself. That is kdb+ territory.
• Databases with no time column: the whole design assumes a time index; without one you lose pruning, the ASOF join, and point-in-time reads.
• Live trading: the backtester never routes a real order. There are no brokerage adapters, no portfolio optimiser and no plotting API; the boundary is simulation and evaluation.

Benchmark

Full methodology and results in benchmarks.

Database

¹ AT (VERSION …) syntax exists but native storage rejects it. ² Experimental join_asof exists but is ~1000× slower, impractical at this scale. ³ Documented single-writer-per-symbol assumption. ⁴ Native ASOF JOIN … MATCH_CONDITION SQL syntax and an asof_join(...) table function (SQL and Python). ⁵ ArcticDB's native time index wins narrow point reads from its own LMDB store; h5i-db's manifest pruning is second and beats every general engine.

Backtesting

Medians of three fresh-process runs after one warm-up; each adapter verifies it saw all 200k events and submitted all 200 orders. The measured boundaries differ, as the column says, and the benchmark checks counts rather than PnL equivalence. ⁶ The other rows never call Python; this one crosses into it per event, as Nautilus does. Callback against callback the gap is 3.1×, not 13×. Derived (native kernel plus measured boundary cost), not timed directly. ⁷ --features wide, off by default; see Precision and range. Derived, not timed directly; method in RESULTS.md.

Development

# One package at a time with capped threads: several suites start DataFusion
# sessions and tokio runtimes, so running the whole workspace at once can
# exhaust memory on a small machine.
for pkg in h5i-db-core h5i-db-query h5i-db-backtest h5i-db-cli h5i-db-ui; do
  cargo test -p $pkg -- --test-threads=2
done
cargo run -p h5i-db-bench --profile bench-fast -- --trades 1000000
cargo run -p h5i-db-bench --profile bench-fast --bin h5i-db-fork-bench
python3 benchmarks/backtest_compare/run.py \
  --output benchmarks/backtest_compare/results.json   # vs NautilusTrader and LEAN

Workspace crates under crates/: core (versioned storage kernel), query (DataFusion layer), backtest (replay kernel, venue models, settlement), venues (Kalshi, Polymarket, Hyperliquid loaders), cli (the agent-facing binary), ui (review surface), observability, python (pip install h5i-db), bench.