Data schema

One row = one second of one market. All timestamps UTC. Prices are probabilities in [0, 1]; sizes are share counts.

Columns

Column	Type	Meaning
ts_iso	string	UTC second this row describes, e.g. 2026-10-30T03:34:00Z
ts_unix	int	The same second as Unix time
slug	string	Market name, e.g. btc-updown-5m-1793331300
symbol	string	btc · eth · sol · xrp
timeframe_minutes	int	5 or 15
window_start_unix	int	Market open time (Unix seconds)
window_end_unix	int	Market settlement time
phase	string	preopen (the 60s before open) or live
yes_bid / yes_ask	float	Best bid and ask on the YES token — captured at the same instant, never crossed
yes_bid_size / yes_ask_size	float	Shares resting at those prices
no_bid / no_ask	float	Best bid and ask on the NO token
no_bid_size / no_ask_size	float	Shares resting at those prices
yes_feed_age_ms / no_feed_age_ms	int	Milliseconds since the last book update when the row was written
yes_crossed / no_crossed	int	1 if bid > ask at snapshot time (legacy rows only — kept verbatim)
schema_version	int	1 = legacy capture, 2 = current collector

Empty values

Empty price cells mean no observation — the token had no book yet, or the connection was re-establishing. They are never filled with a synthetic value.

Use feed_age_ms to filter:

-- only rows observed within the last second
SELECT * FROM read_parquet('.../**/*.parquet', hive_partitioning = true)
WHERE yes_bid IS NOT NULL AND yes_feed_age_ms <= 1000;

The 60-second overlap

Recording starts 60 seconds before a market opens (phase = "preopen"). Consecutive windows therefore share exactly 60 seconds of history — no blind spot at the boundary. When concatenating files, de-duplicate on (slug, ts_unix).

Market details (sidecar)

Each window has a .meta.json sidecar with condition_id, both token IDs, the market question and resolution details — the venues delete these markets shortly after settlement, so this is the durable record of the market's identity.

Legacy rows

Rows with schema_version = 1 come from the earliest collector and can contain crossed quotes (flagged by the *_crossed columns). For modelling work, filter them:

WHERE yes_crossed = 0 AND no_crossed = 0