Calibration City

Resources

Prediction Market FAQ
Scott Alexander describes the basics of prediction markets, their current state, and novel uses.
Prediction Markets are not Polls
Isaac King describes specific advantages prediction markets have over polls.
Wikipedia: Scoring Rule
Wikipedia's article on scoring rules such as Brier scores and Logarithmic scores.
Metaculus Track Record
Metaculus's own excellent calibration page with lots of options and excellent visualizations.
Prediction Market Map
Saul Munn lists several prediction market platforms, organizations, tools, community projects, and more.

Implementation Details

When standardizing things across platforms we ran into some edge cases, I've tried to detail them all here. When in doubt, you can always check the source to see how we compute a specific attribute.

All
To calculate the time-averaged probability, we assume the market opens at 50%. Once the first trade occurs, we track the probability at each trade and the cumulative durations to generate an average.
Kalshi
Notes:
We use the YES price from the most recently executed trade as the probability at any point in time.
The counter for the number of unique traders is currently unimplemented.
Supported market types:
Binary
Multiple-Choice Unlinked
Multiple-Choice Linked
Manifold
Supported market types:
Binary
Pseudo-Numeric
Multiple-Choice Unlinked
Multiple-Choice Linked
Non-CPMM Markets
Metaculus
Notes:
We use the community prediction (history.x2.avg series) for the probability.
Since Metaculus does not have bets, we use the number of forecasts at 10 cents each for the market volume.
Supported market types:
Binary
Multiple-Choice
Polymarket
Notes:
We declare a market has started when the first trade occurs and end at the date noted by end_date_iso. This field is optional and markets without it are not counted.
The counter for the number of unique traders is currently unimplemented.
The counter for market volume is currently unimplemented.
Supported market types:
Single-Market Binary
Multiple-Choice Unlinked
Multiple-Choice Linked
Non-CLOB Markets
Disclaimer
I use Manifold much more than any of the other platforms included in this analysis, and have received bounties from the Manifold team in both mana (play money) and real money. Their contributions did not affect the contents of this site in any way.