AI Models vs Prediction Markets: The Graded Scoreboard

Updated August 13, 2026 · 12 min read · by Jake Hari

Every AI lab has a story about how well its model predicts the world. Almost nobody publishes the ledger. This page is the ledger. For months, our panel of AI models has put a price-blind probability on real Kalshi markets before they settle, every call stored before the outcome was knowable, and then graded against what actually happened. The headline finding is not flattering to the machines, which is exactly why you can trust the rest of it: on the ai models vs prediction markets question, the market is winning. Here is every seat's record anyway, because a scoreboard that only publishes when it is ahead is a press release.

The Quick Answer

Across 8,380 settled Kalshi markets, our multi-model AI panel's blend scores a Brier of 0.171 against the market's 0.163 on the same markets — the market is still the better forecaster overall, and we publish that plainly because the exceptions are where the story is. The closest seat is Gemini Pro (0.115 vs 0.112 on its 7,872 graded markets). The full seat-by-seat board, the calibration table, and named receipts are below.

All numbers as of November 8, 2026, recomputed at build time from 147,283 graded verdict rows across 8,380 settled markets · updated weekly · every verdict is logged before settlement, and nothing is ever quoted from memory.

The Leaderboard: Every Seat, Graded

Each row is one model seat. Its Brier score is computed only on the markets that seat actually priced, and the market's Brier beside it is computed on those exact same markets, so every comparison is apples to apples. Lower is better. The head-to-head column counts, market by market, whether the seat's number or the market's price landed closer to the outcome, written as wins–losses–ties.

Seat	Graded	Seat Brier	Market Brier	Δ	Beat The Market	Win %
Gemini Pro	7,872	0.1152	0.1119	+0.0034	1,860–2,895–3,117	39.1%
Claude Opus	8,235	0.0871	0.0833	+0.0038	2,174–5,835–226	27.1%
Claude Fable	7,579	0.0885	0.0845	+0.0040	2,033–5,065–481	28.6%
Grok	10,903	0.1074	0.1010	+0.0064	2,698–6,116–2,089	30.6%
GPT	8,976	0.1178	0.1097	+0.0081	2,527–5,225–1,224	32.6%
Kimi	6,685	0.1134	0.0991	+0.0142	1,981–3,754–950	34.5%
Claude Sonnet	8,144	0.0976	0.0826	+0.0150	2,135–4,286–1,723	33.3%
GLM	2,089	0.1474	0.1219	+0.0255	636–1,216–237	34.3%
DeepSeek	1,958	0.1570	0.1168	+0.0402	665–964–329	40.8%
Gemini	208	0.2724	0.1371	+0.1353	66–133–9	33.2%

Three honest notes on reading it. First, no seat beats the market on Brier over its full sample; the best seats lose narrowly, and the gap is the price of forecasting from training data against a crowd trading live information. Second, sample sizes differ because the panel's lineup has grown and changed; seats with a few hundred graded markets joined later or run on fewer boards, and small samples move around. Third, ties are markets where the seat's number and the price sat exactly the same distance from the outcome, which happens often when a model agrees with the market and rounds to the same figure.

Hottest Prediction Markets Right Now

• 2028 Democratic presidential nominee · $179M traded
• 2027 Pro Football Champion · $61M traded
• 2028 U.S. Presidential Election winner? · $59M traded
• 2028 Republican presidential nominee · $57M traded
• FedEx St. Jude Championship Winner · $54M traded
• Pro Baseball Champion · $53M traded

Every market above links to our full AI model verdict; browse them all on the OddsShopper prediction markets hub, and see how every settled call actually scored on the full graded scoreboard.

What A Brier Score Actually Is

A Brier score is the squared error of a probability forecast, averaged over every forecast made. Say 90% on things that happen and it rewards you; say 90% on things that do not and it punishes you hard. Zero is a crystal ball. A coin-flip shrug on everything scores 0.25. The market's roughly 0.115 here means Kalshi prices are, on average, very good probabilities, which matches what the accuracy research on prediction markets keeps finding. The interesting question this page tracks is not whether the market is good. It is where a model, or the blend of all of them, closes the distance.

The Record By Category

Category	Graded	Blend Brier	Market Brier	Verdict	Best Seat (n≥20)
Sports	2,152	0.1888	0.1799	market ahead	Claude Opus (0.1476, n=1385)
Politics	323	0.1805	0.1389	market ahead	Grok (0.1203, n=412)
Finance	859	0.1591	0.1474	market ahead	GLM (0.0996, n=225)
Weather	1,910	0.1932	0.1539	market ahead	Grok (0.1330, n=2242)
Entertainment	74	0.0948	0.0637	market ahead	Claude Fable (0.0501, n=72)
Mentions	359	0.2014	0.1659	market ahead	Gemini Pro (0.1236, n=84)
Crypto	1,643	0.1193	0.1638	blend ahead	Claude Fable (0.0213, n=3550)
Other	1,042	0.1734	0.1714	even	Claude Opus (0.1269, n=455)
Sports_News	1	0.0153	0.0841	blend ahead	—
Culture	1	0.1167	0.0064	market ahead	—
Financials	2	0.0002	0.0050	even	—
Elections	7	0.1359	0.0627	market ahead	—
Sports.Nextteam	1	0.0115	0.0090	even	—
Companies	2	0.1604	0.0324	market ahead	—
Sports-Awards	4	0.2186	0.0381	market ahead	—

The pattern to take from this table is uniformity: the market is ahead in essentially every category, but the margin varies a lot. The blend runs closest in the catch-all and finance buckets. It sits furthest behind in politics, on a small sample, and in sports, where lineup news and late information move prices in ways a price-blind model cannot see. Where a single seat's number looks better than the blend in a category, treat it as promising rather than proven; those cells carry smaller samples, and this page re-grades them every week.

Calibration: When The Panel Says 70%, What Happens?

Brier measures error. Calibration measures honesty: when the blend said an event was 70-80% likely, how often did it actually happen?

Panel Said	Markets	Avg Prediction	Actually Happened
0–10%	1,451	5.0%	7.5%
10–20%	1,255	14.7%	14.9%
20–30%	1,022	25.1%	25.9%
30–40%	1,018	34.9%	38.7%
40–50%	917	44.5%	45.0%
50–60%	790	54.7%	55.7%
60–70%	589	64.6%	64.5%
70–80%	533	74.5%	78.2%
80–90%	462	85.0%	85.3%
90–100%	343	93.5%	90.7%

Two things stand out. In the tails, the panel is well calibrated: things it called near-certain happened about as often as advertised. In the 70-90% band, events happened more often than the panel said, meaning the blend has been underconfident about likely outcomes; it hedges toward 50 more than the world does. And in the 10-20% band the opposite shows up, with fewer hits than predicted. That hedging bias is a known large-language-model habit, it is measurable here on real settled markets, and it is one concrete reason the market's sharper numbers keep winning the Brier column.

The Research Findings

• The mentions family is close, and the panel still trails. On 411 graded “will he say it” contracts, the blend's Brier of 0.1780 runs nearer the market's 0.1450 than in most categories, without passing it. These markets reward reading a speaker's verbal habits, which is text analysis, which should be the one thing a language model is built for; the graded rows say the market still reads the room better.
• The always-YES stress test. 51.3% of graded mention contracts settled Yes, so a strategy that mindlessly answers Yes on every one scores a Brier of 0.4866 here. Any seat claiming mention-market skill has to beat that dumb baseline, not just the market; the table above shows which ones do.
• When the panel actually disagrees with the market (blend at least 10 points off the price), it has been right 1062 times and wrong 1425 times on 2487 settled calls. Divergence is an editorial signal, not an automatic edge, and the misses below are part of the record.
• And the harder it disagrees, the worse it does. 42.7% at 10¢ (2487 calls) → 42.8% at 15¢ (1642 calls) → 44.4% at 20¢ (1164 calls) → 53.3% at 30¢ (628 calls). The slope is not monotonic in this rebuild, which is itself worth watching — it was strictly falling when this was first measured.

Receipts: Calls The Panel Got Right

Every receipt below is a settled market where the blend stood at least ten points from the price and the outcome broke the panel's way. The reasoning quotes are the models' own stored rationales, written before settlement, pulled verbatim from the ledger.

• Dallas vs Green Bay: 1st Touchdown: Matthew Golden: 1st Touchdown (settled Yes). Market price 8¢; panel blend 98%. Grok put it at 98% before settlement, reasoning: “Settlement-source and live feeds show Golden already scored the game’s first touchdown; outcome is determined.”
• ETH price on Oct 4, 2026 at 12pm EDT?: Ethereum price at Oct 4, 2026 at 12pm EDT? (leg T2874.99) (settled No). Market price 77¢; panel blend 0%. GPT put it at 0% before settlement, reasoning: “One near-expiry distribution centered at $2,699 with roughly $10 short-horizon sigma makes $2,705 live, lower strikes nearly certain, and higher strikes remote.”
• What will The Procter & Gamble Company say during their next earnings call? (leg CHAR) (settled Yes). Market price 14¢; panel blend 89%. Claude Fable put it at 96% before settlement, reasoning: “Charmin was named repeatedly last quarter, the Bounty/Charmin recovery is management's stated focus, and a major Charmin innovation launched days ago, making a mention near-certain.”
• ETH price on Oct 7, 2026 at 4am EDT?: Ethereum price at Oct 7, 2026 at 4am EDT? (leg T2794.99) (settled No). Market price 71¢; panel blend 0%. GPT put it at 0% before settlement, reasoning: “Using one short-horizon distribution centered near $2,616-$2,618 with elevated intraday uncertainty, lower strikes are nearly settled while $2,625 is the live cutoff.”

The Misses, Because A Scoreboard Without Them Is An Ad

• Will the maximum temperature in Los Angeles be >78° on Oct 19, 2026? (settled Yes). Market price 91¢; panel blend 6%. Grok put it at 3% before settlement, reasoning: “Latest KLAX-specific forecast centers on 73F after a morning downward revision; sunny day lifts above overnight 70F.”
• Generic Ballot: Democrats Up/Down (9/25-10/2)?: Above 8.3% (settled Yes). Market price 94¢; panel blend 14%. Gemini Pro put it at 4% before settlement, reasoning: “With the current margin 1.5 points short and under 24 hours remaining, crossing the threshold would require an unlikely, extreme new poll to dramatically shift the time-weighted average.”
• Will the minimum temperature in New York City be 43-44° on Nov 2, 2026? (settled Yes). Market price 90¢; panel blend 7%.

The LeBron James rows are the most instructive failure in the ledger. The panel was asked while the market already sat at 99 cents, because the news had broken and the market had already repriced. The models, reasoning from a world that ended the day their training data did, coherently argued for a world that no longer existed. That is the whole moral of this page in one market: when the question is about fresh information, the market's edge is structural, and no amount of eloquent reasoning closes it. Model edges, where they exist, live in pattern-reading, not news.

How The Grading Works

The pipeline is the same for every market, every week:

• Price-blind cards. Each model receives a data card with the market's real settlement rules and deterministically fetched facts, and never the market price. It returns a probability and a written rationale.
• A revision round. Seats then read each other's anonymized reasoning and may revise; revised numbers are stored separately, and the blend is the equal-weight mean of the seats.
• Logged before settlement. Every verdict row is written to the ledger with its timestamp and the market price at ask time, before the outcome exists. A market settles Yes or No per its own rules text, an outcome checker grades it, and this page recomputes every statistic from the raw rows on each rebuild.
• Reporting hygiene. Near-certain longshot legs where model and market both said under 3% and the answer was No are excluded from scored tables, because piling up gimme points is how a leaderboard lies. The seat table needs 20 graded markets before a seat appears.

The blend-level version of this record, with the ten biggest wins and live open divergences, lives on the model verdict scoreboard; this page is the seat-level research layer on the same ledger. The panel's current open calls with 3x-the-market upside are on Kalshi longshot picks, the markets where traders themselves are split hardest are in the most contested prediction markets, and the settlement fine print that decides several of these grades gets read weekly in Kalshi Fine Print Watch.

AI Forecasting vs Markets FAQ

Can AI beat prediction markets? Not overall, on our ledger. Every seat trails the market's Brier score on its full graded sample. The blend gets close in some categories, single seats beat the market head-to-head on a quarter to nearly half of the markets they price, and the panel's calls that diverged from the price have been right a minority of the time; the findings section above carries the current counts. That is a real research result, not a sales pitch.

Why grade against the market instead of just reporting accuracy? Because raw accuracy flatters everyone. Most markets resolve the way they were priced to resolve, so a forecaster can look brilliant by agreeing with the price. Grading against the market on the same markets isolates the only question that matters: did the model add information the price did not already have?

What would change this scoreboard's verdict? A seat sustaining a lower Brier than the market over a thousand-plus graded markets, or a category where the blend's lead survives growing samples. The weekly rebuild means either would show up here without anyone deciding to announce it.

The Bottom Line

Prediction markets are the strongest public forecasting machine anyone has built, and this ledger says so with our own models' report card. The value of running the panel anyway is the map it produces: where machines hedge, where they read patterns well, and exactly how large the market's information edge is, measured in Brier points instead of vibes. Leaderboards that launch empty, or gate their grades behind a signup form, are asking you to trust a scoreboard nobody can read. Ours is above, misses included, and it re-grades itself every week whether the numbers flatter us or not.

Event contracts are CFTC-regulated financial products offered on exchanges such as Kalshi, available to eligible U.S. residents 18+. The panel's probabilities are model estimates, not predictions of fact and not financial advice; market prices move constantly, and nothing here suggests any forecast or strategy will be profitable.