The AI Lab · How we test
The Brier score: one number that keeps a forecaster honest
Anyone can claim their predictions are good. The Brier score is how you check — a single number that rewards saying how likely something is and meaning it. It is how we grade our own racing model against the betting market, in public, on every race.
The idea, in plain English
Take any forecast that comes as a percentage — “this horse has a 25% chance”. After the race you know the truth: the horse won (call it 1) or it didn't (call it 0). The Brier score simply measures how far the forecast was from the truth: take the gap between the probability and the outcome, square it, and average that over every runner in every race.
Squaring does two clever things. It makes every miss positive so they can't cancel out, and it punishes confident misses much harder than modest ones. Say a horse loses: if you had given it a 20% chance your penalty is small (0.2² = 0.04); if you had shouted 90% your penalty is huge (0.9² = 0.81). Lower is better, and 0 would be a perfect — omniscient — forecaster.
That is the whole trick: you cannot game it. Hedge everything at 50-50 and the score is mediocre; shout certainties and the misses bury you. The only way to a good Brier score is the boring one — give every runner its honest chance. It was invented in 1950 by the meteorologist Glenn Brier to keep weather forecasts honest, and it does exactly the same job on a racecard.
Our model vs the market — the live example
Our model
0.1023
The market
0.0930
Those are the live all-time scores over 7,290 bets, updated nightly from the model's public ledger— the model's probabilities and the market's own implied probabilities, scored on the same races so the comparison is fair.
Read it honestly: the market is about 10% sharper. Our model is a genuinely well-calibrated forecaster — when it says 25%, that horse wins about a quarter of the time — but the market has everything the model knows and more bet into it. The gap between the two numbers is the whole story of the Lab: a forecaster can be excellent and still not beat the price, because the price already contains the crowd's knowledge plus the bookmaker's margin.
Why this matters if you bet
Because it separates the two things every tipster blurs together. Calibration — do the stated chances match reality? — is what the Brier score measures, and it is checkable. Profitability — do you get paid enough when you are right? — is a different question entirely, decided by the prices you take, not the forecasts you make.
A seller of tips will show you winners. A forecaster who wants to be trusted shows you their Brier score, because it counts every race, weights confidence honestly, and cannot be cherry-picked. That is why it is the number on our Lab's front door rather than a highlight reel of wins — and why the market's sharper score sits right next to ours, on the same races, where it belongs.
Questions
What is a good Brier score in horse racing?
It depends on field size, so compare against benchmarks on the same races rather than an absolute scale. Guessing every runner equally likely scores far worse than any informed forecaster; the betting market — the sharpest public forecaster there is — sets the practical gold standard. A model that gets close to the market's score on the same races is genuinely sharp.
Why not just use strike rate or profit?
Strike rate only checks the top pick and ignores everything the forecaster said about the rest of the field, and profit measures the prices you got as much as the predictions you made. The Brier score grades the whole probability forecast — every runner, every race — so it separates 'reads races well' from 'got lucky' far better than either.
Does a good Brier score mean the model makes money?
No — and that distinction is the honest heart of our whole Lab. Our model's Brier score shows it prices races almost as sharply as the betting market, yet it still loses at Starting Price, because the market's prices already contain everything the model knows plus a built-in margin. Calibration is about telling the truth; profit needs you to know something the market doesn't.
Where do your Brier numbers come from?
From the model's public ledger: every race it prices is scored against the actual result, per runner, and the same races are scored with the market's own implied probabilities so the two are directly comparable. The numbers on this page pull live from that ledger and update as it does — nothing is typed in by hand.
See the score in action on the model's track record, how the same model fares against five chatbots in the Silicon Tipster League, or how we settle everything in the methodology.