Our Model Is Wrong, and It Is Wrong in Your Favour

When our model says a horse has a middling chance of winning, those horses win considerably more often than that.

It is not a small discrepancy and it runs the same way at every level of confidence. The model understates, and the gap widens as it grows more confident.

The market does not do this. Across the same races, what the odds imply and what actually happens sit close together at every point on the scale.

So we have a forecaster that is systematically miscalibrated in the direction punters would like, sitting next to one that is nearly exact. And ours still loses money.

Predicted against actual

Every row understates, and the gap widens as confidence rises.

Now the same test on the market's own prices.

Several of those rows are exact and the rest are within a couple of points. That is what a well-calibrated forecaster looks like, and it is the collective judgement of everybody betting rather than any one clever system.

Why being underconfident does not help

It sounds like it should. If the model's horses win more often than it expects, surely there is money in that.

The trouble is that calibration and profit are separate questions. Being underconfident means the model's probability numbers are wrong. It does not mean it is picking horses the market has mispriced, and it is the second thing that pays.

The overall accuracy score settles it. On the standard measure across 10,478 races, our model reads 0.098 against the market's 0.090, where lower is better. The market is more accurate overall even though the model is generous at the top end, because accuracy is about every runner in every race rather than the confident ones.

And on top of that sits the charge. Backing the model's top pick across 10,297 bets has returned −14.4%, which is −£14,825.

We publish the calibration chart on our track record page and always have. This is not a flaw we discovered and hid. It is a flaw we discovered and put on a chart, because a model whose errors you cannot see is a model you cannot judge.

The standing rule behind that, and the way we invite you to check any of it, is in why we publish every losing bet.

A word on all of this

None of these pages is a tip, and none describes a way to win. They describe what betting costs, which is a different and more reliable subject. If your betting has stopped being fun, BeGambleAware has free, confidential help, and the National Gambling Helpline is on 0808 8020 133.

Common questions

What does it mean that the model is underconfident?

Its stated probabilities are too low. Horses it rates a middling chance win considerably more often than it says, and the same pattern holds at every level of confidence.

Does that make it a good bet?

No. Backing its top pick has returned −14.4% across 10,297 bets. Miscalibration in a helpful direction does not overcome the charge.

Is the market better than the model?

On accuracy, yes: 0.090 against the model's 0.098, where lower is better.

Why publish this at all?

Because a model whose errors are hidden cannot be judged, and because we would rather be the ones to point out our own miscalibration.

Will you fix it?

Recalibrating the output is straightforward and would make the probabilities more honest. It would not change which horses the model likes, so it would not change the returns.