Model Calibration Record
Every GoalPulse club-football forecast is scored against the real result on a held-out test set. This page shows whether, when the model says 70%, it happens about 70% of the time — the definition of calibration. The evaluation is produced by an offline script over historical matches; the same numbers feed the live match pages.
- Held-out matches
- 6,249
- Brier
- 0.5961 vs 0.6494
- skill 8.2%
- RPS
- 0.2029 vs 0.2285
- skill 11.2%
- Weighted mean gap
- 0.0099
The line that matters: The largest gap among trustworthy buckets is the 74% band: the model said 74%, it happened 79.5% of the time, |gap| 0.051 on n=356; 5 of 10 buckets sit within ±2pp.
Calibration curve
Each point is one probability bucket: its horizontal position is the probability the model assigned, its vertical position is how often that actually happened. Points on the dashed line are perfectly calibrated; above the line the model under-estimates those matches, below it the model over-estimates them. Point size is the number of matches in the bucket.
Read across, not along: a point below the dashed line means that band was given more confidence than the results justified.
What to look for
- Distance from the dashed line matters more than the order of the points: a bucket sitting below the line means the model was more confident than the results allow.
- The two hollow points carry fewer than 100 matches each, so they are shown for transparency and never used to claim accuracy on their own.
- The two biggest buckets (16% and 26%) together hold 9,771 of the 18,747 scored 1X2 outcomes (three outcomes per match), and both sit almost exactly on the line — which is why the weighted mean gap is 0.0099.
Predicted probability bucket
| Bucket | n | Predicted | Observed | |gap| | Verdict |
|---|---|---|---|---|---|
| 8% | 314 | 7.6% | 3.2% | 0.044 | Needs care |
| 16% | 2,197 | 16.0% | 14.5% | 0.016 | Calibrated |
| 26% | 7,574 | 25.7% | 26.1% | 0.004 | Calibrated |
| 34% | 3,656 | 34.3% | 33.6% | 0.006 | Calibrated |
| 45% | 2,378 | 44.7% | 45.5% | 0.008 | Calibrated |
| 55% | 1,411 | 54.6% | 53.1% | 0.015 | Calibrated |
| 64% | 770 | 64.5% | 67.7% | 0.032 | Minor drift |
| 74% | 356 | 74.4% | 79.5% | 0.051 | Needs care |
| 83% | 90 | 83.4% | 84.4% | 0.010 | Thin samplen<100 |
| 92% | 1 | 92.3% | 100.0% | 0.077 | Thin samplen<100 |
Per league: how much the blend adds over the model alone
The same held-out rows split by competition, comparing the blended RPS with model-only RPS. Competitions under 200 matches are not given a verdict.
| Competition | n | Blend RPS | Model-only RPS | Gain | Verdict |
|---|---|---|---|---|---|
| mls | 1,035 | 0.2122 | 0.2170 | +0.0048 | Blend better |
| championship | 725 | 0.2157 | 0.2174 | +0.0017 | Blend better |
| brasileirao | 718 | 0.1958 | 0.1992 | +0.0034 | Blend better |
| la-liga | 489 | 0.1924 | 0.1942 | +0.0018 | Blend better |
| serie-a | 480 | 0.1955 | 0.1966 | +0.0011 | Blend better |
| premier-league | 470 | 0.2047 | 0.2071 | +0.0024 | Blend better |
| k-league-1 | 436 | 0.2228 | 0.2228 | +0.0000 | No gain |
| eredivisie | 405 | 0.1961 | 0.1974 | +0.0013 | Blend better |
| primeira-liga | 396 | 0.1766 | 0.1790 | +0.0024 | Blend better |
| ligue-1 | 386 | 0.1992 | 0.1998 | +0.0006 | Blend better |
| bundesliga | 379 | 0.1954 | 0.1977 | +0.0023 | Blend better |
| j1-league | 330 | 0.2120 | 0.2115 | -0.0005 | No gain |
Skill vs baseline
The baseline is the market-consensus-only forecast (no Dixon-Coles model). Positive skill means the blended model beats the baseline on the same matches.
| Brier (model vs baseline) | 0.5961 vs 0.6494 (8.2% better) |
|---|---|
| RPS (model vs baseline) | 0.2029 vs 0.2285 (11.2% better) |
| Accuracy (model vs baseline) | 51.4% vs 43.7% |
Headline metrics (test set)
| Matches | 6,249 |
|---|---|
| Top-1 accuracy (1X2) | 51.4% |
| Brier score (1X2, lower is better) | 0.5961 |
| Log loss (1X2, lower is better) | 0.9973 |
| Ranked probability score (lower is better) | 0.2029 |
| Brier (over/under 2.5) | 0.2409 |
| Mean draw probability predicted | 25.4% |
| Draw rate observed | 25.9% |
| Weighted mean |calibration gap| | 0.0099 |
Full machine-readable evaluation: /data/football/model-evaluation.json (generated by scripts/football/evaluate-football.mjs). Methodology: Dixon-Coles bivariate Poisson with exponential time decay, blended with bookmaker consensus via logarithmic pooling.
How to read these numbers
Layer 1 — the headline
One number is easy to misread. The Brier skill line below compares the blended model against a market-consensus-only baseline on the exact same held-out matches: that gap, not the raw Brier value, is the claim this page makes.
Read the full explanation
One number is easy to misread. The Brier skill line below compares the blended model against a market-consensus-only baseline on the exact same held-out matches: that gap, not the raw Brier value, is the claim this page makes.
Layer 2 — per bucket
Calibration is checked bucket by bucket. A model can win on Brier overall while being over-confident in one band (for example the 65-75% band).
Read the full explanation
Calibration is checked bucket by bucket. A model can win on Brier overall while being over-confident in one band (for example the 65-75% band). Use the tier column to see which bands you can trust for a single match.
Layer 3 — how to use it
These are probabilities, not verdicts: a 62% favourite loses about 38% of the time. Buckets with very few matches carry a large sampling error, so they are marked separately and never used to claim accuracy on their own.
Read the full explanation
These are probabilities, not verdicts: a 62% favourite loses about 38% of the time. Buckets with very few matches carry a large sampling error, so they are marked separately and never used to claim accuracy on their own.
The top strip is equal-width for readability; this one is scaled by sample size, so the narrowest cells are the thin buckets.
Largest gap among trustworthy buckets: 74% → 79.5% (|gap| 0.051, n=356)
Buckets with fewer than 100 matches are marked "thin sample": the gap is shown for transparency but is not statistically meaningful on its own.