← Back to football hub

Model Calibration Record

Every GoalPulse club-football forecast is scored against the real result on a held-out test set. This page shows whether, when the model says 70%, it happens about 70% of the time — the definition of calibration. The evaluation is produced by an offline script over historical matches; the same numbers feed the live match pages.

Held-out matches
6,249
Brier
0.5961 vs 0.6494
skill 8.2%
RPS
0.2029 vs 0.2285
skill 11.2%
Weighted mean gap
0.0099

The line that matters: The largest gap among trustworthy buckets is the 74% band: the model said 74%, it happened 79.5% of the time, |gap| 0.051 on n=356; 5 of 10 buckets sit within ±2pp.

Calibration curve

Each point is one probability bucket: its horizontal position is the probability the model assigned, its vertical position is how often that actually happened. Points on the dashed line are perfectly calibrated; above the line the model under-estimates those matches, below it the model over-estimates them. Point size is the number of matches in the bucket.

Calibration curve: predicted probability against observed frequencyRead across, not along: a point below the dashed line means that band was given more confidence than the results justified.0%0%25%25%50%50%75%75%100%100%perfect calibration8% → 3.2% (n=314, gap 0.044)16% → 14.5% (n=2,197, gap 0.016)26% → 26.1% (n=7,574, gap 0.004)34% → 33.6% (n=3,656, gap 0.006)45% → 45.5% (n=2,378, gap 0.008)55% → 53.1% (n=1,411, gap 0.015)64% → 67.7% (n=770, gap 0.032)74% → 79.5% (n=356, gap 0.051)83% → 84.4% (n=90, gap 0.010)92% → 100.0% (n=1, gap 0.077)Predicted probabilityObserved frequency
at least 100 matches fewer than 100 matches

Read across, not along: a point below the dashed line means that band was given more confidence than the results justified.

What to look for

  • Distance from the dashed line matters more than the order of the points: a bucket sitting below the line means the model was more confident than the results allow.
  • The two hollow points carry fewer than 100 matches each, so they are shown for transparency and never used to claim accuracy on their own.
  • The two biggest buckets (16% and 26%) together hold 9,771 of the 18,747 scored 1X2 outcomes (three outcomes per match), and both sit almost exactly on the line — which is why the weighted mean gap is 0.0099.

Predicted probability bucket

BucketnPredictedObserved|gap|Verdict
8%3147.6%3.2%0.044Needs care
16%2,19716.0%14.5%0.016Calibrated
26%7,57425.7%26.1%0.004Calibrated
34%3,65634.3%33.6%0.006Calibrated
45%2,37844.7%45.5%0.008Calibrated
55%1,41154.6%53.1%0.015Calibrated
64%77064.5%67.7%0.032Minor drift
74%35674.4%79.5%0.051Needs care
83%9083.4%84.4%0.010Thin samplen<100
92%192.3%100.0%0.077Thin samplen<100

Per league: how much the blend adds over the model alone

The same held-out rows split by competition, comparing the blended RPS with model-only RPS. Competitions under 200 matches are not given a verdict.

CompetitionnBlend RPSModel-only RPSGainVerdict
mls1,0350.21220.2170+0.0048Blend better
championship7250.21570.2174+0.0017Blend better
brasileirao7180.19580.1992+0.0034Blend better
la-liga4890.19240.1942+0.0018Blend better
serie-a4800.19550.1966+0.0011Blend better
premier-league4700.20470.2071+0.0024Blend better
k-league-14360.22280.2228+0.0000No gain
eredivisie4050.19610.1974+0.0013Blend better
primeira-liga3960.17660.1790+0.0024Blend better
ligue-13860.19920.1998+0.0006Blend better
bundesliga3790.19540.1977+0.0023Blend better
j1-league3300.21200.2115-0.0005No gain

Skill vs baseline

The baseline is the market-consensus-only forecast (no Dixon-Coles model). Positive skill means the blended model beats the baseline on the same matches.

Brier (model vs baseline)0.5961 vs 0.6494 (8.2% better)
RPS (model vs baseline)0.2029 vs 0.2285 (11.2% better)
Accuracy (model vs baseline)51.4% vs 43.7%

Headline metrics (test set)

Matches6,249
Top-1 accuracy (1X2)51.4%
Brier score (1X2, lower is better)0.5961
Log loss (1X2, lower is better)0.9973
Ranked probability score (lower is better)0.2029
Brier (over/under 2.5)0.2409
Mean draw probability predicted25.4%
Draw rate observed25.9%
Weighted mean |calibration gap|0.0099

Full machine-readable evaluation: /data/football/model-evaluation.json (generated by scripts/football/evaluate-football.mjs). Methodology: Dixon-Coles bivariate Poisson with exponential time decay, blended with bookmaker consensus via logarithmic pooling.

How to read these numbers

Layer 1 — the headline

One number is easy to misread. The Brier skill line below compares the blended model against a market-consensus-only baseline on the exact same held-out matches: that gap, not the raw Brier value, is the claim this page makes.

Read the full explanation

One number is easy to misread. The Brier skill line below compares the blended model against a market-consensus-only baseline on the exact same held-out matches: that gap, not the raw Brier value, is the claim this page makes.

Layer 2 — per bucket

Calibration is checked bucket by bucket. A model can win on Brier overall while being over-confident in one band (for example the 65-75% band).

Read the full explanation

Calibration is checked bucket by bucket. A model can win on Brier overall while being over-confident in one band (for example the 65-75% band). Use the tier column to see which bands you can trust for a single match.

Layer 3 — how to use it

These are probabilities, not verdicts: a 62% favourite loses about 38% of the time. Buckets with very few matches carry a large sampling error, so they are marked separately and never used to claim accuracy on their own.

Read the full explanation

These are probabilities, not verdicts: a 62% favourite loses about 38% of the time. Buckets with very few matches carry a large sampling error, so they are marked separately and never used to claim accuracy on their own.

8162634455564748392

The top strip is equal-width for readability; this one is scaled by sample size, so the narrowest cells are the thin buckets.

Calibrated 5Minor drift 1Needs care 2Thin sample 2

Largest gap among trustworthy buckets: 74% → 79.5% (|gap| 0.051, n=356)

Buckets with fewer than 100 matches are marked "thin sample": the gap is shown for transparency but is not statistically meaningful on its own.