What we tried, and what we rejected
The shipped model came out of 60 proposals across 4 model families, scored on a walk-forward validation season with the locked test seasons opened once. The first 31 rounds (2026-11-07, Claude) produced what we run; of the 29 new rounds here, 4 beat production, 10 tied, 6 were worse and 4 never produced a runnable plugin, and 1 beat production on validation but lost on the locked test, so it was not shipped. A further 5 were restore-only rounds that reproduce an earlier configuration and are not counted twice. All 29 are listed below, including the ones that lost.
What runs in production
- Validation RPS
- 0.2039
- Validation log loss
- 0.9976
- Locked test RPS
- 0.2051
- Test log loss
- 1.0047
On the same test matches: results-only model RPS 0.2087, market-prior-only model 0.2071. Of 10 probability buckets, 5 are well calibrated and 2 are too thin to judge — full buckets in the calibration record.
Injury-intel shadow experiment (pre-registered, no interim result)
Before each kick-off a web-search model reads injury and lineup reports and returns a fact-and-source list. It never touches the published probabilities; it is scored alongside them on the same matches. The adoption rule was fixed before any data arrived: the 90% confidence interval of the paired log-loss difference must sit entirely on the better side, otherwise the idea is archived as a negative. Below 150 scored matches we publish no interim difference.
- Recorded
- 100
- Scored
- 98
- Pre-registered threshold
- 150
52 matches to go; the verdict will appear here either way, positive or negative.
The last 29 proposals and their results
Lower validation RPS is better; production sits at 0.2039. All 60 rounds cost $59.31 (claude-fable-5-1 $24.65, gpt-5.1 $0.67, claude-fable-5-2 $8.48, gpt-6.1-sol $0.86).
Machine-readable: /data/football-model-search.json
| Proposer | Round | Hypothesis | Validation RPS | Test RPS | Verdict |
|---|---|---|---|---|---|
| gpt-5.1 | 31 | Rescale score-only goal observations so each family’s long-run training goals match its own mean of real goals. | 0.2039 | — | Tied |
| gpt-5.1 | 32 | Replace the family-wide goal-level correction with a tier-aware level (tier-1 vs tier-2), blended by recent tier mix. | 0.2039 | — | Tied |
| gpt-5.1 | 33 | Give the most recent match a small extra weight per refit so ratings react faster to genuine strength changes. | 0.2039 | — | Tied |
| gpt-5.1 | 34 | Add a small symmetric rest-days term to the log-strength ratio. | 0.2040 | — | Worse |
| gpt-5.1 | 35 | Reweight short-term form/tempo residuals to give market-observed matches slightly more effective samples. | 0.2040 | — | Worse |
| gpt-5.1 | 36 | Taper the favourite stretch and rho shift in the first ten matchweeks.(plugin longer than 20,000 characters) | — | — | Contract rejection |
| gpt-5.1 | 37 | Ship the round-26 model unchanged.(plugin longer than 20,000 characters) | — | — | Contract rejection |
| claude-fable-5-2 | 31 | Opponent-adjust the fast form and tempo layers by backfitting, with the shrinkage re-derived each pass. | 0.2039 | — | Tied |
| claude-fable-5-2 | 32 | Adjust the form layer for rest asymmetry.(no <plugin> block in the answer) | — | — | Contract rejection |
| claude-fable-5-2 | 33 | Scale the tier-gap correction by the priced share of the current tier-2 teams’ training evidence. | 0.2039 | — | Tied |
| claude-fable-5-2 | 34 | Count the season break as one extra in-season half-life for priced rows. | 0.2040 | — | Worse |
| claude-fable-5-2 | 35 | Split the market residual by league tier.(no <plugin> block in the answer) | — | — | Contract rejection |
| claude-fable-5-2 | 36 | Scale the tier-gap correction by the mean priced share of the family. | 0.2039 | — | Tied |
| claude-fable-5-2 | 37 | Make the rho shift depend on match closeness. | 0.2039 | — | Tied |
| gpt-6.1-sol | 38 | Preserve the predicted goal total when applying strength, tier and form corrections. | 0.2039 | — | Tied |
| gpt-6.1-sol | 39 | Use only priced observations in the fast form and tempo layers, so short-window realised scores cannot add noise to unpriced fixtures. | 0.2038 | 0.2053 | Lost on test |
| gpt-6.1-sol | 40 | Consolidate round 39 unchanged, so no further validation-tuned parameter is added. | 0.2038 | — | Better |
| gpt-6.1-sol | 41 | Remove only the fast tempo adjustment and keep everything else. | 0.2038 | — | Better |
| gpt-6.1-sol | 42 | Restore the market-only fast tempo adjustment on top of round 39. | 0.2038 | — | Better |
| gpt-6.1-sol | 43 | Remove only the fast strength-form adjustment, keeping tempo. | 0.2047 | — | Worse |
| gpt-6.1-sol | 44 | Restore market-only fast strength form while keeping tempo. | 0.2038 | — | Same as round 39 |
| gpt-6.1-sol | 45 | Train priced rows on market-implied goals alone instead of the 75/25 mixture. | 0.2042 | — | Worse |
| gpt-6.1-sol | 46 | Restore the 75% market / 25% compressed-score training mixture. | 0.2038 | — | Same as round 39 |
| gpt-6.1-sol | 47 | Remove blow-out compression and its compensating goal-level correction. | 0.2039 | — | Tied |
| gpt-6.1-sol | 48 | Restore blow-out compression and its goal-level correction. | 0.2038 | — | Same as round 39 |
| gpt-6.1-sol | 49 | Remove only the manual rho offset. | 0.2039 | — | Tied |
| gpt-6.1-sol | 50 | Restore the source-aware rho offset. | 0.2038 | — | Same as round 39 |
| gpt-6.1-sol | 51 | Remove only the tier-gap correction. | 0.2042 | — | Worse |
| gpt-6.1-sol | 52 | Restore the source-scaled tier correction and otherwise ship round 39. | 0.2038 | — | Same as round 39 |
Why these rounds could not win
The shipped plugin already uses every structure that paid: a 0.5 prior, source-specific time decay, reliability weighting, a tier-gap shift, blow-out squashing, a source-aware stretch and rho pair, a self-shrunk 14-day form and tempo layer, and a decompressed market residual. Adding more structure moves the score inside the noise. What is left is information — injuries, lineups, fixture congestion, rotation — which is not in historical scores or closing prices and cannot be programmed into a function. That is why it runs as a separate experiment with the pre-registered rule above.