Evaluating Predictive Models: Brier Score, Log-Loss & Calibrated Probabilities

1. The Flaw of Classification Accuracy in Probabilistic Models

Evaluating a probabilistic model strictly by classification accuracy (e.g., "72% of predicted winners won") hides critical calibration information.

Consider two models predicting 100 matches:

Both models share an identical 70% directional accuracy. However, Model A is dangerously overconfident, whereas Model B is perfectly calibrated. In quantitative research, proper scoring rules are required to measure calibration honesty.

2. Brier Score & Probability Calibration

The Brier Score is a strictly proper scoring rule that measures the mean squared difference between predicted probabilities $f_i$ and actual binary outcomes $y_i \in {0, 1}$:

A lower Brier Score indicates superior probability calibration (0 represents perfect deterministic prediction, while 0.25 represents an uncalibrated 50/50 guess).

Brier Score decomposes mathematically into three constituent components:

3. Logarithmic Loss (Log-Loss)

While Brier Score measures squared error, Logarithmic Loss penalizes confident misclassifications exponentially using the negative log-likelihood:

If a model predicts $f_i = 0.98$ for a player who subsequently loses ($y_i = 0$), Log-Loss imposes a severe mathematical penalty of $-\log(0.02) \approx 3.91$. This property makes Log-Loss the standard loss function for optimizing binary classification algorithms like XGBoost.

4. Expected Calibration Error (ECE) & Market Benchmarks

Expected Calibration Error (ECE) partitions predictions into $M$ equally spaced probability bins and computes the weighted average difference between accuracy and confidence across bins:

In sports analytics, the primary benchmark for probability calibration is the Closing Line Market Consensus (No-Vig Implied Probability). Because closing lines absorb information from global liquidity, their ECE is extremely low. Evaluating an internal predictive model against closing line probabilities provides a quantitative audit of model alpha.