calibration-check

Calibration Check (Sports)

Overview

A model can rank teams well and still be miscalibrated. If the model says 30%, about 30% of those cases should hit.

This skill measures probability reliability for sports models and supports a human-reviewed verdict grounded in evaluation provenance, uncertainty, and segment stability.

Discrimination metrics (AUC, accuracy) do not replace calibration.

When to Use This Skill

Use when:

• Model outputs win/event probabilities
• User asks how confident / how reliable the probs are
• After walk-forward evaluation of a probability model
• Before writing results that quote probability levels
• Comparing raw vs recalibrated probabilities
• sport- and competition-specific probability artifacts

Do not use when:

Installation

The bundled scripts require pandas:

python -m pip install pandas

Parquet input also needs pyarrow or fastparquet. Both scripts accept CSV, Parquet, JSON, JSONL, and NDJSON and expose --help without importing pandas.

What to Measure

1. Reliability / calibration curve — bin predicted prob vs observed rate
2. Expected Calibration Error (ECE)
3. Brier score (+ reliability/resolution decomposition when useful)
4. Log-loss — discrimination + calibration together; not a substitute for ECE
5. Segment calibration — by season, home/away, probability tail
6. Sharpness — are probs informative, not all ~0.5?

Read calibration_metrics.md when interpreting ECE, Brier score, log loss, or sharpness. Read binning.md before choosing bin edges or minimum counts. Read recalibration.md before fitting Platt, isotonic, or another probability correction.

Workflow

1. Confirm predictions come from time-safe / walk-forward folds.
2. Validate probabilities in [0, 1]; account for every excluded row.
3. Pre-declare binning (fixed-width or quantile).
4. Compute curve, ECE, Brier, log-loss.
5. Slice by season and by probability tails.
6. Issue a verdict (table below).
7. If recalibrating, only with nested/train-proper methods — never fit isotonic on the final test fold and call it validated.
8. Write the calibration report into experiment log / results writeup.

Verdict Scale

These verdicts belong to the full methodology, not to a single ECE threshold. Before issuing one, verify that predictions are genuinely held out and walk-forward, inspect uncertainty and populated-bin counts, compare seasons or folds, check decision-relevant segments, and document the prediction source. No universal ECE cutoff establishes that probabilities are trustworthy.

Run on Held-Out Prediction Artifacts

The input must contain one row per evaluated decision, a binary outcome, and a probability in [0, 1].

Preferred portable names from this pack's baseline helper:

• outcome: y_true
• probability: p_pred (also emitted as logistic_probability)

# After baseline-models/scripts/run_baselines.py --predictions-out ...
python /path/to/calibration-check/scripts/calibration_report.py \
  --input baseline-predictions.csv --target y_true --probability p_pred \
  --group-col season --bins 10 --out calibration.json

python /path/to/calibration-check/scripts/calibration_report.py \
  --input predictions.csv --target won --probability win_probability \
  --group-col season --bins 10 --out calibration.json

python /path/to/calibration-check/scripts/segment_calibration.py \
  --input predictions.csv --target won --probability win_probability \
  --segment-col is_home

Use --filter-col is_home --filter-value 1 for a symmetric team-game artifact when one home perspective per game is the actual evaluation unit. Do not double-count both sides of one event.

calibration_report.py returns Brier score, log loss, ECE, reliability-bin counts, row accounting, optional group metrics, and a conservative manual-review-required helper verdict. The script cannot prove held-out provenance or turn a favorable pooled ECE into well-calibrated; the analyst must apply the verdict scale above. segment_calibration.py prints all-row, categorical-segment, probability-tail metrics, and sparse-bin counts.

Binning Guidance

Always report bin counts. Empty bins are not evidence.

Recalibration Rules

Allowed

• Platt scaling / isotonic fit inside training folds only, applied to test fold
• Nested walk-forward recalibration

Not allowed

• Fit isotonic on final test labels and call it validated
• Hand-edit probabilities after seeing outcomes

After recalibration, re-report ECE/Brier on true forward folds.

Hard Constraints

1. Never evaluate calibration on training rows used to fit the same model without nested disclosure.
2. Never present raw scores as probabilities without checking calibration.
3. Never hide segment failures behind a pooled “looks fine.”
4. If sample per bin is tiny, say so — widen bins or reduce claim strength.
5. Accuracy is not calibration.
6. Leakage-invalid evaluations cannot be “well-calibrated.”

Anti-Patterns

• “56% correct, so calibrated”
• One reliability plot with no sample sizes
• Holdout isotonic theater
• Average prob ≈ base rate therefore calibrated (necessary, not sufficient)
• Ignoring 0.05 and 0.95 tails
• Quoting NBA/MLB percents without sport-specific calibration

Reporting Template

Calibration report
Sport/model:
Eval: walk-forward seasons …
n:
Brier:
ECE (bins=…):
Log-loss:
Notes by season:
Tail behavior:
Verdict: well-calibrated | usable-with-caveats | poorly-calibrated | invalid-eval
Actions:
Reproduce:

Output Contract

Done means:

• [ ] Walk-forward probs used
• [ ] ECE/Brier/log-loss reported
• [ ] Bin counts or segment notes present
• [ ] Verdict issued
• [ ] Actions stated

Worked Example

python /path/to/calibration-check/scripts/calibration_report.py \
  --input held-out-predictions.csv --target y_true \
  --probability logistic_probability --group-col season \
  --bins 10 --out calibration.json

python /path/to/calibration-check/scripts/segment_calibration.py \
  --input held-out-predictions.csv --target y_true \
  --probability logistic_probability --segment-col is_home

Example finding: predictions near 0.70 occur in sparse bins and realize near 0.60. Label the tail unstable or overconfident according to its uncertainty; do not infer a durable defect from a handful of rows.

Bundled Resources

references/

scripts/

Related Skills

Quick Command Card

python /path/to/calibration-check/scripts/calibration_report.py \
  --input predictions.csv --target won --probability win_probability \
  --group-col season --bins 10 --out calibration.json

python /path/to/calibration-check/scripts/segment_calibration.py \
  --input predictions.csv --target won --probability win_probability \
  --segment-col is_home