Conformal prediction
- Year:
- 2026
- Status:
- In progress
- Stack:
- Python · NumPy · scikit-learn
Abstract:
Conformal prediction wraps any model in intervals that contain the truth with probability at least 1 − α. I implemented split conformal and CQR from scratch, then tested the guarantee over repeated calibration splits.
Result:
target coverage met on five benchmark datasets
Fig. 1 — Intervals on heteroscedastic data
Scatter plot of 500 test points, x from 0 to 5, where the spread of y grows with x. A dashed line shows the k-nearest-neighbour fit and a shaded band shows the CQR prediction interval for target coverage 0.90. 455 points fall inside the band and 45 fall outside.
CQR, α 0.10: test coverage 0.910, mean over 1,000 splits 0.900, mean width 3.52.
- Test coverage
- 0.910
- Mean over 1,000 splits
- 0.900
- Mean width
- 3.52
CQR on synthetic data where noise grows with x. Filled dots are covered; open circles fall outside. Target: 0.90.
Method:
q̂ is the ⌈(n+1)(1−α)⌉-th smallest calibration score. CQR conformalises quantile estimates instead, so intervals widen where the noise does.
Limitations:
- Coverage is marginal, not conditional.
- Exchangeability fails under time series and covariate shift.
- Next: weighted and adaptive conformal methods.