Fraction of test outcomes inside their intervals.
Coverage on unseen points
Split conformal · exchangeable test data
intervalcoveredmiss
The promised marginal coverage under exchangeability.
Narrower is more useful, if coverage survives.
The finite-sample residual order statistic.
One held-out set converts errors into uncertainty.
Score calibration points
Fit the model without the calibration set. Then measure a nonconformity score for every calibration example, such as the absolute residual.
sᵢ = |yᵢ − ŷᵢ|
Choose a finite-sample quantile
Use the ceiling-adjusted order statistic. The extra one in the rank is what turns a familiar quantile into a finite-sample guarantee.
k = ceil((n + 1)(1 − α))
Wrap the next prediction
For a new point, add and subtract the calibrated radius. Under exchangeability, the outcome lands inside with probability at least one minus alpha.
C(x) = [ŷ(x) − q, ŷ(x) + q]
The guarantee is precise about what it assumes.
Coverage is marginal
Ninety percent coverage over the population does not mean every subgroup or every individual point receives a 90% conditional guarantee. Audit cohorts separately when harm is uneven.
Exchangeability connects calibration and test
The ordering argument relies on calibration and future scores behaving like interchangeable draws. Covariate, label, or concept shift can break that symmetry.
Efficiency still matters
A set that always contains every answer has excellent coverage and no value. Compare width or set size at the same target coverage, then monitor both after deployment.