recalibration module
This section contains the Python API reference for the
uncertainty_toolbox.recalibration module, which contains code for recalibration
procedures, which aim to improve the calibration of a predictor.
uncertainty_toolbox.recalibration Module
Recalibrating uncertainty estimates.
- uncertainty_toolbox.recalibration.get_interval_recalibrator(y_pred, y_std, y_true)
Prediction interval recalibrator.
Fits an isotonic regression recalibration model and returns a function which takes in the mean and standard deviation predictions and a specified centered interval coverage level, and returns the recalibrated interval.
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the recalibration dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.y_true (
ndarray) – 1D array of the true means for the recalibration dataset.
- Return type
Callable[[ndarray,ndarray,Union[float,ndarray]],ndarray]- Returns
A function which outputs the recalibrated prediction interval.
- uncertainty_toolbox.recalibration.get_q_idx(exp_props, q)
Utility function which outputs the array index of an element.
Gets the (approximate) index of a specified probability value, q, in the expected proportions array. Used as a utility function in isotonic regression recalibration.
- Parameters
exp_props (
ndarray) – 1D array of expected probabilities.q (
float) – a specified probability float.
- Return type
int- Returns
An index which specifies the (approximate) index of q in exp_props
- uncertainty_toolbox.recalibration.get_quantile_recalibrator(y_pred, y_std, y_true)
Quantile recalibrator.
Fits an isotonic regression recalibration model and returns a function which takes in the mean and standard deviation predictions and a specified quantile level, and returns the recalibrated quantile.
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the recalibration dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.y_true (
ndarray) – 1D array of the true means for the recalibration dataset.
- Return type
Callable[[ndarray,ndarray,Union[float,ndarray]],ndarray]- Returns
A function which outputs the recalibrated quantile prediction.
- uncertainty_toolbox.recalibration.get_std_recalibrator(y_mean, y_std, y_true, criterion='ma_cal')
Standard deviation recalibrator.
Computes the standard deviation recalibration ratio and returns a function which takes in an array of uncalibrated standard deviations and returns an array of recalibrated standard deviations.
- Parameters
y_mean (
ndarray) – 1D array of the predicted means for the recalibration dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.y_true (
ndarray) – 1D array of the true means for the recalibration dataset.criterion (
str) – calibration metric to optimize for during recalibration; must be one of {“ma_cal”, “rms_cal”, “miscal”}.
- Return type
Callable[[ndarray],ndarray]- Returns
A function which takes uncalibrated standard deviations as input and outputs the recalibrated standard deviations.
- uncertainty_toolbox.recalibration.iso_recal(exp_props, obs_props)
Recalibration algorithm based on isotonic regression.
Fits and outputs an isotonic recalibration model that maps observed probabilities to expected probabilities. This mapping provides the necessary adjustments to produce better calibrated outputs.
- Parameters
exp_props (
ndarray) – 1D array of expected probabilities (values must span [0, 1]).obs_props (
ndarray) – 1D array of observed probabilities.
- Return type
IsotonicRegression- Returns
An sklearn IsotonicRegression recalibration model.
- uncertainty_toolbox.recalibration.optimize_recalibration_ratio(y_mean, y_std, y_true, criterion='ma_cal')
Scale factor which uniformly recalibrates predicted standard deviations.
Searches via black-box optimization the standard deviation scale factor (opt_ratio) which produces the best recalibration, i.e. updated standard deviation can be written as opt_ratio * y_std.
- Parameters
y_mean (
ndarray) – 1D array of the predicted means for the recalibration dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.y_true (
ndarray) – 1D array of the true means for the recalibration dataset.criterion (
str) – calibration metric to optimize for during recalibration; must be one of {“ma_cal”, “rms_cal”, “miscal”}.
- Return type
float- Returns
A single scalar which optimally recalibrates the predicted standard deviations.