recalibration module

This section contains the Python API reference for the uncertainty_toolbox.recalibration module, which contains code for recalibration procedures, which aim to improve the calibration of a predictor.

uncertainty_toolbox.recalibration Module

Recalibrating uncertainty estimates.

uncertainty_toolbox.recalibration.get_interval_recalibrator(y_pred, y_std, y_true)

Prediction interval recalibrator.

Fits an isotonic regression recalibration model and returns a function which takes in the mean and standard deviation predictions and a specified centered interval coverage level, and returns the recalibrated interval.

Parameters
  • y_pred (ndarray) – 1D array of the predicted means for the recalibration dataset.

  • y_std (ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.

  • y_true (ndarray) – 1D array of the true means for the recalibration dataset.

Return type

Callable[[ndarray, ndarray, Union[float, ndarray]], ndarray]

Returns

A function which outputs the recalibrated prediction interval.

uncertainty_toolbox.recalibration.get_q_idx(exp_props, q)

Utility function which outputs the array index of an element.

Gets the (approximate) index of a specified probability value, q, in the expected proportions array. Used as a utility function in isotonic regression recalibration.

Parameters
  • exp_props (ndarray) – 1D array of expected probabilities.

  • q (float) – a specified probability float.

Return type

int

Returns

An index which specifies the (approximate) index of q in exp_props

uncertainty_toolbox.recalibration.get_quantile_recalibrator(y_pred, y_std, y_true)

Quantile recalibrator.

Fits an isotonic regression recalibration model and returns a function which takes in the mean and standard deviation predictions and a specified quantile level, and returns the recalibrated quantile.

Parameters
  • y_pred (ndarray) – 1D array of the predicted means for the recalibration dataset.

  • y_std (ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.

  • y_true (ndarray) – 1D array of the true means for the recalibration dataset.

Return type

Callable[[ndarray, ndarray, Union[float, ndarray]], ndarray]

Returns

A function which outputs the recalibrated quantile prediction.

uncertainty_toolbox.recalibration.get_std_recalibrator(y_mean, y_std, y_true, criterion='ma_cal')

Standard deviation recalibrator.

Computes the standard deviation recalibration ratio and returns a function which takes in an array of uncalibrated standard deviations and returns an array of recalibrated standard deviations.

Parameters
  • y_mean (ndarray) – 1D array of the predicted means for the recalibration dataset.

  • y_std (ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.

  • y_true (ndarray) – 1D array of the true means for the recalibration dataset.

  • criterion (str) – calibration metric to optimize for during recalibration; must be one of {“ma_cal”, “rms_cal”, “miscal”}.

Return type

Callable[[ndarray], ndarray]

Returns

A function which takes uncalibrated standard deviations as input and outputs the recalibrated standard deviations.

uncertainty_toolbox.recalibration.iso_recal(exp_props, obs_props)

Recalibration algorithm based on isotonic regression.

Fits and outputs an isotonic recalibration model that maps observed probabilities to expected probabilities. This mapping provides the necessary adjustments to produce better calibrated outputs.

Parameters
  • exp_props (ndarray) – 1D array of expected probabilities (values must span [0, 1]).

  • obs_props (ndarray) – 1D array of observed probabilities.

Return type

IsotonicRegression

Returns

An sklearn IsotonicRegression recalibration model.

uncertainty_toolbox.recalibration.optimize_recalibration_ratio(y_mean, y_std, y_true, criterion='ma_cal')

Scale factor which uniformly recalibrates predicted standard deviations.

Searches via black-box optimization the standard deviation scale factor (opt_ratio) which produces the best recalibration, i.e. updated standard deviation can be written as opt_ratio * y_std.

Parameters
  • y_mean (ndarray) – 1D array of the predicted means for the recalibration dataset.

  • y_std (ndarray) – 1D array of the predicted standard deviations for the recalibration dataset.

  • y_true (ndarray) – 1D array of the true means for the recalibration dataset.

  • criterion (str) – calibration metric to optimize for during recalibration; must be one of {“ma_cal”, “rms_cal”, “miscal”}.

Return type

float

Returns

A single scalar which optimally recalibrates the predicted standard deviations.