metrics_scoring_rule module
This section contains the Python API reference for the
uncertainty_toolbox.metrics_scoring_rule module, which contains code for uncertainty
metrics involving scoring rules.
uncertainty_toolbox.metrics_scoring_rule Module
Proper Scoring Rules for assessing the quality of predictive uncertainty quantification.
- uncertainty_toolbox.metrics_scoring_rule.check_score(y_pred, y_std, y_true, scaled=True, start_q=0.01, end_q=0.99, resolution=99)
The negatively oriented check score.
Computes the negatively oriented check score for held out data (y_true) given predictive uncertainty with mean (y_pred) and standard-deviation (y_std). Each test point and each quantile is given equal weight in the overall score over the test set and list of quantiles.
The score is computed by scanning over a sequence of quantiles of the predicted distributions, starting at (start_q) and ending at (end_q).
Negatively oriented means a smaller value is more desirable.
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the held out dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the held out dataset.y_true (
ndarray) – 1D array of the true labels in the held out dataset.scaled (
bool) – Whether to scale the score by size of held out set.start_q (
float) – The lower bound of the quantiles to use for computation.end_q (
float) – The upper bound of the quantiles to use for computation.resolution (
int) – The number of quantiles to use for computation.
- Return type
float- Returns
The check score.
- uncertainty_toolbox.metrics_scoring_rule.crps_gaussian(y_pred, y_std, y_true, scaled=True)
The negatively oriented continuous ranked probability score for Gaussians.
Computes CRPS for held out data (y_true) given predictive uncertainty with mean (y_pred) and standard-deviation (y_std). Each test point is given equal weight in the overall score over the test set.
Negatively oriented means a smaller value is more desirable.
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the held out dataset.y_std (
ndarray) – 1D array of he predicted standard deviations for the held out dataset.y_true (
ndarray) – 1D array of the true labels in the held out dataset.scaled (
bool) – Whether to scale the score by size of held out set.
- Return type
float- Returns
The crps for the heldout set.
- uncertainty_toolbox.metrics_scoring_rule.interval_score(y_pred, y_std, y_true, scaled=True, start_p=0.01, end_p=0.99, resolution=99)
The negatively oriented interval score.
Compute the negatively oriented interval score for held out data (y_true) given predictive uncertainty with mean (y_pred) and standard-deviation (y_std). Each test point and each percentile is given equal weight in the overall score over the test set and list of quantiles.
Negatively oriented means a smaller value is more desirable.
This metric is computed by scanning over a sequence of prediction intervals. Where p is the amount of probability captured from a centered prediction interval, intervals are formed starting at p=(start_p) and ending at p=(end_p).
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the held out dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the held out dataset.y_true (
ndarray) – 1D array of the true labels in the held out dataset.scaled (
bool) – Whether to scale the score by size of held out set.start_p (
float) – The lower bound of probability to capture in a prediction interval.end_p (
float) – The upper bound of probability to capture in a prediction interval.resolution (
int) – The number of prediction intervals to use to compute the metric.
- Return type
float- Returns
The interval score.
- uncertainty_toolbox.metrics_scoring_rule.nll_gaussian(y_pred, y_std, y_true, scaled=True)
Negative log likelihood for a gaussian.
The negative log likelihood for held out data (y_true) given predictive uncertainty with mean (y_pred) and standard-deviation (y_std).
- Parameters
y_pred (
ndarray) – 1D array of the predicted means for the held out dataset.y_std (
ndarray) – 1D array of the predicted standard deviations for the held out dataset.y_true (
ndarray) – 1D array of the true labels in the held out dataset.scaled (
bool) – Whether to scale the negative log likelihood by size of held out set.
- Return type
float- Returns
The negative log likelihood for the heldout set.