metrics module

This section contains the Python API reference for the uncertainty_toolbox.metrics module, which contains the main API for uncertainty metrics.

uncertainty_toolbox.metrics Module

Metrics for assessing the quality of predictive uncertainty quantification.

uncertainty_toolbox.metrics.get_all_accuracy_metrics(y_pred, y_true, verbose=True)

Compute all accuracy metrics.

Parameters

• y_pred (ndarray) – 1D array of the predicted means for the held out dataset.

• y_true (ndarray) – 1D array of the true labels in the held out dataset.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, float]

Returns

The evaluations for all accuracy related metrics.

uncertainty_toolbox.metrics.get_all_adversarial_group_calibration(y_pred, y_std, y_true, num_bins, verbose=True)

Compute all metrics for adversarial group calibration.

Parameters

• y_pred (ndarray) – 1D array of the predicted means for the held out dataset.

• y_std (ndarray) – 1D array of he predicted standard deviations for the held out dataset.

• y_true (ndarray) – 1D array of the true labels in the held out dataset.

• num_bins (int) – The number of bins to use for discretization in some metrics.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, Dict[str, ndarray]]

Returns

The evaluations for all metrics relating to adversarial group calibration. Each inner dictionary contains the size of each group and the metrics computed for each group.

uncertainty_toolbox.metrics.get_all_average_calibration(y_pred, y_std, y_true, num_bins, verbose=True)

Compute all metrics for average calibration.

Parameters

• y_pred (ndarray) – 1D array of the predicted means for the held out dataset.

• y_std (ndarray) – 1D array of he predicted standard deviations for the held out dataset.

• y_true (ndarray) – 1D array of the true labels in the held out dataset.

• num_bins (int) – The number of bins to use for discretization in some metrics.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, float]

Returns

The evaluations for all metrics relating to average calibration.

uncertainty_toolbox.metrics.get_all_metrics(y_pred, y_std, y_true, num_bins=100, resolution=99, scaled=True, verbose=True)

Compute all metrics.

Parameters

• y_pred (ndarray) – 1D array of the predicted means for the held out dataset.

• y_std (ndarray) – 1D array of he predicted standard deviations for the held out dataset.

• y_true (ndarray) – 1D array of the true labels in the held out dataset.

• num_bins (int) – The number of bins to use for discretization in some metrics.

• resolution (int) – The number of quantiles to use for computation.

• scaled (bool) – Whether to scale the score by size of held out set.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, Any]

Returns

Dictionary containing all metrics.

uncertainty_toolbox.metrics.get_all_scoring_rule_metrics(y_pred, y_std, y_true, resolution, scaled, verbose=True)

Compute all scoring rule metrics

Parameters

• y_pred (ndarray) – 1D array of the predicted means for the held out dataset.

• y_std (ndarray) – 1D array of he predicted standard deviations for the held out dataset.

• y_true (ndarray) – 1D array of the true labels in the held out dataset.

• resolution (int) – The number of quantiles to use for computation.

• scaled (bool) – Whether to scale the score by size of held out set.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, float]

Returns

The computed scoring rule metrics.

uncertainty_toolbox.metrics.get_all_sharpness_metrics(y_std, verbose=True)

Compute all sharpness metrics

Parameters

• y_std (ndarray) – 1D array of he predicted standard deviations for the held out dataset.

• verbose (bool) – Activate verbose mode.

Return type

Dict[str, float]

Returns

The evaluations for all sharpness metrics.