Benchmarking Uncertainty Quantification in Deep Learning

Detecting Overconfidence and Misclassification in Image Classification

Problem Statement

Despite the growing number of UQ methods in the literature, there is a lack of standardized benchmarks for comparing their effectiveness in real-world scenarios, especially for detecting misclassifications. Existing results are often not directly comparable due to differences in datasets, architectures, or evaluation protocols. This project provides a comprehensive, modular benchmarking framework to evaluate and compare state-of-the-art UQ methods on standardized image classification benchmarks using popular deep learning backbones.

For further reading and implementation details, please refer to the README.md file.

Backbone Architectures

WideResNet

Wide Residual Networks are a variation of ResNet with increased width rather than depth, providing better generalization and faster training.

Variants Used:

  • WideResNet-28-10

Possible Variants:

  • WideResNet-28-4
  • WideResNet-34-10
  • WideResNet-40-4
  • WideResNet-50-2

DenseNet

Dense Convolutional Networks connect each layer to every other layer in a feed-forward fashion, improving feature propagation and reducing parameters.

Variants Used:

  • DenseNet-100

Possible Variants:

  • DenseNet-121
  • DenseNet-169
  • DenseNet-201
  • DenseNet-264

UQ Methods Evaluated

Deep Ensembles

Train multiple models independently and average their outputs.

MC Dropout

Enable dropout at inference to simulate sampling.

Dirichlet Meta-Model

Post-hoc uncertainty learning using Dirichlet distribution parameters.

MAPLE

Meta-model trained on clustered representations with triplet loss.

LP-BNN

Bayesian network using BatchEnsemble layers with latent posterior sampling.

DUE

Deterministic Uncertainty Estimation using Deep Kernel Learning with Gaussian Processes.

DDU

Deep Deterministic Uncertainty using density and distance-based metrics in feature space.

Datasets

CIFAR-10

10 classes, 32x32 images, 50,000 training & 10,000 test samples. Standard benchmark for classification.

CIFAR-100

100 classes (20 superclasses), 32x32 images, 50,000 training & 10,000 test samples. Extended version of CIFAR-10.

Tiny ImageNet

200 classes, 64x64 images, higher complexity and diversity than CIFAR.

MNIST

10 classes (digits 0-9), 28x28 grayscale images, 60,000 training & 10,000 test samples. Handwritten digit recognition.

Fashion MNIST

10 classes (clothing items), 28x28 grayscale images, 60,000 training & 10,000 test samples. More challenging than MNIST.

Results

Dataset: CIFAR-10