Garrus is a python framework for better confidence estimate of deep neural networks. Modern networks are overconfident estimators, that makes themselves unreliable and therefore limits the deployment of them in safety-critical applications.

Garrus provides tools for high quality confidence estimation such as confidence calibration and ordinal ranking methods, helping networks to know correctly what they do not know.

pip install -U garrus

• master v0.2.0
• develop v0.2.0

• Core:
  • Calibration metrics:
    • ECE
    • NLL
    • Brier
  • Ordinal Ranking Metrics:
    • AURC
    • E-AURC
    • AUPRE
    • FPR-n%-TPR
  • Visualizations:
    • Reliability Diagram
    • Confidence Histogram
  • Garrus Profiling
• Confidence Calibration:
  • Scaling:
    • Platt
    • Temperature
  • Binning:
    • Histogram
    • Isotonic Regression
• Confidence Regularization:
  • Losses:
    • Correctness Ranking Loss
    • Focal Entropy Penalized Loss
  • Language Model Beam Search
• Confidence Networks:
  • ConfidNet
  • GarrusNet

Please use this bibtex if you want to cite this repository in your publications:

@misc{garrus,
    author = {Kalashnikov, Alexander},
    title = {Deep neural networks calibration framework},
    year = {2021},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/sleep3r/garrus}},
}