Hi,

Here is a first version on the PR, adding this feature on precision-recall only, once we agree on the integration for this one, I will add the ROC in an analogous way, in this PR.
Thank you in advance for having an initial look on sklearn/metrics/_plot/precision_recall_curve.py

I will add unit tests and more function docstrings in sklearn/metrics/_plot/uncertainty.py soon.

@glemaitre @betatim @lorentzenchr

@stephanecollot This implementation does not match the implementation in MMU and is not as described in the paper.

This code creates a grid of a fixed shape for each P,R point and evaluates the chi2 score.
This means that different thresholds have different resolutions, e.g. 100 bins to cover 0.5 - 0.55 whereas other thresholds might only have a region that covers 0.9 - 0.91 with the same number of bins. You draw a region for each threshold but the overlap is no longer comparable as they are computed with different resolutions.

The reference implementation creates a P, R grid with a set number of points per axis.
For each threshold we evaluate the region in the P,R grid to evaluate and for each predetermined point in this region we compute the chi2 score. We only store the minimum chi2 score for each grid point and draw a single contour for the grid.
Not only is this much faster, this is also consistent as the whole curve has the same resolution.

@stephanecollot This implementation does not match the implementation in MMU and is not as described in the paper.

This code creates a grid of a fixed shape for each P,R point and evaluates the chi2 score. This means that different thresholds have different resolutions, e.g. 100 bins to cover 0.5 - 0.55 whereas other thresholds might only have a region that covers 0.9 - 0.91 with the same number of bins. You draw a region for each threshold but the overlap is no longer comparable as they are computed with different resolutions.

The reference implementation creates a P, R grid with a set number of points per axis. For each threshold we evaluate the region in the P,R grid to evaluate and for each predetermined point in this region we compute the chi2 score. We only store the minimum chi2 score for each grid point and draw a single contour for the grid. Not only is this much faster, this is also consistent as the whole curve has the same resolution.

That is correct, I tried 3 different methods for the grid, fix number grid point per point (the current one), the paper one and an adaptative grid (i.e. "x point per cm²"). They had different pros and cons, and I picked the best in terms for plotting smoothness and execution time. But it is true that I'm not taking the minimum chi2, which can make the plot look different.
I'm going to have a closer look at this.

I‘m not sure, but maybe https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.random_table.html#scipy.stats.random_table could be of help.