What is the mechanism by which this breaks?

It is also possible that this "just works" on master, @anntzer recently put in some PRs to tweak how we handle the window sizing.

See the example in #15040, which shows how it breaks, even on master.

I mean, why does setting the size twice confuse it?

Currently the resize code in Qt5 is:

    def resize(self, width, height):
        # these are Qt methods so they return sizes in 'virtual' pixels
        # so we do not need to worry about dpi scaling here.
        extra_width = self.window.width() - self.canvas.width()
        extra_height = self.window.height() - self.canvas.height()
        self.window.resize(width + extra_width, height + extra_height)

After one resize the window gets resized, but the canvas doesn't. This means when it comes to the second resize, self.canvas.width()/height() don't reflect the previous resize, so extra_width/height are not computed correctly.

My guess as to why it works fine in the usual case with one resize and a draw is because at draw time the canvas is automatically resized to fit the window.

I've just pushed a test that fails without the change in this PR.

I'm not totally convinced this is the correct fix? For example, perhaps set_size_inches should be the one doing the canvas resize, e.g. with

        if forward:
            canvas = getattr(self, 'canvas')
            if canvas is not None:
                dpi_ratio = getattr(canvas, '_dpi_ratio', 1)

                canvas.resize(...)  # <--- HERE

                manager = getattr(canvas, 'manager', None)
                if manager is not None:
                    manager.resize(*(size * self.dpi / dpi_ratio).astype(int))

otherwise, I would guess the other backends also need updating:

    # GTK3
    def resize(self, width, height):
        'set the canvas size in pixels'
        #_, _, cw, ch = self.canvas.allocation
        #_, _, ww, wh = self.window.allocation
        #self.window.resize (width-cw+ww, height-ch+wh)
        self.window.resize(width, height)

    # Tk
    def resize(self, width, height):
        self.canvas._tkcanvas.master.geometry("%dx%d" % (width, height))

        if self.toolbar is not None:
            self.toolbar.configure(width=width)

    # wx
    def resize(self, width, height):
        'Set the canvas size in pixels'
        self.canvas.SetInitialSize(wx.Size(width, height))
        self.window.GetSizer().Fit(self.window)

which appear to all suffer from the same issue? (FigureManagerMac doesn't even seem to define resize()?)

Yes, I think you're right that set_size_inches should handle it, and what you proposed is correct, ie.

1. Change the figure (really canvas) size (currently missing)
2. If there's a window manager, change the window size to match (currently present)

Is there a way to parameterize the test with different backends?

Is there a way to parameterize the test with different backends?

Not sure. You can always generate the functions dynamically in a for loop. I think something like this should work:

for backend in ['Qt5Agg', 'TkAgg']:
    @pytest.mark.backend(backend)
    def _test_x():
        ...
    
    globals()[f'test_x_{backend}'] = _test_x

They basically need to run in different subprocesses (because a given process can only ever use a single interactive backend) -- see tests_backends_interactive.py. (Probably stashing all the tests in a single process invocation, as is done right now, is better for performance.)

With #14703 merged do we still need this?

Can take a look soon-ish, but I think it's worth putting the test in even if it does pass on current master.

Yep, this fix is still needed even with #14703

I've changed my mind and think that it's reasonable for the backend resize methods to also be responsible for resizing the canvas. I'm going to leave this as is as a Qt5 fix, and open new bugs for each other backend that has set_size_inches() issues.

I've also added an explict check for figure width as @timhoffm suggests.