> python -V
Python 2.7.11
> pip freeze | grep matplotlib
matplotlib==1.5.1
> python -m platform -s "print platform()"
Windows-7-6.1.7601-SP2

I have discovered that my application spends a lot of time in Axes creation. Results of class instantiation profiling is below:

> python -m timeit -s "from matplotlib.figure import Figure; from matplotlib.axes import Axes; fig = Figure()" "Axes(fig, (0, 0, 1 ,1))"
10 loops, best of 3: 33.2 msec per loop

Inside Axes.__init__ most of the execution time takes self.cla() call and about the half of it is
this loop below

        for name, spine in six.iteritems(self.spines):
            spine.cla()

After commenting this two lines I have got this:

> python -m timeit -s "from matplotlib.figure import Figure; from matplotlib.axes import Axes; fig = Figure()" "Axes(fig, (0, 0, 1 ,1))"
100 loops, best of 3: 17.9 msec per loop

So we have 33 ms per axes creation, and the half of this time it is used for spines creation that actually may be not used (in my case I have bunch of sharex axes and only one of them have spines).
I see a possible solution where creation of spines, scales (you can see creation of logarithmic scale, which I do not use too) and other stuff is delayed to the point when it actually needed/initialized.

And the fun part. For 20 created Axes we have 60115 calls to Rectangle.stale (3005 call per Axes instantiation)