Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Enh: SymNorm for normalizing symmetrical data around a center #17709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Oct 22, 2020

Conversation

jpmattern
Copy link
Contributor

PR Summary

Adds SymNorm, a new norm that automatically creates linear colormaps that are symmetrical around a center. Useful for symmetrical data and in combination with diverging colormaps.

symnorm

I often create plots where I want the color norm to be symmetric around a center, typically for visualizing anomalies. I tend to use one of the diverging colormaps and I often end up with code like this, where the norm is adjusted after creating the plot to make it symmetric:

pc = ax.pcolormesh(...)
vmax = max(-pc.norm.vmin, pc.norm.vmax)
pc.norm.vmin = -vmax
pc.norm.vmax = vmax

While the above is not a long piece of code, its more elegant to achieve the same goal automatically using a new norm. This approach also has the advantage of easily being able to set a center different from 0 (using, e.g.: SymNorm(vcenter=1.0)).

When I was scrolling through the pull requests here, I noticed that a somewhat similar approach has been discussed here. The proposed SymNorm goes a slightly different direction, in that the norm may extend beyond the data on one side of vcenter. I personally like this, as it, for example, gives the colorbar a symmetrical look (see figure above).

A quick note regarding the new code: The implementation of SymNorm makes use of @property syntax to implement a setter for vcenter. This syntax does not appear in other parts of colors.py and I would be happy to change the implementation, for example by overriding the __call__ method which is currently not overridden.

PR Checklist

  • Has Pytest style unit tests
  • Code is Flake 8 compliant
  • New features are documented, with examples if plot related
  • Documentation is sphinx and numpydoc compliant
  • Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

@jklymak
Copy link
Member

jklymak commented Jun 22, 2020

This seems fine, plus or minus the name which I'm not sure about.

Do you need to lock out setting vmin somehow? What happens if I set norm.vmin=-42 after I've initialized the norm?

@jpmattern
Copy link
Contributor Author

I got the naming idea from the existing SymLogNorm -- and this is basically a symmetrical version of Normalize. Having said that, the name SymLinNorm would also make sense.

And yes, good point about vmin, it would make sense to restrict access to that. Or, alternatively, one could raise an error when the norm is evaluated and it is no longer symmetrical (this behavior is similar to changing vmin to a higher value than vmax for Normalize, it works until the norm is evaluated).

lib/matplotlib/colors.py Outdated Show resolved Hide resolved
lib/matplotlib/colors.py Outdated Show resolved Hide resolved
lib/matplotlib/colors.py Outdated Show resolved Hide resolved
lib/matplotlib/colors.py Outdated Show resolved Hide resolved
@tacaswell
Copy link
Member

Thank you for your work on this @jpmattern ! As you noted, we have had several attempts at getting functionality like this in over the years, hopefully it works this time!

If the __init__ is going to take the center and 1 parameter, why not take the range (rather than the max)? SymNorm(center=50, range=5) feels a bit better than SynNorm(center=50, vmax=55)?

I think that using properties for both vmin/vmax to keep them in sync with each other is reasonable.

This will definitely need tests + documentation.

I'm not wild about the name, I also thought of the analogy to SymLog immediately, but my knee jerk reaction was concern (because of the funny linear / log transition in SymLog).

Could we get the same effect by adding a vcenter and vrange properties and kwargs to Normalize (which some rules to de-conflict them)? Maybe:

  • if you set vmin or vmax, the other does not change and the center / range are re-computed
  • if you set vcenter or vrange the other is held constant and vmin / vmax are re-computed
  • for kwargs you can give up to any 2 and we sort out the rest

but that would lose what I think is the most important feature of this proposal which is that auto scaling works. Maybe we want the above + a very thin sub-class that changes the autoscale rules?

@jpmattern
Copy link
Contributor Author

I like the idea of using range rather than vmax, this would also be a change that could easily be included in the proposed SymNorm changes.

As for changing the name, I would be in favour of that but I haven't come up with a better name so far. One thing that throws me off a bit is the single "m" in the name when both "symmetrical" and "symmetry" have two "m"s.

Changing the entire structure of Normalize could be the most elegant solution: Getters and setters could be used to protect against invalid changes to the properties (e.g. setting vmin to a value greater than vmax). And a Normalize(symmetrical=True) would be pretty self-explanatory. However, that kind of change is quite substantial and would require a lot more testing. I'd be happy to work on a solution if there's desire for a more substantial change.

@jklymak
Copy link
Member

jklymak commented Jun 28, 2020

I’m not a fan of one Normalize method with conflicting kwargs.

CenteredNorm?

@timhoffm
Copy link
Member

timhoffm commented Jun 29, 2020

I'd go for halfrange instead of range. Center +/- halfrange is what one (at least me) typically thinks about.

CenteredNorm is good. SymmetricNorm would also work. SymNorm is a bit of an awkward abbreviation (despite the precedence of SymLogNorm).

@jpmattern
Copy link
Contributor Author

I like the idea of using halfrange and the name makes it clear that it defines half of the range and not the full range which could have been an alternative. The updated code is using halfrange now and the norm has also be renamed.

I am personally not sure if I like CenteredNorm or SymmetricalNorm better but since we had 2 votes for it already, I opted for the former.

@jklymak
Copy link
Member

jklymak commented Jun 30, 2020

You can count my vote as half - if you prefer SymmetricalNorm, that's fine w/ me. I didn't like SymNorm - probably residual trauma from SymLogNorm.

@QuLogic
Copy link
Member

QuLogic commented Jun 30, 2020

This would need a note & example in the What's new document.

@jpmattern
Copy link
Contributor Author

As an example, I'd suggest the image in my first post above, adjusted for the new name. I modified en example from the gallery to create it, code would look something like this:

import numpy as np
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors

delta = 0.1
x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)
Z1 = np.exp(-X**2 - Y**2)
Z2 = np.exp(-(X - 1)**2 - (Y - 1)**2)
Z = (0.9*Z1 - 0.5*Z2) * 2

#cmap = cm.RdYlGn
cmap = cm.seismic

fig, axs = plt.subplots(ncols=3, figsize=(10,2.5))

ax = axs[0]
pc = ax.pcolormesh(Z, norm=mcolors.Normalize(), cmap=cmap)
fig.colorbar(pc, ax=ax)
ax.set_title('Normalize()')

ax = axs[1]
pc = ax.pcolormesh(Z, norm=mcolors.TwoSlopeNorm(vcenter=0.0), cmap=cmap)
fig.colorbar(pc, ax=ax)
ax.set_title('TwoSlopeNorm(vcenter=0.0)')

ax = axs[2]
pc = ax.pcolormesh(Z, norm=mcolors.CenteredNorm(), cmap=cmap)
fig.colorbar(pc, ax=ax)
ax.set_title('CenteredNorm()')

Copy link
Member

@timhoffm timhoffm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example looks good, but I would use the idiom

fig, (ax1, ax2, ax3) = plt.subplots(...)

and use ax1 etc. in the following. This saves the ax = ax[i] statements and makes the example more compact.

lib/matplotlib/colors.py Show resolved Hide resolved
@jpmattern
Copy link
Contributor Author

Where would the example for the code go? Is examples/color/ the right directory and should it be added to a new file?

@jklymak
Copy link
Member

jklymak commented Jul 7, 2020

There is a tutorial on colormap norms. It also has the examples broken out into user demo, but that seems a silly place for them. Without thinking about it too deeply, I'd add the example to the tutorial.

@tacaswell
Copy link
Member

I like where this is heading!

This definitly needs a test, I suggest starting with

@pytest.mark.style(style)
@check_figures_equal(extensions=['png'], st)
def test_centered_norm(fig_ref, fig_test):
    ax_test = fig_test.subplots()
    ax_ref = fig_ref.subplots()

    # make the "same" plot using CenteredNorm and Normalize

I see that at call time it fixes the limits, I think we should also raise or warn if we do change it. If a user does not realize that they have ended up with a CenteredNorm, set norm.vmax = ... we are going to silently discard their change on the next draw. That is the sort of thing that end result in 5 hours of chasing down rabbit holes, growing concerned you are forgetting how to program, and then a lot of rage..... or so friends tell me ;).

@jpmattern
Copy link
Contributor Author

Demo and test code sound good, I can add those. @tacaswell, I think to produce a warning or raise and error is a sensible idea but the only way to implement it I can think of involves changes to the super class Normalize. Currently, there is no control over what the user sets vmin and vmax to, but one could turn them into properties with getter and setter methods which could then be overridden by CenteredNorm.

@timhoffm
Copy link
Member

Closes #1806.

@jpmattern
Copy link
Contributor Author

The prominent use of "Symmetric logarithmic" in the tutorial and the call for a "symmetric" norm in #1806 makes me think that SymmetricalNorm might be a more intuitive name than CenteredNorm. What do others think? I know we have a couple of votes already.

One of the most difficult issues with adding the new norm to the tutorial was where to place it. It seems more commonly applicable than SymLogNorm which is appearing very early on and makes the case for symmetrical data in its description. The TwoSlopeNorm appears quite late in the tutorial, so I have scrapped the use of TwoSlopeNorm from the example and placed it above SymLogNorm for now. I am happy to move it.

@jpmattern
Copy link
Contributor Author

I have added a new gallery example and a few tests to the code. Those appear to run without issues in my virtual environment. I cannot quite wrap my head around the errors reported above. They may be related to me adding code to the gallery but I am not sure what to do about them, can someone help?

This is the error message that CircleCI is showing me:

tar cf doc/build/sphinx-gallery-files.tar.gz doc/api/_as_gen doc/gallery doc/tutorials

tar: doc/build/sphinx-gallery-files.tar.gz: Cannot open: No such file or directory

tar: Error is not recoverable: exiting now

@dstansby
Copy link
Member

😱 I would find this so useful!

In terms of the failing tests, could you rebase this on top of the current master branch, and we can see if the tests are still failing.

@dstansby dstansby self-assigned this Sep 20, 2020
@jpmattern
Copy link
Contributor Author

Sounds good, I will try this in the next few days.

@jpmattern
Copy link
Contributor Author

Ok, I went through the rebasing process and the tests I did on my end succeeded (generating the gallery image). The automated test above fails in an odd place with an error message that is not very helpful to me, it appears to finish building the documentation and then exits with an error:

embedding documentation hyperlinks for tutorials... [ 92%] text_intro.html
embedding documentation hyperlinks for tutorials... [ 95%] text_props.html
embedding documentation hyperlinks for tutorials... [ 97%] sg_execution_times.html
embedding documentation hyperlinks for tutorials... [100%] annotations.html

make: *** [Makefile:33: html] Error 1

Exited with code exit status 2

@QuLogic
Copy link
Member

QuLogic commented Sep 25, 2020

I'm not sure the rebase was quite right, as you shouldn't have so many unrelated commits in the PR. I see the last commit is a merge, which shouldn't be there. That means you pulled the old commits from GitHub back into your local copy. After rebasing, you'll want to force push back to GitHub to delete the old commits, instead of pulling.

The doc error is earlier in the log:

/home/circleci/project/doc/tutorials/colors/colormapnorms.rst:99: WARNING: py:obj reference target not found: colors.CenteredNorm

@jpmattern
Copy link
Contributor Author

Thanks @QuLogic, that was helpful. The doc error was related to a missing entry for CenteredNorm in doc/api/colors_api.rst. I have never worked with Sphinx before but that issue seems to be resolved.

I don't know what to do about the rebase though. This is my first one and it appeared to go well until I accidentally performed a git pull and now I have a bunch of extra commits in my branch. My attempts to undo them have been unsuccessful so far.

@QuLogic
Copy link
Member

QuLogic commented Oct 1, 2020

Yes, after a rebase, you don't want to pull, you want to force push to replace the old versions on GitHub. You can see this using git log --graph --pretty=fuller master..symnorm. The left-most branchline contains commits authored in July, and committed in September, i.e., the rebased commits. The middle branchline is a merge path which doesn't really matter. The right-most branchline contains commits authored in July and committed in July, i.e., the original commits.

To drop the old commits, do the following:

$ git checkout symnorm
# Back to the last commit made of the _new_ commits
$ git reset --hard de6de1950eeb49177b3d320548f154edab4c6e92
# Replace everything on GitHub
$ git push --force-with-lease origin symnorm

PS, git on your computer says your name is Paul home desktop, which doesn't match your name in GitHub, and seems more like a description of your computer than your actual name. You'll probably want to fix that.

@jpmattern
Copy link
Contributor Author

True, I changed the author names to the commits in question.

norm = mcolors.CenteredNorm(vcenter=0.5, halfrange=0.5)
assert_array_almost_equal(x, norm(x))
norm = mcolors.CenteredNorm(vcenter=1, halfrange=1)
assert_array_almost_equal(x, 2 * norm(x))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you test the setters and getters in here? I know they are trivial, but....

@@ -69,6 +70,46 @@
fig.colorbar(pcm, ax=ax[1], extend='max')
plt.show()

###############################################################################
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should definitely be moved under TwoSlopeNorm, please, as it is a specialization of that norm.

lib/matplotlib/colors.py Outdated Show resolved Hide resolved
lib/matplotlib/tests/test_colors.py Show resolved Hide resolved
tutorials/colors/colormapnorms.py Outdated Show resolved Hide resolved
@QuLogic
Copy link
Member

QuLogic commented Oct 3, 2020

This would need a What's new entry as well.

jpmattern and others added 3 commits October 4, 2020 12:41
Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>
Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>
@jpmattern
Copy link
Contributor Author

Ok, I

  • added a what's new entry with a new example plot (creating a heatmap plot)
  • slightly modified and expanded the tests to include usage of the setters
  • added the seed to the tests, though it may make more sense to set the seed before every individual check

@jklymak, I agree that it would make sense to place the tutorial entry below or closer to TwoSlopeNorm but I like that it currently sits just above SymLogNorm and the flow of text works quite well in my opinion: LogNorm sets the stage for a logarithmic transformation, CenteredNorm introduces the concept of symmetry and a center, and then SymLogNorm combines the two, starting out with: "Similarly, it sometimes happens that there is data that is positive and negative, but we would still like a logarithmic scaling applied to both."
Happy to change the order but I think it works quite well right now.

self.vmax = None
else:
self._halfrange = abs(halfrange)
#self._set_vmin_vmax()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this intentionally commented out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch again! Yes, left it in to check and meant to delete the line as it is unnecessary and may sometimes lead to undesired results (when invoking the setter for halfrange first and then the setter for vcenter). I'll delete that line now and include that change in my next commit.

@jpmattern
Copy link
Contributor Author

In my most recent commit, I expanded testing to cover 100% of the CenteredNorm code. I may be wrong but I don't think the most recent failing check is due to my commits to the code. Are there any other requests for features?

@QuLogic QuLogic requested a review from jklymak October 20, 2020 04:13
Copy link
Member

@QuLogic QuLogic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave it to @jklymak to decide on the doc position.

Copy link
Member

@jklymak jklymak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree w/ the doc position, but I can always change in a follow up PR ;-)

@jklymak jklymak merged commit 03c293b into matplotlib:master Oct 22, 2020
@jklymak
Copy link
Member

jklymak commented Oct 22, 2020

Thanks @jpmattern - looks like a useful addition!

@jpmattern
Copy link
Contributor Author

Awesome, thanks! Great to see that this feature its making its way into the code now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.