Suggested by @GaelVaroquaux , we should use the sphinx extension Napoleon to avoid the Numpydoc dependency.
http://www.sphinx-doc.org/en/stable/ext/napoleon.html
It's used in dirty-cat for example : https://github.com/dirty-cat/dirty_cat