Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

x64 and Windows support for ndarray#37

Closed
classner wants to merge 9 commits into
ndarray:masterndarray/ndarray:masterfrom
classner:masterclassner/ndarray:masterCopy head branch name to clipboard
Closed

x64 and Windows support for ndarray#37
classner wants to merge 9 commits into
ndarray:masterndarray/ndarray:masterfrom
classner:masterclassner/ndarray:masterCopy head branch name to clipboard

Conversation

@classner

@classner classner commented Oct 9, 2014

Copy link
Copy Markdown

Dear ndarray team,

we are using Boost.NumPy and the ndarray project in a big data context at our lab. So I made the effort and made both projects x64 Python compatible, as well as the ndarrays 64bit indexed (similar to eigen3). I was mainly working on Windows and had to test the projects, so at the same time I extended the SCons build system to work with Windows.

After all changes, both projects build without any internal warnings (the numpy interface deprecation warning left aside) and are working flawlessly on Windows and Linux. All tests are passing.

The changes here are extensive, but focus on x64 indexing for the arrays. The number of dimensions is still left to be an integer, as well as the int fftw-interface is still used (but can now also easily be replaced by the x64 interface).

I am ready to answer questions and hope it is a useful contribution to the project!

Best,
Christoph

@TallJimbo

Copy link
Copy Markdown
Member

Sorry I've let this linger so long.

I'm going to merge over your Windows build-support changes largely as they are, but I'm going to spend some time re-doing the 64-bit size changes - I think we need to use std::ptrdiff_t for strides in order to support reversed arrays, even if we switch to std::size_t for shapes. And that will be a bit more of a disruptive change, as it means they won't have the same Index Vector typedefs anymore.

I'll also be adding some assert statements to verify that ndarray::Array objects never get created with sizes larger than std::ptrdiff_t's max, as that could cause problems all over the library. I'd be happy to accept a pull request in the future that moved that checking from array creation down to the places where those problems could occur, to allow such arrays to be created and used as long as certain operations were avoided, but for now I don't think that will be worth the extra effort. Note that this should also avoid any problems converting ndarray::Array objects to Python, assuming Py_ssize_t is equivalent to std::ptrdiff_t, as I believe it should be on most modern systems.

@TallJimbo

Copy link
Copy Markdown
Member

Build system changes finally merged. I'll do the integer type changes next (but after the Boost.NumPy integer type changes).

@TallJimbo

Copy link
Copy Markdown
Member

@ChrisLS I think I finally have everything working (as described in my previous comment) and ready to merge to master on branch #37. Would you mind taking a look and verifying that this works for you (especially on Windows)?

@classner

classner commented Mar 5, 2015

Copy link
Copy Markdown
Author

Ok, great! :) Sure, I'll have a look at it as well as re-verifying the tests on Windows and Linux

@classner

Copy link
Copy Markdown
Author

I went through the code and find the changes elegant and well-working. It nearly worked directly with Windows, with one nasty bug. Let's go through it:

  1. The build requires m4, which is not by default loaded as tool on Windows. This can, however, be easily achieved by adding env.Tool('m4') after line 36 to SConstruct.

  2. The header cxxabi.h is not available on Windows, but is used in views.cc. It is not required on Windows at all, so I just guarded it:

    #ifndef _MSC_VER
    #include "cxxabi.h"
    #endif
    
  3. The really nasty bug was in include/ndarray/swig/PyConverter.h: on line 257, you are extracting the value of an unsigned long long from the Python variable. For some reason, this function is broken on Windows (at least on my compiled Anaconda version of Python) and leads to an immediate crash. I just replaced it with the signed counterpart (PyLong_AsLongLong) to have it working flawlessly (of course, only as long as the bounds are preserved). I'm not sure on how to handle this issue best.

With these changes, compilation and tests worked.

Looking forward to have the x64 version from master on Windows! :)

@TallJimbo

Copy link
Copy Markdown
Member

I've put in fixes for all three issues you found, with the last one just really a workaround - I've switched to PyLong_AsLongLong with some additional error checking on Windows only.

Could you verify that this now works for you? And, if you have some time in the future, I may open up a new issue for the PyLong_AsUnsignedLongLong problem and ask you to take a closer look at what triggers it.

@classner

classner commented May 3, 2015

Copy link
Copy Markdown
Author

Ok, I went over this again, and found your implementation of PyConverter.h clean and safe. There's just one compilation issue: in PyConverter.h, line 265 currently is:

            PyErr_Set(PyExc_OverflowError, "Negative integer in conversion to unsigned");

and should be

            PyErr_SetString(PyExc_OverflowError, "Negative integer in conversion to unsigned");

With that change, everything works as expected!

As much as I'd like to, I don't expect to have the time to dig deeper into the PyLong_AsUnsignedLongLong issue. With some luck, this gets resolved by the Python developers in a future release anyway.

TallJimbo added a commit that referenced this pull request May 30, 2015
@TallJimbo

Copy link
Copy Markdown
Member

Merged to master!

Once again, sorry this took so incredibly long, and thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.