numpy · ngoldbaum · Nov 28, 2023 · Nov 2, 2023 · Nov 14, 2023 · Nov 14, 2023
diff --git a/doc/release/upcoming_changes/25149.c_api.rst b/doc/release/upcoming_changes/25149.c_api.rst
@@ -0,0 +1,10 @@
+Lager ``NPY_MAXDIMS`` and ``NPY_MAXARGS``, ``NPY_RAVEL_AXIS`` introduced
+------------------------------------------------------------------------
+
+``NPY_MAXDIMS`` is now 64, you may want to review its use.  This is usually
+used in stack allocation, where the increase should be safe.
+However, we do encourage generally to remove any use of ``NPY_MAXDIMS`` and
+``NPY_MAXARGS`` to eventually allow removing the constraint completely.
+In some cases, ``NPY_MAXDIMS`` was passed (and returned) to mean ``axis=None``
+these must be replaced with ``NPY_RAVEL_AXIS``.
+See also :ref:`migration_maxdims`.
diff --git a/doc/release/upcoming_changes/25149.change.rst b/doc/release/upcoming_changes/25149.change.rst
@@ -0,0 +1,8 @@
+Out-of-bound axis not the same as ``axis=None``
+-----------------------------------------------
+In some cases ``axis=32`` or for concatenate any large value
+was the same as ``axis=None``.
+Except for ``concatenate`` this was deprecate.
-Except for ``concatenate`` this was deprecate.
+Except for ``concatenate`` this was deprecated in Numpy 1.23.
-Except for ``concatenate`` this was deprecate.
+Except for ``concatenate`` this was deprecated in Numpy 1.23.
+Any out of bound axis value will now error, make sure to use
+``axis=None``.
+
diff --git a/doc/source/numpy_2_0_migration_guide.rst b/doc/source/numpy_2_0_migration_guide.rst
@@ -41,8 +41,30 @@ Some are defined in ``numpy/_core/include/numpy/npy_2_compat.h``
 (for example ``NPY_DEFAULT_INT``) which can be vendored in full or part
 to have the definitions available when compiling against NumPy 1.x.

+If necessary, ``PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION`` can be
+used to explicitly implement different behavior on NumPy 1.x and 2.0.
+(The compat header defines it in a way compatible with such use.)
+
 Please let us know if you require additional workarounds here.

+.. _migration_maxdims:
+
+Increased maximum number of dimensions
+--------------------------------------
+The maximum number of dimensions (and arguments) was increased to 64, this
+affects the ``NPY_MAXDIMS`` and ``NPY_MAXARGS`` macros.
+It may be good to review their use, and we generally encourage you to
+not use this macros (especially ``NPY_MAXARGS``), so that a future version of
+NumPy can remove this limitation on the number of dimensions.
+
+``NPY_MAXDIMS`` was also used to signal ``axis=None`` in the C-API, including
+the ``PyArray_AxisConverter``.
+If you run into this problem, you will see ``-2147483648``
+(the minimum integer value) or ``64`` being used as an invalid axis.
+Wherever you do, you must replace ``NPY_MAXDIMS`` with ``NPY_RAVEL_AXIS``.
+This is defined in the ``npy_2_compat.h`` header and runtime dependent.
+
+
 Namespace changes
 =================


diff --git a/doc/source/reference/c-api/array.rst b/doc/source/reference/c-api/array.rst
@@ -1949,7 +1949,7 @@ Calculation

 .. tip::

-    Pass in :c:data:`NPY_MAXDIMS` for axis in order to achieve the same
+    Pass in :c:data:`NPY_RAVEL_AXIS` for axis in order to achieve the same
    effect that is obtained by passing in ``axis=None`` in Python
    (treating the array as a 1-d array).

@@ -2968,7 +2968,7 @@ to.
    Convert a Python object, *obj*, representing an axis argument to
    the proper value for passing to the functions that take an integer
    axis. Specifically, if *obj* is None, *axis* is set to
-    :c:data:`NPY_MAXDIMS` which is interpreted correctly by the C-API
+    :c:data:`NPY_RAVEL_AXIS` which is interpreted correctly by the C-API
    functions that take axis arguments.

 .. c:function:: int PyArray_BoolConverter(PyObject* obj, npy_bool* value)
@@ -3411,11 +3411,26 @@ Other constants

 .. c:macro:: NPY_MAXDIMS

-    The maximum number of dimensions allowed in arrays.
+    The maximum number of dimensions that may be used by NumPy.
+    This is set to 64 and was 32 before NumPy 2.
+
+    .. note::
+        We encourage you to avoid ``NPY_MAXDIMS``.  A future version of NumPy
+        may wish to remove any dimension limitation (and thus the constant).
+        The limitation was created mainly a simplification for the liberal use
+        of small stack allocations as scratch space.
+
+        If your algorithm has a reasonable maximum number of dimension you
+        could check and use that locally.

 .. c:macro:: NPY_MAXARGS

-    The maximum number of array arguments that can be used in functions.
+    The maximum number of array arguments that can be used in in some
+    functions.  This used to be 32 before NumPy 2 and is now 64.
+
+    .. note::
+        You should never use this.  We may remove it in future versions of
+        NumPy.

 .. c:macro:: NPY_FALSE

@@ -3435,6 +3450,14 @@ Other constants
    The return value of successful converter functions which are called
    using the "O&" syntax in :c:func:`PyArg_ParseTuple`-like functions.

+.. c:macro:: NPY_RAVEL_AXIS
+
+    Some NumPy functions (mainly the C-entrypoints for Python functions)
+    have an ``axis`` argument.  This macro may be passed for ``axis=None``.
+
+    .. note::
+        This macro is a NumPy version dependent at runtime.
+

 Miscellaneous Macros
 ~~~~~~~~~~~~~~~~~~~~

diff --git a/doc/source/reference/c-api/types-and-structures.rst b/doc/source/reference/c-api/types-and-structures.rst
@@ -119,8 +119,11 @@ PyArray_Type and PyArrayObject
       array. When nd is 0, the array is sometimes called a rank-0
       array. Such arrays have undefined dimensions and strides and
       cannot be accessed. Macro :c:data:`PyArray_NDIM` defined in
-       ``ndarraytypes.h`` points to this data member. :c:data:`NPY_MAXDIMS`
-       is the largest number of dimensions for any array.
+       ``ndarraytypes.h`` points to this data member.
+       Although most operations may be limited in dimensionality, we do not
+       advertise a maximum dimension.  Anyone explicitly relying on one
+       must check for it.  Before NumPy 2.0, NumPy used 32 dimensions at most
+       after it, the limit is currently 64.

   .. c:member:: npy_intp *dimensions

@@ -986,11 +989,11 @@ PyArrayIter_Type and PyArrayIterObject
          int   nd_m1;
          npy_intp  index;
          npy_intp  size;
-          npy_intp  coordinates[NPY_MAXDIMS];
-          npy_intp  dims_m1[NPY_MAXDIMS];
-          npy_intp  strides[NPY_MAXDIMS];
-          npy_intp  backstrides[NPY_MAXDIMS];
-          npy_intp  factors[NPY_MAXDIMS];
+          npy_intp  coordinates[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  strides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  backstrides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  factors[NPY_MAXDIMS_LEGACY_ITERS];
          PyArrayObject *ao;
          char  *dataptr;
          npy_bool  contiguous;
@@ -1086,8 +1089,8 @@ PyArrayMultiIter_Type and PyArrayMultiIterObject
          npy_intp size;
          npy_intp index;
          int nd;
-          npy_intp dimensions[NPY_MAXDIMS];
-          PyArrayIterObject *iters[NPY_MAXDIMS];
+          npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
+          PyArrayIterObject *iters[NPY_MAXDIMS_LEGACY_ITERS];
      } PyArrayMultiIterObject;

   .. c:macro: PyObject_HEAD
@@ -1141,20 +1144,20 @@ PyArrayNeighborhoodIter_Type and PyArrayNeighborhoodIterObject
          PyObject_HEAD
          int nd_m1;
          npy_intp index, size;
-          npy_intp coordinates[NPY_MAXDIMS]
-          npy_intp dims_m1[NPY_MAXDIMS];
-          npy_intp strides[NPY_MAXDIMS];
-          npy_intp backstrides[NPY_MAXDIMS];
-          npy_intp factors[NPY_MAXDIMS];
+          npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS]
+          npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS];
          PyArrayObject *ao;
          char *dataptr;
          npy_bool contiguous;
-          npy_intp bounds[NPY_MAXDIMS][2];
-          npy_intp limits[NPY_MAXDIMS][2];
-          npy_intp limits_sizes[NPY_MAXDIMS];
+          npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+          npy_intp limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+          npy_intp limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
          npy_iter_get_dataptr_t translate;
          npy_intp nd;
-          npy_intp dimensions[NPY_MAXDIMS];
+          npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
          PyArrayIterObject* _internal_iter;
          char* constant;
          int mode;

diff --git a/numpy/__init__.cython-30.pxd b/numpy/__init__.cython-30.pxd
@@ -187,9 +187,8 @@ cdef extern from "numpy/arrayobject.h":
        NPY_ARRAY_UPDATE_ALL

    cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE
+        NPY_MAXDIMS  # 64 on NumPy 2.x and 32 on NumPy 1.x
+        NPY_RAVEL_AXIS  # Used for functions like PyArray_Mean

    ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *,  void *)


diff --git a/numpy/__init__.pxd b/numpy/__init__.pxd
@@ -182,9 +182,8 @@ cdef extern from "numpy/arrayobject.h":
        NPY_ARRAY_UPDATE_ALL

    cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE
+        NPY_MAXDIMS  # 64 on NumPy 2.x and 32 on NumPy 1.x
+        NPY_RAVEL_AXIS  # Used for functions like PyArray_Mean

    ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *,  void *)


diff --git a/numpy/_core/include/numpy/ndarrayobject.h b/numpy/_core/include/numpy/ndarrayobject.h
@@ -103,7 +103,6 @@ extern "C" {

 #define PyArray_FILLWBYTE(obj, val) memset(PyArray_DATA(obj), val, \
                                           PyArray_NBYTES(obj))
-#define NPY_MAX_ELSIZE (2 * NPY_SIZEOF_LONGDOUBLE)

 #define PyArray_ContiguousFromAny(op, type, min_depth, max_depth) \
        PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \

diff --git a/numpy/_core/include/numpy/ndarraytypes.h b/numpy/_core/include/numpy/ndarraytypes.h
@@ -35,10 +35,17 @@
 * The array creation itself could have arbitrary dimensions but all
 * the places where static allocation is used would need to be changed
 * to dynamic (including inside of several structures)
+ *
+ * As of NumPy 2.0, we strongly discourage the downstream use of NPY_MAXDIMS,
+ * but since auditing everything seems a big ask, define it as 64.
+ * A future version could:
+ * - Increase or remove the limit and require recompilation (like 2.0 did)
+ * - Deprecate or remove the macro but keep the limit (at basically any time)
 */
-
-#define NPY_MAXDIMS 32
-#define NPY_MAXARGS 32
+#define NPY_MAXDIMS 64
+/* We cannot change this as it would break ABI: */
+#define NPY_MAXDIMS_LEGACY_ITERS 32
+#define NPY_MAXARGS 64

 /* Used for Converter Functions "O&" code in ParseTuple */
 #define NPY_FAIL 0
@@ -1095,18 +1102,18 @@ struct PyArrayIterObject_tag {
        PyObject_HEAD
        int               nd_m1;            /* number of dimensions - 1 */
        npy_intp          index, size;
-        npy_intp          coordinates[NPY_MAXDIMS];/* N-dimensional loop */
-        npy_intp          dims_m1[NPY_MAXDIMS];    /* ao->dimensions - 1 */
-        npy_intp          strides[NPY_MAXDIMS];    /* ao->strides or fake */
-        npy_intp          backstrides[NPY_MAXDIMS];/* how far to jump back */
-        npy_intp          factors[NPY_MAXDIMS];     /* shape factors */
+        npy_intp          coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
+        npy_intp          dims_m1[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->dimensions - 1 */
+        npy_intp          strides[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->strides or fake */
+        npy_intp          backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
+        npy_intp          factors[NPY_MAXDIMS_LEGACY_ITERS];     /* shape factors */
        PyArrayObject     *ao;
        char              *dataptr;        /* pointer to current item*/
        npy_bool          contiguous;

-        npy_intp          bounds[NPY_MAXDIMS][2];
-        npy_intp          limits[NPY_MAXDIMS][2];
-        npy_intp          limits_sizes[NPY_MAXDIMS];
+        npy_intp          bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+        npy_intp          limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+        npy_intp          limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
        npy_iter_get_dataptr_t translate;
 } ;

@@ -1230,7 +1237,7 @@ typedef struct {
        npy_intp             size;                    /* broadcasted size */
        npy_intp             index;                   /* current index */
        int                  nd;                      /* number of dims */
-        npy_intp             dimensions[NPY_MAXDIMS]; /* dimensions */
+        npy_intp             dimensions[NPY_MAXDIMS_LEGACY_ITERS]; /* dimensions */
        PyArrayIterObject    *iters[NPY_MAXARGS];     /* iterators */
 } PyArrayMultiIterObject;

@@ -1335,18 +1342,18 @@ typedef struct {
     */
    int               nd_m1;            /* number of dimensions - 1 */
    npy_intp          index, size;
-    npy_intp          coordinates[NPY_MAXDIMS];/* N-dimensional loop */
-    npy_intp          dims_m1[NPY_MAXDIMS];    /* ao->dimensions - 1 */
-    npy_intp          strides[NPY_MAXDIMS];    /* ao->strides or fake */
-    npy_intp          backstrides[NPY_MAXDIMS];/* how far to jump back */
-    npy_intp          factors[NPY_MAXDIMS];     /* shape factors */
+    npy_intp          coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
+    npy_intp          dims_m1[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->dimensions - 1 */
+    npy_intp          strides[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->strides or fake */
+    npy_intp          backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
+    npy_intp          factors[NPY_MAXDIMS_LEGACY_ITERS];     /* shape factors */
    PyArrayObject     *ao;
    char              *dataptr;        /* pointer to current item*/
    npy_bool          contiguous;

-    npy_intp          bounds[NPY_MAXDIMS][2];
-    npy_intp          limits[NPY_MAXDIMS][2];
-    npy_intp          limits_sizes[NPY_MAXDIMS];
+    npy_intp          bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+    npy_intp          limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+    npy_intp          limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
    npy_iter_get_dataptr_t translate;

    /*
@@ -1355,7 +1362,7 @@ typedef struct {
    npy_intp nd;

    /* Dimensions is the dimension of the array */
-    npy_intp dimensions[NPY_MAXDIMS];
+    npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];

    /*
     * Neighborhood points coordinates are computed relatively to the

diff --git a/numpy/_core/include/numpy/npy_2_compat.h b/numpy/_core/include/numpy/npy_2_compat.h
@@ -30,6 +30,20 @@
 #ifndef NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_
 #define NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_

+/*
+ * Allow users to use `PyArray_RUNTIME_VERSION` when vendoring the file for
+ * compilation with NumPy 1.x.
+ * Simply do not define when compiling with 2.x.  It must be defined later
+ * as it is set during `import_array()`.
+ */
+#if !defined(PyArray_RUNTIME_VERSION) && NPY_ABI_VERSION < 0x02000000
+  /*
+   * If we are compiling with NumPy 1.x, PyArray_RUNTIME_VERSION so we
+   * pretend the `PyArray_RUNTIME_VERSION` is `NPY_FEATURE_VERSION`.
+   */
+  #define PyArray_RUNTIME_VERSION NPY_FEATURE_VERSION
+#endif
+
 /*
 * New macros for accessing real and complex part of a complex number can be
 * found in "npy_2_complexcompat.h".
@@ -52,4 +66,14 @@
 #endif


+#if NPY_FEATURE_VERSION >= NPY_2_0_API_VERSION
+    #define NPY_RAVEL_AXIS NPY_MIN_INT
+#elif NPY_ABI_VERSION < 0x02000000
+    #define NPY_RAVEL_AXIS 32
+#else
+    #define NPY_RAVEL_AXIS  \
+        (PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION ? -1 : 32)
+#endif
+
+
 #endif  /* NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_ */
-Original file line number
+Diff line change
@@ -0,0 +1,10 @@
+    Lager ``NPY_MAXDIMS`` and ``NPY_MAXARGS``, ``NPY_RAVEL_AXIS`` introduced
+    ------------------------------------------------------------------------
+    ``NPY_MAXDIMS`` is now 64, you may want to review its use.  This is usually
+    used in stack allocation, where the increase should be safe.
                   seberg marked this conversation as resolved.
              

          
            Show resolved
            Hide resolved
        

                
  
            
  

  
    
  
    
      

              Uh oh!

              
There was an error while loading. Please reload this page.
+    However, we do encourage generally to remove any use of ``NPY_MAXDIMS`` and
+    ``NPY_MAXARGS`` to eventually allow removing the constraint completely.
+    In some cases, ``NPY_MAXDIMS`` was passed (and returned) to mean ``axis=None``
             Copy link

  
      
    
  

  
      

  
  Member


      

  

  
    
      

      
            ngoldbaum
  

      

      

      


        Nov 22, 2023


      
    

  


        
      
  
  
  
    

    There was a problem hiding this comment.


  

 
  
    

    Choose a reason for hiding this comment

    
      The reason will be displayed to describe this comment to others. Learn more.
    

    
      
      


  


  
    
      Can you make "In some cases" a little more specific? Does this only impact things that interact with numpy via the C API, or could you trigger this behavior from the python API too?
    
  
  


    

        
      
  
  
    
  
  
  
    
    Sorry, something went wrong.
  

  
    
  
    
      

              Uh oh!

              
There was an error while loading. Please reload this page.


  
  


          
      
  
    
    
      
        
            
    All reactions
  


          
          
        
      
    

    



    
        
  
    
        
    
  


      
          
  
      
            Copy link

  
      
    
  

  
      

  
  Member


      

  Author


  

  
    
      

      
            seberg
  

      

      

      


        Nov 24, 2023


      
    

  


        
      
  
  
  
    

    There was a problem hiding this comment.


  

 
  
    

    Choose a reason for hiding this comment

    
      The reason will be displayed to describe this comment to others. Learn more.
    

    
      
      


  


  
    
      The python side places are already mentioned in the other one, but made it a bit more concrete.
    
  
  


    

        
      
  
  
    
  
  
  
    
    Sorry, something went wrong.
  

  
    
  
    
      

              Uh oh!

              
There was an error while loading. Please reload this page.


  
  


          
      
  
    
    
      
        
            
    All reactions
+    these must be replaced with ``NPY_RAVEL_AXIS``.
+    See also :ref:`migration_maxdims`.