BUG: avoid possible stack overflow in arraydescr_dealloc by ngoldbaum · Pull Request #32133 · numpy/numpy

ngoldbaum · Jul 28, 2026

PR summary

The Py_DECREF(lself->subarray->base) we currently have can lead to situations where the DECREF causes a deallocation, which then recursively calls into arraydescr_dealloc. With a properly setup dtype, this can lead to a stack overflow on newer Python versions. See python/cpython#142253 for the upstream issue that was opened about that and #30370 on the NumPy side.

My fix is to check for cases when decrefing will deallocate and in those cases explicitly detach the base before decrefing, avoiding the recursive call.

AI Disclosure

I used an AI model to understand the problem and debug the fix and new test.

kumaraditya303 · Jul 29, 2026

        PyArray_free(lself->subarray);
+        while (base != NULL && Py_REFCNT(base) == 1 && PyDataType_HASSUBARRAY(base)) {
+            _PyArray_LegacyDescr *lbase = (_PyArray_LegacyDescr *)base;
+            base = lbase->subarray->base;


Use Py_CLEAR instead.

I don't think that's right. I'm stealing the reference without deallocating immediately on purpose.

I could INCREF then Py_CLEAR to avoid that but that adds an unnecessary incref. Would you prefer that for clarity? I can also add a comment explaining that I'm setting the point to NULL and explicitly not deallocating yet.

The CLEAR macro does exactly that, though.

It would DECREF base, not lbase. I added a comment explaining the reference stealing so hopefully it's clearer.

Yeah, nvm. I thought the concern was Py_CLEAR not being careful enough, but this isn't a Py_CLEAR pattern at all.

Oh, I missed this as well. thanks for adding the comment

seberg · Jul 29, 2026

+         */
+        PyArray_Descr *base = lself->subarray->base;
        PyArray_free(lself->subarray);
+        while (base != NULL && Py_REFCNT(base) == 1 && PyDataType_HASSUBARRAY(base)) {


Not quite sure, but the Py_REFCNT doesn't smell strictly thread-safe to me (if true maybe you are OK with ignoring that, but considering that this is a niche failure)?

Just enabling GC tracking feels reasonable to me (we should do this more anyway, maybe even for arrays, I even did once but that PR just withered away), but I am not sure about implications... dtypes are practically immutable (unless metadata is an actual dict as that could contain mutable objects), though, you'd want to untrack them mostly.
(We can do that now, although I think the NumPy internal singletons may still be static structs.)

For the Py_REFCNT==1, I looked at using PyUnstable_Object_IsUniquelyReferenced instead but that is actually strictly worse because it also has a check that the current thread is the owning thread and it's possible (if weird) for a remote thread to call the deallocator.

The Py_REFCNT==1 check is fine because this is in a deallocator, the base array is not exposed to Python, and because dtypes aren't GC-tracked and don't support weakrefs. All of those facts combine to mean that it's not possible to observe Py_REFCNT==1 here and not be genuinely the only possible reference.

We could make dtypes be GC types and add trashcan tracking here but we would also have to figure out how to manage a migration story for user dtypes and their deallocators, which would need to be updated. We would also need to figure out how to create all the statically initialized singletons like LONG_Descr. All do-able but a much bigger -- and I'd argue riskier -- change. Also not backportable like this PR.

the base array is not exposed to Python

That part isn't strictly true, though?

import numpy as np d = np.dtype("f", metadata={"a": 3}) # just to have a unique one. d1 = np.dtype((d, 234)) d2 = np.dtype((d1, 235)) assert d2.base is d1 assert d1.base is d

The same thing can probably also happen for fields, fields that get cleared could recurse just the same.

All do-able but a much bigger -- and I'd argue riskier -- change. Also not backportable like this PR.

Yap, agreed, should be doable, but probably tedious.

You're right, I overstated my case. The thing that matters here is because we're in a deallocator and there's no gc tracking and no weakref support, it's not possible to observe a refcnt of 1 if there are external references, even on remote threads.

Your example is like the held variable in the test I added.

For fields, you're right that they can create recursion, but all the objects in the fields dict are types that CPython handles via the trashcan mechanism. The subarray field has to deal with the statically defined dtype types.

If we do migrate dtypes to heap types (maybe a numpy 3.0 thing along with some of the issues @prathamhole14 is identifying with static data in the C API), we'd need to update this code path to use the trashcan macros. I'll try to add comments so future grep searches find it.

seberg

Should apply Kumar's suggestion to use CLEAR, CLEAR uses the right pattern.
But otherwise fine to merge I think.

I agree now that it seems safe (if confusing...). Feels all unfortunate and bit of code smell.
Seems like we should mark them as GC allocated even if we untrack them always (when there is no metadata). I don't think the static instances matter for that? Because those must be immortal anyway (you can't free them after all)!
But even then, I'll believe that is annoying to just do, so...

seberg · Jul 29, 2026

        PyArray_free(lself->subarray);
+        while (base != NULL && Py_REFCNT(base) == 1 && PyDataType_HASSUBARRAY(base)) {
+            _PyArray_LegacyDescr *lbase = (_PyArray_LegacyDescr *)base;
+            base = lbase->subarray->base;


The CLEAR macro does exactly that, though.

seberg · Jul 29, 2026

+            t.start()
+            t.join()
+        """)
+        run_subprocess([sys.executable, "-c", script], timeout=180)


I don't really see the point of the subprocess. This seems much like the previous one (but yes won't fail due to recursion).
But the previous test already segfaults without the fix? Although I guess this one may segfault on more platforms :).

BUG: avoid possible stack overflow in arraydescr_dealloc (#32133)

This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [numpy](https://github.com/numpy/numpy) ([changelog](https://numpy.org/doc/stable/release)) | dependencies | patch | `2.5.0` → `2.5.2` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/numpy/numpy/badge)](https://securityscorecards.dev/viewer/?uri=github.com/numpy/numpy) | | [uvicorn](https://github.com/Kludex/uvicorn) ([changelog](https://uvicorn.dev/release-notes)) | dependencies | patch | `0.52.1` → `0.52.3` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Kludex/uvicorn/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Kludex/uvicorn) | --- ### Release Notes <details> <summary>numpy/numpy (numpy)</summary> ### [`v2.5.2`](https://github.com/numpy/numpy/releases/tag/v2.5.2): (Aug 9, 2026) [Compare Source](numpy/numpy@v2.5.1...v2.5.2) ### NumPy 2.5.2 Release Notes The NumPy 2.5.2 is a patch release that fixes bugs discovered after the 2.5.1 release. The big news is that it includes wheels for the newly released Python 3.15.0rc1. This release supports Python versions 3.12-3.15 #### C API changes ##### `PyArray_StringDTypeObject` is opaque under the abi3t stable ABI The `PyArray_StringDTypeObject` was accidentally exposed in NumPy 2.5 when targeting the free-threading-compatible stable ABI (`Py_TARGET_ABI3T`). `PyArray_StringDTypeObject` is now an opaque struct: extensions compiled that way cannot access its fields, since the struct layout depends on the size of the object header. Any code that accessed `PyArray_StringDTypeObject` fields in an abi3t build would have crashed, so we are making this API change in a bugfix release. The `NpyString` allocator API remains usable by passing the descriptor object pointer, e.g. `NpyString_acquire_allocator((PyArray_StringDTypeObject *)descr)`. ([gh-31771](numpy/numpy#31771)) #### Contributors A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Abhijeetsingh Meena + - Charalampos Stratakis - Charles Harris - Chris Ninham + - David Woods - Geonho + - Gopu Yeshwanth Reddy + - Iason Krommydas - Ijtihed Kilani - Jelle Zijlstra + - Joren Hammudoglu - Kumar Aditya - Mike Boyle - Nathan Goldbaum - Raghuveer Devulapalli - Sebastian Berg #### Pull requests merged A total of 28 pull requests were merged for this release. - [#31864](numpy/numpy#31864): MAINT: Prepare 2.5.x for further development - [#31889](numpy/numpy#31889): TYP: Backport multiple static typing fixes 1. - [#31900](numpy/numpy#31900): TST: add tests for stable ABI numpy extensions ([#31822](numpy/numpy#31822)) - [#31901](numpy/numpy#31901): BUG: fix `StringDType` coerce flag in binary ufunc promotion... - [#31902](numpy/numpy#31902): BLD: fix meson deprecation warnings ([#31892](numpy/numpy#31892)) - [#31921](numpy/numpy#31921): TYP: Backport multiple typing fixes 2. - [#31947](numpy/numpy#31947): MAINT: Update x86-simd-sort subproject ([`5adb334`](numpy/numpy@5adb334) → [`fa944ef`](numpy/numpy@fa944ef)) ([#31908](numpy/numpy#31908)) - [#31949](numpy/numpy#31949): BUG: fix crash on 32 bit systems using abi3t ([#31771](numpy/numpy#31771)) - [#31950](numpy/numpy#31950): MNT: remove some obsolete string to bool workarounds ([#31859](numpy/numpy#31859)) - [#31952](numpy/numpy#31952): BUG: centralized helper for output coerce and na\_object in stringdtype... - [#31953](numpy/numpy#31953): BUG: fix CPU feature env diagnostic buffer overruns ([#31905](numpy/numpy#31905)) - [#31954](numpy/numpy#31954): BUG: restore ndarray.conjugate() for legacy user-defined dtypes... - [#31955](numpy/numpy#31955): TYP: Avoid shadowed dtype annotations - [#32077](numpy/numpy#32077): MAINT: Update verdored-meson/meson to match main. - [#32114](numpy/numpy#32114): BUG: fix refcount leak on overlapping copyto with where=False - [#32115](numpy/numpy#32115): BUG: fix swallowed cast error in fancy indexing assignment ([#31975](numpy/numpy#31975)) - [#32116](numpy/numpy#32116): BUG: Fix buffered iterator stride after removing multi-index - [#32117](numpy/numpy#32117): BUG: fix `np.fromiter` corruption when reusing a `StringDType`... - [#32119](numpy/numpy#32119): BUG: add a special case for StringDType in np.isdtype ([#32030](numpy/numpy#32030)) - [#32121](numpy/numpy#32121): BUG: reference leak in `simd_sequence_from_iterable` ([#32038](numpy/numpy#32038)) - [#32122](numpy/numpy#32122): BUG: ensure lock is held when accessing or writing to RNG state... - [#32123](numpy/numpy#32123): BUG: fully reset cached RNG state for non-MT19937 RNGs ([#32062](numpy/numpy#32062)) - [#32135](numpy/numpy#32135): TYP: type capabilities max dimensions - [#32158](numpy/numpy#32158): BUG: avoid possible stack overflow in arraydescr\_dealloc ([#32133](numpy/numpy#32133)) - [#32206](numpy/numpy#32206): MAINT: Update cibuildwheel to v4.2.0 - [#32214](numpy/numpy#32214): MAINT: Skip limited\_api tests on some platforms. - [#32220](numpy/numpy#32220): TYP: `isclose` shape-typing fix for 2d array-likes ([#32205](numpy/numpy#32205)) - [#32221](numpy/numpy#32221): BUG: avoid segfaults when legacy copyswap slot is not defined... ### [`v2.5.1`](https://github.com/numpy/numpy/releases/tag/v2.5.1): (July 4, 2026) [Compare Source](numpy/numpy@v2.5.0...v2.5.1) ### NumPy 2.5.1 Release Notes The NumPy 2.5.1 is a patch release that fixes bugs discovered after the 2.5.0 release. The most noticeable is the fix is to the numpy datetime cython API which should allow downstream to support NumPy versions older than 2.5. Preparation for Python 3.15 continues along with typing improvements. This release supports Python versions 3.12-3.14 #### Changes - The minimum supported GCC version has been updated from 9.3.0 to 10.3.0 ([gh-31843](numpy/numpy#31843)) #### Contributors A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Adhyan Gupta + - Ankit Ahlawat - Charles Harris - Iason Krommydas - Joren Hammudoglu - Kumar Aditya - Nathan Goldbaum - Sebastian Berg - Ties Jan Hefting + - Vineet Kumar #### Pull requests merged A total of 20 pull requests were merged for this release. - [#31707](numpy/numpy#31707): MAINT: Prepare 2.5.x for further development - [#31721](numpy/numpy#31721): CI: fix new `cython-lint` errors ([#31711](numpy/numpy#31711)) - [#31723](numpy/numpy#31723): MAINT: Update meson to match main - [#31729](numpy/numpy#31729): TST: use setup-sde instead of curl to get SDE binaries ([#31727](numpy/numpy#31727)) - [#31829](numpy/numpy#31829): BUG: Relax finfo to be easier accessible for all user dtypes... - [#31831](numpy/numpy#31831): TYP: Fix `flatiter.__next__` return type for `object_` and... - [#31832](numpy/numpy#31832): BUG: avoid deadlocks using NpyString API ([#31682](numpy/numpy#31682)) - [#31833](numpy/numpy#31833): BUG: fix out array leak in reduceat and accumulate when dtype... - [#31835](numpy/numpy#31835): BUG: fix numpy datetime cython APIs to be compatible with older... - [#31836](numpy/numpy#31836): TYP: Fix incorrect dtype inference of `asarray([])` ([#31732](numpy/numpy#31732)) - [#31837](numpy/numpy#31837): TYP: Fix `np.ma.masked_array` 2.5.0 regression - [#31838](numpy/numpy#31838): FIX: Refactor error handling in array\_setstate to prevent typecode... - [#31839](numpy/numpy#31839): TST: xfail multithreaded BLAS test more generously - [#31840](numpy/numpy#31840): MAINT: Rename subroutine for crackfortran tests - [#31842](numpy/numpy#31842): BUG: fix leak in reductions when a ufunc override errors or is... - [#31849](numpy/numpy#31849): BLD: set minimum required gcc version to 10.3 ([#31843](numpy/numpy#31843)) - [#31855](numpy/numpy#31855): CI: fix hangs on MacOS ASan CI ([#31853](numpy/numpy#31853)) - [#31856](numpy/numpy#31856): BUG: fix several bugs in StringDType operations ([#31846](numpy/numpy#31846)) - [#31857](numpy/numpy#31857): BUG: Fix segfault in MT19937 by preventing recursive seed lists... - [#31858](numpy/numpy#31858): BUG: Fix signed integer overflow in datetime.c ([#31688](numpy/numpy#31688)) </details> <details> <summary>Kludex/uvicorn (uvicorn)</summary> ### [`v0.52.3`](https://github.com/Kludex/uvicorn/releases/tag/0.52.3): Version 0.52.3 [Compare Source](Kludex/uvicorn@0.52.2...0.52.3) ##### Changed - Update `zttp` to 0.0.24 and use its combined receive path, improving HTTP/1.1 request parsing performance ([#3067](Kludex/uvicorn#3067)) **Full Changelog**: <Kludex/uvicorn@0.52.2...0.52.3> ### [`v0.52.2`](https://github.com/Kludex/uvicorn/releases/tag/0.52.2): Version 0.52.2 [Compare Source](Kludex/uvicorn@0.52.1...0.52.2) ##### Fixed - Update `zttp` to 0.0.22, fixing bodyless request receives and improving HTTP/1 request parsing performance ([#3063](Kludex/uvicorn#3063)) **Full Changelog**: <Kludex/uvicorn@0.52.1...0.52.2> </details> --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  See merge request swiss-armed-forces/cyber-command/cea/loom!639 Co-authored-by: Loom MR Pipeline Trigger <group_103951964_bot_9504bb8dead6d4e406ad817a607f24be@noreply.gitlab.com>

BUG: avoid possible stack overflow in arraydescr_dealloc

d369ce1

github-actions Bot added the 00 - Bug label Jul 28, 2026

ngoldbaum added the 09 - Backport-Candidate PRs tagged should be backported label Jul 28, 2026

ngoldbaum mentioned this pull request Jul 28, 2026

NumPy test failure after upgrading to 3.14.1t on MacOS x86_64 python/cpython#142253

Closed

kumaraditya303 reviewed Jul 29, 2026

View reviewed changes

seberg reviewed Jul 29, 2026

View reviewed changes

seberg approved these changes Jul 29, 2026

View reviewed changes

ngoldbaum force-pushed the fix-descr-dealloc-overflow branch from 366db47 to 0c06127 Compare July 29, 2026 20:22

MAINT: add more comments, don't use subprocess in test

fbf433f

ngoldbaum force-pushed the fix-descr-dealloc-overflow branch from 0c06127 to fbf433f Compare July 29, 2026 20:34

ngoldbaum added 2 commits July 29, 2026 14:45

TST: disable test on wasm

7624fac

MAINT: tweak comments and fix use of stack_size global

84b5f44

ngoldbaum requested a review from kumaraditya303 July 29, 2026 21:27

kumaraditya303 approved these changes Jul 30, 2026

View reviewed changes

ngoldbaum merged commit 5d011d8 into numpy:main Jul 30, 2026
91 checks passed

charris mentioned this pull request Jul 31, 2026

BUG: avoid possible stack overflow in arraydescr_dealloc (#32133) #32158

Merged

charris removed the 09 - Backport-Candidate PRs tagged should be backported label Jul 31, 2026

charris pushed a commit that referenced this pull request Jul 31, 2026

BUG: avoid possible stack overflow in arraydescr_dealloc (#32133)

b2e4f97

charris added a commit that referenced this pull request Jul 31, 2026

Merge pull request #32158 from charris/backport-32133

988d94d

BUG: avoid possible stack overflow in arraydescr_dealloc (#32133)

Uh oh!

Uh oh!

Conversation

ngoldbaum commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

AI Disclosure

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ngoldbaum commented Jul 28, 2026 •

edited

Loading