ENH: add NEP-50 style semantics for string scalars and StringDType by ngoldbaum · Pull Request #32040 · numpy/numpy

ngoldbaum · Jul 19, 2026

PR summary

While working on ByteStringDType, I noticed that trailing nulls are currently stripped when StringDType operations use python scalars. For example:

>>> np.array(["hello", "world"], dtype="T") + "hello\0\0"
array(['hellohello', 'worldhello'], dtype=StringDType())

This is happening because "hello\0\0" is coerced to a fixed-width unicode array, which strips the trailing NULLs.

@seberg let me know that the infrastructure in NumPy to do this is already in NumPy to support NEP-50. Somewhat differently from the other NEP-50 cases, I don't think it's necessary to add a new abstract dtype to represent python strings because we don't want string scalars and string arrays to have different semantics.

While working on this I realized that partition and rpartition need a promoter to be able to work with this new functionality, so this includes that as well.

AI Disclosure

I iterated on the implementation with an AI model.

ngoldbaum · Jul 19, 2026

+                || (inputs_tup != NULL
+                    && (PyArray_FLAGS(operands[i])
+                        & NPY_ARRAY_WAS_PYTHON_STR))) {


format like the stanza above

seberg

Thanks, this looks nice and straight-forward enough. As you said, dunno if that one test has much use, but I don't mind keeping it.

May make sense to do that (yes annoying) ternary to avoid double-conversion, I think, but otherwise this seems good and pragmatic.

seberg · Jul 19, 2026

-                /* `resolve_descriptors_with_scalars` decides the descr */
+            if ((PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL) ||
+                /* str is only replaced if the resolver was passed the scalar */
+                (input_scalars[i] != NULL &&


Is this needed (since the above doesn't have it)?

The numeric case uses NULL to indicate that the value isn't available. The string case needs the original string so NULL doesn't make sense here.

But actually looking closer there's possible NULL-pointer dereferences associated with this behavior in the numeric cases. Inside npy_update_operand_for_scalar:

numpy/numpy/_core/src/multiarray/abstractdtypes.c

Lines 386 to 400 in e503c0e

Py_TYPE(scalar)->tp_name, descr);

return -1;

}

Py_INCREF(descr);

PyArrayObject *new = (PyArrayObject *)PyArray_NewFromDescr(

&PyArray_Type, descr, 0, NULL, NULL, NULL, 0, NULL);

Py_SETREF(*operand, new);

if (*operand == NULL) {

return -1;

}

if (scalar == NULL) {

/* The ufunc.resolve_dtypes paths can go here. Anything should go. */

return 0;

}

In the error path, scalar is accessed to get the name of the scalar type, but that happens before the null-check below.

Also here:

numpy/numpy/_core/src/umath/ufunc_object.c

Lines 4215 to 4232 in e503c0e

safety = ufuncimpl->resolve_descriptors_with_scalars(

ufuncimpl, signature, original_descrs, input_scalars,

dtypes, &view_offset

);

/* For scalars, replace the operand if needed (scalars can't be out) */

for (int i = 0; i < nin; i++) {

if ((PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL)) {

/* `resolve_descriptors_with_scalars` decides the descr */

if (npy_update_operand_for_scalar(

&operands[i], input_scalars[i], dtypes[i],

/* ignore cast safety for this op (resolvers job) */

NPY_SAFE_CASTING) < 0) {

goto finish;

}

}

}

goto check_safety;

The safety should be checked before accessing dtypes[i], since it might not be initialized if resolve_descriptors_with_scalars fails.

Those are both preexisting issues thoush so I'll send in a followup for those...

…e review.

ngoldbaum

I think I responded to all your comments. I also fixed some additional issues I missed on the first pass for unary ufuncs, ufunc.at, and ufunc.outer.

ngoldbaum · Jul 21, 2026

-                /* `resolve_descriptors_with_scalars` decides the descr */
+            if ((PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL) ||
+                /* str is only replaced if the resolver was passed the scalar */
+                (input_scalars[i] != NULL &&


The numeric case uses NULL to indicate that the value isn't available. The string case needs the original string so NULL doesn't make sense here.

But actually looking closer there's possible NULL-pointer dereferences associated with this behavior in the numeric cases. Inside npy_update_operand_for_scalar:

numpy/numpy/_core/src/multiarray/abstractdtypes.c

Lines 386 to 400 in e503c0e

Py_TYPE(scalar)->tp_name, descr);

return -1;

}

Py_INCREF(descr);

PyArrayObject *new = (PyArrayObject *)PyArray_NewFromDescr(

&PyArray_Type, descr, 0, NULL, NULL, NULL, 0, NULL);

Py_SETREF(*operand, new);

if (*operand == NULL) {

return -1;

}

if (scalar == NULL) {

/* The ufunc.resolve_dtypes paths can go here. Anything should go. */

return 0;

}

In the error path, scalar is accessed to get the name of the scalar type, but that happens before the null-check below.

Also here:

numpy/numpy/_core/src/umath/ufunc_object.c

Lines 4215 to 4232 in e503c0e

safety = ufuncimpl->resolve_descriptors_with_scalars(

ufuncimpl, signature, original_descrs, input_scalars,

dtypes, &view_offset

);

/* For scalars, replace the operand if needed (scalars can't be out) */

for (int i = 0; i < nin; i++) {

if ((PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL)) {

/* `resolve_descriptors_with_scalars` decides the descr */

if (npy_update_operand_for_scalar(

&operands[i], input_scalars[i], dtypes[i],

/* ignore cast safety for this op (resolvers job) */

NPY_SAFE_CASTING) < 0) {

goto finish;

}

}

}

goto check_safety;

The safety should be checked before accessing dtypes[i], since it might not be initialized if resolve_descriptors_with_scalars fails.

Those are both preexisting issues thoush so I'll send in a followup for those...

ngoldbaum · Jul 21, 2026

+    result = np.strings.replace(a, old, new)
+    assert_array_equal(result, np.array(["a+b"], dtype=dt))
+    assert a.array_calls == old.array_calls == new.array_calls == 1
+    assert a.ufunc_calls == old.ufunc_calls == new.ufunc_calls == 0


let me know if you think this test is over-engineered

I quite like it, though perhaps would just raise NotImplementedError in __array_ufunc__, in which case you do not need ufunc_calls and its checks.

mhvk

@ngoldbaum - this is rather nice! But while reviewing the special-casing for .outer, I realized there was a bigger problem there (see #32076).

I also somewhat wonder whether it would not be better to treat the python str scalars more similar to the numerical ones (see comment about the tests).

mhvk · Jul 23, 2026

        out_op_DTypes[i] = NPY_DTYPE(PyArray_DESCR(out_op[i]));
        Py_INCREF(out_op_DTypes[i]);

+        /* Does not affect promotion, only conversion after resolution. */


I think this can be moved inside the scalar-case if block on original l.688, since it would be a scalar and does not seem relevant for a ufunc with nin=1. Has the advantage of not causing any delay for non-string inputs.

mhvk · Jul 23, 2026

        /*
         * If we are working with Python literals/scalars, deal with them.
         * If needed, we create new array with the right descriptor.
+         * A str operand is only replaced when `inputs_tup` provides the


This describes what the if statement does, but not why, and I'm slightly confused. Why should it be skipped when not from inputs_tup, while other scalar cases are not skipped? (I probably mostly do not understand in what case we get here for any scalar when inputs_tup == NULL...)

mhvk · Jul 23, 2026

            goto fail;
        }
-        Py_SETREF(full_args.in, new_in);
+        original_inputs = full_args.in;


Hmm, this had me really think why it is necessary. The reason would seem to be that now a scalar is promoted to an array before testing whether it is a python scaler. But this is not only a problem for strings:

a = np.arange(10.0, dtype="f4").reshape(5, 2) np.add(1.0, a).dtype # dtype('float32') np.add.outer(1.0, a).dtype # dtype('float64')

See #32076.

It may be an idea to try to fix this separately from this PR, and solve it together...

mhvk · Jul 23, 2026

        goto fail;
    }

+    if (original_inputs != NULL) {


See comment above; would seem better to fix this with the general problem of ufunc.outer

mhvk · Jul 23, 2026

    if (resolve_descriptors(nop, ufunc, ufuncimpl,
            operands, operation_descrs, signature, operand_DTypes,
-            full_args.in, casting) < 0) {
+            original_inputs != NULL ? original_inputs : full_args.in,


mhvk · Jul 23, 2026

+    assert_array_equal(np.strings.find(arr, "c\0"), [2, -1])
+    assert_array_equal(np.strings.replace(arr, "\0", "!"), ["abc!", "abc"])
+
+    outer = np.add.outer(np.array(["x"], dtype="T"), "y\0")


Total nitpick, but I'd make this a different test function (and same for at below), especially as the test itself does not use the same input arrays.

mhvk · Jul 23, 2026

+        sep = ArrayLike("-")
+        result = func(a, sep)
+        for actual, expected in zip(result, [["a"], ["-"], ["b"]]):
+            assert_array_equal(actual, np.array(expected, dtype=dt))


Do you want to add strict=True in the comparison to be sure the dtype itself is checked too?

mhvk · Jul 23, 2026

+    result = np.strings.replace(a, old, new)
+    assert_array_equal(result, np.array(["a+b"], dtype=dt))
+    assert a.array_calls == old.array_calls == new.array_calls == 1
+    assert a.ufunc_calls == old.ufunc_calls == new.ufunc_calls == 0


I quite like it, though perhaps would just raise NotImplementedError in __array_ufunc__, in which case you do not need ufunc_calls and its checks.

mhvk · Jul 23, 2026


-    if np.result_type(arr, old, new).char == "T":
-        return _replace(arr, old, new, count)
+    if np.result_type(arr, old_arr, new_arr).char == "T":


Hmm, it would be nice if one could avoid asanyarray in case of str input and then let the rest still work. But right now,

np.result_type(np.array(123, "f4"), 1.0) # dtype('float32') np.result_type(np.array("123", "T"), "1") # TypeError: data type '' not understood

Is this because of not introducing an abstract dtype class for python strings?

Or could this be fixed more easily? (Anyway, not the biggest deal...)

ENH: add NEP-50 style semantics for string scalars and StringDType

7a893d7

ngoldbaum added the 01 - Enhancement label Jul 19, 2026

ngoldbaum added 2 commits July 19, 2026 14:50

DOC: add release notes

c54cd95

DOC: simplify release notes

0d4b710

ngoldbaum commented Jul 19, 2026

View reviewed changes

seberg reviewed Jul 19, 2026

View reviewed changes

MNT: fix unary ufuncs, ufunc.at, and ufunc.outer. Also respond to cod…

2ce5efc

…e review.

ngoldbaum commented Jul 21, 2026

View reviewed changes

mhvk mentioned this pull request Jul 23, 2026

BUG: NEP 50 promotion for python scalars is not followed for <ufunc>.outer and <ufunc>.at #32076

Open

mhvk reviewed Jul 23, 2026

View reviewed changes

	Py_TYPE(scalar)->tp_name, descr);
	return -1;
	}

	Py_INCREF(descr);
	PyArrayObject new = (PyArrayObject )PyArray_NewFromDescr(
	&PyArray_Type, descr, 0, NULL, NULL, NULL, 0, NULL);
	Py_SETREF(*operand, new);
	if (*operand == NULL) {
	return -1;
	}
	if (scalar == NULL) {
	/* The ufunc.resolve_dtypes paths can go here. Anything should go. */
	return 0;
	}

	safety = ufuncimpl->resolve_descriptors_with_scalars(
	ufuncimpl, signature, original_descrs, input_scalars,
	dtypes, &view_offset
	);

	/* For scalars, replace the operand if needed (scalars can't be out) */
	for (int i = 0; i < nin; i++) {
	if ((PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL)) {
	/* `resolve_descriptors_with_scalars` decides the descr */
	if (npy_update_operand_for_scalar(
	&operands[i], input_scalars[i], dtypes[i],
	/* ignore cast safety for this op (resolvers job) */
	NPY_SAFE_CASTING) < 0) {
	goto finish;
	}
	}
	}
	goto check_safety;

Search code, repositories, users, issues, pull requests...

Uh oh!

Uh oh!

Conversation

ngoldbaum commented Jul 19, 2026

PR summary

AI Disclosure

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants