Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

gh-133465: Allow PyErr_CheckSignals to be called without holding the GIL. #133466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
Loading
from
Draft

gh-133465: Allow PyErr_CheckSignals to be called without holding the GIL. #133466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c0f5de9
Allow PyErr_CheckSignals to be called without holding the GIL.
zackw Apr 30, 2025
3e25a93
Take advantage of PyErr_CheckSignals now being callable without the GIL.
zackw May 5, 2025
1c3af66
Add a new function instead of changing PyErr_CheckSignals semantics.
zackw May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 43 additions & 19 deletions 62 Doc/c-api/exceptions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,8 @@ Querying the error indicator
Signal Handling
===============

See the :mod:`signal` module for an overview of signals and signal
handling.

.. c:function:: int PyErr_CheckSignals()

Expand All @@ -639,29 +641,51 @@ Signal Handling
single: SIGINT (C macro)
single: KeyboardInterrupt (built-in exception)

This function interacts with Python's signal handling.
This function is to be called by long-running C code that wants to
be interruptible by user requests, such as by pressing Ctrl-C.

When it is called from the main thread and under the main Python
interpreter, it checks whether any signals have been delivered to
the process, and if so, invokes the corresponding Python-level
signal handler (if there is one) for each signal.

:c:func:`PyErr_CheckSignals()` attempts to call signal handlers
for each signal that has been delivered since the last time it
was called. If all signal handlers complete successfully, it
returns ``0``. However, if a signal handler raises an exception,
that exception is stored in the error indicator for the main thread,
and :c:func:`PyErr_CheckSignals()` immediately returns ``-1``.
(When this happens, some of the pending signals may not have had
their signal handlers called; they will be called the next time
:c:func:`PyErr_CheckSignals()` is called.)

Callers of :c:func:`PyErr_CheckSignals()` should treat a ``-1``
return value the same as any other failure of a C-API function;
they should immediately cease work, clean up (deallocating
resources, etc.) and propagate the failure status to their
callers.

When this function is called from other than the main thread, or
other than the main Python interpreter, it does not invoke any
signal handlers, and it always returns ``0``.

Regardless of context, calling this function may have the side
effect of running the cyclic garbage collector.

If the function is called from the main thread and under the main Python
interpreter, it checks whether a signal has been sent to the processes
and if so, invokes the corresponding signal handler. If the :mod:`signal`
module is supported, this can invoke a signal handler written in Python.

The function attempts to handle all pending signals, and then returns ``0``.
However, if a Python signal handler raises an exception, the error
indicator is set and the function returns ``-1`` immediately (such that
other pending signals may not have been handled yet: they will be on the
next :c:func:`PyErr_CheckSignals()` invocation).

If the function is called from a non-main thread, or under a non-main
Python interpreter, it does nothing and returns ``0``.

This function can be called by long-running C code that wants to
be interruptible by user requests (such as by pressing Ctrl-C).
.. warning::
This function may execute arbitrary Python code before returning
to its caller.

.. note::
The default Python signal handler for :c:macro:`!SIGINT` raises the
:exc:`KeyboardInterrupt` exception.
This function can be called without an :term:`attached thread state`
(see :ref:`threads`). However, this function may internally
attach (and then release) a thread state (only if it has any
work to do); it must be safe to do that at each point where
this function is called.

.. note::
The default Python signal handler for :c:macro:`!SIGINT` raises
the :exc:`KeyboardInterrupt` exception.

.. c:function:: void PyErr_SetInterrupt()

Expand Down
5 changes: 5 additions & 0 deletions 5 Doc/c-api/init.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1045,6 +1045,11 @@ to be released to attach their thread state, allowing true multi-core parallelis
For example, the standard :mod:`zlib` and :mod:`hashlib` modules detach the
:term:`thread state <attached thread state>` when compressing or hashing data.

.. note::
Any code that executes for a long time without returning to the
Python interpreter should call :c:func:`PyErr_CheckSignals()`
at reasonable intervals (at least once a millisecond) so that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is too frequent for my taste. Use interactions feel "delay-less" under 10 msec, and for ^C 100 even msec feels very quick, and 1sec would still be acceptable. After a few seconds I would hit ^C again.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is briefly discussed in the talk. If you add enough PyErr_CheckSignals calls to your extension module that every long-running loop can be interrupted, in practice this makes you do the check way too often, once every few tens of micro seconds. This is fine with the hypothetical new PyErr_CheckSignals_Detached, but with the old version, where you have to reclaim the GIL, it's way too costly. So in the talk I recommended looking at the actual system clock (with clock_gettime) and only calling PyErr_CheckSignals if a millisecond or more had gone by. That's faster than required for human responsiveness, yes, but the remaining overhead is the overhead of looking at the clock and you can't reduce that by doing the actual calls even less often. So that's where "at least once a millisecond" came from.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe explain that better in the docs then.

it can be interrupted by the user.

.. _gilstate:

Expand Down
1 change: 1 addition & 0 deletions 1 Include/pyerrors.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ PyAPI_FUNC(void) PyErr_WriteUnraisable(PyObject *);

/* In signalmodule.c */
PyAPI_FUNC(int) PyErr_CheckSignals(void);
PyAPI_FUNC(int) PyErr_CheckSignals_Detached(void);
PyAPI_FUNC(void) PyErr_SetInterrupt(void);
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000
PyAPI_FUNC(int) PyErr_SetInterruptEx(int signum);
Expand Down
7 changes: 7 additions & 0 deletions 7 Misc/NEWS.d/next/C_API/2025-05-05-15-20-11.gh-issue-133465.ZksxSw.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
:c:func:`PyErr_CheckSignals` has been changed to acquire the global
interpreter lock (GIL) itself, only when necessary (i.e. when it has work to
do). This means that modules that perform lengthy computations with the GIL
released may now call :c:func:`PyErr_CheckSignals` during those computations
without re-acquiring the GIL first. (However, it must be *safe to* acquire
the GIL at each point where :c:func:`PyErr_CheckSignals` is called. Also,
keep in mind that it can run arbitrary Python code before returning to you.)
Comment on lines +1 to +7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A NEWS entry should be more concise, users can refer to docs for in depth explanations.

Copy link
Author

@zackw zackw May 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this better?

:c:func:`PyErr_CheckSignals` has been made safe to call without holding the GIL.
It will acquire the GIL itself when it needs it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A NEWS entry should be more concise, users can refer to docs for in depth explanations.

@StanFromIreland, I disagree. AFAIK you aren't a core dev or triager, I wish you wouldn't give other contributors questionable advice without a reference in such an authoritive-sounding way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would update this and focus on how the new API differs from the old one: IIUC, "can be called without the GIL [or whatever PC phrasing] and doesn't acquire it unless a signal's handler needs to be called."

4 changes: 2 additions & 2 deletions 4 Modules/_io/fileio.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,16 +403,16 @@ _io_FileIO___init___impl(fileio *self, PyObject *nameobj, const char *mode,

errno = 0;
if (opener == Py_None) {
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
#ifdef MS_WINDOWS
self->fd = _wopen(widename, flags, 0666);
#else
self->fd = open(name, flags, 0666);
#endif
Py_END_ALLOW_THREADS
} while (self->fd < 0 && errno == EINTR &&
!(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS

if (async_err)
goto error;
Expand Down
2 changes: 0 additions & 2 deletions 2 Modules/_io/winconsoleio.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,9 +647,7 @@ read_console_w(HANDLE handle, DWORD maxlen, DWORD *readlen) {
if (WaitForSingleObjectEx(hInterruptEvent, 100, FALSE)
== WAIT_OBJECT_0) {
ResetEvent(hInterruptEvent);
Py_BLOCK_THREADS
sig = PyErr_CheckSignals();
Py_UNBLOCK_THREADS
if (sig < 0)
break;
}
Expand Down
8 changes: 4 additions & 4 deletions 8 Modules/_multiprocessing/posixshmem.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ _posixshmem_shm_open_impl(PyObject *module, PyObject *path, int flags,
PyErr_SetString(PyExc_ValueError, "embedded null character");
return -1;
}
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
fd = shm_open(name, flags, mode);
Py_END_ALLOW_THREADS
} while (fd < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS

if (fd < 0) {
if (!async_err)
Expand Down Expand Up @@ -102,11 +102,11 @@ _posixshmem_shm_unlink_impl(PyObject *module, PyObject *path)
PyErr_SetString(PyExc_ValueError, "embedded null character");
return NULL;
}
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
rv = shm_unlink(name);
Py_END_ALLOW_THREADS
} while (rv < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS

if (rv < 0) {
if (!async_err)
Expand Down
4 changes: 2 additions & 2 deletions 4 Modules/_multiprocessing/semaphore.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,19 +356,19 @@ _multiprocessing_SemLock_acquire_impl(SemLockObject *self, int blocking,

if (res < 0 && errno == EAGAIN && blocking) {
/* Couldn't acquire immediately, need to block */
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
if (!use_deadline) {
res = sem_wait(self->handle);
}
else {
res = sem_timedwait(self->handle, &deadline);
}
Py_END_ALLOW_THREADS
err = errno;
if (res == MP_EXCEPTION_HAS_BEEN_SET)
break;
} while (res < 0 && errno == EINTR && !PyErr_CheckSignals());
Py_END_ALLOW_THREADS
}

if (res < 0) {
Expand Down
32 changes: 16 additions & 16 deletions 32 Modules/fcntlmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,11 @@ fcntl_fcntl_impl(PyObject *module, int fd, int code, PyObject *arg)
}
}

Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = fcntl(fd, code, (int)int_arg);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
if (ret < 0) {
return !async_err ? PyErr_SetFromErrno(PyExc_OSError) : NULL;
}
Expand All @@ -103,11 +103,11 @@ fcntl_fcntl_impl(PyObject *module, int fd, int code, PyObject *arg)
memcpy(buf + len, guard, GUARDSZ);
PyBuffer_Release(&view);

Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = fcntl(fd, code, buf);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
if (ret < 0) {
return !async_err ? PyErr_SetFromErrno(PyExc_OSError) : NULL;
}
Expand Down Expand Up @@ -195,11 +195,11 @@ fcntl_ioctl_impl(PyObject *module, int fd, unsigned long code, PyObject *arg,
}
}

Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = ioctl(fd, code, int_arg);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
if (ret < 0) {
return !async_err ? PyErr_SetFromErrno(PyExc_OSError) : NULL;
}
Expand All @@ -219,11 +219,11 @@ fcntl_ioctl_impl(PyObject *module, int fd, unsigned long code, PyObject *arg,
memcpy(buf + len, guard, GUARDSZ);
ptr = buf;
}
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = ioctl(fd, code, ptr);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
if (ret < 0) {
if (!async_err) {
PyErr_SetFromErrno(PyExc_OSError);
Expand Down Expand Up @@ -261,11 +261,11 @@ fcntl_ioctl_impl(PyObject *module, int fd, unsigned long code, PyObject *arg,
memcpy(buf + len, guard, GUARDSZ);
PyBuffer_Release(&view);

Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = ioctl(fd, code, buf);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
if (ret < 0) {
return !async_err ? PyErr_SetFromErrno(PyExc_OSError) : NULL;
}
Expand Down Expand Up @@ -308,11 +308,11 @@ fcntl_flock_impl(PyObject *module, int fd, int code)
}

#ifdef HAVE_FLOCK
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = flock(fd, code);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
#else

#ifndef LOCK_SH
Expand All @@ -335,11 +335,11 @@ fcntl_flock_impl(PyObject *module, int fd, int code)
return NULL;
}
l.l_whence = l.l_start = l.l_len = 0;
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = fcntl(fd, (code & LOCK_NB) ? F_SETLK : F_SETLKW, &l);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
}
#endif /* HAVE_FLOCK */
if (ret < 0) {
Expand Down Expand Up @@ -439,11 +439,11 @@ fcntl_lockf_impl(PyObject *module, int fd, int code, PyObject *lenobj,
return NULL;
}
l.l_whence = whence;
Py_BEGIN_ALLOW_THREADS
do {
Py_BEGIN_ALLOW_THREADS
ret = fcntl(fd, (code & LOCK_NB) ? F_SETLK : F_SETLKW, &l);
Py_END_ALLOW_THREADS
} while (ret == -1 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
Py_END_ALLOW_THREADS
}
if (ret < 0) {
return !async_err ? PyErr_SetFromErrno(PyExc_OSError) : NULL;
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.