Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit c0f5de9

Browse filesBrowse files
zackwzackw
committed
Allow PyErr_CheckSignals to be called without holding the GIL.
Compiled-code modules that implement time-consuming operations that don’t require manipulating Python objects, are supposed to call PyErr_CheckSignals frequently throughout each such operation, so that if the user interrupts the operation with control-C, it is cancelled promptly. In the normal case where no signals are pending, PyErr_CheckSignals is cheap; however, callers must hold the GIL, and compiled-code modules that implement time-consuming operations are also supposed to release the GIL during each such operation. The overhead of reclaiming the GIL in order to call PyErr_CheckSignals, and then releasing it again, sufficiently often for reasonable user responsiveness, can be substantial. If my understanding of the thread-state rules is correct, PyErr_CheckSignals only *needs* the GIL if it has work to do. *Checking* whether there is a pending signal, or a pending request to run the cycle collector, requires only a couple of atomic loads. Therefore: Reorganize the logic of PyErr_CheckSignals and its close relatives (_PyErr_CheckSignals and _PyErr_CheckSignalsTstate) so that all the “do we have anything to do” checks are done in a batch before anything that needs the GIL. If any of them are true, acquire the GIL, repeat the check (because another thread could have stolen the event while we were waiting for the GIL), and then actually do the work, enabling callers to *not* hold the GIL. (There are some fine details here that I’d really appreciate a second pair of eyes on — see the comments in the new functions _PyErr_CheckSignalsHoldingGIL and _PyErr_CheckSignalsNoGIL.)
1 parent 9434709 commit c0f5de9
Copy full SHA for c0f5de9

File tree

Expand file treeCollapse file tree

4 files changed

+125
-44
lines changed
Filter options
Expand file treeCollapse file tree

4 files changed

+125
-44
lines changed

‎Doc/c-api/exceptions.rst

Copy file name to clipboardExpand all lines: Doc/c-api/exceptions.rst
+43-19Lines changed: 43 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -631,6 +631,8 @@ Querying the error indicator
631631
Signal Handling
632632
===============
633633
634+
See the :mod:`signal` module for an overview of signals and signal
635+
handling.
634636
635637
.. c:function:: int PyErr_CheckSignals()
636638
@@ -639,29 +641,51 @@ Signal Handling
639641
single: SIGINT (C macro)
640642
single: KeyboardInterrupt (built-in exception)
641643
642-
This function interacts with Python's signal handling.
644+
This function is to be called by long-running C code that wants to
645+
be interruptible by user requests, such as by pressing Ctrl-C.
646+
647+
When it is called from the main thread and under the main Python
648+
interpreter, it checks whether any signals have been delivered to
649+
the process, and if so, invokes the corresponding Python-level
650+
signal handler (if there is one) for each signal.
651+
652+
:c:func:`PyErr_CheckSignals()` attempts to call signal handlers
653+
for each signal that has been delivered since the last time it
654+
was called. If all signal handlers complete successfully, it
655+
returns ``0``. However, if a signal handler raises an exception,
656+
that exception is stored in the error indicator for the main thread,
657+
and :c:func:`PyErr_CheckSignals()` immediately returns ``-1``.
658+
(When this happens, some of the pending signals may not have had
659+
their signal handlers called; they will be called the next time
660+
:c:func:`PyErr_CheckSignals()` is called.)
661+
662+
Callers of :c:func:`PyErr_CheckSignals()` should treat a ``-1``
663+
return value the same as any other failure of a C-API function;
664+
they should immediately cease work, clean up (deallocating
665+
resources, etc.) and propagate the failure status to their
666+
callers.
667+
668+
When this function is called from other than the main thread, or
669+
other than the main Python interpreter, it does not invoke any
670+
signal handlers, and it always returns ``0``.
671+
672+
Regardless of context, calling this function may have the side
673+
effect of running the cyclic garbage collector.
643674
644-
If the function is called from the main thread and under the main Python
645-
interpreter, it checks whether a signal has been sent to the processes
646-
and if so, invokes the corresponding signal handler. If the :mod:`signal`
647-
module is supported, this can invoke a signal handler written in Python.
648-
649-
The function attempts to handle all pending signals, and then returns ``0``.
650-
However, if a Python signal handler raises an exception, the error
651-
indicator is set and the function returns ``-1`` immediately (such that
652-
other pending signals may not have been handled yet: they will be on the
653-
next :c:func:`PyErr_CheckSignals()` invocation).
654-
655-
If the function is called from a non-main thread, or under a non-main
656-
Python interpreter, it does nothing and returns ``0``.
657-
658-
This function can be called by long-running C code that wants to
659-
be interruptible by user requests (such as by pressing Ctrl-C).
675+
.. warning::
676+
This function may execute arbitrary Python code before returning
677+
to its caller.
660678
661679
.. note::
662-
The default Python signal handler for :c:macro:`!SIGINT` raises the
663-
:exc:`KeyboardInterrupt` exception.
680+
This function can be called without an :term:`attached thread state`
681+
(see :ref:`threads`). However, this function may internally
682+
attach (and then release) a thread state (only if it has any
683+
work to do); it must be safe to do that at each point where
684+
this function is called.
664685
686+
.. note::
687+
The default Python signal handler for :c:macro:`!SIGINT` raises
688+
the :exc:`KeyboardInterrupt` exception.
665689
666690
.. c:function:: void PyErr_SetInterrupt()
667691

‎Doc/c-api/init.rst

Copy file name to clipboardExpand all lines: Doc/c-api/init.rst
+5Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1045,6 +1045,11 @@ to be released to attach their thread state, allowing true multi-core parallelis
10451045
For example, the standard :mod:`zlib` and :mod:`hashlib` modules detach the
10461046
:term:`thread state <attached thread state>` when compressing or hashing data.
10471047
1048+
.. note::
1049+
Any code that executes for a long time without returning to the
1050+
Python interpreter should call :c:func:`PyErr_CheckSignals()`
1051+
at reasonable intervals (at least once a millisecond) so that
1052+
it can be interrupted by the user.
10481053
10491054
.. _gilstate:
10501055

‎Misc/NEWS.d/next/C_API/2025-05-05-15-20-11.gh-issue-133465.ZksxSw.rst

Copy file name to clipboard
+7Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
:c:func:`PyErr_CheckSignals` has been changed to acquire the global
2+
interpreter lock (GIL) itself, only when necessary (i.e. when it has work to
3+
do). This means that modules that perform lengthy computations with the GIL
4+
released may now call :c:func:`PyErr_CheckSignals` during those computations
5+
without re-acquiring the GIL first. (However, it must be *safe to* acquire
6+
the GIL at each point where :c:func:`PyErr_CheckSignals` is called. Also,
7+
keep in mind that it can run arbitrary Python code before returning to you.)

‎Modules/signalmodule.c

Copy file name to clipboardExpand all lines: Modules/signalmodule.c
+70-25Lines changed: 70 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1759,12 +1759,22 @@ _PySignal_Fini(void)
17591759
Py_CLEAR(state->ignore_handler);
17601760
}
17611761

1762-
1763-
/* Declared in pyerrors.h */
1764-
int
1765-
PyErr_CheckSignals(void)
1762+
/* Subroutine of _PyErr_CheckSignalsNoGIL. Does all the work that
1763+
needs the GIL. When called, 'tstate' must be the thread state for
1764+
the current thread, and the current thread must hold the GIL. */
1765+
static int
1766+
_PyErr_CheckSignalsHoldingGIL(PyThreadState *tstate, bool cycle_collect)
17661767
{
1767-
PyThreadState *tstate = _PyThreadState_GET();
1768+
#if defined(Py_REMOTE_DEBUG) && defined(Py_SUPPORTS_REMOTE_DEBUG)
1769+
_PyRunRemoteDebugger(tstate);
1770+
#endif
1771+
1772+
/* It is necessary to repeat all of the checks of global flags
1773+
that were done in _PyErr_CheckSignalsNoGIL. At the time of
1774+
those checks, we might not have held the GIL; in between those
1775+
checks and when we acquired the GIL, some other thread may have
1776+
processed the events that were flagged. Since we now hold the
1777+
GIL, a check now will be valid until we release it again. */
17681778

17691779
/* Opportunistically check if the GC is scheduled to run and run it
17701780
if we have a request. This is done here because native code needs
@@ -1777,24 +1787,8 @@ PyErr_CheckSignals(void)
17771787
_Py_RunGC(tstate);
17781788
}
17791789

1780-
#if defined(Py_REMOTE_DEBUG) && defined(Py_SUPPORTS_REMOTE_DEBUG)
1781-
_PyRunRemoteDebugger(tstate);
1782-
#endif
1783-
1784-
if (!_Py_ThreadCanHandleSignals(tstate->interp)) {
1785-
return 0;
1786-
}
1787-
1788-
return _PyErr_CheckSignalsTstate(tstate);
1789-
}
1790-
1791-
1792-
/* Declared in cpython/pyerrors.h */
1793-
int
1794-
_PyErr_CheckSignalsTstate(PyThreadState *tstate)
1795-
{
1796-
_Py_CHECK_EMSCRIPTEN_SIGNALS();
1797-
if (!_Py_atomic_load_int(&is_tripped)) {
1790+
if (!_Py_ThreadCanHandleSignals(tstate->interp)
1791+
|| !_Py_atomic_load_int(&is_tripped)) {
17981792
return 0;
17991793
}
18001794

@@ -1877,15 +1871,66 @@ _PyErr_CheckSignalsTstate(PyThreadState *tstate)
18771871
return 0;
18781872
}
18791873

1874+
/* Subroutine of the PyErr_CheckSignals family:
1875+
Determine whether there is actually any work needing to be done.
1876+
If so, acquire the GIL if necessary, and do that work. */
1877+
static int
1878+
_PyErr_CheckSignalsNoGIL(PyThreadState *tstate, bool cycle_collect)
1879+
{
1880+
/* If this thread does not have a thread state at all, then it has
1881+
never been associated with the Python runtime, so it should not
1882+
attempt to handle signals or run the cycle collector. */
1883+
if (!tstate) {
1884+
return 0;
1885+
}
1886+
1887+
_Py_CHECK_EMSCRIPTEN_SIGNALS();
1888+
1889+
/* Don't acquire the GIL if we don't have anything to do.
1890+
VERIFYME: I *think* every piece of this expression is safe to
1891+
execute without holding the GIL and is already sufficiently
1892+
atomic. */
1893+
if ((!_Py_ThreadCanHandleSignals(tstate->interp)
1894+
|| !_Py_atomic_load_int(&is_tripped))
1895+
&& (!cycle_collect
1896+
|| !_Py_eval_breaker_bit_is_set(tstate, _PY_GC_SCHEDULED_BIT))) {
1897+
return 0;
1898+
}
18801899

1900+
/* FIXME: Given that we already have 'tstate', is there a more efficient
1901+
way to do this? */
1902+
PyGILState_STATE st = PyGILState_Ensure();
1903+
int err = _PyErr_CheckSignalsHoldingGIL(tstate, cycle_collect);
1904+
PyGILState_Release(st);
18811905

1906+
return err;
1907+
}
1908+
1909+
/* Declared in pycore_pyerrors.h.
1910+
This function may be called without the GIL. */
1911+
int
1912+
_PyErr_CheckSignalsTstate(PyThreadState *tstate)
1913+
{
1914+
return _PyErr_CheckSignalsNoGIL(tstate, true);
1915+
}
1916+
1917+
/* Declared in pycore_pyerrors.h.
1918+
This function may be called without the GIL. */
18821919
int
18831920
_PyErr_CheckSignals(void)
18841921
{
1885-
PyThreadState *tstate = _PyThreadState_GET();
1886-
return _PyErr_CheckSignalsTstate(tstate);
1922+
PyThreadState *tstate = PyGILState_GetThisThreadState();
1923+
return _PyErr_CheckSignalsNoGIL(tstate, false);
18871924
}
18881925

1926+
/* Declared in pyerrors.h.
1927+
This function may be called without the GIL. */
1928+
int
1929+
PyErr_CheckSignals(void)
1930+
{
1931+
PyThreadState *tstate = PyGILState_GetThisThreadState();
1932+
return _PyErr_CheckSignalsNoGIL(tstate, true);
1933+
}
18891934

18901935
/* Simulate the effect of a signal arriving. The next time PyErr_CheckSignals
18911936
is called, the corresponding Python signal handler will be raised.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.