Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit c0b81c4

Browse filesBrowse files
gh-107305: Update the C-API Docs for PEP 684 (gh-107324)
1 parent ecc05e2 commit c0b81c4
Copy full SHA for c0b81c4

File tree

Expand file treeCollapse file tree

2 files changed

+197
-15
lines changed
Filter options
Expand file treeCollapse file tree

2 files changed

+197
-15
lines changed

‎Doc/c-api/init.rst

Copy file name to clipboardExpand all lines: Doc/c-api/init.rst
+194-15Lines changed: 194 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1226,7 +1226,95 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
12261226
function. You can create and destroy them using the following functions:
12271227
12281228
1229-
.. c:function:: PyThreadState* Py_NewInterpreter()
1229+
.. c:type:: PyInterpreterConfig
1230+
1231+
Structure containing most parameters to configure a sub-interpreter.
1232+
Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
1233+
never modified by the runtime.
1234+
1235+
.. versionadded:: 3.12
1236+
1237+
Structure fields:
1238+
1239+
.. c:member:: int use_main_obmalloc
1240+
1241+
If this is ``0`` then the sub-interpreter will use its own
1242+
"object" allocator state.
1243+
Otherwise it will use (share) the main interpreter's.
1244+
1245+
If this is ``0`` then
1246+
:c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
1247+
must be ``1`` (non-zero).
1248+
If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
1249+
must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
1250+
1251+
.. c:member:: int allow_fork
1252+
1253+
If this is ``0`` then the runtime will not support forking the
1254+
process in any thread where the sub-interpreter is currently active.
1255+
Otherwise fork is unrestricted.
1256+
1257+
Note that the :mod:`subprocess` module still works
1258+
when fork is disallowed.
1259+
1260+
.. c:member:: int allow_exec
1261+
1262+
If this is ``0`` then the runtime will not support replacing the
1263+
current process via exec (e.g. :func:`os.execv`) in any thread
1264+
where the sub-interpreter is currently active.
1265+
Otherwise exec is unrestricted.
1266+
1267+
Note that the :mod:`subprocess` module still works
1268+
when exec is disallowed.
1269+
1270+
.. c:member:: int allow_threads
1271+
1272+
If this is ``0`` then the sub-interpreter's :mod:`threading` module
1273+
won't create threads.
1274+
Otherwise threads are allowed.
1275+
1276+
.. c:member:: int allow_daemon_threads
1277+
1278+
If this is ``0`` then the sub-interpreter's :mod:`threading` module
1279+
won't create daemon threads.
1280+
Otherwise daemon threads are allowed (as long as
1281+
:c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
1282+
1283+
.. c:member:: int check_multi_interp_extensions
1284+
1285+
If this is ``0`` then all extension modules may be imported,
1286+
including legacy (single-phase init) modules,
1287+
in any thread where the sub-interpreter is currently active.
1288+
Otherwise only multi-phase init extension modules
1289+
(see :pep:`489`) may be imported.
1290+
1291+
This must be ``1`` (non-zero) if
1292+
:c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
1293+
1294+
.. c:member:: int gil
1295+
1296+
This determines the operation of the GIL for the sub-interpreter.
1297+
It may be one of the following:
1298+
1299+
.. c:namespace:: NULL
1300+
1301+
.. c:macro:: PyInterpreterConfig_DEFAULT_GIL
1302+
1303+
Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
1304+
1305+
.. c:macro:: PyInterpreterConfig_SHARED_GIL
1306+
1307+
Use (share) the main interpreter's GIL.
1308+
1309+
.. c:macro:: PyInterpreterConfig_OWN_GIL
1310+
1311+
Use the sub-interpreter's own GIL.
1312+
1313+
If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
1314+
:c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
1315+
1316+
1317+
.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
12301318
12311319
.. index::
12321320
pair: module; builtins
@@ -1246,16 +1334,47 @@ function. You can create and destroy them using the following functions:
12461334
``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
12471335
file descriptors).
12481336
1249-
The return value points to the first thread state created in the new
1337+
The given *config* controls the options with which the interpreter
1338+
is initialized.
1339+
1340+
Upon success, *tstate_p* will be set to the first thread state
1341+
created in the new
12501342
sub-interpreter. This thread state is made in the current thread state.
12511343
Note that no actual thread is created; see the discussion of thread states
1252-
below. If creation of the new interpreter is unsuccessful, ``NULL`` is
1253-
returned; no exception is set since the exception state is stored in the
1254-
current thread state and there may not be a current thread state. (Like all
1255-
other Python/C API functions, the global interpreter lock must be held before
1256-
calling this function and is still held when it returns; however, unlike most
1257-
other Python/C API functions, there needn't be a current thread state on
1258-
entry.)
1344+
below. If creation of the new interpreter is unsuccessful,
1345+
*tstate_p* is set to ``NULL``;
1346+
no exception is set since the exception state is stored in the
1347+
current thread state and there may not be a current thread state.
1348+
1349+
Like all other Python/C API functions, the global interpreter lock
1350+
must be held before calling this function and is still held when it
1351+
returns. Likewise a current thread state must be set on entry. On
1352+
success, the returned thread state will be set as current. If the
1353+
sub-interpreter is created with its own GIL then the GIL of the
1354+
calling interpreter will be released. When the function returns,
1355+
the new interpreter's GIL will be held by the current thread and
1356+
the previously interpreter's GIL will remain released here.
1357+
1358+
.. versionadded:: 3.12
1359+
1360+
Sub-interpreters are most effective when isolated from each other,
1361+
with certain functionality restricted::
1362+
1363+
PyInterpreterConfig config = {
1364+
.use_main_obmalloc = 0,
1365+
.allow_fork = 0,
1366+
.allow_exec = 0,
1367+
.allow_threads = 1,
1368+
.allow_daemon_threads = 0,
1369+
.check_multi_interp_extensions = 1,
1370+
.gil = PyInterpreterConfig_OWN_GIL,
1371+
};
1372+
PyThreadState *tstate = Py_NewInterpreterFromConfig(&config);
1373+
1374+
Note that the config is used only briefly and does not get modified.
1375+
During initialization the config's values are converted into various
1376+
:c:type:`PyInterpreterState` values. A read-only copy of the config
1377+
may be stored internally on the :c:type:`PyInterpreterState`.
12591378
12601379
.. index::
12611380
single: Py_FinalizeEx()
@@ -1290,19 +1409,79 @@ function. You can create and destroy them using the following functions:
12901409
.. index:: single: close() (in module os)
12911410
12921411
1412+
.. c:function:: PyThreadState* Py_NewInterpreter(void)
1413+
1414+
.. index::
1415+
pair: module; builtins
1416+
pair: module; __main__
1417+
pair: module; sys
1418+
single: stdout (in module sys)
1419+
single: stderr (in module sys)
1420+
single: stdin (in module sys)
1421+
1422+
Create a new sub-interpreter. This is essentially just a wrapper
1423+
around :c:func:`Py_NewInterpreterFromConfig` with a config that
1424+
preserves the existing behavior. The result is an unisolated
1425+
sub-interpreter that shares the main interpreter's GIL, allows
1426+
fork/exec, allows daemon threads, and allows single-phase init
1427+
modules.
1428+
1429+
12931430
.. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
12941431
12951432
.. index:: single: Py_FinalizeEx()
12961433
1297-
Destroy the (sub-)interpreter represented by the given thread state. The given
1298-
thread state must be the current thread state. See the discussion of thread
1299-
states below. When the call returns, the current thread state is ``NULL``. All
1300-
thread states associated with this interpreter are destroyed. (The global
1301-
interpreter lock must be held before calling this function and is still held
1302-
when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
1434+
Destroy the (sub-)interpreter represented by the given thread state.
1435+
The given thread state must be the current thread state. See the
1436+
discussion of thread states below. When the call returns,
1437+
the current thread state is ``NULL``. All thread states associated
1438+
with this interpreter are destroyed. The global interpreter lock
1439+
used by the target interpreter must be held before calling this
1440+
function. No GIL is held when it returns.
1441+
1442+
:c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
13031443
haven't been explicitly destroyed at that point.
13041444
13051445
1446+
A Per-Interpreter GIL
1447+
---------------------
1448+
1449+
Using :c:func:`Py_NewInterpreterFromConfig` you can create
1450+
a sub-interpreter that is completely isolated from other interpreters,
1451+
including having its own GIL. The most important benefit of this
1452+
isolation is that such an interpreter can execute Python code without
1453+
being blocked by other interpreters or blocking any others. Thus a
1454+
single Python process can truly take advantage of multiple CPU cores
1455+
when running Python code. The isolation also encourages a different
1456+
approach to concurrency than that of just using threads.
1457+
(See :pep:`554`.)
1458+
1459+
Using an isolated interpreter requires vigilance in preserving that
1460+
isolation. That especially means not sharing any objects or mutable
1461+
state without guarantees about thread-safety. Even objects that are
1462+
otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
1463+
because of the refcount. One simple but less-efficient approach around
1464+
this is to use a global lock around all use of some state (or object).
1465+
Alternately, effectively immutable objects (like integers or strings)
1466+
can be made safe in spite of their refcounts by making them "immortal".
1467+
In fact, this has been done for the builtin singletons, small integers,
1468+
and a number of other builtin objects.
1469+
1470+
If you preserve isolation then you will have access to proper multi-core
1471+
computing without the complications that come with free-threading.
1472+
Failure to preserve isolation will expose you to the full consequences
1473+
of free-threading, including races and hard-to-debug crashes.
1474+
1475+
Aside from that, one of the main challenges of using multiple isolated
1476+
interpreters is how to communicate between them safely (not break
1477+
isolation) and efficiently. The runtime and stdlib do not provide
1478+
any standard approach to this yet. A future stdlib module would help
1479+
mitigate the effort of preserving isolation and expose effective tools
1480+
for communicating (and sharing) data between interpreters.
1481+
1482+
.. versionadded:: 3.12
1483+
1484+
13061485
Bugs and caveats
13071486
----------------
13081487
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Add documentation for :c:type:`PyInterpreterConfig` and
2+
:c:func:`Py_NewInterpreterFromConfig`. Also clarify some of the nearby docs
3+
relative to per-interpreter GIL.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.