@@ -1226,7 +1226,95 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
1226
1226
function. You can create and destroy them using the following functions:
1227
1227
1228
1228
1229
- .. c:function:: PyThreadState* Py_NewInterpreter()
1229
+ .. c:type:: PyInterpreterConfig
1230
+
1231
+ Structure containing most parameters to configure a sub-interpreter.
1232
+ Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
1233
+ never modified by the runtime.
1234
+
1235
+ .. versionadded:: 3.12
1236
+
1237
+ Structure fields:
1238
+
1239
+ .. c:member:: int use_main_obmalloc
1240
+
1241
+ If this is ``0`` then the sub-interpreter will use its own
1242
+ " object" allocator state.
1243
+ Otherwise it will use (share) the main interpreter's.
1244
+
1245
+ If this is ``0`` then
1246
+ :c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
1247
+ must be ``1`` (non-zero).
1248
+ If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
1249
+ must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
1250
+
1251
+ .. c:member:: int allow_fork
1252
+
1253
+ If this is ``0`` then the runtime will not support forking the
1254
+ process in any thread where the sub-interpreter is currently active.
1255
+ Otherwise fork is unrestricted.
1256
+
1257
+ Note that the :mod:`subprocess` module still works
1258
+ when fork is disallowed.
1259
+
1260
+ .. c:member:: int allow_exec
1261
+
1262
+ If this is ``0`` then the runtime will not support replacing the
1263
+ current process via exec (e.g. :func:`os.execv`) in any thread
1264
+ where the sub-interpreter is currently active.
1265
+ Otherwise exec is unrestricted.
1266
+
1267
+ Note that the :mod:`subprocess` module still works
1268
+ when exec is disallowed.
1269
+
1270
+ .. c:member:: int allow_threads
1271
+
1272
+ If this is ``0`` then the sub-interpreter's :mod:`threading` module
1273
+ won't create threads.
1274
+ Otherwise threads are allowed.
1275
+
1276
+ .. c:member:: int allow_daemon_threads
1277
+
1278
+ If this is ``0`` then the sub-interpreter's :mod:`threading` module
1279
+ won't create daemon threads.
1280
+ Otherwise daemon threads are allowed (as long as
1281
+ :c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
1282
+
1283
+ .. c:member:: int check_multi_interp_extensions
1284
+
1285
+ If this is ``0`` then all extension modules may be imported,
1286
+ including legacy (single-phase init) modules,
1287
+ in any thread where the sub-interpreter is currently active.
1288
+ Otherwise only multi-phase init extension modules
1289
+ (see :pep:`489`) may be imported.
1290
+
1291
+ This must be ``1`` (non-zero) if
1292
+ :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
1293
+
1294
+ .. c:member:: int gil
1295
+
1296
+ This determines the operation of the GIL for the sub-interpreter.
1297
+ It may be one of the following:
1298
+
1299
+ .. c:namespace:: NULL
1300
+
1301
+ .. c:macro:: PyInterpreterConfig_DEFAULT_GIL
1302
+
1303
+ Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
1304
+
1305
+ .. c:macro:: PyInterpreterConfig_SHARED_GIL
1306
+
1307
+ Use (share) the main interpreter's GIL.
1308
+
1309
+ .. c:macro:: PyInterpreterConfig_OWN_GIL
1310
+
1311
+ Use the sub-interpreter's own GIL.
1312
+
1313
+ If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
1314
+ :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
1315
+
1316
+
1317
+ .. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
1230
1318
1231
1319
.. index::
1232
1320
pair: module; builtins
@@ -1246,16 +1334,47 @@ function. You can create and destroy them using the following functions:
1246
1334
``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
1247
1335
file descriptors).
1248
1336
1249
- The return value points to the first thread state created in the new
1337
+ The given *config* controls the options with which the interpreter
1338
+ is initialized.
1339
+
1340
+ Upon success, *tstate_p* will be set to the first thread state
1341
+ created in the new
1250
1342
sub-interpreter. This thread state is made in the current thread state.
1251
1343
Note that no actual thread is created; see the discussion of thread states
1252
- below. If creation of the new interpreter is unsuccessful, ``NULL`` is
1253
- returned; no exception is set since the exception state is stored in the
1254
- current thread state and there may not be a current thread state. (Like all
1255
- other Python/C API functions, the global interpreter lock must be held before
1256
- calling this function and is still held when it returns; however, unlike most
1257
- other Python/C API functions, there needn't be a current thread state on
1258
- entry.)
1344
+ below. If creation of the new interpreter is unsuccessful,
1345
+ *tstate_p* is set to ``NULL``;
1346
+ no exception is set since the exception state is stored in the
1347
+ current thread state and there may not be a current thread state.
1348
+
1349
+ Like all other Python/C API functions, the global interpreter lock
1350
+ must be held before calling this function and is still held when it
1351
+ returns. Likewise a current thread state must be set on entry. On
1352
+ success, the returned thread state will be set as current. If the
1353
+ sub-interpreter is created with its own GIL then the GIL of the
1354
+ calling interpreter will be released. When the function returns,
1355
+ the new interpreter's GIL will be held by the current thread and
1356
+ the previously interpreter's GIL will remain released here.
1357
+
1358
+ .. versionadded:: 3.12
1359
+
1360
+ Sub-interpreters are most effective when isolated from each other,
1361
+ with certain functionality restricted::
1362
+
1363
+ PyInterpreterConfig config = {
1364
+ .use_main_obmalloc = 0,
1365
+ .allow_fork = 0,
1366
+ .allow_exec = 0,
1367
+ .allow_threads = 1,
1368
+ .allow_daemon_threads = 0,
1369
+ .check_multi_interp_extensions = 1,
1370
+ .gil = PyInterpreterConfig_OWN_GIL,
1371
+ };
1372
+ PyThreadState *tstate = Py_NewInterpreterFromConfig(&config);
1373
+
1374
+ Note that the config is used only briefly and does not get modified.
1375
+ During initialization the config's values are converted into various
1376
+ :c:type:`PyInterpreterState` values. A read-only copy of the config
1377
+ may be stored internally on the :c:type:`PyInterpreterState`.
1259
1378
1260
1379
.. index::
1261
1380
single: Py_FinalizeEx()
@@ -1290,19 +1409,79 @@ function. You can create and destroy them using the following functions:
1290
1409
.. index:: single: close() (in module os)
1291
1410
1292
1411
1412
+ .. c:function:: PyThreadState* Py_NewInterpreter(void)
1413
+
1414
+ .. index::
1415
+ pair: module; builtins
1416
+ pair: module; __main__
1417
+ pair: module; sys
1418
+ single: stdout (in module sys)
1419
+ single: stderr (in module sys)
1420
+ single: stdin (in module sys)
1421
+
1422
+ Create a new sub-interpreter. This is essentially just a wrapper
1423
+ around :c:func:`Py_NewInterpreterFromConfig` with a config that
1424
+ preserves the existing behavior. The result is an unisolated
1425
+ sub-interpreter that shares the main interpreter's GIL, allows
1426
+ fork/exec, allows daemon threads, and allows single-phase init
1427
+ modules.
1428
+
1429
+
1293
1430
.. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
1294
1431
1295
1432
.. index:: single: Py_FinalizeEx()
1296
1433
1297
- Destroy the (sub-)interpreter represented by the given thread state. The given
1298
- thread state must be the current thread state. See the discussion of thread
1299
- states below. When the call returns, the current thread state is ``NULL``. All
1300
- thread states associated with this interpreter are destroyed. (The global
1301
- interpreter lock must be held before calling this function and is still held
1302
- when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
1434
+ Destroy the (sub-)interpreter represented by the given thread state.
1435
+ The given thread state must be the current thread state. See the
1436
+ discussion of thread states below. When the call returns,
1437
+ the current thread state is ``NULL``. All thread states associated
1438
+ with this interpreter are destroyed. The global interpreter lock
1439
+ used by the target interpreter must be held before calling this
1440
+ function. No GIL is held when it returns.
1441
+
1442
+ :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
1303
1443
haven't been explicitly destroyed at that point.
1304
1444
1305
1445
1446
+ A Per-Interpreter GIL
1447
+ ---------------------
1448
+
1449
+ Using :c:func:`Py_NewInterpreterFromConfig` you can create
1450
+ a sub-interpreter that is completely isolated from other interpreters,
1451
+ including having its own GIL. The most important benefit of this
1452
+ isolation is that such an interpreter can execute Python code without
1453
+ being blocked by other interpreters or blocking any others. Thus a
1454
+ single Python process can truly take advantage of multiple CPU cores
1455
+ when running Python code. The isolation also encourages a different
1456
+ approach to concurrency than that of just using threads.
1457
+ (See :pep:`554`.)
1458
+
1459
+ Using an isolated interpreter requires vigilance in preserving that
1460
+ isolation. That especially means not sharing any objects or mutable
1461
+ state without guarantees about thread-safety. Even objects that are
1462
+ otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
1463
+ because of the refcount. One simple but less-efficient approach around
1464
+ this is to use a global lock around all use of some state (or object).
1465
+ Alternately, effectively immutable objects (like integers or strings)
1466
+ can be made safe in spite of their refcounts by making them " immortal" .
1467
+ In fact, this has been done for the builtin singletons, small integers,
1468
+ and a number of other builtin objects.
1469
+
1470
+ If you preserve isolation then you will have access to proper multi-core
1471
+ computing without the complications that come with free-threading.
1472
+ Failure to preserve isolation will expose you to the full consequences
1473
+ of free-threading, including races and hard-to-debug crashes.
1474
+
1475
+ Aside from that, one of the main challenges of using multiple isolated
1476
+ interpreters is how to communicate between them safely (not break
1477
+ isolation) and efficiently. The runtime and stdlib do not provide
1478
+ any standard approach to this yet. A future stdlib module would help
1479
+ mitigate the effort of preserving isolation and expose effective tools
1480
+ for communicating (and sharing) data between interpreters.
1481
+
1482
+ .. versionadded:: 3.12
1483
+
1484
+
1306
1485
Bugs and caveats
1307
1486
----------------
1308
1487
0 commit comments