Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 7d83f7b

Browse filesBrowse files
gvanrossumgvanrossum
authored
gh-118335: Configure Tier 2 interpreter at build time (#118339)
The code for Tier 2 is now only compiled when configured with `--enable-experimental-jit[=yes|interpreter]`. We drop support for `PYTHON_UOPS` and -`Xuops`, but you can disable the interpreter or JIT at runtime by setting `PYTHON_JIT=0`. You can also build it without enabling it by default using `--enable-experimental-jit=yes-off`; enable with `PYTHON_JIT=1`. On Windows, the `build.bat` script supports `--experimental-jit`, `--experimental-jit-off`, `--experimental-interpreter`. In the C code, `_Py_JIT` is defined as before when the JIT is enabled; the new variable `_Py_TIER2` is defined when the JIT *or* the interpreter is enabled. It is actually a bitmask: 1: JIT; 2: default-off; 4: interpreter.
1 parent 9c468e2 commit 7d83f7b
Copy full SHA for 7d83f7b
Expand file treeCollapse file tree

32 files changed

+181
-42
lines changed

‎Doc/whatsnew/3.13.rst

Copy file name to clipboardExpand all lines: Doc/whatsnew/3.13.rst
+23-7Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -888,7 +888,7 @@ Experimental JIT Compiler
888888
=========================
889889

890890
When CPython is configured using the ``--enable-experimental-jit`` option,
891-
a just-in-time compiler is added which can speed up some Python programs.
891+
a just-in-time compiler is added which may speed up some Python programs.
892892

893893
The internal architecture is roughly as follows.
894894

@@ -905,19 +905,35 @@ The internal architecture is roughly as follows.
905905
before it is interpreted or translated to machine code.
906906

907907
* There is a Tier 2 interpreter, but it is mostly intended for debugging
908-
the earlier stages of the optimization pipeline. If the JIT is not
909-
enabled, the Tier 2 interpreter can be invoked by passing Python the
910-
``-X uops`` option or by setting the ``PYTHON_UOPS`` environment
911-
variable to ``1``.
908+
the earlier stages of the optimization pipeline.
909+
The Tier 2 interpreter can be enabled by configuring Python
910+
with ``--enable-experimental-jit=interpreter``.
912911

913-
* When the ``--enable-experimental-jit`` option is used, the optimized
912+
* When the JIT is enabled, the optimized
914913
Tier 2 IR is translated to machine code, which is then executed.
915-
This does not require additional runtime options.
916914

917915
* The machine code translation process uses an architecture called
918916
*copy-and-patch*. It has no runtime dependencies, but there is a new
919917
build-time dependency on LLVM.
920918

919+
The ``--enable-experimental-jit`` flag has the following optional values:
920+
921+
* ``no`` (default) -- Disable the entire Tier 2 and JIT pipeline.
922+
923+
* ``yes`` (default if the flag is present without optional value)
924+
-- Enable the JIT. To disable the JIT at runtime,
925+
pass the environment variable ``PYTHON_JIT=0``.
926+
927+
* ``yes-off`` -- Build the JIT but disable it by default.
928+
To enable the JIT at runtime, pass the environment variable
929+
``PYTHON_JIT=1``.
930+
931+
* ``interpreter`` -- Enable the Tier 2 interpreter but disable the JIT.
932+
The interpreter can be disabled by running with
933+
``PYTHON_JIT=0``.
934+
935+
(On Windows, use ``PCbuild/build.bat --enable-jit`` to enable the JIT.)
936+
921937
See :pep:`744` for more details.
922938

923939
(JIT by Brandt Bucher, inspired by a paper by Haoran Xu and Fredrik Kjolstad.

‎Include/internal/pycore_opcode_metadata.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_opcode_metadata.h
+2-2Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_uop_metadata.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_uop_metadata.h
+1-1Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/dis.py

Copy file name to clipboardExpand all lines: Lib/dis.py
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ def _get_code_array(co, adaptive):
216216
if op == ENTER_EXECUTOR:
217217
try:
218218
ex = get_executor(co, i)
219-
except ValueError:
219+
except (ValueError, RuntimeError):
220220
ex = None
221221

222222
if ex:

‎Lib/test/support/__init__.py

Copy file name to clipboardExpand all lines: Lib/test/support/__init__.py
+4-4Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2539,17 +2539,17 @@ def exceeds_recursion_limit():
25392539
# Decorator to disable optimizer while a function run
25402540
def without_optimizer(func):
25412541
try:
2542-
import _testinternalcapi
2542+
from _testinternalcapi import get_optimizer, set_optimizer
25432543
except ImportError:
25442544
return func
25452545
@functools.wraps(func)
25462546
def wrapper(*args, **kwargs):
2547-
save_opt = _testinternalcapi.get_optimizer()
2547+
save_opt = get_optimizer()
25482548
try:
2549-
_testinternalcapi.set_optimizer(None)
2549+
set_optimizer(None)
25502550
return func(*args, **kwargs)
25512551
finally:
2552-
_testinternalcapi.set_optimizer(save_opt)
2552+
set_optimizer(save_opt)
25532553
return wrapper
25542554

25552555

‎Lib/test/test_capi/test_opt.py

Copy file name to clipboardExpand all lines: Lib/test/test_capi/test_opt.py
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ def clear_executors(func):
3434

3535

3636
@requires_specialization
37+
@unittest.skipUnless(hasattr(_testinternalcapi, "get_optimizer"),
38+
"Requires optimizer infrastructure")
3739
class TestOptimizerAPI(unittest.TestCase):
3840

3941
def test_new_counter_optimizer_dealloc(self):
@@ -136,6 +138,8 @@ def get_opnames(ex):
136138

137139

138140
@requires_specialization
141+
@unittest.skipUnless(hasattr(_testinternalcapi, "get_optimizer"),
142+
"Requires optimizer infrastructure")
139143
class TestExecutorInvalidation(unittest.TestCase):
140144

141145
def setUp(self):
@@ -215,6 +219,8 @@ def f():
215219

216220

217221
@requires_specialization
222+
@unittest.skipUnless(hasattr(_testinternalcapi, "get_optimizer"),
223+
"Requires optimizer infrastructure")
218224
@unittest.skipIf(os.getenv("PYTHON_UOPS_OPTIMIZE") == "0", "Needs uop optimizer to run.")
219225
class TestUops(unittest.TestCase):
220226

@@ -579,6 +585,8 @@ def testfunc(n):
579585

580586

581587
@requires_specialization
588+
@unittest.skipUnless(hasattr(_testinternalcapi, "get_optimizer"),
589+
"Requires optimizer infrastructure")
582590
@unittest.skipIf(os.getenv("PYTHON_UOPS_OPTIMIZE") == "0", "Needs uop optimizer to run.")
583591
class TestUopsOptimization(unittest.TestCase):
584592

‎Lib/test/test_monitoring.py

Copy file name to clipboardExpand all lines: Lib/test/test_monitoring.py
+6-4Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1831,15 +1831,17 @@ class TestOptimizer(MonitoringTestBase, unittest.TestCase):
18311831

18321832
def setUp(self):
18331833
_testinternalcapi = import_module("_testinternalcapi")
1834-
self.old_opt = _testinternalcapi.get_optimizer()
1835-
opt = _testinternalcapi.new_counter_optimizer()
1836-
_testinternalcapi.set_optimizer(opt)
1834+
if hasattr(_testinternalcapi, "get_optimizer"):
1835+
self.old_opt = _testinternalcapi.get_optimizer()
1836+
opt = _testinternalcapi.new_counter_optimizer()
1837+
_testinternalcapi.set_optimizer(opt)
18371838
super(TestOptimizer, self).setUp()
18381839

18391840
def tearDown(self):
18401841
super(TestOptimizer, self).tearDown()
18411842
import _testinternalcapi
1842-
_testinternalcapi.set_optimizer(self.old_opt)
1843+
if hasattr(_testinternalcapi, "get_optimizer"):
1844+
_testinternalcapi.set_optimizer(self.old_opt)
18431845

18441846
def test_for_loop(self):
18451847
def test_func(x):

‎Lib/test/test_opcache.py

Copy file name to clipboardExpand all lines: Lib/test/test_opcache.py
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616

1717
def disabling_optimizer(func):
1818
def wrapper(*args, **kwargs):
19+
if not hasattr(_testinternalcapi, "get_optimizer"):
20+
return func(*args, **kwargs)
1921
old_opt = _testinternalcapi.get_optimizer()
2022
_testinternalcapi.set_optimizer(None)
2123
try:

‎Lib/test/test_optimizer.py

Copy file name to clipboardExpand all lines: Lib/test/test_optimizer.py
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,8 @@ def func(x=0):
8080

8181
class TestOptimizerSymbols(unittest.TestCase):
8282

83+
@unittest.skipUnless(hasattr(_testinternalcapi, "uop_symbols_test"),
84+
"requires _testinternalcapi.uop_symbols_test")
8385
def test_optimizer_symbols(self):
8486
_testinternalcapi.uop_symbols_test()
8587

‎Lib/test/test_weakref.py

Copy file name to clipboardExpand all lines: Lib/test/test_weakref.py
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
from test.support import gc_collect
1818
from test.support import import_helper
1919
from test.support import threading_helper
20+
from test.support import is_wasi, Py_DEBUG
2021

2122
# Used in ReferencesTestCase.test_ref_created_during_del() .
2223
ref_from_del = None
@@ -960,6 +961,7 @@ def test_hashing(self):
960961
self.assertEqual(hash(a), hash(42))
961962
self.assertRaises(TypeError, hash, b)
962963

964+
@unittest.skipIf(is_wasi and Py_DEBUG, "requires deep stack")
963965
def test_trashcan_16602(self):
964966
# Issue #16602: when a weakref's target was part of a long
965967
# deallocation chain, the trashcan mechanism could delay clearing

‎Misc/NEWS.d/next/Core and Builtins/2024-04-26-14-06-18.gh-issue-118335.SRFsxO.rst

Copy file name to clipboard
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Change how to use the tier 2 interpreter. Instead of running Python with
2+
``-X uops`` or setting the environment variable ``PYTHON_UOPS=1``, this
3+
choice is now made at build time by configuring with
4+
``--enable-experimental-jit=interpreter``.

‎Modules/_opcode.c

Copy file name to clipboardExpand all lines: Modules/_opcode.c
+6Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -367,7 +367,13 @@ _opcode_get_executor_impl(PyObject *module, PyObject *code, int offset)
367367
Py_TYPE(code)->tp_name);
368368
return NULL;
369369
}
370+
#ifdef _Py_TIER2
370371
return (PyObject *)PyUnstable_GetExecutor((PyCodeObject *)code, offset);
372+
#else
373+
PyErr_Format(PyExc_RuntimeError,
374+
"Executors are not available in this build");
375+
return NULL;
376+
#endif
371377
}
372378

373379
static PyMethodDef

‎Modules/_testinternalcapi.c

Copy file name to clipboardExpand all lines: Modules/_testinternalcapi.c
+12-1Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -985,6 +985,8 @@ get_co_framesize(PyObject *self, PyObject *arg)
985985
return PyLong_FromLong(code->co_framesize);
986986
}
987987

988+
#ifdef _Py_TIER2
989+
988990
static PyObject *
989991
new_counter_optimizer(PyObject *self, PyObject *arg)
990992
{
@@ -1012,7 +1014,10 @@ set_optimizer(PyObject *self, PyObject *opt)
10121014
static PyObject *
10131015
get_optimizer(PyObject *self, PyObject *Py_UNUSED(ignored))
10141016
{
1015-
PyObject *opt = (PyObject *)PyUnstable_GetOptimizer();
1017+
PyObject *opt = NULL;
1018+
#ifdef _Py_TIER2
1019+
opt = (PyObject *)PyUnstable_GetOptimizer();
1020+
#endif
10161021
if (opt == NULL) {
10171022
Py_RETURN_NONE;
10181023
}
@@ -1045,6 +1050,8 @@ invalidate_executors(PyObject *self, PyObject *obj)
10451050
Py_RETURN_NONE;
10461051
}
10471052

1053+
#endif
1054+
10481055
static int _pending_callback(void *arg)
10491056
{
10501057
/* we assume the argument is callable object to which we own a reference */
@@ -2020,12 +2027,14 @@ static PyMethodDef module_functions[] = {
20202027
{"iframe_getline", iframe_getline, METH_O, NULL},
20212028
{"iframe_getlasti", iframe_getlasti, METH_O, NULL},
20222029
{"get_co_framesize", get_co_framesize, METH_O, NULL},
2030+
#ifdef _Py_TIER2
20232031
{"get_optimizer", get_optimizer, METH_NOARGS, NULL},
20242032
{"set_optimizer", set_optimizer, METH_O, NULL},
20252033
{"new_counter_optimizer", new_counter_optimizer, METH_NOARGS, NULL},
20262034
{"new_uop_optimizer", new_uop_optimizer, METH_NOARGS, NULL},
20272035
{"add_executor_dependency", add_executor_dependency, METH_VARARGS, NULL},
20282036
{"invalidate_executors", invalidate_executors, METH_O, NULL},
2037+
#endif
20292038
{"pending_threadfunc", _PyCFunction_CAST(pending_threadfunc),
20302039
METH_VARARGS | METH_KEYWORDS},
20312040
{"pending_identify", pending_identify, METH_VARARGS, NULL},
@@ -2072,7 +2081,9 @@ static PyMethodDef module_functions[] = {
20722081
{"py_thread_id", get_py_thread_id, METH_NOARGS},
20732082
#endif
20742083
{"set_immortalize_deferred", set_immortalize_deferred, METH_VARARGS},
2084+
#ifdef _Py_TIER2
20752085
{"uop_symbols_test", _Py_uop_symbols_test, METH_NOARGS},
2086+
#endif
20762087
{NULL, NULL} /* sentinel */
20772088
};
20782089

‎Objects/codeobject.c

Copy file name to clipboardExpand all lines: Objects/codeobject.c
+6Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1496,6 +1496,8 @@ PyCode_GetFreevars(PyCodeObject *code)
14961496
return _PyCode_GetFreevars(code);
14971497
}
14981498

1499+
#ifdef _Py_TIER2
1500+
14991501
static void
15001502
clear_executors(PyCodeObject *co)
15011503
{
@@ -1515,6 +1517,8 @@ _PyCode_Clear_Executors(PyCodeObject *code)
15151517
clear_executors(code);
15161518
}
15171519

1520+
#endif
1521+
15181522
static void
15191523
deopt_code(PyCodeObject *code, _Py_CODEUNIT *instructions)
15201524
{
@@ -1739,9 +1743,11 @@ code_dealloc(PyCodeObject *co)
17391743

17401744
PyMem_Free(co_extra);
17411745
}
1746+
#ifdef _Py_TIER2
17421747
if (co->co_executors != NULL) {
17431748
clear_executors(co);
17441749
}
1750+
#endif
17451751

17461752
Py_XDECREF(co->co_consts);
17471753
Py_XDECREF(co->co_names);

‎Objects/object.c

Copy file name to clipboardExpand all lines: Objects/object.c
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2281,9 +2281,11 @@ static PyTypeObject* static_types[] = {
22812281
&_PyBufferWrapper_Type,
22822282
&_PyContextTokenMissing_Type,
22832283
&_PyCoroWrapper_Type,
2284+
#ifdef _Py_TIER2
22842285
&_PyCounterExecutor_Type,
22852286
&_PyCounterOptimizer_Type,
22862287
&_PyDefaultOptimizer_Type,
2288+
#endif
22872289
&_Py_GenericAliasIterType,
22882290
&_PyHamtItems_Type,
22892291
&_PyHamtKeys_Type,
@@ -2304,8 +2306,10 @@ static PyTypeObject* static_types[] = {
23042306
&_PyPositionsIterator,
23052307
&_PyUnicodeASCIIIter_Type,
23062308
&_PyUnion_Type,
2309+
#ifdef _Py_TIER2
23072310
&_PyUOpExecutor_Type,
23082311
&_PyUOpOptimizer_Type,
2312+
#endif
23092313
&_PyWeakref_CallableProxyType,
23102314
&_PyWeakref_ProxyType,
23112315
&_PyWeakref_RefType,

‎PCbuild/_testinternalcapi.vcxproj

Copy file name to clipboardExpand all lines: PCbuild/_testinternalcapi.vcxproj
+6Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,12 @@
108108
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
109109
</ProjectReference>
110110
</ItemGroup>
111+
<ItemDefinitionGroup>
112+
<ClCompile>
113+
<PreprocessorDefinitions Condition="'$(UseJIT)' == 'true'">_Py_JIT;%(PreprocessorDefinitions)</PreprocessorDefinitions>
114+
<PreprocessorDefinitions Condition="'$(UseTIER2)' != '0'">_Py_TIER2=$(UseTIER2);%(PreprocessorDefinitions)</PreprocessorDefinitions>
115+
</ClCompile>
116+
</ItemDefinitionGroup>
111117
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
112118
<ImportGroup Label="ExtensionTargets">
113119
</ImportGroup>

‎PCbuild/build.bat

Copy file name to clipboardExpand all lines: PCbuild/build.bat
+9-2Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,9 @@ echo. overrides -c and -d
3636
echo. --disable-gil Enable experimental support for running without the GIL.
3737
echo. --test-marker Enable the test marker within the build.
3838
echo. --regen Regenerate all opcodes, grammar and tokens.
39-
echo. --experimental-jit Enable the experimental just-in-time compiler.
39+
echo. --experimental-jit Enable the experimental just-in-time compiler.
40+
echo. --experimental-jit-off Ditto but off by default (PYTHON_JIT=1 enables).
41+
echo. --experimental-interpreter Enable the experimental Tier 2 interpreter.
4042
echo.
4143
echo.Available flags to avoid building certain modules.
4244
echo.These flags have no effect if '-e' is not given:
@@ -66,6 +68,7 @@ set verbose=/nologo /v:m /clp:summary
6668
set kill=
6769
set do_pgo=
6870
set pgo_job=-m test --pgo
71+
set UseTIER2=0
6972

7073
:CheckOpts
7174
if "%~1"=="-h" goto Usage
@@ -86,7 +89,10 @@ if "%~1"=="--disable-gil" (set UseDisableGil=true) & shift & goto CheckOpts
8689
if "%~1"=="--test-marker" (set UseTestMarker=true) & shift & goto CheckOpts
8790
if "%~1"=="-V" shift & goto Version
8891
if "%~1"=="--regen" (set Regen=true) & shift & goto CheckOpts
89-
if "%~1"=="--experimental-jit" (set UseJIT=true) & shift & goto CheckOpts
92+
if "%~1"=="--experimental-jit" (set UseJIT=true) & (set UseTIER2=1) & shift & goto CheckOpts
93+
if "%~1"=="--experimental-jit-off" (set UseJIT=true) & (set UseTIER2=3) & shift & goto CheckOpts
94+
if "%~1"=="--experimental-interpreter" (set UseTIER2=4) & shift & goto CheckOpts
95+
if "%~1"=="--experimental-interpreter-off" (set UseTIER2=6) & shift & goto CheckOpts
9096
rem These use the actual property names used by MSBuild. We could just let
9197
rem them in through the environment, but we specify them on the command line
9298
rem anyway for visibility so set defaults after this
@@ -179,6 +185,7 @@ echo on
179185
/p:DisableGil=%UseDisableGil%^
180186
/p:UseTestMarker=%UseTestMarker% %GITProperty%^
181187
/p:UseJIT=%UseJIT%^
188+
/p:UseTIER2=%UseTIER2%^
182189
%1 %2 %3 %4 %5 %6 %7 %8 %9
183190

184191
@echo off

‎PCbuild/pythoncore.vcxproj

Copy file name to clipboardExpand all lines: PCbuild/pythoncore.vcxproj
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@
105105
<PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_BUILD_CORE_BUILTIN;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
106106
<PreprocessorDefinitions Condition="$(IncludeExternals)">_Py_HAVE_ZLIB;%(PreprocessorDefinitions)</PreprocessorDefinitions>
107107
<PreprocessorDefinitions Condition="'$(UseJIT)' == 'true'">_Py_JIT;%(PreprocessorDefinitions)</PreprocessorDefinitions>
108+
<PreprocessorDefinitions Condition="'$(UseTIER2)' != '0'">_Py_TIER2=$(UseTIER2);%(PreprocessorDefinitions)</PreprocessorDefinitions>
108109
</ClCompile>
109110
<Link>
110111
<AdditionalDependencies>version.lib;ws2_32.lib;pathcch.lib;bcrypt.lib;%(AdditionalDependencies)</AdditionalDependencies>

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.