Overview

Over in faster-cpython/ideas we've been exploring the idea of generating the interpreter from a set of instruction definitions. This will eventually enable having multiple versions of the interpreter (e.g. with and without tracing enabled), and it will allow us to automatically combine instructions using a powerful notation (e.g. super(LOAD_FAST__LOAD_FAST) = LOAD_FAST + LOAD_FAST;). It will also let us auto-generate things like stack_effect().

We are planning to land at least an early version of this work in 3.12. We have a tentative grammar for instruction definition DSL -- which will undoubtedly undergo several iterations before we've settled. We have a first draft of the tooling ready for review (which currently reproduces the status quo) done.

Once the first version of the tooling has landed we expect to iterate quickly, using this issue an umbrella issue for our PRs to link to.

References

• Generating code for the ceval loop/switch faster-cpython/ideas#5
• Super-instructions faster-cpython/ideas#16
• Define opcodes in terms of smaller "micro-ops" faster-cpython/ideas#454
• Change the interpreter definition DSL to be legal C syntax faster-cpython/ideas#477
• Mechanics of interpreter generation from DSL faster-cpython/ideas#479
• Stack effect of jumping -- how to express in DSL faster-cpython/ideas#481

• PR: GH-98831: "Generate" the interpreter #98830

• PR: GH-98831: Remove redundant extract_cases.py script #99065

• PR: GH-98831: Implement super-instruction generation #99084

• PR: GH-98831: Add some macros definitions to bytecodes.c to reduce IDE warnings. #99093

• PR: GH-98831: Auto-generate PREDICTED() macro calls #99102

• PR: GH-98831: Add regen-cases to regen-all #99107

• PR: GH-98831: Simple input-output stack effects #99120

• PR: GH-98831: Remove all remaining DISPATCH() calls from bytecodes.c #99271

• PR: GH-98831: Implement basic cache effects #99313

• PR: GH-98831: Refactor generate_cases.py #99408

• PR: GH-98831: Add macro and op and their implementation to DSL #99495

• PR: GH-98831: Refactor and fix cases generator #99526

• PR: GH-98831: Support cache effects in super- and macro instructions #99601

• PR: GH-98831: Typed stack effects, and more instructions converted #99764

• PR: GH-98831: Get rid of super(), just use macro() #100095

• PR: GH-98831: Generate things in the input order #100123

• PR: GH-98831: Remove super-instruction definitions, use macro instructions instead #100124

• PR: GH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input #100205

• PR: GH-98831: Modernize a ton of simpler instructions #100545

• PR: GH-98831: Add some tests for generate_cases.py #100763

• PR: GH-98831: Update generate_cases.py: register inst, opcode_metadata.h #100735

• PR: GH-98831: Revert "GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)" #100777

• PR: gh-98831: Regenerate Python/opcode_metadata.h #100778

• PR: GH-98831: Refactor instr format code and change to enum #100895

• PR: GH-98831: Implement array support in cases generator #100912

• PR: GH-98831: Identify instructions that don't use oparg #100957

• PR: GH-98831: Move assorted macros from ceval.h to a new header #101116

• PR: gh-98831: register instructions have 0 pushes and pops #101163

• PR: gh-98831: rewrite CHECK_EG_MATCH opcode in the instruction definition DSL #101269

• PR: gh-98831: rewrite pattern matching opcodes in the instruction definition DSL #101287

• PR: GH-98831: Elaborate some cases_generator tests #101299

• PR: gh-98831: rewrite RAISE_VARARGS in the instruction definition DSL #101306

• PR: gh-98831: add variable stack effect support to cases generator #101309

• PR: gh-98831: Support conditional effects; use for LOAD_ATTR #101333

• PR: gh-98831: Fix two bugs in case generator #101349

• PR: gh-98831: Clean up and add cache size static_assert to macro #101442

• PR: gh-98831: rewrite GET_LEN, GET_ITER, BEFORE_WITH and a few simple opcodes in the instruction definition DSL #101443

• PR: gh-98831: rewrite BEFORE_ASYNC_WITH and END_ASYNC_FOR in the instruction definition DSL #101458

• PR: gh-98831: rewrite PUSH_EXC_INFO and conditional jumps in the instruction definition DSL #101481

• PR: gh-98831: Modernize the LOAD_ATTR family #101488

• PR: gh-98831: Modernize the LOAD_GLOBAL family #101502

• PR: gh-98831: rewrite RERAISE and CLEANUP_THROW in the instruction definition DSL #101511

• PR: gh-98831: Modernize CALL and family #101508

• PR: gh-98831: rewrite SEND, GET_YIELD_FROM_ITER, RETURN_GENERATOR in the instruction definition DSL #101516

• PR: gh-98831: rewrite MAKE_FUNCTION and BUILD_SLICE in the instruction definition DSL #101529

• PR: gh-98831: rewrite COPY and SWAP in the instruction definition DSL #101620

• PR: gh-98831: Modernize the FOR_ITER family of instructions #101626

• PR: gh-98831: Modernize CALL_FUNCTION_EX #101627

• PR: gh-98831: Modernize FORMAT_VALUE #101628

• PR: gh-98831: Move DSL documentation here from ideas repo #101629

• PR: gh-98831: rewrite UNPACK_EX, UNPACK_SEQUENCE, UNPACK_SEQUENCE_TWO_TUPLE in the instruction definition DSL  #101641

• PR: gh-98831: Finish the UNPACK_SEQUENCE family #101666

• PR: gh-98831: Use opcode metadata for stack_effect() #101704

• PR: gh-98831: Use DECREF_INPUTS() more #102409