Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit f201628

Browse filesBrowse files
ericsnowcurrentlyericsnowcurrently
authored
gh-117953: Other Cleanups in the Extensions Machinery (gh-118206)
This change will make some later changes simpler.
1 parent f8290df commit f201628
Copy full SHA for f201628

File tree

Expand file treeCollapse file tree

3 files changed

+410
-102
lines changed
Filter options
Expand file treeCollapse file tree

3 files changed

+410
-102
lines changed

‎Lib/test/test_import/__init__.py

Copy file name to clipboardExpand all lines: Lib/test/test_import/__init__.py
+109-7Lines changed: 109 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2285,6 +2285,107 @@ def test_disallowed_reimport(self):
22852285

22862286

22872287
class TestSinglePhaseSnapshot(ModuleSnapshot):
2288+
"""A representation of a single-phase init module for testing.
2289+
2290+
Fields from ModuleSnapshot:
2291+
2292+
* id - id(mod)
2293+
* module - mod or a SimpleNamespace with __file__ & __spec__
2294+
* ns - a shallow copy of mod.__dict__
2295+
* ns_id - id(mod.__dict__)
2296+
* cached - sys.modules[name] (or None if not there or not snapshotable)
2297+
* cached_id - id(sys.modules[name]) (or None if not there)
2298+
2299+
Extra fields:
2300+
2301+
* summed - the result of calling "mod.sum(1, 2)"
2302+
* lookedup - the result of calling "mod.look_up_self()"
2303+
* lookedup_id - the object ID of self.lookedup
2304+
* state_initialized - the result of calling "mod.state_initialized()"
2305+
* init_count - (optional) the result of calling "mod.initialized_count()"
2306+
2307+
Overridden methods from ModuleSnapshot:
2308+
2309+
* from_module()
2310+
* parse()
2311+
2312+
Other methods from ModuleSnapshot:
2313+
2314+
* build_script()
2315+
* from_subinterp()
2316+
2317+
----
2318+
2319+
There are 5 modules in Modules/_testsinglephase.c:
2320+
2321+
* _testsinglephase
2322+
* has global state
2323+
* extra loads skip the init function, copy def.m_base.m_copy
2324+
* counts calls to init function
2325+
* _testsinglephase_basic_wrapper
2326+
* _testsinglephase by another name (and separate init function symbol)
2327+
* _testsinglephase_basic_copy
2328+
* same as _testsinglephase but with own def (and init func)
2329+
* _testsinglephase_with_reinit
2330+
* has no global or module state
2331+
* mod.state_initialized returns None
2332+
* an extra load in the main interpreter calls the cached init func
2333+
* an extra load in legacy subinterpreters does a full load
2334+
* _testsinglephase_with_state
2335+
* has module state
2336+
* an extra load in the main interpreter calls the cached init func
2337+
* an extra load in legacy subinterpreters does a full load
2338+
2339+
(See Modules/_testsinglephase.c for more info.)
2340+
2341+
For all those modules, the snapshot after the initial load (not in
2342+
the global extensions cache) would look like the following:
2343+
2344+
* initial load
2345+
* id: ID of nww module object
2346+
* ns: exactly what the module init put there
2347+
* ns_id: ID of new module's __dict__
2348+
* cached_id: same as self.id
2349+
* summed: 3 (never changes)
2350+
* lookedup_id: same as self.id
2351+
* state_initialized: a timestamp between the time of the load
2352+
and the time of the snapshot
2353+
* init_count: 1 (None for _testsinglephase_with_reinit)
2354+
2355+
For the other scenarios it varies.
2356+
2357+
For the _testsinglephase, _testsinglephase_basic_wrapper, and
2358+
_testsinglephase_basic_copy modules, the snapshot should look
2359+
like the following:
2360+
2361+
* reloaded
2362+
* id: no change
2363+
* ns: matches what the module init function put there,
2364+
including the IDs of all contained objects,
2365+
plus any extra attributes added before the reload
2366+
* ns_id: no change
2367+
* cached_id: no change
2368+
* lookedup_id: no change
2369+
* state_initialized: no change
2370+
* init_count: no change
2371+
* already loaded
2372+
* (same as initial load except for ns and state_initialized)
2373+
* ns: matches the initial load, incl. IDs of contained objects
2374+
* state_initialized: no change from initial load
2375+
2376+
For _testsinglephase_with_reinit:
2377+
2378+
* reloaded: same as initial load (old module & ns is discarded)
2379+
* already loaded: same as initial load (old module & ns is discarded)
2380+
2381+
For _testsinglephase_with_state:
2382+
2383+
* reloaded
2384+
* (same as initial load (old module & ns is discarded),
2385+
except init_count)
2386+
* init_count: increase by 1
2387+
* already loaded: same as reloaded
2388+
"""
22882389

22892390
@classmethod
22902391
def from_module(cls, mod):
@@ -2901,17 +3002,18 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29013002
# * module's global state was initialized but cleared
29023003

29033004
# Start with an interpreter that gets destroyed right away.
2904-
base = self.import_in_subinterp(postscript='''
2905-
# Attrs set after loading are not in m_copy.
2906-
mod.spam = 'spam, spam, mash, spam, eggs, and spam'
2907-
''')
3005+
base = self.import_in_subinterp(
3006+
postscript='''
3007+
# Attrs set after loading are not in m_copy.
3008+
mod.spam = 'spam, spam, mash, spam, eggs, and spam'
3009+
''')
29083010
self.check_common(base)
29093011
self.check_fresh(base)
29103012

29113013
# At this point:
29123014
# * alive in 0 interpreters
29133015
# * module def in _PyRuntime.imports.extensions
2914-
# * mod init func ran again
3016+
# * mod init func ran for the first time (since reset)
29153017
# * m_copy is NULL (claered when the interpreter was destroyed)
29163018
# * module's global state was initialized, not reset
29173019

@@ -2923,7 +3025,7 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29233025
# At this point:
29243026
# * alive in 1 interpreter (interp1)
29253027
# * module def still in _PyRuntime.imports.extensions
2926-
# * mod init func ran again
3028+
# * mod init func ran for the second time (since reset)
29273029
# * m_copy was copied from interp1 (was NULL)
29283030
# * module's global state was updated, not reset
29293031

@@ -2935,7 +3037,7 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29353037
# At this point:
29363038
# * alive in 2 interpreters (interp1, interp2)
29373039
# * module def still in _PyRuntime.imports.extensions
2938-
# * mod init func ran again
3040+
# * mod init func did not run again
29393041
# * m_copy was copied from interp2 (was from interp1)
29403042
# * module's global state was updated, not reset
29413043

‎Modules/_testsinglephase.c

Copy file name to clipboardExpand all lines: Modules/_testsinglephase.c
+167-1Lines changed: 167 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,172 @@
11

22
/* Testing module for single-phase initialization of extension modules
3-
*/
3+
4+
This file contains 5 distinct modules, meaning each as its own name
5+
and its own init function (PyInit_...). The default import system will
6+
only find the one matching the filename: _testsinglephase. To load the
7+
others you must do so manually. For example:
8+
9+
```python
10+
name = '_testsinglephase_base_wrapper'
11+
filename = _testsinglephase.__file__
12+
loader = importlib.machinery.ExtensionFileLoader(name, filename)
13+
spec = importlib.util.spec_from_file_location(name, filename, loader=loader)
14+
mod = importlib._bootstrap._load(spec)
15+
```
16+
17+
Here are the 5 modules:
18+
19+
* _testsinglephase
20+
* def: _testsinglephase_basic,
21+
* m_name: "_testsinglephase"
22+
* m_size: -1
23+
* state
24+
* process-global
25+
* <int> initialized_count (default to -1; will never be 0)
26+
* <module_state> module (see module state below)
27+
* module state: no
28+
* initial __dict__: see common initial __dict__ below
29+
* init function
30+
1. create module
31+
2. clear <global>.module
32+
3. initialize <global>.module: see module state below
33+
4. initialize module: set initial __dict__
34+
5. increment <global>.initialized_count
35+
* functions
36+
* (3 common, see below)
37+
* initialized_count() - return <global>.module.initialized_count
38+
* import system
39+
* caches
40+
* global extensions cache: yes
41+
* def.m_base.m_copy: yes
42+
* def.m_base.m_init: no
43+
* per-interpreter cache: yes (all single-phase init modules)
44+
* load in main interpreter
45+
* initial (not already in global cache)
46+
1. get init function from shared object file
47+
2. run init function
48+
3. copy __dict__ into def.m_base.m_copy
49+
4. set entry in global cache
50+
5. set entry in per-interpreter cache
51+
6. set entry in sys.modules
52+
* reload (already in sys.modules)
53+
1. get def from global cache
54+
2. get module from sys.modules
55+
3. update module with contents of def.m_base.m_copy
56+
* already loaded in other interpreter (already in global cache)
57+
* same as reload, but create new module and update *it*
58+
* not in any sys.modules, still in global cache
59+
* same as already loaded
60+
* load in legacy (non-isolated) interpreter
61+
* same as main interpreter
62+
* unload: never (all single-phase init modules)
63+
* _testsinglephase_basic_wrapper
64+
* identical to _testsinglephase except module name
65+
* _testsinglephase_basic_copy
66+
* def: static local variable in init function
67+
* m_name: "_testsinglephase_basic_copy"
68+
* m_size: -1
69+
* state: same as _testsinglephase
70+
* init function: same as _testsinglephase
71+
* functions: same as _testsinglephase
72+
* import system: same as _testsinglephase
73+
* _testsinglephase_with_reinit
74+
* def: _testsinglephase_with_reinit,
75+
* m_name: "_testsinglephase_with_reinit"
76+
* m_size: 0
77+
* state
78+
* process-global state: no
79+
* module state: no
80+
* initial __dict__: see common initial __dict__ below
81+
* init function
82+
1. create module
83+
2. initialize temporary module state (local var): see module state below
84+
3. initialize module: set initial __dict__
85+
* functions: see common functions below
86+
* import system
87+
* caches
88+
* global extensions cache: only if loaded in main interpreter
89+
* def.m_base.m_copy: no
90+
* def.m_base.m_init: only if loaded in the main interpreter
91+
* per-interpreter cache: yes (all single-phase init modules)
92+
* load in main interpreter
93+
* initial (not already in global cache)
94+
* (same as _testsinglephase except step 3)
95+
1. get init function from shared object file
96+
2. run init function
97+
3. set def.m_base.m_init to the init function
98+
4. set entry in global cache
99+
5. set entry in per-interpreter cache
100+
6. set entry in sys.modules
101+
* reload (already in sys.modules)
102+
1. get def from global cache
103+
2. call def->m_base.m_init to get a new module object
104+
3. replace the existing module in sys.modules
105+
* already loaded in other interpreter (already in global cache)
106+
* same as reload (since will only be in cache for main interp)
107+
* not in any sys.modules, still in global cache
108+
* same as already loaded
109+
* load in legacy (non-isolated) interpreter
110+
* initial (not already in global cache)
111+
* (same as main interpreter except skip steps 3 & 4 there)
112+
1. get init function from shared object file
113+
2. run init function
114+
...
115+
5. set entry in per-interpreter cache
116+
6. set entry in sys.modules
117+
* reload (already in sys.modules)
118+
* same as initial (load from scratch)
119+
* already loaded in other interpreter (already in global cache)
120+
* same as initial (load from scratch)
121+
* not in any sys.modules, still in global cache
122+
* same as initial (load from scratch)
123+
* unload: never (all single-phase init modules)
124+
* _testsinglephase_with_state
125+
* def: _testsinglephase_with_state,
126+
* m_name: "_testsinglephase_with_state"
127+
* m_size: sizeof(module_state)
128+
* state
129+
* process-global: no
130+
* module state: see module state below
131+
* initial __dict__: see common initial __dict__ below
132+
* init function
133+
1. create module
134+
3. initialize module state: see module state below
135+
4. initialize module: set initial __dict__
136+
5. increment <global>.initialized_count
137+
* functions: see common functions below
138+
* import system: same as _testsinglephase_basic_copy
139+
140+
Module state:
141+
142+
* fields
143+
* <PyTime_t> initialized - when the module was first initialized
144+
* <PyObject> *error
145+
* <PyObject> *int_const
146+
* <PyObject> *str_const
147+
* initialization
148+
1. set state.initialized to the current time
149+
2. set state.error to a new exception class
150+
3. set state->int_const to int(1969)
151+
4. set state->str_const to "something different"
152+
153+
Common initial __dict__:
154+
155+
* error: state.error
156+
* int_const: state.int_const
157+
* str_const: state.str_const
158+
* _module_initialized: state.initialized
159+
160+
Common functions:
161+
162+
* look_up_self() - return the module from the per-interpreter "by-index" cache
163+
* sum() - return a + b
164+
* state_initialized() - return state->initialized (or None if m_size == 0)
165+
166+
See Python/import.c, especially the long comments, for more about
167+
single-phase init modules.
168+
*/
169+
4170
#ifndef Py_BUILD_CORE_BUILTIN
5171
# define Py_BUILD_CORE_MODULE 1
6172
#endif

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.