Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Implement PEP 788 #133110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 62 commits into
base: main
Choose a base branch
Loading
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
1828b9a
Implement interpreter state reference counting.
ZeroIntensity Apr 24, 2025
d0895f9
Add a test for interpreter reference counts.
ZeroIntensity Apr 24, 2025
5c3ee8d
Add thread state daemon-ness that doesn't work yet.
ZeroIntensity Apr 25, 2025
7b9ac59
Add untested implementation of non-daemon native threads.
ZeroIntensity Apr 25, 2025
0868c15
Add a test for PyThreadState_SetDaemon().
ZeroIntensity Apr 25, 2025
e9ea644
Add untested implementation of Ensure()/Release() that probably doesn…
ZeroIntensity Apr 25, 2025
0ebdca4
Change some comments.
ZeroIntensity Apr 25, 2025
40989de
Add a test that I'm sure doesn't work.
ZeroIntensity Apr 25, 2025
d501f35
Use the interpreter's reference count and native thread countdown as …
ZeroIntensity Apr 25, 2025
c5ec89c
Fix the countdown decrement.
ZeroIntensity Apr 26, 2025
4e1f599
Remove unused variable.
ZeroIntensity Apr 26, 2025
3127a3f
Test for PyGILState_Ensure()
ZeroIntensity Apr 26, 2025
82b5b9f
Fix the test for the new reference counting.
ZeroIntensity Apr 26, 2025
fda9886
Add PyInterpreterState_Lookup()
ZeroIntensity Apr 26, 2025
f7723c0
Fix a few bugs and add a test.
ZeroIntensity Apr 26, 2025
62e9549
Add a test for PyThreadState_Ensure() across interpreters.
ZeroIntensity Apr 27, 2025
bc60630
Remove an artifact from old approach.
ZeroIntensity Apr 28, 2025
9d8d526
Fix test from earlier semantics.
ZeroIntensity Apr 28, 2025
54b0ce0
Remove 'daemonness' as a property of a thread.
ZeroIntensity May 22, 2025
5955de6
Add strong interpreter reference functions.
ZeroIntensity May 22, 2025
92cf906
Implement weak references.
ZeroIntensity May 22, 2025
b0d0673
Fix some thread safety issues regarding interpreter deletion.
ZeroIntensity May 22, 2025
0a15beb
Merge from main branch.
ZeroIntensity May 22, 2025
16d79de
Implement new version of PyThreadState_Ensure() and PyThreadState_Rel…
ZeroIntensity May 22, 2025
c2bffcd
Use the new APIs in the tests.
ZeroIntensity May 22, 2025
911c6b5
Fix _testcapi.
ZeroIntensity May 22, 2025
b84fa90
Fix the ensure counter.
ZeroIntensity May 22, 2025
9ccf6bd
Add the test to test_embed.
ZeroIntensity May 22, 2025
481caf5
Allow the wait to be interrupted by CTRL+C.
ZeroIntensity May 22, 2025
71e1aec
Print the error before bailing out.
ZeroIntensity May 22, 2025
d661578
Updates for the new proposal.
ZeroIntensity May 28, 2025
4249c5d
Bikeshedding.
ZeroIntensity May 28, 2025
71f2fd7
Apply suggestions from code review
ZeroIntensity May 28, 2025
03ccb38
Fix failing build.
ZeroIntensity May 29, 2025
bca65fb
Rename parameter.
ZeroIntensity May 29, 2025
510ade1
Fix formatting.
ZeroIntensity May 29, 2025
3971408
Add tstate check.
ZeroIntensity May 29, 2025
6c4c52b
Move to pycore_pystate.h
ZeroIntensity May 29, 2025
64920a8
Revert "Move to pycore_pystate.h"
ZeroIntensity May 29, 2025
05436f3
Use an exponential wait time for the event.
ZeroIntensity May 29, 2025
02bc2d7
Mark function as static.
ZeroIntensity May 29, 2025
03fa2af
Update fatal error message.
ZeroIntensity May 29, 2025
b955d85
Remove incorrect assertion.
ZeroIntensity May 29, 2025
5236700
Add a comment regarding PyMem_RawMalloc()
ZeroIntensity May 29, 2025
a277130
Add a comment.
ZeroIntensity Jun 1, 2025
dac0c1a
Update Include/cpython/pystate.h
ZeroIntensity Jun 1, 2025
ab9e3b5
Merge branch 'pep-788-impl' of https://github.com/ZeroIntensity/cpyth…
ZeroIntensity Jun 1, 2025
79a1852
Move some tests around to prevent exposure of the private API.
ZeroIntensity Jun 1, 2025
47957b8
Move weakref test to internal C API.
ZeroIntensity Jun 1, 2025
771d7ed
Improve reference counting tests.
ZeroIntensity Jun 1, 2025
0c3c1c7
Remove dead function.
ZeroIntensity Jun 1, 2025
531928e
Add some more tests.
ZeroIntensity Jun 1, 2025
08a8af6
Remove unused variables.
ZeroIntensity Jun 1, 2025
02f93bc
Fix some thread state attachment problems.
ZeroIntensity Jun 1, 2025
d6c82bd
Only delete thread states created by PyThreadState_Ensure()
ZeroIntensity Jun 1, 2025
082fd69
Add a test for crossinterpreter ensures.
ZeroIntensity Jun 1, 2025
61f70ae
Add a test for weak interpreter references.
ZeroIntensity Jun 1, 2025
fa961e9
Fix concurrent shutdown races in PyGILState_Ensure().
ZeroIntensity Jun 1, 2025
ea1da77
Add a test for PyInterpreterRef_Main().
ZeroIntensity Jun 1, 2025
6f19384
Use PyErr_FormatUnraisable to show signals.
ZeroIntensity Jun 1, 2025
b702da2
Fix a re-entrancy deadlock.
ZeroIntensity Jun 1, 2025
40e7e68
Remove stupid IDE imports.
ZeroIntensity Jun 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review
Co-authored-by: Victor Stinner <vstinner@python.org>
  • Loading branch information
ZeroIntensity and vstinner authored May 28, 2025
commit 71f2fd7d46831209a31dc18b22f493ba58852e9d
6 changes: 3 additions & 3 deletions 6 Include/cpython/pystate.h
Original file line number Diff line number Diff line change
Expand Up @@ -279,11 +279,11 @@ PyAPI_FUNC(PyInterpreterState *) PyInterpreterRef_AsInterpreter(PyInterpreterRef
#define PyInterpreterRef_Close(ref) do { \
PyInterpreterRef_Close(ref); \
ref = 0; \
} while (0);
} while (0)

/* Weak interpreter references */

typedef struct _interpreter_weakref {
typedef struct _PyInterpreterWeakRef {
int64_t id;
Py_ssize_t refcount;
} _PyInterpreterWeakRef;
Expand All @@ -298,7 +298,7 @@ PyAPI_FUNC(void) PyInterpreterWeakRef_Close(PyInterpreterWeakRef wref);
#define PyInterpreterWeakRef_Close(ref) do { \
PyInterpreterWeakRef_Close(ref); \
ref = 0; \
} while (0);
} while (0)

// Exports for '_testcapi' shared extension
PyAPI_FUNC(Py_ssize_t) _PyInterpreterState_Refcount(PyInterpreterState *interp);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move it to the internal C API? (Include/internal/pycore_pystate.h).

Same remark for _PyInterpreterState_Incref().

Copy link
Member Author

@ZeroIntensity ZeroIntensity May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_testcapi doesn't allow importing from pycore headers. I only need these for those tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move tests to _testinternalcapi in this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we're not testing the internal C API, we're testing the public C API. We just use the internal C API to write those tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't add private functions (ex: _PyInterpreterState_Refcount()) to the public C API. So move it to the internal C API. Sadly, it means moving your tests to the internal C API tests, unless a test only uses public functions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split up the tests, so now the internal stuff is in _testinternalcapi and the internal functions are in the private headers.

Expand Down
4 changes: 3 additions & 1 deletion 4 Programs/_testembed.c
Original file line number Diff line number Diff line change
Expand Up @@ -2313,7 +2313,9 @@
return result;
}

const char *THREAD_CODE = "import time\n"
const char *THREAD_CODE = \
"import time\n"
...

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Ubuntu (free-threading) / build and test (ubuntu-24.04)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Ubuntu / build and test (ubuntu-24.04)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Ubuntu (bolt) / build and test (ubuntu-24.04)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Ubuntu (free-threading) / build and test (ubuntu-24.04-arm)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Cross build Linux

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Hypothesis tests on Ubuntu

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Ubuntu / build and test (ubuntu-24.04-arm)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Windows / Build and test (x64)

syntax error: missing ';' before '...' [D:\a\cpython\cpython\PCbuild\_testembed.vcxproj]

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / Build and test (x64)

syntax error: missing ';' before '...' [D:\a\cpython\cpython\PCbuild\_testembed.vcxproj]

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Address sanitizer (ubuntu-24.04)

expected ‘,’ or ‘;’ before ‘...’ token

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Windows / Build and test (arm64)

syntax error: missing ';' before '...' [C:\a\cpython\cpython\PCbuild\_testembed.vcxproj]

Check failure on line 2318 in Programs/_testembed.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / Build and test (arm64)

syntax error: missing ';' before '...' [C:\a\cpython\cpython\PCbuild\_testembed.vcxproj]
vstinner marked this conversation as resolved.
Show resolved Hide resolved
"time.sleep(0.2)\n"
"def fib(n):\n"
" if n <= 1:\n"
Expand Down
10 changes: 5 additions & 5 deletions 10 Python/pystate.c
Original file line number Diff line number Diff line change
Expand Up @@ -3141,8 +3141,7 @@ Py_ssize_t
_PyInterpreterState_Refcount(PyInterpreterState *interp)
{
assert(interp != NULL);
Py_ssize_t refcount = _Py_atomic_load_ssize_relaxed(&interp->threads.finalizing.countdown);
return refcount;
return _Py_atomic_load_ssize_relaxed(&interp->threads.finalizing.countdown);
}

int
Expand All @@ -3168,7 +3167,7 @@ ref_as_interp(PyInterpreterRef ref)
{
PyInterpreterState *interp = (PyInterpreterState *)ref;
if (interp == NULL) {
Py_FatalError("Got a null interpreter reference, likely due to use after close.");
Py_FatalError("Got a null interpreter reference, likely due to use after PyInterpreterRef_Close()");
}

return interp;
Expand Down Expand Up @@ -3235,7 +3234,7 @@ wref_handle_as_ptr(PyInterpreterWeakRef wref_handle)
{
_PyInterpreterWeakRef *wref = (_PyInterpreterWeakRef *)wref_handle;
if (wref == NULL) {
Py_FatalError("Got a null weak interpreter reference, likely due to use after close.");
Py_FatalError("Got a null weak interpreter reference, likely due to use after PyInterpreterWeakRef_Close()");
}

return wref;
Expand Down Expand Up @@ -3269,7 +3268,8 @@ try_acquire_strong_ref(PyInterpreterState *interp, PyInterpreterRef *strong_ptr)
/* Interpreter has already finished threads */
*strong_ptr = 0;
return -1;
} else {
}
else {
_Py_atomic_add_ssize(&finalizing->countdown, 1);
}
PyMutex_Unlock(mutex);
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.