bpo-37788: Fix reference leak when Thread is never joined#26103
Merged
miss-islington merged 1 commit intoMay 14, 2021
python:mainpython/cpython:mainfrom
pitrou:bpo-37788-thread-join-leakpitrou/cpython:bpo-37788-thread-join-leakCopy head branch name to clipboard
Merged
bpo-37788: Fix reference leak when Thread is never joined#26103miss-islington merged 1 commit intopython:mainpython/cpython:mainfrom pitrou:bpo-37788-thread-join-leakpitrou/cpython:bpo-37788-thread-join-leakCopy head branch name to clipboard
miss-islington merged 1 commit into
python:mainpython/cpython:mainfrom
pitrou:bpo-37788-thread-join-leakpitrou/cpython:bpo-37788-thread-join-leakCopy head branch name to clipboard
Conversation
Member
Author
|
Note the unit test is originally from PR #25226. |
Member
Author
|
I started a buildbot run here: https://buildbot.python.org/all/#/changes/4460 (edit: previous run was on a bogus changeset) |
Member
Author
|
I also ran @vstinner 's regression test and it seems to work fine here: |
When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.
dc79edc to
16edc6a
Compare
shihai1991
approved these changes
May 14, 2021
| # does not leak. Test written for reference leak checks. | ||
| def noop(): pass | ||
| with threading_helper.wait_threads_exit(): | ||
| threading.Thread(target=noop).start() |
Member
There was a problem hiding this comment.
Would you mind to add a test case that the target function isn't just only the pass
Member
Author
There was a problem hiding this comment.
Hmm, what do you mean exactly? I'm not sure I understand your suggestion.
Member
There was a problem hiding this comment.
Oh, sorry. I didn't explain it clearly. I suggest that the target of Thread could be more complex ;)
Something like:
def noop():
# we don't what code will run in the thread in fact.
time.sleep(random.random())
with threading_helper.wait_threads_exit():
threading.Thread(target=noop).start()
Member
Author
There was a problem hiding this comment.
But what would that change? What matters is what happens after the thread ends.
Member
There was a problem hiding this comment.
Hm, give me a second.
No matter how complex of the target of thread, the thread will finish in time in python level. So we don't adding a complex testcase in here is fine. right?
Member
Thanks for checking with the buildbots! Seems that the failures are unrelated to this, so I think we are good to go |
Member
Author
|
Ok, merging then. |
Contributor
|
Thanks @pitrou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.8, 3.9. |
Contributor
|
Sorry, @pitrou, I could not cleanly backport this to |
|
GH-26138 is a backport of this pull request to the 3.10 branch. |
Contributor
|
Sorry @pitrou, I had trouble checking out the |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2021
…6103) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
Member
|
Thanks for the fix @pitrou, it looks simple :-) |
pitrou
added a commit
to pitrou/cpython
that referenced
this pull request
May 15, 2021
…onGH-26103) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
miss-islington
added a commit
that referenced
this pull request
May 15, 2021
…26103) (GH-26138) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org>
miss-islington
pushed a commit
that referenced
this pull request
May 15, 2021
…6103) (GH-26142) When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.. (cherry picked from commit c10c2ec) Co-authored-by: Antoine Pitrou <antoine@python.org> Automerge-Triggered-By: GH:pitrou
|
|
Hey there! |
jpmckinney
added a commit
to open-contracting/kingfisher-process
that referenced
this pull request
Jan 27, 2022
jpmckinney
added a commit
to open-contracting/yapw
that referenced
this pull request
Jan 28, 2022
Note that the previous code was not thread-safe. A thread-safe version would have been:
for t in list(threads):
if not t.is_alive():
# A set is used for `threads`, because `set.discard()` is O(1) but `list.remove()` is O(n).
threads.discard(t)
# Python 3.8 and below has a memory leak. python/cpython#26103
t.join()
jpmckinney
added a commit
to open-contracting/kingfisher-process
that referenced
this pull request
Jan 28, 2022
…thon/cpython#26103" This reverts commit cd4a74a.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.
https://bugs.python.org/issue37788
Automerge-Triggered-By: GH:pitrou