Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

bpo-37788: Fix a reference leak if a thread is not joined#15228

Merged
vstinner merged 1 commit into
python:masterpython/cpython:masterfrom
vstinner:thread_join_leakCopy head branch name to clipboard
Aug 19, 2019
Merged

bpo-37788: Fix a reference leak if a thread is not joined#15228
vstinner merged 1 commit into
python:masterpython/cpython:masterfrom
vstinner:thread_join_leakCopy head branch name to clipboard

Conversation

@vstinner

@vstinner vstinner commented Aug 12, 2019 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Add threading.Thread.del() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.

https://bugs.python.org/issue37788

@vstinner
bpo-37788: Fix a reference leak if a thread is not joined
30dd4cf 
Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
@vstinner

vstinner commented Aug 12, 2019

Copy link
Copy Markdown
Member Author

@akruis proposed a different approach: his PR #15175 modify the sentinel callback to call something like:

with _shutdown_locks_lock:
    _shutdown_locks.discard(tstate_lock)

@vstinner vstinner mentioned this pull request Aug 12, 2019
Closed
@akruis

akruis commented Aug 13, 2019

Copy link
Copy Markdown

I have two points:

  1. Our fixes are semantically different: Probably the del approach allows for a shutdown race condition, if the thread raises an exception.
  2. Adding __del__ to Thread is surely an option for 3.8 and master, but is it also OK for 3.7.5?

@vstinner

vstinner commented Aug 19, 2019

Copy link
Copy Markdown
Member Author

@pitrou, @pablogsal, @methane, @serhiy-storchaka: my https://bugs.python.org/issue36402 fix introduced a memory leak when a thread is started but never joined explicitly. This PR adds a destructor to thread.Thread to fix the regression. What do you think? We need to fix 3.7, 3.8 and master branches, I would prefer to use the same fix in all branches.

@akruis proposed a different fix: PR #15175.

pitrou
pitrou approved these changes Aug 19, 2019
View reviewed changes

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks @vstinner !

@pitrou

pitrou commented Aug 19, 2019

Copy link
Copy Markdown
Member

@akruis proposed a different fix: PR #15175.

That fix is much more complicated. I prefer this PR.

@serhiy-storchaka

serhiy-storchaka commented Aug 19, 2019

Copy link
Copy Markdown
Member

__del__ makes an object non-collectable. Is it possible to use weakref.finalize() here?

@pitrou

pitrou commented Aug 19, 2019

Copy link
Copy Markdown
Member

del makes an object non-collectable

That hasn't been the case in a long time, except if you're talking about Python 2.7.

@serhiy-storchaka

serhiy-storchaka commented Aug 19, 2019

Copy link
Copy Markdown
Member

So we can now get rid of weakref.finalize() in the tempfile module?

@pitrou

pitrou commented Aug 19, 2019

Copy link
Copy Markdown
Member

Hmm... probably? I'll let you figure out :-)

@serhiy-storchaka

serhiy-storchaka commented Aug 19, 2019

Copy link
Copy Markdown
Member

That code was added after 2.7, it was not the first attempt to fix some issues with tempfile, and only it was successful.

@vstinner

vstinner commented Aug 19, 2019

Copy link
Copy Markdown
Member Author

del makes an object non-collectable

That's no longer the case since @pitrou's PEP 442 https://www.python.org/dev/peps/pep-0442/ (finalizers).

@vstinner vstinner merged commit d3dcc92 into python:master Aug 19, 2019
@vstinner vstinner deleted the thread_join_leak branch August 19, 2019 22:37
@miss-islington

miss-islington commented Aug 19, 2019

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8.
🐍🍒⛏🤖

@miss-islington

miss-islington commented Aug 19, 2019

Copy link
Copy Markdown
Contributor

I'm having trouble backporting to 3.8. Reason: 'Error 110 while writing to socket. Connection timed out.'. Please retry by removing and re-adding the needs backport to 3.8 label.

@bedevere-bot

bedevere-bot commented Aug 19, 2019

Copy link
Copy Markdown

GH-15336 is a backport of this pull request to the 3.7 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 19, 2019
@vstinner @miss-islington
bpo-37788: Fix a reference leak if a thread is not joined (pythonGH-1…
2d61971 
…5228)

Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
(cherry picked from commit d3dcc92)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
@miss-islington

miss-islington commented Aug 19, 2019

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 19, 2019
@vstinner @miss-islington
bpo-37788: Fix a reference leak if a thread is not joined (pythonGH-1…
24f2303 
…5228)

Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
(cherry picked from commit d3dcc92)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
@bedevere-bot

bedevere-bot commented Aug 19, 2019

Copy link
Copy Markdown

GH-15337 is a backport of this pull request to the 3.8 branch.

@vstinner

vstinner commented Aug 19, 2019

Copy link
Copy Markdown
Member Author

Oh, my change is wrong: it introduced a regression :-(

vstinner@apu$ ./python -m test -v test_threading -m test_threads_join_2 -F
(...)
0:00:00 load avg: 0.85 [  4] test_threading
test_threads_join_2 (test.test_threading.SubinterpThreadingTests) ... Fatal Python error: Py_EndInterpreter: not the last thread

vstinner added a commit that referenced this pull request Aug 19, 2019
@vstinner
Revert "bpo-37788: Fix a reference leak if a thread is not joined (GH…
453c352 
…-15228)"

This reverts commit d3dcc92.
@vstinner vstinner mentioned this pull request Aug 19, 2019
Merged
vstinner added a commit that referenced this pull request Aug 19, 2019
@vstinner
Revert "bpo-37788: Fix a reference leak if a thread is not joined (GH…
d11c2c6 
…-15228)" (GH-15338)

This reverts commit d3dcc92.
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@vstinner @lisroach
bpo-37788: Fix a reference leak if a thread is not joined (pythonGH-1…
d1942cd 
…5228)

Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@vstinner @lisroach
Revert "bpo-37788: Fix a reference leak if a thread is not joined (py…
c240c2f 
…thonGH-15228)" (pythonGH-15338)

This reverts commit d3dcc92.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@vstinner @DinoV
bpo-37788: Fix a reference leak if a thread is not joined (pythonGH-1…
80ca192 
…5228)

Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@vstinner @DinoV
Revert "bpo-37788: Fix a reference leak if a thread is not joined (py…
60aae42 
…thonGH-15228)" (pythonGH-15338)

This reverts commit d3dcc92.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
@vstinner @websurfer5
bpo-37788: Fix a reference leak if a thread is not joined (pythonGH-1…
3870c0b 
…5228)

Add threading.Thread.__del__() method to ensure that the thread state
lock is removed from the _shutdown_locks list when a thread
completes.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
@vstinner @websurfer5
Revert "bpo-37788: Fix a reference leak if a thread is not joined (py…
bdd8416 
…thonGH-15228)" (pythonGH-15338)

This reverts commit d3dcc92.
@harmon

harmon commented Sep 24, 2020 •
edited
Loading

Copy link
Copy Markdown

Any movement on fixing this leak when threads aren't "join"-ed? I couldn't upgrade to 3.7 because of this leak, and now this leak was backported to 3.6, so I can't upgrade past 3.6.9... (nevermind, I looked at the wrong commit)

Thanks for all your work on Python, I see you've been quick active in making it better.

@shihai1991 shihai1991 mentioned this pull request Apr 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitrou pitrou pitrou approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@vstinner @akruis @pitrou @serhiy-storchaka @miss-islington @bedevere-bot @harmon @the-knights-who-say-ni
Morty Proxy This is a proxified and sanitized view of the page, visit original site.