Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

GH-126491: GC: Mark objects reachable from roots before doing cycle collection #126502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Nov 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
2ec8d8a
GC experiment: mark almost all reachable objects before doing collect…
markshannon Nov 4, 2024
1fdf00e
Add stats for objects marked
markshannon Nov 4, 2024
5e813c5
Start with mark phase
markshannon Nov 4, 2024
8bd7606
Add stats for visits during marking
markshannon Nov 5, 2024
3513da2
Visit new frames before each increment
markshannon Nov 5, 2024
ab1faec
Redo stats
markshannon Nov 6, 2024
9e2d93c
Fix freezing and GC untracking
markshannon Nov 6, 2024
3c18fc8
Don't untrack dicts
markshannon Nov 6, 2024
94da963
Remove lazy dict tracking from no-gil build
markshannon Nov 6, 2024
659fd1e
Remove unused variable
markshannon Nov 6, 2024
4cfbc4f
Add news
markshannon Nov 6, 2024
8c92ca6
Fix use after free
markshannon Nov 6, 2024
12d7f7c
Attempt more careful fix of use-after-free
markshannon Nov 7, 2024
1f619d7
Typo
markshannon Nov 7, 2024
b55fe37
Fix use of uninitialized variable
markshannon Nov 7, 2024
73b7f52
Fix compiler warnings
markshannon Nov 7, 2024
33f6386
Tweak test
markshannon Nov 7, 2024
8574d00
Add section to internal docs
markshannon Nov 11, 2024
70007b0
Rephrase new docs
markshannon Nov 11, 2024
f043080
Use symbolic constant
markshannon Nov 13, 2024
db2e173
Update section on untracking
markshannon Nov 13, 2024
6a50c2f
Merge branch 'main' into mark-first-gc
markshannon Nov 14, 2024
b9467ec
Update docs
markshannon Nov 14, 2024
14ae8d7
A few more edits
markshannon Nov 14, 2024
3337512
Update comment
markshannon Nov 14, 2024
3ae87fa
Address doc review comments
markshannon Nov 14, 2024
a2d9e3e
Merge branch 'main' into mark-first-gc
markshannon Nov 15, 2024
1452378
Avoid repeated collection of the young gen
markshannon Nov 15, 2024
595b14c
Clearer calculation of work to do.
markshannon Nov 15, 2024
278059b
Make sure tuples are untracked and avoid quadratic time validation
markshannon Nov 15, 2024
f186b4a
Update InternalDocs/garbage_collector.md
markshannon Nov 18, 2024
5f6d04e
Remove unused variable
markshannon Nov 18, 2024
9cfb5f0
Tweak work to do calculation
markshannon Nov 18, 2024
c7683a4
Explain work to do calculation
markshannon Nov 18, 2024
170ea6d
Initialize field to prevent code analyzer warning.
markshannon Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update docs
  • Loading branch information
markshannon committed Nov 14, 2024
commit b9467ec138ca1eeac2b40165bcd4151ad7a227eb
60 changes: 37 additions & 23 deletions 60 InternalDocs/garbage_collector.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,29 +352,6 @@ follows these steps in order:
objects.


Optimization: marking
=====================

An object cannot be garbage if it can be reached.

To avoid performing the complex algorithm above on the whole heap, we first
mark all objects that can be reached from any frame stack or from global
objects like the modules or builtin classes.

This marking step does much less work per object, so reduces the time spent
performing garbage collection by at least half.

This mark phase marks all object that are transitively reachable from the
roots as follows:
* All objects directly referred by any builtin class, the `sys` module, the `builtins`
module or any frame stack are added to a working set of reachable objects.
* Until this working set is empty:
* Pop an object from the set and move it to the reachable set
* For each object directly reachable from that object:
* If it is not already reachable and it is a GC object, then move it to
the working set


Optimization: incremental collection
====================================

Expand Down Expand Up @@ -508,6 +485,43 @@ specifically in a generation by calling `gc.collect(generation=NUM)`.
```


Optimization: visiting reachable objects
========================================

An object cannot be garbage if it can be reached.

To avoid having to identify reference cycles across the whole heap, we can
reduce the amount of work done considerably by first moving most reachable objects
to the `visited` space. Empirically, most reachable objects can be reached from a
small set of global objects and local variables.
This step does much less work per object, so reduces the time spent
performing garbage collection by at least half.

> [!NOTE]
> Objects that are not determined to be reachable by this pass are not necessarily
> unreachable. We still need to perform the main algorithm to determine which objects
> are actually unreachable.

We use the same technique of forming a transitive closure as the incremental
collector does to find reachable objects, seeding the list with some global
objects and the current frame of each stack.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
objects and the current frame of each stack.
objects and the currently executing frames.

?


This mark phase moves all objects `visited` space, as follows:

1. All objects directly referred by any builtin class, the `sys` module, the `builtins`
module and all objects directly referred to from stack frames are added to a working
set of reachable objects.
2. Until this working set is empty:
1. Pop an object from the set and move it to the `visited` space
2. For each object directly reachable from that object:
* If it is not already in `visited` space and it is a GC object,
then move it to the working set

Before each increment of collection is performed, any stack frames that have been created
since the last increment are added to the working set and above algorithm is repeated,
starting from step 2.


Optimization: reusing fields to save memory
===========================================

Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.