Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

bpo-32596: Lazy import concurrent.futures.process and thread #5241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
methane merged 2 commits into from
Jan 20, 2018
Merged

bpo-32596: Lazy import concurrent.futures.process and thread #5241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0fa144f
bpo-32596: Lazy import concurrent.futures.process and thread
methane Jan 19, 2018
38d55ac
Add NEWS entry
methane Jan 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions 35 Lib/concurrent/futures/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,36 @@
Executor,
wait,
as_completed)
from concurrent.futures.process import ProcessPoolExecutor
from concurrent.futures.thread import ThreadPoolExecutor

__all__ = (
'FIRST_COMPLETED',
'FIRST_EXCEPTION',
'ALL_COMPLETED',
'CancelledError',
'TimeoutError',
'BrokenExecutor',
'Future',
'Executor',
'wait',
'as_completed',
'ProcessPoolExecutor',
'ThreadPoolExecutor',
)


def __dir__():
return __all__ + ('__author__', '__doc__')


def __getattr__(name):
global ProcessPoolExecutor, ThreadPoolExecutor

@asvetlov asvetlov Jan 19, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't follow why do you need globals here?

@methane methane Jan 19, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once ProcessPoolExecutor is accessed, it's assigned to module global.
Next time, __getattr__ is not called, because futures has ProcessPoolExecutor now.

@methane methane Jan 19, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

>>> import concurrent.futures
>>> concurrent.futures.__dict__.keys()
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__path__', '__file__', '__cached__', '__builtins__', '__author__', '_base', 'FIRST_COMPLETED', 'FIRST_EXCEPTION', 'ALL_COMPLETED', 'CancelledError', 'TimeoutError', 'BrokenExecutor', 'Future', 'Executor', 'wait', 'as_completed', '__all__', '__dir__', '__getattr__'])
>>> from concurrent.futures import *
>>> concurrent.futures.__dict__.keys()
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__path__', '__file__', '__cached__', '__builtins__', '__author__', '_base', 'FIRST_COMPLETED', 'FIRST_EXCEPTION', 'ALL_COMPLETED', 'CancelledError', 'TimeoutError', 'BrokenExecutor', 'Future', 'Executor', 'wait', 'as_completed', '__all__', '__dir__', '__getattr__', 'process', 'ProcessPoolExecutor', 'thread', 'ThreadPoolExecutor'])

@asvetlov asvetlov Jan 19, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat trick!

@ilevkivskyi ilevkivskyi Jan 20, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this pattern is explicitly discouraged and my not work in other implementations, see https://docs.python.org/3.7/reference/simple_stmts.html#the-global-statement

Names listed in a global statement must not be defined as formal parameters or in a for loop control target, class definition, function definition, import statement, or variable annotation.
CPython implementation detail: The current implementation does not enforce some of these restrictions, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program.

There is however no recommendations on what is the recommended pattern instead.

@methane methane Jan 25, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ilevkivskyi Thanks, I didn't know that.
I'll fix it before 3.7b1


if name == 'ProcessPoolExecutor':
from .process import ProcessPoolExecutor
return ProcessPoolExecutor

if name == 'ThreadPoolExecutor':

@asvetlov asvetlov Jan 19, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if/elif/else would be a little faster maybe?

@methane methane Jan 19, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dis shows exactly same bytecode for both.

@asvetlov asvetlov Jan 19, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

from .thread import ThreadPoolExecutor
return ThreadPoolExecutor

raise AttributeError(f"module {__name__} has no attribute {name}")
4 changes: 4 additions & 0 deletions 4 Misc/NEWS.d/next/Library/2018-01-19-19-57-45.bpo-32596.4aVIie.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
``concurrent.futures`` imports ``ThreadPoolExecutor`` and
``ProcessPoolExecutor`` lazily (using :pep:`562`).
It makes ``import asyncio`` about 15% faster because asyncio
uses only ``ThreadPoolExecutor`` by default.
Morty Proxy This is a proxified and sanitized view of the page, visit original site.