4,941 questions
3
votes
3
answers
169
views
Calls to Numpy routines from within multiprocessing tasks are very slow
I am calling numpy.linalg.inv from within a multiprocessing task forked from the Python multiprocessing module. I compare the Numpy routine to my own inverse routine my_inv.
What I see is that the ...
- 1,670
1
vote
0
answers
82
views
Pickling error work-around when passing decorators to multiprocessing module
I use the "decorator" idea as way to parameterize functions, but ran into pickling errors when passing these functions as targets to the multiprocessing module. I found a work-around by ...
- 1,670
1
vote
0
answers
83
views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
- 459
0
votes
0
answers
35
views
flask-scoketio: emitting from an external process without a message queue
I am developing a project using flask, flask-socketio, and the Multiprocessing python libraries.
My site runs on localhost (127.0.0.1:5000).
The goal is to update the html template every second from ...
- 1
1
vote
0
answers
84
views
Parent process unexpectedly exits during debugging when child receives SIGTERM (works fine outside debugger)
I'm experiencing an issue where the parent Python process terminates unexpectedly when debugging, but only when a child process is sent SIGTERM. The same code works perfectly when run normally (...
- 35
3
votes
1
answer
132
views
Isolation of a custom multiprocessing manager and how to update internal state
I try to use a custom multiprocessing manager, mostly following the example from the docs. The main difference is that my class updates internal state. It looks like this:
class IdIndex:
def ...
- 15.7k
3
votes
1
answer
61
views
Python multiprocessing: Responsibility for terminating daemons
I am using pythons multiprocessing library to create some subprocesses. I implemented a way to terminate my child-processes gracefully.
To terminate them even if my main-process has a crash, I made ...
- 43
0
votes
1
answer
45
views
How to Get the Return Value of a Function in Python Multithreading [duplicate]
I’m writing a Python web crawler, and I’m using multithreading to execute a function. The function returns a link when it finishes.
Here’s the general structure of the function:
def download_link(...
- 11
2
votes
2
answers
76
views
How to use multiprocessing module with interdependent tasks?
The existing examples of the multiprocessing-module assume, the tasks are independent, but my situation is "hairier".
I need to process several hundred thousands of widgets. These are ...
- 4,312
0
votes
1
answer
31
views
Python: problem with multiprocessing.Pool and deleting files (WindowsError: [Error 32])
I'm writing code in Python 2.7 (bigger framework compatibility reasons) that does the following:
Takes an ID_something.py file, opens it, reads the lines from it, then closes the file.
Creates an ...
- 13
1
vote
0
answers
53
views
Multiprocessing pool with stop flag
I am trying to implement a stop flag that gracefully prevents new parallel jobs from being started.
Specifically, I am running a large number of simulations that each takes a few to many hours; in ...
- 468
0
votes
1
answer
81
views
How to concurrently remove lines from a file in Python?
I have a cluster of compute nodes, each node with many CPUs. I want them to execute commands located in a file, one command per line. The drive where the command file is located is mounted on all ...
- 2,842
2
votes
1
answer
56
views
Running a python script in the background, then passing an event to end it after a given amount of time
In short, I'm trying to run a python script from another python script but in the background, allowing the parent script to continue executing, and then to pass a CTRL_C_EVENT to the child script to ...
- 29
0
votes
1
answer
77
views
How should CPU time be computed for calculations parallelized with the multiprocessing module?
I am trying to measure the processing time, or CPU time, of a CPU-intensive computation that has been parallelized with multiprocessing. However, simply bookending the parallelization of the ...
- 437
1
vote
1
answer
84
views
What are the basic rules for propagating lists and dictionaries across processes using the Manager of the multiprocessing module?
I am trying to use the multiprocessing module to parallelize a CPU-intensive piece code over multiple cores. This module looks terrific in several respects, but when I try to pass lists and ...
- 437