Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Creating multiprocessing.Queues operates strangely with Process's created in classes #123208

Copy link
Copy link
Open
@Moosems

Description

@Moosems
Issue body actions

Bug report

Bug description:

The following from the docs works:

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()

The following also works:

from multiprocessing import Process, Queue
from time import sleep

def f(q):
    pass

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    sleep(1)

This works just fine:

class Foo:
    def __init__(self, q) -> None:
        pass

if __name__ == '__main__':
    q = Queue()
    p = Process(target=Foo, args=(q,))
    p.start()
    sleep(1)

But the following does NOT work:

from multiprocessing import Process, Queue
from time import sleep

class Foo:
    def __init__(self, q) -> None:
        pass

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=Foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar()
    sleep(1)

Which now gives this traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

The following does work, though:

from multiprocessing import Process, Queue
from time import sleep

class Foo:
    def __init__(self, q) -> None:
        q.put("INIT COMPLETE")

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=Foo, args=(q,))
        p.start()
        q.get()

if __name__ == '__main__':
    Bar()
    sleep(1)

This should either be documented or fixed, preferably the latter. This behavior is strange and, quite frankly, unexpected. It took me quite a while to figure this out and fix my code but adding an init queue item is not ideal. Note that the following also breaks with the same error:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar()
    sleep(1)

Moving the Queue outside of the class (when calling it) also doesn't fix the issue:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self, q) -> None:
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar(Queue())
    sleep(1)

But the following works just fine:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self, q) -> None:
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    q = Queue()
    Bar(q)
    sleep(1)

This may be related to #116526 but I don't personally think this should be classified as the same issue. May be related to #23376 as well but this may still be a separate issue. Neither address this kind of behavior, however.

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.