green.zmq program freeze with multiprocessing

[orestis-z]

I have a python application which spans multiple workers as sub processes (multiprocessing.Process).
In order to communicate with my sub process workers I'm spanning a server thread in the main process which listenes to requests from the worker sub processes using pyzmq. The worker sub processes are being created at any point of time.

But using gevent and zmq.green I encounter a freeze.

I managed to trace down the problem with this minimal snippet:

from gevent import monkey
monkey.patch_all()

import sys
import logging
from threading import Thread
from multiprocessing import Process

import zmq.green as zmq


logging.basicConfig(level=logging.DEBUG)


def subprocess():
    with zmq.Context().socket(zmq.REQ) as socket:
        logging.debug("Socket connecting to 5556")
        socket.connect('tcp://0.0.0.0:5556')
        socket.send_string("Hello from child proc client")
        reply = socket.recv_string()
        print("Received reply: {}".format(reply))
    logging.debug("Socket disconnecting from 5556")


def target():
    with zmq.Context().socket(zmq.REP) as socket:
        logging.debug("Socket binding to 5556")
        socket.bind("tcp://*:5556")
        while True:
            msg = socket.recv_string()
            print("Received msg: {}".format(msg))
            socket.send_string("Hi from parent proc server")

if sys.argv[-1] == "1":
    # start server subprocess
    proc = Process(target=subprocess)
    proc.start()

    thread = Thread(target=target)
    thread.start()
else:
    thread = Thread(target=target)
    thread.start()

    # start server subprocess
    proc = Process(target=subprocess)
    proc.start()

When passing 1 as an argument in the end (python3 <script>.py 1) it passes (spanning first the process and then the server thread seems to work). But python3 <script>.py freezes the program execution.

Sidenote: If removing the monkey-patching and changing zmq.green to zmq it works too

Tested with Python 3.7.3, gevent==1.4.0 and pyzmq==18.1.0

I'd appreciate any hints to solve the freeze while keeping gevent

[orestis-z]

Could be related to this: gevent/gevent#1268
(Got BlockingIOError when using Pipes instead of 0MQ for IPC)

Possible solution: https://github.com/karellen/geventmp, but still very experimental at the time of writing

[orestis-z]

Replacing 0MQ with Pipes works for both tests after all for this example (with geventmp). So still a green.zmq-related problem after all:

from gevent import monkey
monkey.patch_all()

from multiprocessing import Process, Pipe
from threading import Thread
import sys


child_conn, parent_conn = Pipe()


def subprocess(pipe):
    pipe.send("Hello from child proc client")
    reply = pipe.recv()
    print("Received reply: {}".format(reply))


def target(pipe):
    while True:
        msg = pipe.recv()
        print("Received msg: {}".format(msg))
        pipe.send("Hi from parent proc server")


if sys.argv[-1] == "1":
    # start server subprocess
    proc = Process(target=subprocess, args=(child_conn,))
    proc.start()

    thread = Thread(target=target, args=(parent_conn,))
    thread.start()
else:
    thread = Thread(target=target, args=(parent_conn,))
    thread.start()

    # start server subprocess
    proc = Process(target=subprocess, args=(child_conn,))
    proc.start()

[wannabesrevenge]

I'm seeing something similar in this issue with asyncio: #1333