Using zmq.asyncio, getting NotImplementedError when creating a REQ socket

[daenglis]

Had a simple REQ-REP application working fine. I went to convert the python client code to asyncio and ran into a NotImplemented error.

I tried to find it. Does this mean that AbstractEventLoop virtual methods are getting called?

pyzmq 22.0.3
python 3.9
win10

Traceback

Traceback (most recent call last):
  File "..\minimal_example.py", line 33, in <module>
    asyncio.run(main())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "..\minimal_example.py", line 27, in main
    bl_1300 = ServerConnection(addr='127.0.0.1', port=9999)
  File "..\minimal_example.py", line 21, in __init__
    self._zmq_socket = _zmq_context.socket(zmq.REQ)
  File "..\venv\lib\site-packages\zmq\sugar\context.py", line 235, in socket
    s = self._socket_class(self, socket_type, **kwargs)
  File "..\venv\lib\site-packages\zmq\_future.py", line 150, in __init__
    self._init_io_state()
  File "..\venv\lib\site-packages\zmq\asyncio.py", line 51, in _init_io_state
    self.io_loop.add_reader(self._fd, lambda: self._handle_events(0, 0))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\asyncio\events.py", line 504, in add_reader
    raise NotImplementedError
NotImplementedError
Exception ignored in: <function Socket.__del__ at 0x000002ECFB11C280>
Traceback (most recent call last):
  File "..\venv\lib\site-packages\zmq\sugar\socket.py", line 66, in __del__
  File "..\venv\lib\site-packages\zmq\_future.py", line 168, in close
  File "..\venv\lib\site-packages\zmq\asyncio.py", line 58, in _clear_io_state
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\asyncio\events.py", line 507, in remove_reader
NotImplementedError: 

Process finished with exit code 1

minimal_example.py

#minmal_example.py
#Using zmq.asyncio, NotImplementedError when creating a REQ socket

import asyncio
import zmq
import zmq.asyncio

_zmq_context = None


class ServerConnection:
    """Represents a remote system, running a zmq server."""
    _zmq_socket = None

    def __init__(self, addr='127.0.0.1', port=9999):
        """Contructor. Supply the zmq address string, addr, to reach this endstation."""

        global _zmq_context

        # the first server object will create the global zmq context
        if not _zmq_context:
            _zmq_context = zmq.asyncio.Context()

        self._zmq_socket = _zmq_context.socket(zmq.REQ)

        # ^^^^^^^^^^^^^^ fails NotImplementedError


async def main():
    bl_1300 = ServerConnection(addr='127.0.0.1', port=9999)

    print(await bl_1300.request('server status', {}))


if __name__ == '__main__':
    asyncio.run(main())

[bestdani]

I ran into the same issue on Win10 using Python 3.8+ and the reason seems to be the default asyncio event loop policy changed for the Windows platform. Manually changing it to the supported one resolved the issue for me.

Therefore I just added:

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

after the asyncio imports (wrapped into this if block to keep things platform independent)

[daenglis]

pyzmq doesn't support the proactor event loop? OK. Thanks for the work-around.

[minrk]

No, pyzmq can't support proactor because asyncio decided to exclude FD support from their proactor implementation, and libzmq's FD support is (mostly) incompatible with IOCP. Tornado 6.1 has a decent workaround, but frustrating that it's still required (and there are some performance costs associated with using threads).

Until that's addressed, either in libzmq or in asyncio, libzmq (not really up to pyzmq) is incompatible with ProactorEventLoop, as far as I can tell. The only way for pyzmq to take responsibility for supporting proactor itself, would be to duplicate tornado's threads workaround to not actually use Proactor and use select in a separate thread to wake proactor in the main thread.

[Netzeband]

Please add the workaround to your examples and documentation.