[FIXED] Multi-thread support for python asyncio gRPC clients

Issue

I have an asyncio gRPC client that is used in a multithreaded environment. When multiple threads connect to the service via the client simultaneously, I see a stream of the following errors:

2021-01-27 09:33:56,937 ERROR [asyncio] [thread_0] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable
2021-01-27 09:33:56,937 ERROR [asyncio] [thread_1] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable

The requests appear to be completing successfully, however, the messages are flooding my logs and making me nervous!

In my tests, each thread creates its own channel and submits its own async requests. The errors occur regardless of the load on the service. The errors do not occur if the clients are running in different processes.

My setup:

• Python version: 3.8.6
• grpcio version: 1.35.0

Any insight is appreciated!

Solution

The gRPC AsyncIO uses a UDS to communicate between C extension and Python. From your log, there is a race condition for the fd access. The AsyncIO API supports multi-threading, but this looks like a new issue (it helps to create an issue on https://github.com/grpc/grpc/issues).

The fix of the race condition could be tricky, since AsyncIO uses AsyncIO locks which are not thread-safe. If we protect the fd with thread-safe locks, it might block the AsyncIO loop. Feel free to propose or contribute solutions.

AsyncIO performs best if you let all clients run on one single thread. The event loop will handle the execution of coroutines well without thread hops. If the goal is to saturate all computational power on the machine, as you mentioned, it's better to use multi-processing.

Link to basic gRPC AsyncIO example: https://github.com/grpc/grpc/blob/master/examples/python/helloworld/async_greeter_client.py

Answered By - Lidi Zheng