[FIXED] Can I use asyncio to read from and write to a multiprocessing.Pipe?

Issue

I need to communicate between processes in Python and am using asyncio in each of the processes for concurrent network IO.

Currently I'm using multiprocessing.Pipe to send and recv significantly large amounts of data between the processes, however I do so outside of asyncio and I believe I'm spending a lot of cpu time in IO_WAIT because of it.

It seems like asyncio can and should be used to handle the Pipe IO between processes, however I can't find an example for anything but piping STDIN/STDOUT.

From what I read it seems like I should register the pipe with loop.connect_read_pipe(PROTOCOL_FACTORY, PIPE) and likewise for write. However I don't understand the purpose of protocol_factory as it would relate to a multiprocessing.Pipe. It's not even clear if I should be creating a multiprocessing.Pipe or whether I can create a pipe within asyncio.

Solution

aiopipe seems to do what you want! It can be used with the builtin multiprocessing module, and provides a similar API to the regular blocking pipes.

Answered By - kmaork