[FIXED] How to start a subprocess using a specific Executor

Issue

I am creating a subprocess from my FastAPI application as follows:

proc = await asyncio.create_subprocess_shell(
    cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
)

I am using the asyncio.create_subprocess_shell module function for the purpose of capturing the program's stdout line by line.

How can I make it so that the process uses a specific executor? I tried this:

pool = ProcessPoolExecutor(max_workers=10)
loop = asyncio.get_running_loop()

task = partial(
    asyncio.create_subprocess_shell,
    cmd,
    stdout=asyncio.subprocess.PIPE,
    stderr=asyncio.subprocess.PIPE,
)

proc = await loop.run_in_executor(pool, task)

But it fails with this error:

TypeError: cannot pickle 'coroutine' object

Solution

The .create_subprocess_shell will give you direct, detailed, control over that single sub-process, including the capability of interacting with it via pipe.

A process executor pool, on the other hand, does a different thing altogether: its processes are a specialized type of process running Python code, and being able to import your project modules, and each of them will work as a server of sort, waiting for the .submit (or .map) methods in the coordinating process to pass then Python tasks in the form of callables + arguments. They are not a "general case sub-process" that can run any arbitrary program.

In this case, what you want is simply create several processes wrapped in tasks, with the create_subproces_shell in a container object (a set or list will do), and iterate over this container to get to them. Use some manually coded logic to limit the number of concurrent processes if you need - the logic built-in concurrent.futures.Executors won't apply for this use case.

Answered By - jsbueno