[FIXED] Extract stdout from long-running process

Issue

Disclaimer: I have seen many similar questions, but either they do not use asyncio, or they don't seem to do exactly what I do.

I am trying to get the output of a long-running command (it's actually a server that logs out to stdout) with the following:

    proc = await asyncio.create_subprocess_shell(
        "my_long_running_command",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        limit=10
    )

    stdout = ""
    try:
        stdout, stderr = await asyncio.wait_for(proc.communicate(), 2.0)
    except asyncio.exceptions.TimeoutError:
        pass
    print(stdout)

But I don't get anything. If I use ls instead of my_long_running_command, it works. The only difference I see is that ls returns, and not my command.

I am using wait_for because the documentation says:

the communicate() and wait() methods don’t have a timeout parameter: use the wait_for() function;

I tried with limit=10, hoping it would help with the buffering, and without it. It seems to have no effect at all.

Though I don't really understand how they differ, I tried both asyncio.create_subprocess_shell and asyncio.create_subprocess_exec without success.

Is there a way to extract stdout from the process before it returns?

Solution

Try to use await proc.stdout.readline() for reading from standard output in realtime. For example:

import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        "top",                        # <--- some long running task
        stdout=asyncio.subprocess.PIPE,
    )

    line = await proc.stdout.readline()
    while line:
        print(line.decode())
        line = await proc.stdout.readline()


asyncio.run(main())

---

EDIT: Using await proc.stdout.read():

import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        "watch", "-n", "2", "ls", "-alF", "/",
        stdout=asyncio.subprocess.PIPE,
    )

    char = await proc.stdout.read(1)
    while char:
        print(char.decode(), end='')
        char = await proc.stdout.read(1)


asyncio.run(main())

---

EDIT 2: When two above solutions don't work, try to turn-off the buffering in terminal:

stdbuf -o0 <your command here>

Also, running this Python script without buffering. (for example, using the -u python command line flag)

Answered By - Andrej Kesely