[FIXED] Python subprocess not printing real time

Issue

test.py

import time

for x in range(0, 10, 1):

    print(x)
    time.sleep(1)

python test.py prints real time i.e a number every second

0
1
2
3
4
5
6
7
8
9

Now, I wanted to run test.py in another script called run.py via subprocess like below

run.py

import subprocess
from subprocess import PIPE, STDOUT

proc = subprocess.Popen(
    'python test.py',
    stdout=PIPE,
    stderr=STDOUT,
    shell=True,
    encoding="utf-8",
    errors="replace",
    universal_newlines=True,
    text=True,
    bufsize=1,
)

while (realtime_output := proc.stdout.readline()) != "" or proc.poll() is None:
    print(realtime_output.strip(), flush=True)

when I run python run.py, the output is not real time (no number being printed every second)

Strangely, if i modify the code in test.py to print(x, flush=True) then python run.py prints every second

Is there a way to have real time output via subprocess in run.py without modifying test.py print statement?

Solution

bufsize=1 is changing the input buffer in the parent process's proc.stdout file handle. It doesn't affect the child process's output buffer in any way. And when Python's stdout is connected to a pipe, rather than a terminal, it automatically goes to block buffering mode.

Change test.py to use print(x, flush=True) and it will flush the output buffer on each print, whether connected to terminal, file, or pipe, and your work to disable the buffering in the parent process will take effect as expected.

Alternatively, change the run of the child process to pass -u to python to make it unbuffered, e.g. 'python -u test.py'. That will disable output buffering entirely, removing the need for flush=True (but it would make programs that only need flush=True sometimes, but write out a lot without it, run much slower).

---

As a side-note: There's almost never a reason to use shell=True (the only time you can/should use it is if no components of the command line come from untrusted data sources, e.g. the user, and you need to use a shell built-in and/or shell metacharacters, and you're sure you're not using any such metacharacters by accident). Your child process would avoid an unnecessary layer of shell wrapping (that could introduce additional buffering complications), by changing 'python test.py' to ['python', 'test.py'] (['python', '-u', 'test.py'] for unbuffered output), and removing shell=True,, and everything should run more efficiently and more safely.

You can also drop universal_newlines=True,, as text=True means exactly the same thing; if you're only supporting versions of Python that allow text=True, then universal_newlines=True is useless to you.

Answered By - ShadowRanger