[FIXED] show output from subprocess.Popen as if run in terminal

Issue

  subprocess.Popen('dstat --output stats.csv', shell=True, stdout=subprocess.PIPE).stdout.read()

I'm working with ipython and when I run the command above nothing is printed on the console. Is there a way to see the same output in the console window as I would see if I ran the command directly in a linux terminal?

Solution

After writing up the answer below, I remembered that there was a way to break out of IPython and run like a shell. Specifically, if you start your line with an exclamation mark (!), IPython will execute the command as if you were on the command-line.

For example, if I run !dstat --output stats.csv, I get the following:

I am still keeping around the custom code-based approach below, because it was interesting to write, but obviously, the built-in solution (i.e., prefixing commands with !) would likely be better for your usage.

---

Custom Solution Below This Point

How about this?

"""Simulate command-line execution."""
import os
import sys
import subprocess
import signal
from time import sleep


def local_command(command):
    """Simulate command execution as if on command-line."""
    env_to_pass = dict(os.environ)
    kwargs_for_popen = {
        'shell': True,
        'bufsize': 1,
        'stdin': sys.stdin.fileno(),
        'stdout': sys.stdout.fileno(),
        'stderr': sys.stderr.fileno(),
        'env': env_to_pass
    }
    # Determine which keyword we should use for putting the process I/O into
    # text-only mode.
    if sys.hexversion >= 0x3070000:
        # "text" was added as a keyword argument alias for "universal_newlines"
        # in Python 3.7, and "universal_newlines" is provided still only for
        # backwards compatibility. Let's do this right if we're going to do it.
        kwargs_for_popen['text'] = True
    else:
        # For systems with python before 3.7, use "universal_newlines"
        kwargs_for_popen['universal_newlines'] = True

    sp = subprocess.Popen(command, **kwargs_for_popen)
    while True:
        try:
            while sp.poll() is None:
                sleep(0.02)
        except KeyboardInterrupt:
            sp.send_signal(signal.SIGINT)
            sleep(0.02)
        if sp.poll() is not None:
            # Process has terminated.
            # Exit event loop.
            break
    # end while

    sp_stdout_data, sp_stderr_data = sp.communicate()
    print(sp_stdout_data)
    return sp.returncode

Output from IPython running Python 3.7.3 on Ubuntu:

The code isn't particularly pretty, but you can easily put that in its own module and then call the function. I am fairly pleased with the results inside of IPython. (Output feels fairly natural.)

Answered By - Spencer D