[FIXED] Running interactive command line code from Jupyter notebook

Issue

There is an interesting option in Ipython Jupyter Notebook to execute command line statements directly from the notebook. For example:

! mkdir ...
! python file.py

Moreover - this code can be run using os:

import os
os.system('cmd command')

but how do I run interactive shell commands. For example:

!conda install package

may require future input ([Y]/N) or folder location, but won't accept further input.

Solution

Assuming you are asking about interactivity, there is something you can try.

If you ever wondered how Jupyter knows when the output of a cell ends: well, it apparently does not know, it just dumps any captured output into the most recently active cell:

import threading,time
a=5
threading.Thread(target=lambda:[print(a),time.sleep(20),print(a)]).start()

(deliberately shorter-than-nice example, as this is just side-info. While the 20-second wait is running you have time to activate another cell, perhaps via issuing an a=6)

This can be used to get the output of some console code to the screen, while controlling it from the main thread:

import sys,threading,subprocess

proc=subprocess.Popen('/bin/sh',stdout=subprocess.PIPE,stdin=subprocess.PIPE,stderr=subprocess.STDOUT)
pout=proc.stdout
pin=proc.stdin

def outLoop():
    running=True
    while(running):
        line=pout.readline().decode(sys.stdout.encoding)
        print(line,end='')
        running='\n' in line
    print('Finished')

threading.Thread(target=outLoop).start()

Then you can isssue commands, like

pin.write(b'ls -l\n')
pin.flush()

and

pin.write(b'exit\n')
pin.flush()

Even b'ls\nexit\n' works, that is why outLoop is so long (a simple while(proc.poll() is None)-print(...) loop would finish sooner than it has grabbed all output.

Then the whole thing can be automated as:

while(proc.poll() is None):
    inp=bytearray(input('something: ')+'\n',sys.stdin.encoding)
    if(proc.poll() is None):
        pin.write(inp)
        pin.flush()

This works well on https://try.jupyter.org/, but obviously I did not want to try installing conda packages there, so I do not know what happens when conda asks a question.

A lucky thing is that the input field stays at the bottom of the cell (tested with ls;sleep 10;ls). An unlucky thing is that the input field needs an extra entry at the end to disappear (and that is already the 'nice' way, when it was a simple while(...)-write(bytearray(input()))-flush() 3-liner, it was exiting with an exception.

If someone wants to try this on Windows, it works with 'cmd', but I suggest using a hardcoded 'windows-1252' instead of sys.stdin/out.encoding: they say UTF-8, but a simple dir command already produces output which is neither UTF-8 nor ASCII (the non-breakable space between the 3-digit groups in sizes is a 0xA0 character). Or just remove the decode part (and use running=0xA in line)

Answered By - tevemadar