[FIXED] Provide remote shell for Python script

Issue

I want to create a convenient simple way to connect to my running Python script remotely (via file sockets, TCP or whatever) to get a remote interactive shell.

I thought that this would be easy via IPython or so. However, I didn't really found any good example. I tried to start IPython.embed_kernel(), but that is blocking. So I tried to run that in another thread but that had many strange side effects on the rest of my script and I don't want any side effects (no replacement of sys.stdout, sys.stderr, sys.excepthook or whatever) and it also didn't worked - I could not connect. I found this related bug report and this code snippet which suggest to use mock.patch('signal.signal') but that also didn't worked. Also, why do I need that - I also don't want IPython to register any signal handlers.

There are also hacks such as pyringe and my own pydbattach to attach to some running Python instance but they seem to be too hacky.

Maybe QdbRemotePythonDebugger can help me?

Solution

My current solution is to setup an IPython ZMQ kernel. I don't just use

IPython.embed_kernel()

because that has many side effects, such as messing around with sys.stdout, sys.stderr, sys.excepthook, signal.signal, etc and I don't want these side effects. Also, embed_kernel() is blocking and doesn't really work out-of-the-box in a separate thread (see here).

So, I came up with this code, which is far too complicated in my opinion. (That is why I created a feature request here.)

def initIPythonKernel():
  # You can remotely connect to this kernel. See the output on stdout.
  try:
    import IPython.kernel.zmq.ipkernel
    from IPython.kernel.zmq.ipkernel import Kernel
    from IPython.kernel.zmq.heartbeat import Heartbeat
    from IPython.kernel.zmq.session import Session
    from IPython.kernel import write_connection_file
    import zmq
    from zmq.eventloop import ioloop
    from zmq.eventloop.zmqstream import ZMQStream
    IPython.kernel.zmq.ipkernel.signal = lambda sig, f: None  # Overwrite.
  except ImportError, e:
    print "IPython import error, cannot start IPython kernel. %s" % e
    return
  import atexit
  import socket
  import logging
  import threading

  # Do in mainthread to avoid history sqlite DB errors at exit.
  # https://github.com/ipython/ipython/issues/680
  assert isinstance(threading.currentThread(), threading._MainThread)
  try:
    connection_file = "kernel-%s.json" % os.getpid()
    def cleanup_connection_file():
      try:
        os.remove(connection_file)
      except (IOError, OSError):
        pass
    atexit.register(cleanup_connection_file)

    logger = logging.Logger("IPython")
    logger.addHandler(logging.NullHandler())
    session = Session(username=u'kernel')

    context = zmq.Context.instance()
    ip = socket.gethostbyname(socket.gethostname())
    transport = "tcp"
    addr = "%s://%s" % (transport, ip)
    shell_socket = context.socket(zmq.ROUTER)
    shell_port = shell_socket.bind_to_random_port(addr)
    iopub_socket = context.socket(zmq.PUB)
    iopub_port = iopub_socket.bind_to_random_port(addr)
    control_socket = context.socket(zmq.ROUTER)
    control_port = control_socket.bind_to_random_port(addr)

    hb_ctx = zmq.Context()
    heartbeat = Heartbeat(hb_ctx, (transport, ip, 0))
    hb_port = heartbeat.port
    heartbeat.start()

    shell_stream = ZMQStream(shell_socket)
    control_stream = ZMQStream(control_socket)

    kernel = Kernel(session=session,
                    shell_streams=[shell_stream, control_stream],
                    iopub_socket=iopub_socket,
                    log=logger)

    write_connection_file(connection_file,
                          shell_port=shell_port, iopub_port=iopub_port, control_port=control_port, hb_port=hb_port,
                          ip=ip)

    print "To connect another client to this IPython kernel, use:", \
          "ipython console --existing %s" % connection_file
  except Exception, e:
    print "Exception while initializing IPython ZMQ kernel. %s" % e
    return

  def ipython_thread():
    kernel.start()
    try:
      ioloop.IOLoop.instance().start()
    except KeyboardInterrupt:
      pass

  thread = threading.Thread(target=ipython_thread, name="IPython kernel")
  thread.daemon = True
  thread.start()

Note that this code is outdated now. I have made a package here which should contain a more recent version, and which can be installed via pip.

---

Other alternatives to attach to running CPython process without having it prepared beforehand. Those usually use the OS debugging capabilities (or use gdb/lldb) to attach to the native CPython process and then inject some code or just analyze the native CPython thread stacks.

• pyringe
• pyrasite
• pystuck
• pdb-clone

Here are other alternatives where you prepare your Python script beforehand to listen on some (tcp/file) socket to provide an interface for remote debugging and/or just a Python shell / REPL.

• winpdb (cross platform) remote debugger
• PyCharm IDE remote debugger, doc
• PyDev IDE remote debugger
• Twisted Conch Manhole, official example, lothar.com example, lysator.liu.se example, related StackOverflow question, blog.futurefoundries.com (2013)
• very simple manhole, has also some overview over related projects
• ispyd
• Eric IDE
• Trepan (based on pydb)
• rpdb
• rconsole (part of rfoo)

Some overviews and collected code examples:

• (QGIS) Example code for PyDev, Winpdb, Eric
• Python Wiki: Python debugging tools, Python Wiki: Python debuggers

(This overview is from here.)

Answered By - Albert