[FIXED] IPython detect automagics

Issue

If I write

ls *.txt

into a cell in an IPython notebook, then it correctly executes. However, if I try to transform the cell using TransformerManager().transform_cell, nothing happens, and I get invalid Python syntax:

>>> from IPython.core.inputtransformer2 import TransformerManager
>>> import ast
>>> TransformerManager().transform_cell('ls *.txt')
'ls *.txt\n'
>>> ast.parse('ls *.txt\n')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ignoring_gravity/miniconda3/envs/tmp/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    ls *.txt
        ^
SyntaxError: invalid syntax

Is there a way to transform automagics in a way that returns valid Python code? The equivalent code, without the automagic, would get transformed as follows:

>>> TransformerManager().transform_cell('!ls *.txt')
"get_ipython().system('ls *.txt')\n"

---

What I'm looking for is a way of detecting automagics without running the code

Solution

Automagics are a feature of a running kernel, not the syntax. For example, cd by itself is a valid automagic unless it's shadowed by a Python name, or for that matter if %automagic is disabled.

In [1]: cd
/home/wja

In [2]: cd = 'CD'

In [3]: cd
Out[3]: 'CD'

In [4]: del cd

In [5]: cd
/home/wja

In [6]: %automagic 0

Automagic is OFF, % prefix IS needed for line magics.

In [7]: cd
Traceback (most recent call last):
  File "<ipython-input-7-9c6465b4471e>", line 1, in <module>
    cd
NameError: name 'cd' is not defined

Under the hood, from what I understand, when a cell throws certain errors like SyntaxError or NameError, it's sent to the prefilters, and if it could be transformed into a magic, it's caught by the prefilter AutoMagicChecker and transformed. My understanding is mostly based on this comment on an IPython GitHub issue:

Input transformers are applied line-by-line, but prefilters are only applied when the code is run. So 'invalid' [an invalid line] triggers an attempt to execute, and then prefilters step in and may transform it into valid code.

_{-- Thomas Kluyver, 11 Jul 2015}

Now, if you do have a running kernel, you could use the prefilters, something like this:

In [1]: ip = get_ipython()  # The running kernel

In [3]: source = ip.prefilter('cd')  # Transform

In [4]: source
Out[4]: "get_ipython().run_line_magic('cd', '')"

In [5]: exec(source)  # Run, just to prove it works
/home/wja

Or, the long way:

In [2]: from IPython.core.splitinput import LineInfo

In [3]: line_info = LineInfo('cd')  # Parse

In [4]: ip = get_ipython()

In [5]: ip.prefilter_manager.checkers  # List of prefilters
Out[5]:
[<EmacsChecker(priority=100, enabled=False)>,
 <MacroChecker(priority=250, enabled=True)>,
 <IPyAutocallChecker(priority=300, enabled=True)>,
 <AssignmentChecker(priority=600, enabled=True)>,
 <AutoMagicChecker(priority=700, enabled=True)>,
 <PythonOpsChecker(priority=900, enabled=True)>,
 <AutocallChecker(priority=1000, enabled=True)>]

In [6]: for checker in ip.prefilter_manager.checkers:
   ...:     handler = checker.check(line_info)
   ...:     if handler:  # Find the first one that matches
   ...:         break
   ...:

In [7]: handler
Out[7]: <IPython.core.prefilter.MagicHandler at 0x7f01e8ccc7f0>

In [10]: handler.handle(line_info)  # Transform
Out[10]: "get_ipython().run_line_magic('cd', '')"

Answered By - wjandrea