[FIXED] What does CompileError/LinkerError: "command 'gcc' failed with exit status 1" mean, when running %%cython-magic cell in IPython

Issue

Sometimes, when I run a %%cython-cell in an IPython-notebook I get a long backtrace with ends with a quite short error message:

CompileError: command 'gcc' failed with exit status 1

or

LinkError: command 'gcc' failed with exit status 1

On Windows the corresponding messages are:

CompileError: command 'C:.\Microsoft Visual Studio\..\cl.exe' failed with exit status X

and

LinkError: command 'C:..\Microsoft Visual Studio\..\link.exe' failed with exit status YYYY

Is it possible to get more precise information about the underlying error?

An example of such a %%cython-cell follows:

[1] %load_ext Cython
[2] %%cython
    from sklearn.tree._tree cimport Node
    print("loaded")

Solution

The %%cython magic uses distutils to build the Cython-extension under the hood and IPython doesn't not capture the output gcc or other compilers/linkers log to standard error/output.

In order to see the errors and warnings logged by the compiler/linker, one must go to the place where the errors are logged to by the compiler, which depends on the way the IPython was started.

On Linux there is another possibility: to install wurlitzer package and activate it via %load_ext wurlitzer, which would also capture the output from gcc and show it in the notebook, see also this answer.

Sadly, wurlitzer works only on Linux, while the options bellow work for any OS.

IPython/Jupiter notebook:

When the notebook was started from the terminal, e.g. via ipython notebook or similar, then the compiler output can be seen in this terminal - we see that the problem with the above cell is:

/home/ed/.cache/ipython/cython/_cython_magic_5f6d267a6f541c572b4517a74d5c9aad.c:607:31: fatal error: numpy/arrayobject.h: No such file or directory compilation terminated.

missing numpy-headers, which can be found in numpy.get_include().

IPython-console

If IPython is started from a terminal, the errors are logged directly to the IPython console . But be aware: the compiler/linker outputs will come directly at the beginning of the error-trace:

 >>> ipython
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.                                                                       

In [1]: %load_ext Cython                                                        

In [2]: %%cython 
   ...: from sklearn.tree._tree cimport Node 
   ...: print("loaded") 
   ...:  
   ...: 
/home/ed/.cache/ipython/cython/_cython_magic_1182f410e5c0a56b03b28dd88700704d.c:607:31: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
....

CompileError: command 'gcc' failed with exit status 1

The first lines tell us everything we need to know!

Spyder:

At least since Spyder 3.3.3, the output of the compiler/linker is seen in the IPython console (the same as in a standalone IPython-console).

---

The example %%cython-cell can be fixed as follows:

%%cython -I <path from numpy.get_include()>
from sklearn.tree._tree cimport Node
print("loaded")

Answered By - ead