[FIXED] Invoking pyspark with `ipython` specified results in jupyter notebook being launched

Issue

I prefer using the ipython enhanced REPL over the python and so am specifying as such in the pyspark command line:

Result: jupyter notebook launched (surprising):

 PYSPARK_DRIVER_PYTHON=ipython MASTER="local[*]" $SPARK_HOME/bin/pyspark

This approach was working for several years - but at the present it is causing jupyter notebook to be launched.

That result would make sense if the explicit command notebook were also present:

Result: jupyter notebook launched (expected):

 PYSPARK_DRIVER_PYTHON=ipython notebook MASTER="local[*]" $SPARK_HOME/bin/pyspark

Given that the notebook is not specified then why is the notebook launched instead of the REPL ?

Note: when using ipython alone on the command line via

Result: ipython REPL launched (expected):

    ipython

the REPL is launched (and not the jupyter notebook).

Solution

you might have set PYSPARK_DRIVER_PYTHON_OPTS variable to "notebook". this will cause to open jupyter notebook

Answered By - InAFlash