[FIXED] How to limit the resources on PySpark cluster for Jupyter Notebook?

Issue

I want to restrict the usage of resources for my PySpark code running on Jupyter Notebook. I tried

%%configure -f {'driverMemory': '1000M', "executorMemory": "2000M", "executorCores": 1, 'numExecutors': 10}

But it's throwing the following error:

UsageError: %%configure is a cell magic, but the cell body is empty.

What does this error mean and how do I resolve the issue?

Solution

You need to put the settings in a new line, otherwise Jupyter can't find anything in the cell. Also avoid mixing single and double quotes.

%%configure -f
{
    "driverMemory": "1000M",
    "executorMemory": "2000M",
    "executorCores": 1,
    "numExecutors": 10
}

Answered By - mck