[FIXED] Send To Spark Cell Magic Dataframe size configuration

Issue

https://github.com/jupyter-incubator/sparkmagic/blob/master/examples/Send%20local%20data%20to%20Spark.ipynb

I've looked at the document above on how to send from local to spark kernel.

However, I've encountered a problem where I have local pandas df fo 60,000 rows but when I try send_to_spark cell magic on this dataframe, only 2500 rows are sent. Is there anyway I can send the whole 60,000 rows without splitting up the df in local ?

I know for sending from spark to local, %%spark -o df -n 60000 will do the job.

Solution

Use %%send_to_spark -i df_sending -t df -n df_sending -m 60000

-i : file I'm sending

-t : type I'm sending

-n : variable name assigned to the file

-m : max number of rows that I will send

use %%help spark cell magic

Answered By - Joon