[FIXED] double percent spark sql in jupyter notebook

Issue

I'm using a Jupyter Notebook on a Spark EMR cluster, want to learn more about a certain command but I don't know what the right technology stack is to search. Is that Spark? Python? Jupyter special syntax? Pyspark?

When I try to google it, I get only a couple results and none of them actually include the content I quoted. It's like it ignores the %%.

What does "%%spark_sql" do, what does it originate from, and what are arguments you can pass to it like -s and -n?

An example might look like

%%spark_sql -s true
select 
*
from df

Solution

These are called magic commands/functions. Try running %pinfo %%spark_sql or %pinfo2 %%spark_sqlin a Jupyter cell and see if it gives you a detailed information about %%spark_sql.

Answered By - AdibP