[FIXED] Glue PySpark kernel not showing in VS Code

Issue

I have a Window 10 VDI and have set-up AWS Glue Interactive session using this link. Python version set-up for this is Python 3.9.7

When I start the notebook from command prompt, I see the various kernels in the browser

I have VSCode ( updated to v1.77 ) and have installed the Python and Jupyter extensions as well and trying to set-up VSCode to use the Glue Interactive sessions using this .

In VSCode, I do not see Glue PySpark as kernel Option, though see Glue Spark. I have also added python path the kernel.json as described here

But I still do not see Glue PySpark as an option in VSCode. Am I missing any set-up ?

Solution

I was able to resolve my issue by adding the below information found at the link Pyspark Kernel not showing

{
  "argv": [
    "D:\\venvs3\\.venv\\Scripts\\python.exe", <-- Full path to python from venv 
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_pyspark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "PySpark",
  "language": "python"
}

I also uninstalled and reinstalled the set-up a few times as trial and error, but don't know if that made any difference. The only reason I am mentioning it here is - during that process I changed different versions for packages and also installing some of them as --user as I was getting permissions issues otherwise.

That may or may not have anything to do with why it started working, but the above update to kernel.json is definitely needed.

Answered By - adbdkb