[FIXED] Jupyter starting a kernel in a docker container?

Issue

I want to switch my notebook easily between different kernels. One use case is to quickly test a piece of code in tensorflow 2, 2.2, 2.3, and there are many similar use cases. However I prefer to define my environments as dockers these days, rather than as different (conda) environments.

Now I know that you can start jupyter in a container, but that it not what I want. I would like to just click Kernel > use kernel > TF 2.2 (docker), and let jupyter connect to a kernel running in this container.

Is something like that around? I have used livy to connect to remote spark kernels via ssh, so it feels like this should be possible.

Solution

Full disclosure: I'm the author of Dockernel.

By using Dockernel

Put the following in a file called Dockerfile, in a separate directory.

FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel
CMD python -m ipykernel_launcher -f $DOCKERNEL_CONNECTION_FILE

Then issue the following commands:

docker build --tag my-docker-image /path/to/the/dockerfile/dir
pip install dockernel
dockernel install my-docker-image

You should now see "my-docker-image" option when creating a new notebook in Jupyter.

Manually

It is possible to do this kind of thing without much additional implementation/tooling, it just requires a bit of manual work:

1. Use the following Dockerfile:

FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel

2. Build the image using docker build --tag my-docker-image .

3. Create a directory for your kernelspec, e.g. ~/.local/share/jupyter/kernels/docker_test (%APPDATA%\jupyter\kernels\docker_test on Windows)

4. Put the following kernelspec into kernel.json file in the directory you created (Windows users might need to change argv a bit)

{
 "argv": [
  "/usr/bin/docker",
  "run",
  "--network=host",
  "-v",
  "{connection_file}:/connection-spec",
  "my-docker-image",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "/connection-spec"
 ],
 "display_name": "docker-test",
 "language": "python"
}

Jupyter should now be able spin up a container using the docker image specified above.

Answered By - Błażej Michalik