[FIXED] How to deploy custom Docker containers as Jenkins slaves from a remote Kubernetes installation?

Issue

Utilizing Kubernetes Plugin: https://wiki.jenkins.io/display/JENKINS/Kubernetes+Plugin

Most examples I have read through describe deploying a Jenkins master via a Kubernetes installation in the same environment/machine. However for testing purposes I have a minikube environment running on a separate VM (i.e. not on the same machine the Jenkins master is running). I am connecting to the VM successfully after doing some port forwarding, setting up the certs as described in this blog and configuring the Kubernetes cloud. Also if I try to deploy the default JNLP container, that slave will be deployed, and my job will run in it successfully.

So now I would like to deploy 2 custom containers in a single pod that each run their own applications (which are basic web services using Flask), that will send an HTTP post from port 5001 on the frontend container to port 5002 backend container. I have mainly just attempted setup for this in the web UI in the Jenkins global config, I have not yet tried to setup the pod templates within a Jenkinsfile yet.

A little background is that I would like to be able to deploy a separate and isolated testing environment every time a new job is triggered. I want to have Kubernetes deploy a separate Pod that contains the frontend and backend containers and then runs tests on how they interact. I'm hoping deploying these pods as Jenkins slaves would give me the right isolated environments for each build to accomplish that.

Below are screenshots of the initial Kubernetes plugin configs I've tried. Would I need to expose ports 50000 and run the JNLP process in each container in order to connect to the master?

Solution

I found out that Jenkins deploys a JNLP slave by default to drive the JNLP connection in the Pod, so there will be 3 containers in my pod: the JNLP Slave, frontend, and backend. Info on that JNLP Image can be found here: https://github.com/jenkinsci/docker-jnlp-slave. The default configuration of this slave container doesn't do exactly what I want, but it does handle the connection to the Jenkins master. So I will likely modify it or overwrite it with a new container as mentioned in the doc to drive distribution of my tests to the frontend and backend containers.

Also watch out when setting the working dir. I had this pointing to where my application's python files were and they were getting overwritten. I'm assuming this is the Jenkins workspace directory and will overwrite anything already there.

I also removed the Commands for the container as my Docker image already defines some initial commands, so anything further isn't necessary for the frontend and backend containers.

Below are the working container configurations:

Answered By - Cheen