Jenkins, Github, Docker and peace of mind
I've been using Docker a lot recently in my development environments. I find it very useful to create fast and disposable containers for testing purposes. The ability to create application environment without cluttering my desktop or server with unnecessary packages, daemons and libraries is what Docker provides and why I started liking it after all.
Docker lets me easily create an staging environment for an application I want to create and perform any tests I want on my own desktop, without the need to fire up a VPS somewhere or worry about dependencies. All I have to do, is to prepare a Dockerfile, build an image and fire up tests. Then, if everything works as expected, I can push changes to GitHub (or the image straight to a Docker Registry) for Docker Hub Automated Builds prepare a working image I can pull later wherever I want to deploy it.
This workflow, however, is a little tedious and cumbersome; after all, I have to perform all the steps by hand (or a script that would do it for me) and monitor everything on the go. Docker Hub automated builds lack feature of running tests before pushing a commit to the registry, so I had to rely on my own setup.
That's where Jenkins came to the rescue. Whenever I push a change to Git repository, Jenkins automatically attempts to build a new Docker container from it. If all tests pass, this image is pushed either to my private registry or Docker Hub. The workflow is now much simpler and very configurable. All I have to worry about is writing a Dockerfile and pushing changes to Git repository; Jenkins will do the rest.
How to prepare Jenkins to work with GutHub and perform Docker automated builds?
If you are starting from scratch, you have to make sure the machine you want to setup this workflow can run docker engine. In short, this means, your machine must support virtualization and run 64bit OS. More information on how to install docker, can be found in the official documentation.
After Docker is up and running, you should install Jenkins. I won't go into details here as well, as there are plenty resources on how to install Jenkins. If you cannot deploy Jenkins locally and need a good hosting solution, I can recommend Layershift Jelastic PaaS platform where you can easily deploy a Jenkins instance with just one click! Just sign up below for a free account and an automatic Jenkins deployment.
Before you can start firing Docker builds, you need to configure Jenkins to work with Github. Install Github Jenkins plugins from Jenkins > Manage Jenkins > Manage Plugins > Available. Search for GitHub plugin and install it:
This plugin will allow Github to fire a webhook to Jenkins to start a build job when a change is pushed to repository.
Now we can configure a build slave that is going to execute all the build jobs. You mac configure as many slaves as you want and your resources allow, the more you have, the more Docker builds can be performed at once. To create a slave node, head to Manage Jenkins > Manage Nodes > New Node. Provide a node name, e.g. docker-slave-1 and tick Dumb Slave.
On the next screen, provide:
- Remote root directory - it's directory where slave node will be created, e.g. your home directory;
docker- we will need it later;
- Launch method: choose 'Launch slave agents on Unix machines via SSH' - Host: the host where you want to deploy Jenkins slave
- Credentials: ssh key or username/password combination to access the deployment machine
- Under node properties if you are using your private docker registry, you may define environment variable with your registry name which can be later passed to build script to make things easier.
After this has been set up, we can head to Github to set up repository to work with our Jenkins. Choose a project with Dockerfile, open it's settings > Webhooks & Services. Under Services, search for Jenkins (GitHub plugin).
In Jenkins hook url field provide addres to your Jenkins server followed by /github-webhook/ endpoint:
Once added, we can go back to our Jenkins and configure a build.
On Jenkins main page, click 'create new jobs' (or New Item from right-hand side menu bar), give it a name, e.g. the same as Github repository and select Freestyle project.
On the project configuration page tick GitHub project, and paste your repository address. To make the builds run only on previously created build slave, we can make use of option Restrict where this project can be run and specify for the builds only to run on nodes that we tag as docker. This way, we do not have to add new nodes to build configuration if we decide to create additional nodes:
Scroll down, and under Source Code Management choose Git, provide your Git repository URL:
Under Build Triggers, we want to tell Jenkins to start a build when a change is pushed to GitHub as well as to periodically check for changes in repository, e.g. every 5 minutes:
Last part, is to specify the command to run after Jenkins pulls the repository. Here's where our Docker image would be built, tested and pushed to repository. Under Build, choose 'Execute shell' as build step and paste the following:
DOCKERTAG=`echo $GIT_BRANCH|cut -d'/' -f 2-` if [ $DOCKERTAG == "master" ] then docker build --pull=true -t $DOCKER_REGISTRY/$JOB_NAME:latest . docker run -i -rm $DOCKER_REGISTRY/$JOB_NAME:latest ./script/test docker push $DOCKER_REGISTRY/$JOB_NAME:latest else docker build --pull=true -t $DOCKER_REGISTRY/$JOB_NAME:$DOCKERTAG . docker run -i -rm $DOCKER_REGISTRY/$JOB_NAME:$DOCKERTAG ./script/test docker push $DOCKER_REGISTRY/$JOB_NAME:$DOCKERTAG fi
To explain, when creating builds from github, I decided not to use github tag master as my Docker tag, but rather latest, which seems to be used with Docker more often. I'm using a custom variable DOCKERTAG, that extracts branch name from Jenkins Github plugin variable GIT_BRANCH.
JOB_NAME is the name of our build job, in my case, rainbowstream.
The if statement takes care of running the build, tests and pushing the image to Registry with appropriate tag. If the git branch is master, the Docker tag applied will be latest. Otherwise, git branch name will be preserved.
Since I'm using a private Registry, I'm also using the previously mentioned DOCKER__REGISTRY variable I've set up on my build slave. If you want the image to be pushed to your Docker Hub private registry, simply omit the DOCKER__REGISTRY variable in the above script.
docker run -i -rm $DOCKER_REGISTRY/$JOB_NAME:$DOCKERTAG ./script/test
is responsible for running tests on the container.
Once job added, as soon as you push a change to your github repository, a build will start, tests run and image will be pushed to registry if everything goes well.
On the build page you will se current build status as well as build details you may review.
Setting up Jenkins automated builds for Docker add the missing feature of Docker Public Registry that I was looking for, improves my workflow and saves a lot of time!