Continuous Integration

The primary goal of Continuous Integration (CI) is to enable multiple developers to work on the same code base while ensuring the quality of the final product.

This involves the following challenges:

  • No one person has complete knowledge of the entire system.

  • Multiple changes can be happening at the same time. Even if the changes are made in different components, it is possible for something to break when they are integrated.

Enabling Large Teams to Work on a System Simultaneously

An “integration server” (or “build server”) is a dedicated server (or VM) that prepares software for release. The server automates common tasks, including:

  • Building software binaries from source code (for compiled languages)

  • Running tests

  • Creating images, installers, or other artifacts

  • Deploying/installing the software

We are ultimately aiming for the following “Continuous Integration” work flow or process; this mirrors the process used by a number of teams working on “large” software systems, both in academia and industry:

  • Developers (i.e., you) check out code onto a machine where they will do their work. This could be a VM somewhere or their local laptop.

  • They make changes to the code to add a feature or fix a bug.

  • Once their work is done they add any additional tests as needed and then run all unit tests “locally” (i.e., on the same machine).

  • Assuming the tests pass, the develop commits their changes and pushes to the remote origin (in this case, GitHub).

  • A pre-established build server gets a message from the origin that a new commit was pushed.

  • The build server:

    • Checks out the latest version

    • Executes any build steps to create the software

    • Runs unit tests

    • Starts an instance of the system

    • Runs integration tests

    • Deploys the software to a staging environment

If any one of the steps above fails, the process stops. In such a situation, the code defect should be addressed as soon as possible.

GitHub-Docker Hub Integration

A quick-and-easy way to implement some parts of CI for our app is to use a handy GitHub-Docker Hub Integration.

Rather than commit to GitHub AND push to Docker Hub each time you want to release a new version of code, you can set up an integration between the two services that automates it. The key benefit is you only have to commit to one place (GitHub), and you know the image available on Docker Hub is always in sync.

To set up the integration, navigate to an existing Docker repository that you own in a web browser, which should be at an address similar to:

https://hub.docker.com/repository/docker/YOUR-DOCKER-USERNAME/pssp-app-api

Tip

You can click + New Repository if it doesn’t exist yet.

Click on Builds => Link to GitHub. (If this is your first time connecting a Docker repo to a GitHub repo, you will need to set it up. Press the ‘Connect’ link to the right of ‘GitHub’. If you are already signed in to both Docker and GitHub in the same browser, it takes about 15 seconds to set up).

Once you reach the Build Configurations screen, you will select your GitHub username and repository named, e.g., pssp-api.

Leaving all the defaults selected will cause this Docker image to rebuild every time you push code to the master branch of your GitHub repo. For this example, set the build to to trigger whenever a new release is tagged:

../_images/docker-git-integration.png

Click ‘Save and Build’ and check the ‘Timeline’ tab on Docker Hub to see if it is working as expected.

EXERCISE

In your own GitHub account, fork this repository:

https://github.com/wjallen/pssp-api

Create the same GitHub - Docker Hub integration shown above, write a new feature and tag a new release as described below to trigger a build.

Commit to GitHub

To trigger the build in a real-world scenario, make some changes to your source code, push your modified code to GitHub and tag the release as X.Y.Z (whatever new tag is appropriate) to trigger another automated build:

$ git add *
$ git commit -m "added a new route to delete jobs"
$ git push
$ git tag -a 0.1.4 -m "release version 0.1.4"
$ git push origin 0.1.4

By default, the git push command does not transfer tags, so we are explicitly telling git to push the tag we created (0.1.4) to the remote (origin).

Now, check the online GitHub repo to make sure your change / tag is there, and check the Docker Hub repo to see if your image is automatically rebuilding.

Deploy to Kubernetes

The final step in our example is to update the image tag in our deployment yaml files in both test and prod, and apply them all. Apply to test (staging) first as one final check that things are working as expected. Then, deploy to prod. Because the old containers are Running right up until the moment the new containers are deployed, there is virtually no disruption in service.

Note

Some CI / CD services can even handle the deployment to Kubernetes following Docker image builds and passing tests.