I’ve just released this open source build framework: https://gitlab.com/smcphee/build-flow-example. It is an example of a CI/CD pipeline for gitlab.com projects, it uses the .gitlab-ci.yaml CI/CD file to build projects on gitlab CI runners and deploy the resulting docker images to gitlab’s internal docker registry (other repositories are possible to configure).

It implements a preferred workflow for CI/CD that we (at my work) have developed over some time using a self-hosted gitlab. We developed this workflow using Makefiles. The goal of the new project was to take the concepts we use there and completely re-implement them from scratch, in my spare time (and thus freeing them of any work-related copyright claims), and all the while completely eliminating the need for a Makefile (personally, I don’t mind Makefiles but some people hate them). Instead this example uses pure .gitlab-ci.yml format.

It results in your production Docker images being created as repo:1.1.1, i.e. with semver tags. The version number generation for production releases is achieved by using the python library bumpversion. Gitlab really needs to add some type of native semver generation into its build pipeline, if it did, then half of this project would never have been necessary, however, until then, here it is.

caveats

• This isn’t a functional project that you can just import into your gitlab repo and set a couple of variables and forget about it.
• It is designed to show a set of techniques for CI/CD pipelines that you can borrow and adapt.
• It is opinionated.
  • Production containers are released from tags which are created off the master branch (so-called golden-master).
  • It expects that your non-prod code is driven from a development branch.
• It probably will not work well for NPM projects. NPM is a stupid release tool that expects its production software dependencies will be installed in the running production container at boot time (aka npm install). Who thought that was a great idea? Let yourself out, thanks.

features

There are a number of features to this CI/CD pipeline example.

1. The final artefact is a Docker container with the service inside it.
   1. This example doesn’t really care what that container has got inside it.
   2. For the sake of simplicity the example is running a Java application with the Corretto11 OpenJDK.
   3. Adapt as you see fit for your particular preferred technology.
2. Two protected branches: develop and master.
   1. The default branch is develop.
   2. Production builds occur when develop is merged to master (more on this in a minute).
   3. Any tag starting with v* is a version tag for a production release and also protected.
3. Environments are expected to be prd (production) and npr (non-production).
   1. prd runs the master branch artefact.
   2. npr runs the develop branch artefact.
   3. Additional npr-environments are created from feature branches.
   4. There’s no example here of an actual deployment of the run-time: that’s left as an exercise to the reader.
4. Developer workflow works as follows:
   1. Clone the develop branch to the local development environment.
   2. Create a ‘feature branch’ e.g. git checkout -b my-awesome-feature
   3. Make changes as desired and push the feature branch up to the gitlab repository.
   4. The software in the feature branch is pushed to the Docker repository as repo:my-awesome-feature-abcdef09 (where my-awesome-feature is the branch name and abcdef09 is the short git hash of the commit).
   5. Keep doing steps 4.3 and 4.4 until complete.
   6. Merge to develop. Typically, a repository Maintainer, or at least a different Developer, would do this, and review the code before accepting the merge request.
   7. The develop branch will create repo:develop-abcdef09 and repo:develop-latest tags for the Docker artefact.
5. Production release process works as follows:
   1. When ready for production (however you want to determine this, perhaps after automated tests pass) create a Merge Request to merge develop into master.
   2. When that’s ready (e.g. after review), accept the merge request.
6. The build on master does two things:
   1. Performs a light sanity check, e.g. mvn clean compile test, and;
   2. Creates a protected tag, which is vN.N.N where N.N.N is a semver compliant version number.
   3. After this, the master branch build is complete.
7. The build of the protected tag creates the production artefact:
   1. Builds and tests the code and assembles the code artefact (e.g. the jar file).
   2. Assembles the Docker image and tags and publishes it as repo:vN.N.N and repo:stable.
   3. Increments the semver patch number (using bumpversion) and pushes this change to the develop branch (note: the develop branch).
   4. The .bumpversion.cfg file controls what files bumpversion changes (itself included!). Currently it changes itself and .mvn/maven.config (which tells maven what version it is building).

Both tagging the release from master and pushing the updated semver into the develop branch from the tag are done with gitlab api calls. The neat thing about this is that you do not need to escalate git privileges inside the running docker builder image, which is what you’d have to do in order to perform a git push from inside the running build. What you do need to do is to create a protected, masked variable inside your build (or its parent group) called API_ACCESS_TOKEN. This token requires full API access to your gitlab project (you create this token in gitlab).

Currently the docker artefacts are published to the gitlab registry. In order to build the images and publish them I have used the Kaniko project image gcr.io/kaniko-project/executor:debug. This is all documented on gitlab as a sensible alternative to using docker-in-docker. The latter method is a frightful way to build docker images.

to use

Please clone and detach the repository and then modify to your heart’s content.

If you are using it on gitlab.com, and publishing the images back into the projects’ gitlab-hosted docker registry, you can use it almost unchanged. You will have to modify the process of compiling the software (it’s Java 11 and uses maven 3.6). You’ll have to modify the Dockerfile to suit your application.

In the gitlab-ci.yml you may have to modify the following lines:

1. line 11, 12, 13. This is only needed for maven. Even if you are using Java and maven you might want to change them.

2. line 14. If you want some branch other than develop to get the new version pushed into it after the production release container is created.

3. line 20. This caches the local maven repository between runs. You may want to cache something else, or nothing.

4. line 27 to 32. This specifies the custom container (based off the Amazon Linux Corretto11 container) that I use for compiling the software. If you’re not using Java, use some other container and alter these lines appropriately. It also specifies the jar files as build artefacts which are attached to each build run. If you not using Java you’ll want some other artefacts.

5. line 163

   echo "{ \"branch\":\"$PBRANCH\", \"commit_message\":\"$COMMITMESSAGE\", \"actions\":[ \
   { \"action\":\"update\", \"file_path\":\".bumpversion.cfg\", \"encoding\":\"base64\", \"content\":\"$CONTENT_BUMPVER\" }, \
   { \"action\":\"update\", \"file_path\":\".mvn/maven.config\", \"encoding\":\"base64\", \"content\":\"$CONTENT_MVNCFG\" } \
   ] }" > payload.log

   This line specifically creates a json payload which commits two files into the $PBRANCH branch (default is develop) of the git repository using the gitlab API. It’s currently the ugliest part of the entire pipeline (look at all that horrific quote escaping!) and the part I am most likely to change. If you’re not using Java you will definitely not require .mvn/maven.config.

6. Also look at line 161 and line 162 which gather the new content (and base64 encode it) of those two files respectively and put them into the environment variables $CONTENT_BUMPVER and $CONTENT_MVNCFG for inclusion in the API call.

7. The following lines invoke mvn to compile the Java:

   1. line 39,
   2. line 52,
   3. line 115,
   4. line 128.

license

This code is released with the GNU GENERAL PUBLIC LICENSE, Version 3 (GPL3) license. See the LICENSE.

issues

If you have issues or suggestions please raise an issue, or preferably submit a merge request.