Mastering Continuous Integration and Continuous Deployment With GitLab

In the last part we learned what GitLab is and how you can manage your git-repositories with it. This time, I will show you how to use the ‘GitLab CI Multi Runner’ to get continuous integration and continuous deployment to go.

We can start right away with the installation of the runner since we already configured GitLab locally in the last part. There are several ways to install it, you can check GitLab CI Multi Runner Repository for more information. We will run one “Multi Runner” based on Docker and another based on bash.

For this example I imported our ‘rails4ruby2example’ app to my local GitLab instance.

Table of Contents

• Register the GitLab Ci Multi Runner
  • Docker Runner
• Configuring .gitlab-ci.yml
  • Bash Runner

Register the GitLab Ci Multi Runner

Docker Runner

You will need the GitLab CI Coordinator URL, find it out at your project >> settings >> runner, there you can also find the registration token.

1. sudo apt-get install docker
   install Docker
2. curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash
   add GitLab’s official repository
3. apt-get install gitlab-ci-multi-runner
   install GitLab CI Multi Runner
4. sudo gitlab-ci-multi-runner register
   register your first runner.
5. Enter GitLab-CI Coordinator URL
6. Enter GitLab Token
7. Enter GitLab-CI description for the runner
8. ruby
   Enter tags for your runner, these are used to specify the runner which should build your commit later
9. docker
   Enter the executor
10. ruby:2.2
    Please enter the Docker Image (of which your runner should be built)
11. Press enter for all other questions

**!!! If you got green runners that do not build anything, please double check that you registered them with sudo in front of the command, otherwise it will not work. !!!
**
Refresh the “runner” page, you should see your new runner now.

Configuring .gitlab-ci.yml

With the ‘.gitlab-ci.yml’ you can configure the CI, but before we go into detail with our specific ‘.gitlab-ci.yml’ we will talk about some important key-words you can use inside your configuration.

Typically a build run in GitLab is separated into stages (default: build, test, deploy). These stages can be configured by you. The stages will run one after another. Inside every stage you define so-called ‘jobs’. A job has to contain the script tag, inside which you can configure actions which will run when the job is executed. Jobs that belong to the same stage will be executed simultaneously. Only when all jobs of a stage have passed their builds, the jobs of the next stage will be executed. That’s enough for the moment, I will explain the other options right when we will be about to use them. For further information browse the official Gitlab-CI Docs.

Before we prepare our project to use CI, let’s have a quick talk about what I have in mind.

We will have two repositories, one with a Dockerfile and one with an application. We will push some changes in the Dockerfile to our repository with the Dockerfile, we will then automatically build a Docker Image and push it to Docker Hub. After that, the application repository is triggered because something changed in the Dockerfile repository and the application in the second repository will run its tests. Afterwards, it’ll be deployed to anynines.

Bash Runner

So, let’s start with our Dockerfile repository. (I already added it to the local GitLab instance, if you got questions how to do that, please check out my introduction to GitLab.)

It is based on Docker’s official ruby Dockerfile I just changed the version from 2.2.4 to 2.2.2.

FROM buildpack-deps:jessie

ENV RUBY_MAJOR 2.2 ENV RUBY_VERSION 2.2.2 ENV RUBYGEMS_VERSION 2.5.1

skip installing gem documentation

RUN echo 'install: --no-document\nupdate: --no-document' >> "$HOME/.gemrc"

some of ruby's build scripts are written in ruby

we purge this later to make sure our final image uses what we just built

RUN apt-get update \ && apt-get install -y bison libgdbm-dev ruby \ && rm -rf /var/lib/apt/lists/* \ && mkdir -p /usr/src/ruby \ && curl -fSL -o ruby.tar.gz "https://cache.ruby-lang.org/pub/ruby/$RUBY_MAJOR/ruby-$RUBY_VERSION.tar.gz" \ && tar -xzf ruby.tar.gz -C /usr/src/ruby --strip-components=1 \ && rm ruby.tar.gz \ && cd /usr/src/ruby \ && autoconf \ && ./configure --disable-install-doc \ && make -j"$(nproc)" \ && make install \ && apt-get purge -y --auto-remove bison libgdbm-dev ruby \ && gem update --system $RUBYGEMS_VERSION \ && rm -r /usr/src/ruby

install things globally, for great justice

ENV GEM_HOME /usr/local/bundle ENV PATH $GEM_HOME/bin:$PATH

ENV BUNDLER_VERSION 1.11.2

RUN gem install bundler --version "$BUNDLER_VERSION" \ && bundle config --global path "$GEM_HOME" \ && bundle config --global bin "$GEM_HOME/bin" \ && bundle config --global silence_root_warning true

don't create ".bundle" in all our apps

ENV BUNDLE_APP_CONFIG $GEM_HOME

CMD [ "irb" ]

As soon as when we have this, we need another runner to get the Docker image built and pushed to Docker Hub. This time we need a ‘bash runner’. However, before we can start with registering the runner, we need to do some small preparations hence our runner can use Docker.

Please enter your Vagrant VM with ‘vagrant ssh’ (make sure you started it before with ‘vagrant up’) and enter

sudo usermod -aG docker gitlab-runner

this will add a user named gitlab-runner to Docker.

You can check if the command worked by entering this:

sudo -u gitlab-runner -H docker info

This should give you an output similar to this:

Containers: 0 Images: 60 Server Version: 1.9.1 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 64 Dirperm1 Supported: false Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.13.0-74-generic Operating System: Ubuntu 14.04.3 LTS CPUs: 1 Total Memory: 3.861 GiB Name: 192.168.33.10 ID: WTJ7:Q77O:T2Q4:FTE7:GHXF:J3NI:CWXF:IBQW:Y54X:MRML:4LHW:MJ5K

Let’s register our bash runner now:

1. sudo gitlab-ci-multi-runner register
2. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci):
   https://192.168.33.10/ci
   You should enter your own coordinator url here
3. Please enter the gitlab-ci token for this runner:
   -p_e6jCpKf7xAsQkd41Y
   Once again, please enter your own token here.
4. Please enter the gitlab-ci description for this runner:
   private-bash-runner
5. Please enter the gitlab-ci tags for this runner (comma separated):
   docker
6. Please enter the executor: docker-ssh, virtualbox, ssh, shell, parallels, docker:
   shell
7. Runner registered successfully. Feel free to start it, but if it’s running already the config should be automatically reloaded!

Browse to the runner’s overview page (Settings -> Runners), there you should see your new ‘private-bash-runner’. As you can see on the picture, you can add runners that are used in your other projects to any other project just by clicking ‘enable for this project’.

Next step is to tell the runner what to do by adding the .gitlab-ci.yml to our Dockerfile repository.

This is what our GitLab-CI config file is looking like at the moment:

stages:

• build

build_docker_image: stage: build script: - docker build -t bguttmannavtq/anynines_blog_Docker image_gitlab . - docker login -u $DOCKER_USER -e $DOCKER_MAIL -p $DOCKER_PASSWORD - docker push bguttmannavtq/anynines_blog_Docker image_gitlab:latest tags: - docker only: - master

For an overview of all docker commands you can use a GitLab CI Multi Runner please have a look at the GitLab Documentation.

GitLab got a very nice feature, which I haven’t introduced yet. In the ‘settings’ part of a project you can define ‘variables’, you can use this variables in your gitlab-ci.yml and the values will not show up in the build logs. This is a good way to hide credentials from the logs. These variables could be accessed by writing the name of the variable with a $ in front. Perhaps you already recognized that I am actually using this in the config above.

docker login -u $DOCKER_USER -e $DOCKER_MAIL -p $DOCKER_PASSWORD

There are some other ways to include variables into your gitlab-ci.yml and a list of predefined variables. For more information check out GitLab’s Documentation about variables.

After adding the variables, we are ready to let our runner run. Push the changes to the repository, lean back and watch the runner do its thing. Go back to your project, click ‘Builds’, there should be one build marked with ‘running’, click the build and have a look at what is happening.

Finally …

Now let’s prepare our project to use CI is the next step.

Create a ‘.gitlab-ci.yml’ in the root directory of your project folder.

We will start with the definition of the stages. As I mentioned at the beginning, stages divide the builds in different parts. Everything within one stage will be executed at the same time (if there are enough runners), if everything passed in one stage, the next one will be then executed and so on. If something fails, the following stage will not be entered.

stages:

• test_branches
• test_master
• deploy

As you can see, we will have three different stages, first for branch tests, second for master tests and the third will deploy our app.

Our app will use a postgres database, because of this we need to add a ‘postgres’ service. That is how we will add this service.

services:

• postgres

Very easy and simple, isn’t it? . The official documentation is the source to get more information about the different services and how to get them going.

Since I used a certain value for the database name, user name and the password, I will tell the postgres service which values it should use for that. I will do so by adding these lines:

variables: POSTGRES_DB: db/test POSTGRES_USER: testuser POSTGRES_PASSWORD: test

Following this, the interesting part starts. Adding jobs is up next. We will add several jobs to show you how this works. Different jobs will look quite similar so I will not explain every single one in detail.

branch_test:2.3: image: ruby:2.3 stage: test_branches script: - rake test only: - branches except: - master tags: - ruby allow_failure: true

First, we define the name of the job ‘branch_test:2.3’, this will show up in the build overview.

Secondly, we define the Docker image our runner will be build with. We will create 4 jobs, every job uses another ruby image (2.0 – 2.3). If you need just one image you can define the image right at the beginning of your gitlab-ci.yml with ‘image: ruby:2.2’. Check out Docker Hub for available Docker images.

The keyword ‘stage’ indicates to which stage our job belongs to (you also can use type instead of stage). You can find the commands which will be executed within the ‘script’ part. There you can write your sequence of commands, one by one with a ‘-’ in front, or you can put them into a script which will make it more handy to manage.

‘Only’ defines a list of git refs for which builds are created; on the contrary, ‘except’ defines a list of git refs for which builds are not created.

The ‘tags’ section tells GitLab CI which runner to use. Only if the tags match the tags of a runner, the runner will execute the build.

Last but not least, ‘allow_failure’ defines if the next stage will be entered even if the build fails. By default it is set to false, in such case you can just leave it out.

To get postgres working, we need a little trick as well. GitLab is using the host ‘postgres’ instead of a localhost or something like this. We can change the host: localhost to host: postgres in our ‘database.yml’ just before executing with a ‘sed’ command.

sed -i 's/host: localhost/host: postgres/g' config/database.yml

In order to ensure that this will be executed before every job, we can insert a ‘before_script’ part. We also install the ‘bundler’ gem and execute a ‘bundle install’ before every job.

before_script:

• "sed -i 's/host: localhost/host: postgres/g' config/database.yml"
• gem install bundler
• bundle install

What’s up next? The master_test job, which looks like this:

master_test:2.2.2: image: bguttmannavtq/anynines_blog_Docker image_gitlab stage: test_master script: - source ./scripts/master_script only: - master tags: - ruby

Three things are a little bit different from our branch tests. First, we are now in stage ‘test_master’ this stage will be executed after the ‘test_branches’ stage. Another, we use the Docker image I pushed to Docker Hub some minutes ago. Third, in the script part we use a script file. Its content is just ‘rake test’.

With that, continuous integration is done.

Next step on our list is adding continuous deployment.

deploy_to_anynines: stage: deploy script:

• gem install dpl
• dpl --provider=anynines --username=$ANYNINES_USER --password=$ANYNINES_PASS \ --organization=$ANYNINES_USER_ORGA --space=test --manifest=manifest.yml.v6 only:
• master

We use the gem ‘dpl’ to deploy our app to anynines. ‘Dpl’ is the gem Travis-CI uses to deploy apps. That’s means for us that all providers that are supported by Travis-CI are supported by GitLab-CI.
In the script part we first install the gem, then we use the gem with the different options required. Please notice that I am once again using variables for the credentials.

The whole gitlab-ci.yml looks like this

before_script:

• "sed -i 's/host: localhost/host: postgres/g' config/database.yml"
• gem install bundler
• bundle install

stages:

• test_branches
• test_master
• deploy

services:

• postgres

variables: POSTGRES_DB: db/test POSTGRES_USER: testuser POSTGRES_PASSWORD: test

branch_test:2.0: image: ruby:2.0 stage: test_branches script: - rake test only: - branches except: - master tags: - ruby

branch_test:2.1: image: ruby:2.1 stage: test_branches script: - rake test only: - branches except: - master tags: - ruby

branch_test:2.2: image: ruby:2.2 stage: test_branches script: - rake test only: - branches except: - master tags: - ruby

branch_test:2.3: image: ruby:2.3 stage: test_branches script: - rake test only: - branches except: - master tags: - ruby allow_failure: true

master_test:2.2.2: image: bguttmannavtq/anynines_blog_Docker image_gitlab stage: test_master script: - source ./scripts/master_script only: - master tags: - ruby

deploy_to_anynines: stage: deploy script: - gem install dpl - dpl --provider=anynines --username=$ANYNINES_USER --password=$ANYNINES_PASS \ --organization=$ANYNINES_USER_ORGA --manifest=manifest.yml.v6 --space=test only: - master

Before we commit and push our changes to our repository, I will show you one further setting you can change, if you wish to. We can define the concurrent builds a runner can execute. To edit this, we go back to your vagrant VM and enter

vim /etc/gitlab-runner/config.toml

This folder is only visible to you as root. There you can change the number of concurrent jobs in the first line. For me, I will choose 2.

To show you the whole pipeline, I will add a branch to the project and push everything to the repository.

As you can see, our single runner starts with stage test_branches and executes two jobs at once.

In GitLab you can automatically merge a ‘merge request’ if the build passes. Then, first the branches will get tested and after merging the master will be tested and deployed. Isn’t that really awesome?

The test that was allowed to fail, failed but every stage was executed and passed. What is more, deploying the app to anynines worked.

There is one last thing left. Let’s add a trigger to the gitlab-ci.yml of our Dockerfile so that every time it pushes a new version of the Docker image to Docker Hub, our application’s master will be tested automatically. To me, that sounds quite nice, let’s then add that feature.

In our sample application repository, go to ‘Settings->Triggers’ and press ‘Add Trigger’.

We need just two more steps to get the trigger working. Adding the token and the ref_name to the variables section of our Dockerfile repository is the first one.

Now add another stage, called ‘trigger’ to the .gitlab-ci.yml of our Dockerfile repo and add a job for this stage.

trigger_builds: stage: trigger script: - "curl -X POST -F token=$TOKEN -F ref=$REF_NAME https://192.168.33.10/api/v3/projects/2/trigger/builds"

Notice the ‘$TOKEN’ and the ‘$REF_NAME’ , these are the variables. That’s all, commit and push, and have a look at how the magic happens ;-)

The Dockerfile builds passed successfully, in the stage trigger the runner triggered the master branch of our sample application and if we look at its builds, they are already running with the tag ‘triggered’.

With this part working, we got our whole list done. There are some services you can add to GitLab as well, perhaps this is something for another article in the future.

To end this all up let me quickly explain how to update your GitLab Community Edition or the GitLab CI Multi Runner to a new version in your Vagrant VM. Just enter ‘sudo apt-get -y upgrade’.

Thanks for your attention. I hope you enjoyed this tutorial and you are now able to master CI and CD with GitLab. If you have some remarks or questions, just let me know. I hope to see you next time :)