Getting Started With Concourse CI and Docker – Part 3

Setting up a local Concourse CI and creating a Dockerfile pipeline.

Welcome to the third part of our series concerning Concourse CI and Docker.
In the first one you got a brief overview about Concourse CI in general and a very quick ‘getting started’ guide for Docker.
The second part was about writing an independent Dockerfile, building Docker images and pushing them to Docker Hub. 

Well, that’s enough of Docker for the moment, let’s get back to Concourse CI.

Table of Contents

• Setting up a local Concourse CI
• Introduction to tasks, jobs and resources
  • Jobs
  • Tasks
  • Resources
• Setting up our first pipeline

Setting up a local Concourse CI

We will use Vagrant to get our Concourse instance running, so please get Vagrant and install it.
So, let’s go. Open a terminal and create a new directory.

1. mkdir concourse
2. create a Vagrantfile with the concourse/lite pre-built box
vagrant init concourse/lite
3. setting up the VM ( I am using VirtualBox)
vagrant up —provider virtualbox

After that, you can access the web server via the IP 192.168.100.4:8080

On the main page you can download the Fly CLI for your operating system, please do so.

One small remark, if you’re on Linux or OS X, you will have to chmod +x the downloaded binary and put it in your $PATH.

As you might see, there are currently no pipelines configured, we will change this in a moment.

Just to make it clear, this is a very simple example without fancy Dockerfiles and so on, the goal is to make it clear how Concourse CI works in general, please keep this in mind.

For this part I made small changes to the Dockerfile of part two, I removed the install git command and the app will not be cloned and started now. You can get the new Dockerfile from Github.

Introduction to tasks, jobs and resources

Before we start with our pipeline, it’s good to know some facts about the base concepts of Concourse CI. The three core concepts are named tasks, jobs and resources, let’s take a brief look at each one.

Jobs

A job describes some actions to be performed, when dependent resources change or when they are triggered manually. You can define jobs that run unit tests whenever a new code is pushed to a specified repository.
Jobs are functions which automatically run when there are new inputs available. A job can depend on the outputs of another job, that is why we need to build a pipeline.

The actions that need to be performed are defined in a so-called Build Plan. In such a plan you can express everything, e. g.  running simple unit tests or execute a huge amount of tasks and aggregate the results.

Tasks

In the Concourse CI world the execution of a script in an isolated environment is called task. That means the script will be executed in a Docker Container with some parameters.
The execution of the script decides if the task succeeds or not, i.e. exit code 0 is success, otherwise it is fail.

You can use the “Fly” command line tool to execute a task manually or you can let a job do that for you. Both the job and “Fly” will execute the same configuration that will give you the guarantee that if you execute tasks locally with Fly, the tasks will run the same way as they would in your pipeline.

In general, tasks run as unprivileged users but you can let Fly or your job execute the task as root user. A task cannot configure itself to be run as root, please take a note of that fact.

Resources

In general, a resource in Concourse doesn’t have to be something special. There are just a few requirements to meet in order to be a resource in Concourse: it must be possible to check for updated versions of the entity, it should be possible to pull down some specific version and/or push up to create new versions. An example for such a resource that most of us may know would be a git repository.

The best way to get to know which resources you can use, is to take a look at the Concourse GitHub organization and search through..

Setting up our first pipeline

Well, enough of the boring (but important) basics, let’s get our first pipeline started.

First of all, we need to make some preparations before we can finally start to create our pipeline.

1. Upload your Dockerfile to a Github-Repo
2. Create a new repo at Docker Hub for your Docker Image

Now, we will start with our „pipeline.yml“, first thing we will do is to add a resource.

1. Add the resources keyword

resources:

1. Add a name for this resource

-name: anynines_blog_dockerfile

1. Add the type of your resource

type: git

1. Tell concourse where to find your resource

source:

1. Add the name of the Github repository where your Dockerfile is located

uri: https://github.com/benjaminguttmann-avtq/concourse_ci_blog.git
 or if you want to check out via ssh
 git@github.com:benjaminguttmann-avtq/concourse_ci_blog.git

1. Add the branch you want to look at

branch: master

1. Add the path to your Dockerfile in your repository

paths: [Dockerfile]

Please double-check the indentation of your file because it is a YAML file and it is very ruthless when it comes to unnecessary white spaces and things like that.

Your pipeline.yml should now look something like this:

resources:

• name: anynines_blog_dockerfile type: git source: uri: https://github.com/benjaminguttmann-avtq/concourse_ci_blog.git branch: master paths: [Dockerfile]

We want to push our Docker image to Docker Hub, after it was build for this reason we need to define another resource directly under the git resource.

1. Add a name for this resource

– name: anynines_blog_build_image

1. Add the type of your resource

type: docker-image

1. Tell concourse where to find your resource

source:

1. Add the name of the Docker Hub repository where your Docker Image is located

repository: bguttmannavtq/anynines-blog-build-image

1. Add the credentials for your Docker Hub account (we will save them into a credentials.yml, more on this later)

email: {{DOCKER_MAIL}}
username: {{DOCKER_USER}}
password: {{DOCKER_PASS}}

As I mentioned before, there are different types of resources in Concourse. The first resource we used was a git-resource, that is a git repository where you can get code from or push code to. The second resource is of docker-image type, that means we specify a repository at Docker Hub where we can get a Docker Image from or push a Docker Image to. If you want to know more about the resources just click on the types above and you will be redirected to the Github repositories.

This is what your pipeline.yml is now looking like:

resources:

• name: anynines_blog_dockerfile type: git source: uri: https://github.com/benjaminguttmann-avtq/concourse_ci_blog.git branch: master paths: [Dockerfile]
• name: anynines_blog_build_image type: docker-image source: repository: bguttmannavtq/anynines-blog-build-image email: {{docker_mail}} username: {{docker_user}} password: {{docker_pass}}

The next step is adding a job that builds our Docker Image from our Dockerfile.

1. Add the jobs keyword:

jobs:

1. Give a name to the job

– name: anynines_blog_build_job

1. Add our build-plan

plan:

1. Get our resource (we defined the resource right above)

– get: anynines_blog_dockerfile

1. Set a trigger, that means the pipeline should be started every time our resource is changed

trigger: true

1. Define where to put our result

– put: anynines-blog-build-image

1. Add some params (for Docker Images you can just use the build command to build it)

params: {build: anynines-blog-dockerfile}

Once again please keep an eye on the indentations of the file.

At this point,  your file should look something similar to this:

jobs:

• name: anynines_blog_build_job plan:
  • get: anynines_blog_dockerfile trigger: true
  • put: anynines_blog_build_image params: {build: anynines_blog_dockerfile} resources:
• name: anynines_blog_dockerfile type: git source: uri: https://github.com/benjaminguttmann-avtq/concourse_ci_blog.git branch: master private_key: {{git_private_key}} paths: [Dockerfile]
• name: anynines_blog_build_image type: docker-image source: repository: bguttmannavtq/anynines-blog-build-image email: {{docker_mail}} username: {{docker_user}} password: {{docker_pass}}

This kind of YAML would work perfectly with a public Github repository, but what if you want to use a private Github repository as a resource? Not a problem, there are just a few things you have to add to get this to work as well.

In your pipeline.yml add a private_key to the source, you can do this in two ways:

1. Just put it into your pipeline.yml, the result will look something like this

resources:

• name: anynines_blog_dockerfile type: git source: uri: git@github.com:benjaminguttmann-avtq/concourse_ci_blog.git private_key: -----BEGIN RSA PRIVATE KEY----- klajslkdjalksjdlkjaslkdjlkajslkd < lots more text > lanslkdaklsjdljalksjdksdkföokeop -----END RSA PRIVATE KEY----- paths: [Dockerfile]

1. Put a credentials.yml in the same folder as your pipeline.yml and refer to the variable name, the result of this way should look like this:

resources:

• name: anynines_blog_dockerfile type: git source: uri: git@github.com:benjaminguttmann-avtq/concourse_ci_blog.git private_key: {{git-private-key}} paths: [Dockerfile]

The credentials.yml should look something like this:

DOCKER_USER: dockerhub-username DOCKER_MAIL: dockerhub-email DOCKER_PASS: dockerhub-password

git-private-key: | -----BEGIN RSA PRIVATE KEY----- Add your private key here -----END RSA PRIVATE KEY——

If you want to use a ssh key please make sure that your key has no passphrase; keys with passphrases are not supported by Concourse CI.

After we finished writing our pipeline let’s get it to work, switch to your command line (cd to the directory where your pipeline.yml is) and enter the words

fly set-pipeline -p docker-images-pipeline -c pipeline.yml -l credentials.yml

If you got an error like „failed to load variables from file (  ): yaml: line 1: found character that cannot start any token“, please check if you used white spaces instead of tabs for indentation in your YAML files. Perhaps you used some „special“ symbols, e.g. in your password please set them into quotes. This should solve the issue.

Refresh the Concourse main page (192.168.100.4:8080) and now you should see your pipeline.

Unpause your pipeline by pressing the play-button in the UI or by writing:

fly unpause-pipeline -p docker-images-pipeline

to the command line.

After pushing something new to your resource repo the pipeline will start and build the Docker Image and afterwards, push the image to Docker Hub. I think that is more than enough as for this time.

In the last part of our series about Concourse CI and Docker we will take a look at testing an application in a Docker container built of our Docker Image and ways  to deploy it afterwards.

<< Read Part 2 | Read Part 4 >>