Concourse: Learn to fly (Part 1)
Introduction
Concourse? Planes now, huh?
Well, as much as I would love to fly again, this is not about planes or even real flying. This is about Concourse CI, an open-source tool out of the VMware Tanzu product family aiming to deliver automation in form of CI/CD pipelines, not only to build applications but to build the whole infrastructure too.
This is going to be a multipart series, where I show a bit about Concourse. In this, the first part, I’ll just explain what Concourse is, what it consists of, and what its primitives are.
The next part (probably split in two) will show a simple Pipeline, that gets automatically triggered whenever a specific Git Repository is updated. It will then get the Dockerfile from this Repo, build a container-image from it, push this container-image into my local Registry and update the Kubernetes deployment with the new image.
What is Concourse?
Concourse was developed in 2014 at Pivotal by just two engineers (Alex Suraci and Chris Brown). They were simply not satisfied by the CI/CD tools that were out there at the time. So, what could be more obvious than creating your own? Said and done. This is how Concourse was born.
Concourse is aiming to automate not only your App deployments but also your infrastructure. It does this with so-called Pipelines. Of course, there are other tools out there that claim to do the same (e.g. GitLab, Jenkins,…) and they actually do. But Concourse is a bit different than all the others.
Concourse is completely decoupled from the rest of your environment. It doesn’t save anything within itself (except the actual Pipelines of course). Everything it needs is provided on demand by so-called Resources and they are always external. Another differentiator is that Concourse spins up Containers to take care of each Task and each new Task gets a new Container. With that, you always have a clean and predictable deployment environment.
Concourse Infrastructure
From an infrastructure perspective, Concourse consists of three parts:
Web Node
The Web node provides the Web UI (obvious huh? =D). This is where all the pretty Pipelines are visualized. But besides pure beauty, it also does all the scheduling and provides the API.
Worker Nodes
They do the work (Captain Obvious once more =D). The workers do only what they were told to by the Web Node. They are the place where the containers are started to perform some Tasks. After the Task is done (failed or succeeded), the container is gone, and with it all the in-flight data (again, Concourse doesn’t save it). In that way, even the workers are kind of stateless. You could just rip out a worker and everything would continue, as long as you have remaining workers (except some in-flight processes, that were just running on this worker).
Postgres Database
This is where the Concourse internal (!) stuff gets stored, for example, build logs, pipeline configurations, metadata, …
Concourse Primitives
In one sentence: Concourse consists of Pipelines, which reach out to external Resources, in order to run Jobs, each containing a single Plan with one or more Steps to perform.
Pipeline
A Pipeline combines Jobs with Resources.
Resources
Resources provide all the inputs (like Code from a Git Repo) or are destinations for the outputs (like a Kubernetes Cluster for a deployment). There are close to 100 different resource types. And since Concourse is OpenSource and built in a modular way, there will probably be more.
You can find all official resource-types here: https://resource-types.concourse-ci.org/
Jobs
Jobs simply define what to do with a resource. Each Job has a single Plan. You can have multiple Jobs, depending on each other (like being triggered only when another specific Job has passed).
Plans
A plan contains one or more different Steps.
Steps
Steps are the actual actions. There are a few different steps like Get, Put, Tasks, …
Tasks
A task is the smallest configurable unit in Concourse. It is always run in a container. This container is gone after the Task has succeeded or failed. A new Task (or even a new run of the same Task) gets a new container. This way, you always have a nice and clean environment. The Container Image itself is specified in the Pipeline and can basically be any Container. It simply depends on what you wanna run in this Container.
Concourse Installation methods
There are basically four different ways, how concourse is deployed.
The Quick’n’Dirty Way
There is a docker-compose file available. Just run it, and you are good to go. (Spoilers – this is what I gonna use for this series).
You can find the compose file here: https://concourse-ci.org/docker-compose.yml
The HELM Way
Of course, there is a HELM Chart to deploy Concourse. Just be aware, that Concourse manages its own containers, which means privileged containers have to be allowed in the K8s cluster. This might be a security concern. Though it’s on the Roadmap to find a way without privileged containers, it’s definitely not there yet and maybe never will.
You can download the HELM Chart from the Tanzu Network: https://network.pivotal.io/products/p-concourse/
The BOSH Way
This is the traditional way. Since Pivotal was using BOSH for a lot of stuff, they also used it to deploy Concourse. Just note, it is directly deployed via BOSH Director and not from within the Ops Manager. This means that it will not be available as Tile in Ops Manager either.
But since Ops Manager is not part of this, you can also use a standalone BOSH installation.
Detailed Installation steps can be found here: https://docs.pivotal.io/p-concourse/v6/installation/install-concourse-bosh/
Installation as Services
It is also possible to run everything as traditional services on a traditional Linux Server (VM or Bare Metal).
Concourse-CI itself provides the instructions on how to achieve this: https://concourse-ci.org/install.html
Next up: Start Building the Pipeline
