Concourse: Learn to fly (Part 3)

• Tweet it
• Pin it
• Share it
• Email

This is part 3 of my little Concourse-CI series.
Part 1 has shown what Concourse actually is, and in Part 2 I’ve created a first Pipeline that pulled a Git Repository, created a Container Image based on the Dockerfile within that Repo, and finally pushed that image into my local Harbor Registry.
Part 3 however will show how to add the functionality to update my running Kubernetes Deployment with that new image too.
To do this, I need to add another Resource (for Kubernetes) and add another Job that updates the Deployment on my Kubernetes Cluster.

Kubernetes Preparation

Before I can continue with Concourse, I need to do some prep-work on K8s – creating the actual Deployment and creating a ServiceAccount for Concourse.

K8s Deployment

My Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: learn-to-fly
  name: learn-to-fly
spec:
  replicas: 1
  selector:
    matchLabels:
      app: learn-to-fly
  template:
    metadata:
      labels:
        app: learn-to-fly
    spec:
      containers:
      - image: harbor.vraccoon.lab/library/learn-to-fly@IMAGE
        name: learn-to-fly

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: learn-to-fly
  name: svc-learn-to-fly
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: learn-to-fly
  type: LoadBalancer

It’s a very simple Deployment file.
Just note Line 18 – the actual image digest is missing, because I’m replacing this line during runtime with whatever is the newest digest of my Harbor Image.

Concourse Service Account

Creating the Service Account is again pretty straight forward. I’ll assign it cluster-admin permissions, just because it’s already there and easier that way.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-concourse

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: crb-concourse-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: sa-concourse
  namespace: default

Get the Service Account’s secret:

vraccoon@ubu:~$ kubectl get sa sa-concourse -o json | jq .secrets[0].name
"sa-concourse-token-h85dd"

Next, get the Token and decode it:

vraccoon@ubu:~$ kubectl get secrets sa-concourse-token-h85dd -o json | jq .data.token -r | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik5FcFR0aE9YR2pacFlKWFB4Sm5MNFBLVW9HaHB.....

I’ll take a note of that token. I’ll need it in a bit for the K8s Resource in Concourse.

Continue the Pipeline

The preparations are done, let’s get back to our Pipeline.

Kubernetes Resource Type

Concourse doesn’t bring a K8s resource type out of the box, so I have to define the resource “myself”. I’m going to use jgriff’s k8s-resource type for this.

resource_types:
  - name: k8s-resource
    type: docker-image
    source:
      repository: jgriff/k8s-resource

Line 2 – Name of this Resource type, this is how I name it
Line 5 – This is the reference to the repository. It basically maps to a specific container image

Kubernetes Resource

The Kubernetes Resource from its build structure looks very similar to what I have already with Harbor:

  - name: r-k8s-cluster
    type: k8s-resource
    icon: kubernetes
    source:
      url: https://tkg-c1.vraccoon.lab:8443
      token: eyJhbGciOiJSUzI1NiIsImtpZCI6Ik5FcFR0aE9YR2pacFlKWFB4Sm5MNFBLVW9HaH <truncated-for-brevity>
      certificate_authority: |
                                -----BEGIN CERTIFICATE-----
                                MIIC+zCCAeOgAwIBAgIUTk0x/4jOgNuKF9L+CHwIo6xP9EkwDQYJKoZIhvcNAQEL
                                <truncated-for-brevity>
                                uqGvpS5fz6Y0bgX6TrNsqeFh2ADwZIPcj7/8EerophOGIqD7eJ76GiKlRaLx+0M=
                                -----END CERTIFICATE-----

Line 1 – Just the Name of my resource
Line 2 – The name of the resource-type, I’ve specified earlier
Line 3 – Just an Icon (they are from Material Design Icons)
Line 5 – The url of my Kubernetes Cluster. Make sure, that this FQDN/IP is covered in the Certificate in line 7
Line 6 – The Service account Token, I’ve collected earlier
Line 7 – The Certificate which has to be trusted for this K8s Cluster

Job

This Job is a bit more nested than the one for Harbor in my last post.
This time, the Job consists of a Plan with three steps:

1. Get the Git Repo
   Similar to before, I need to get the Git Repo. This feels kind of redundant, and it technically is, because I’m pulling the Repo again. In theory I could hand it over from my first job. But in order to pass files from one job to another, I need an S3 compatible storage set up between them. And this felt a bit to much for a Git Repo with just a hand full of text files.
2. Task to modify the deployment
   This task fires up a script, that checks my local Harbor for the latest image digest and then replace the image value in the k8s deployment file.
   If I’d only write “image: learn-to-fly:latest” in my deployment yaml, Kubernetes would not recognize any change (even though the image version in Harbor were updated). This because K8s only compares these strings.
   An alternative could be to use a versioning resource in concourse to set proper version tags on the image (already during the push), instead of working with the actual digest as I do.
3. “Put” the deployment in Place
   This is where the Kubernetes resource comes into play. It simply applies the modified deployment file to Kubernetes.

- name: j-update-deployment
    serial: true
    plan:
      - get: r-git
        trigger: true
        passed:
          - j-create-and-push-image
      - task: t-modify-deployment
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: {repository: mwendler/jq}
          inputs:
            - name: r-git
          outputs:
            - name: e-modified-deployment
          run:
            path: r-git/k8s/get-digest.sh
      - put: r-k8s-cluster
        params:
          kubectl: apply -f e-modified-deployment/deployment.yml
          namespace: default

Line 1 – The name of the Job. This is also what is displayed in the GUI.
Line 4 – Get the Git Resource (aka clone the Repo)
Lines 6-7 – This step will only trigger, when the previous Job (from part 2 of this series) has successfully passed.
Line 8 – Name of the step from type “task”
Lines 10-12 – Define the generell config of this task
Line 13 – Defines the actual container in which this task is going to run. In my case, I simply needed a container image with jq installed, as this is required by my script.
Lines 14-15 – This defines the input to this task, which is whatever the step called “r-git” outputs. This output is then mounted into the container of this task.
Lines 16-17 – This is the relative directory where this task puts its output to.
Lines 18-19 – Command to run my script. The git repo was cloned and handed over in the input “r-git” directory. Within this directory, you’d find the repo structure. In my repo is a folder called k8s, which contains the script to modify the output –> the relative path in my case is <input-name>/<git-structure>.
The get-digest script itself just queries harbor for the latest digest of our container image and uses sed to replace the image value in the deployment file (which is sitting in the very same directory). After that, it’ll put the deployment.yml into the output directory, which is definied as e-modified-deployment
Lines 20 – Using put on the Kubernetes resource
Lines 22 – running kubectl apply on the deployment.yml in my output directory.

Create the Pipeline

I’ve saved my entire Pipeline in a file called pipe-create-and-push-image.yml. Let’s push it to Concourse and unpause the pipeline.

vraccoon@ubu:~$ fly -t local-concourse set-pipeline -p create-and-push-image -c pipe-create-and-push-image -n
resources:
  resource r-git has been added:
+ icon: github
+ name: r-git
+ source:
+   branch: master
+   uri: https://github.com/vRaccoon/learn-to-fly.git
+ type: git

  resource r-harbor has been added:
+ icon: oci
+ name: r-harbor
+ source:
+   ca_certs:
+   - cert: |
+       -----BEGIN CERTIFICATE-----
+       MIIDUTCCAjmgAwIBAgIVAM62WHzfLCadYf0ugejAyBO0eCaaMA0GCSqGSIb3DQEB
+       CwUAMB8xCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdQaXZvdGFsMB4XDTIwMTEyMDIw
+       MTEzOFoXDTI0MTEyMTIwMTEzOFowHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bp
+       dm90YWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0EeKE1SMb58cH
+       HHtavZN5cY0Ia4eCzPBlpUap7YZqKKUFI5BSUw4fXpGIqKYSzH3EVefrBx7JvqSh
+       247ADN92RoiWaN31NGey1Hhk35m63tVQ+336AORP0vQzHexD1CeNGtE8w7GBsOEI
+       XNuuvbY/zFbsxy29XpUYpUuBpQPGUwNR3fI5IgeTjf1HsRYq49EnAh8LfnYyH1r+
+       I5OIUsNYx85fMS+/QEla+XfjvjAS6omwtr4A+VFQQJMMAnykHXial0J7+JjiyYyd
+       GgmbSnrcXBSaWPmuTv7QbCtYSqO7eCs6GV9/lokR2q3ZyNtELo5vFPns06v4slMB
+       CTKc4d+xAgMBAAGjgYMwgYAwHQYDVR0OBBYEFKhM9OHRYvNFFGgyYKR29zkvzVN5
+       MB8GA1UdIwQYMBaAFKhM9OHRYvNFFGgyYKR29zkvzVN5MB0GA1UdJQQWMBQGCCsG
+       AQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB
+       BjANBgkqhkiG9w0BAQsFAAOCAQEAl0Rnl1nRDCSUdHWy6BPxSWWEQwYoPd/Z2QBO
+       vizn8AFn6YG0thscNfsuJW4Cz4UoDbkazmt9tybB9Wv8BVROfJdeucOdLtGd+4QU
+       jWabPWoI0v2QqBg9Y/1h9w99Cm77i/57O+vtDpgbP1RL6yDdv3sznsx4xpor59Gi
+       2crv5RwexJp/TzGfrlwfLWzgFJ1t2hhoTBZQXTXpOKA8/dP3p3+3W8uat3UvfnNL
+       dF3mvm7+giRIk2Kl80sCyp5Poubuq+Qb95MUsR+CukpTUdpKFVBQhzWWUTt9cm5E
+       PLCaQHLMrcjlpQlMzMu+EB4XB4gm1IeZnpxE2Gyio3AmWApfHQ==
+       -----END CERTIFICATE-----
+     domain: harbor.vraccoon.lab
+   password: Password123!
+   repository: harbor.vraccoon.lab/library/learn-to-fly
+   username: admin
+ type: docker-image

  resource r-k8s-cluster has been added:
+ icon: kubernetes
+ name: r-k8s-cluster
+ source:
+   certificate_authority: |
+     -----BEGIN CERTIFICATE-----
+     MIIC+zCCAeOgAwIBAgIUTk0x/4jOgNuKF9L+CHwIo6xP9EkwDQYJKoZIhvcNAQEL
+     BQAwDTELMAkGA1UEAxMCY2EwHhcNMjAxMTIxMjEwMTU5WhcNMjQxMTIxMjEwMTU5
+     WjANMQswCQYDVQQDEwJjYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
+     AM6AU+fod1Y6+UIgxV7TumyWl9KA18KPrAPNubbmXWtmgbODwwMHyAN2ILxey54N
+     k5FQbyPzpoRXw9QuM4CXesDTddnD9HeazW9pAz4QqBbZpWNG3pm+7VmCcgLcaNGg
+     aeWA8+Wzu/W8M4QDUOViemEMwapRsL+pYRLSHqiXhai5UiISlOSClaR3HhNWDFwj
+     756awTKA46/nOTzGtZ82mY9Mdyo+/maAKbQIEpEoexeMVq20Q0wZPhhG90/u4ABL
+     EVr4M8LN2wJLDDGj4R905OY2Va0/wMyKZM/gY80vvrYMn/Ggy9MuU3W3c0E57vXM
+     59W3LXZN+KFu3ciNSBxmfyECAwEAAaNTMFEwHQYDVR0OBBYEFLpcrAFza6UHTSUz
+     usNtvI4ZrayYMB8GA1UdIwQYMBaAFLpcrAFza6UHTSUzusNtvI4ZrayYMA8GA1Ud
+     EwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAHos9S9l36kq7VPT7XUfPzUS
+     YkfF2BoJvBSRVe+UuWoKbF9oeASpzUBvEZmi6nNYHgsX6ZbBVnkDPhenKs/n7YAN
+     9svtxeG0lWYG982zFOaUTZxfXKE9eijA5yDrzEM1j3SsazuDG+yi5q240Hxozz23
+     b17nD0OX7RZOhNuHfBgwO62rHzSL10ElXigCApva+KeEtUPJPK/JourSRQju9Edw
+     5JauqehF8Jgua6EcIA5KFujBfmipfhM6iu8LPetgd8DGm+GuaxVPn1u2WhwHQaZJ
+     uqGvpS5fz6Y0bgX6TrNsqeFh2ADwZIPcj7/8EerophOGIqD7eJ76GiKlRaLx+0M=
+     -----END CERTIFICATE-----
+   token: eyJhbGciOiJSUzI1NiIsImtpZCI6Ik5FcFR0aE9YR2pacFlKWFB4Sm5MNFBLVW9HaHBPNGlPdlFfMGFoRUg3aFUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNhLWNvbmNvdXJzZS10b2tlbi1oODVkZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzYS1jb25jb3Vyc2UiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmNDc0Mzk0NS0xNTQ4LTQ2MDQtYTY2YS03Nzk3YTIxNzhmOGEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpzYS1jb25jb3Vyc2UifQ.ZNMh65KCr5_tdciA-yIoMGPVY7TQ-xrpveQ3tLDb8Tv-Ys-EBXg2B7H7vjeYDwI6y5rlzQy7t4iGO798C5KpkRtzET9NrclCzzyzTPXazZdWqvfq4vkIexyOQBESPLV5VJ05eN-nsG-krgp07ADrqoRCPXW4T3hQGtHgBYIdhE5T-WrGuDgjQMq5AVY-UWYh-vdj1WANd0ePhjtCXlIWi5hKFXGtxHOgPEWBXdm45RGte8kPkEncNXxggU28MdaP1QWdd2CNGFkAeZq1Rqi_WuTvO1Q_6clLNYzz7mZTdlKqW8d5JsMy5J-HJ1qddEtGfAgJh3sJ4Pe_2Fl0CUjwiQ
+   url: https://tkg-c1.vraccoon.lab:8443
+ type: k8s-resource

resource types:
  resource type k8s-resource has been added:
+ name: k8s-resource
+ source:
+   repository: jgriff/k8s-resource
+ type: docker-image

jobs:
  job j-create-and-push-image has been added:
+ name: j-create-and-push-image
+ plan:
+ - get: r-git
+   trigger: true
+ - params:
+     build: r-git/docker
+   put: r-harbor
+ public: true
+ serial: true

  job j-update-deployment has been added:
+ name: j-update-deployment
+ plan:
+ - get: r-git
+   passed:
+   - j-create-and-push-image
+   trigger: true
+ - config:
+     image_resource:
+       source:
+         repository: mwendler/jq
+       type: docker-image
+     inputs:
+     - name: r-git
+     outputs:
+     - name: e-modified-deployment
+     platform: linux
+     run:
+       path: r-git/k8s/get-digest.sh
+   task: t-modify-deployment
+ - params:
+     kubectl: apply -f e-modified-deployment/deployment.yml
+     namespace: default
+   put: r-k8s-cluster
+ public: true
+ serial: true

pipeline created!
you can view your pipeline here: http://cmgmt.vraccoon.lab:8080/teams/main/pipelines/create-and-push-image

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t local-concourse unpause-pipeline -p create-and-push-image
  - click play next to the pipeline in the web ui



vraccoon@ubu:~$ fly -t local-concourse unpause-pipeline -p create-and-push-image
unpaused 'create-and-push-image'

The Pipeline should start automatically after a few seconds:

Checking the results

Let’s check if the deployment was successful. And if so, which specific image was used:

vraccoon@ubu:~$ kubectl get deployment learn-to-fly
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
learn-to-fly   1/1     1            1           113m

vraccoon@ubu:~$ kubectl get deployment learn-to-fly -o json | jq .spec.template.spec.containers[0].image
"harbor.vraccoon.lab/library/learn-to-fly@sha256:4024d59c1f26037e5a50f7db8e78b404c6801a85b794da6dadc1d1bf7ae093a8"

Compare this to what I have in Harbor:

As you can see, there are multiple artifacts already. So it might make sense to set up some kind of retention policy here too, otherwise Harbor might get overwhelmed at some time.
But the important part is the digest of the artifact with the label “latest”. This digest is the same as the one seen in Kubernetes. Which means, the Pipeline works as intended =D

This concludes my mini series on Concourse. I hope it was helpful =)