Velero – Backup and Restore Example

Containers are ephemeral. They are stateless.
I’ve heard these sentences so often. Though, they are not false, but if that would be the whole truth, we would never have to consider backups anymore 😉 Sadly, that’s not what reality is like, as you probably know.

If you want to backup your containers (more specifically on K8s), you have to follow a different approach than on your VMs. Here, you have to backup basically two things:

  1. The desired state of your Objects (Deployments, Pods, Services, …)
  2. Your Persistent Volumes

For the first point, you could argue that there is no backup needed, since you are running a fully automated CI/CD Pipeline, which enables you to simply re-trigger all your deployments from your Git … don’t you? 😉
But even if, you would still care about point number two – your Persistent Volumes, since they are storing the data you want to keep.

So how do you to this? One possible solution is Velero.
Velero (previously Ark from Heptio) is an OpenSource Tool, which is now part of the VMware Tanzu family (https://github.com/vmware-tanzu/velero).
Velero can backup your applications (deployment/pods/…), your persistent volumes, whole namespaces or even the entire cluster.
While doing this, Velero does not interact directly with the etcd database (like some other tools). Instead it queries the kube-apiserver.

In this Article, I want to demonstrate how you can backup and restore a simple mysql database, including a persistent volume (where the actual database is saved).

Prerequisites

Kubernetes Cluster

Obvious, isn’t it? 😉 I’m using my Tanzu Kubernetes Cluster.

Object Store

Velero requires an S3 compatible object store. I don’t have access to any of the cloud providers, so I’m using Minio running on a VM. You could also run Minio in your K8s Cluster. In fact, if you check the Velero Git Repo, you’ll see that it actually brings a deployment file to run Minio within K8s.
To install Minio, checkout the Quickstart Guide.

An Application to backup

My example application is a simple mysql deployment, using a pvc to store its database. To validate that the backup worked, I’ve created a Database, a Table and insert some simple information. Following the individual parts of my almost-rocket-science-database.

Persistent Volume Claim (PVC)
A simple PVC. Since I’m using Tanzu Kubernetes Cluster, I don’t have to create the PV manually beforehand. K8s/vSphere take care of it automatically 🙂

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-mysql
spec:
  storageClassName: vsan-default-storage-policy
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Service (LoadBalancer)
The LoadBalancer is actually not needed for the functionality of the mysql server. I only wanted to expose the mysql port, to simplify access to the database later. Also, why not using the NSX-T capabilities, since its there anyway? 🙂

apiVersion: v1
kind: Service
metadata:
  name: svc-mysql
spec:
  type: LoadBalancer
  ports:
  - port: 3306
  selector:
    app: mysql

Deployment
And finally, a simple Deployment File, which only starts a mysql container and mounts a volume.

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: d-mysql
  labels:
    application: mysql-app
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: Password123!
        ports:
        - containerPort: 3306
          name: port-mysql
        volumeMounts:
        - name: vol-mysql
          mountPath: /var/lib/mysql
      volumes:
      - name: vol-mysql
        persistentVolumeClaim:
          claimName: pvc-mysql

Create some DB Entries
Check, which IP got assigned to my LoadBalancer.

vraccoon@ubu:~$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP          27h
supervisor   ClusterIP      None            <none>          6443/TCP         27h
svc-mysql    LoadBalancer   10.107.12.123   172.31.60.195   3306:32137/TCP   7m56s

Next, create a Database, a Table and insert some entries using mycli.

mycli -h 172.31.60.195 -u root -p Password123! -e 'CREATE DATABASE TestDB;'
mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'CREATE TABLE TestTable (id INT(4) AUTO_INCREMENT PRIMARY KEY,firstname VARCHAR(30),lastname VARCHAR(30),reg_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);'
mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'INSERT INTO TestTable(firstname,lastname) VALUES ("Aulus","Augerius")'
mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'INSERT INTO TestTable(firstname,lastname) VALUES ("Numerius","Negidius")'
mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'INSERT INTO TestTable(firstname,lastname) VALUES ("John","Doe")'

And double-check that everything is there.

vraccoon@ubu:~$mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'SELECT * FROM TestTable'
id      firstname      lastname        reg_date
1       Aulus          Augerius        2020-09-06 08:29:21
2       Numerius       Negidius        2020-09-06 08:29:21
3       John           Doe             2020-09-06 08:29:23

So, now we are all set to backup our tremendously complex application =D

Install Velero

First, we have to download the velero binary. I’m using GitHub page to download velero-v1.4.2-linux-amd64.tar.gz. But depending on your OS, you could also use brew (macOS) or choco (Windows).

vraccoon@ubu:~$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.4.2/velero-v1.4.2-linux-amd64.tar.gz
Saving to: ‘velero-v1.4.2-linux-amd64.tar.gz’

velero-v1.4.2-linux-amd64.tar.gz           100%[===========================================>]  22.72M  5.70MB/s    in 5.1s

vraccoon@ubu:~$ tar -xvzf velero-v1.4.2-linux-amd64.tar.gz
velero-v1.4.2-linux-amd64/LICENSE
velero-v1.4.2-linux-amd64/examples/README.md
velero-v1.4.2-linux-amd64/examples/minio
velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
velero-v1.4.2-linux-amd64/examples/nginx-app
velero-v1.4.2-linux-amd64/examples/nginx-app/README.md
velero-v1.4.2-linux-amd64/examples/nginx-app/base.yaml
velero-v1.4.2-linux-amd64/examples/nginx-app/with-pv.yaml
velero-v1.4.2-linux-amd64/velero

vraccoon@ubu:~$ sudo cp velero-v1.4.2-linux-amd64/velero /usr/local/bin/velero

Next, we need to create a credentials file, for our S3 Storage. in my case it looks as follows:

vraccoon@ubu:~$ cat credentials-velero-minio
[default]
aws_access_key_id = minio
aws_secret_access_key = Password123!

Now, as we have the binary and our S3 credentials file, we can continue and deploy velero to our K8s Cluster. Make sure, you have your kubeconfig setup 😉

velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.1.0 \
--bucket velero \
--secret-file ./credentials-velero-minio \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.vraccoon.lab:9000 \
--use-restic

Let’s explain that command a bit more:
–provider aws and –plugins velero/velero-plugin-for-aws:v1.1.0
Sadly, the vSphere Plugin does not have object store support. Hence, we have to use aws.
–bucket velero
The bucket name inside your Opject Store
–secret-file ./credentials-velero-minio
The credentials file, we created before, to access our object store.
–use-volume-snapshots=false
For not setting the snapshot location automatically.
–backup-location-config region=minio,s3ForcePathStyle=”true”,s3Url=http://minio.vraccoon.lab:9000

Some S3 Parameter, including my Minio Url as the target.
–use-restic
To be able to backup PVCs too.

After running the command, you’ll see how velero install a bunch of objects into Kubernetes, including additional CRDs. If everything goes well, it ends with the following line:
Velero is installed! ⛵ Use ‘kubectl logs deployment/velero -n velero’ to view the status.

Create the Backup

Now, we have everything we need. Our Object Store is up, Velero is installed and we have deployed an example application. So, let’s back it up!
Since our MySQL application has an PVC attached, which we want to backup too, there is one more step we have to do – annotate the volume. Restic will only backup volumes, which are annotated.

vraccoon@ubu:~$ kubectl annotate pod d-mysql-64cdbd97c4-nq9lx backup.velero.io/backup-volumes=vol-mysql
pod/d-mysql-64cdbd97c4-ml5s8 annotated

Next, create the backup. We will backup the whole namespace, which is the “default” namespace.

vraccoon@ubu:~$ velero backup create my-mysql-backup --include-namespaces default
Backup request "my-mysql-backup" submitted successfully.
Run `velero backup describe my-mysql-backup` or `velero backup logs my-mysql-backup` for more details.

Seems successfully. Let’s check it in more detail:

vraccoon@ubu:~$ velero backup describe my-mysql-backup
Name:         my-mysql-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.17.8+vmware.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=17

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-09-06 08:33:17 +0000 UTC
Completed:  2020-09-06 08:33:20 +0000 UTC

Expiration:  2020-10-06 08:33:17 +0000 UTC

Total items to be backed up:  33
Items backed up:              33

Velero-Native Snapshots: <none included>

Restic Backups (specify --details for more information):
  Completed:  1

As you can see, we have basically included every Ressource in the namespace “default” into our backup. The last two lines are referring to restic, which has backed up our PV. You can use the same command with “–details” to get more information about it.
Also, Velero (and restic) have created a bunch of objects in our object store, which you could explore.

Delete the Application
Now, as the backup is created, I’m confident enough to delete my Business-critical MySQL Application.

vraccoon@ubu:~$ kubectl delete -f mysql.yaml
persistentvolumeclaim "pvc-mysql" deleted
service "svc-mysql" deleted
deployment.apps "d-mysql" deleted

And confirm, that everything is gone.

vraccoon@ubu:~$ kubectl get deploy,pods,pv,pvc,svc
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    40h
service/supervisor   ClusterIP   None         <none>        6443/TCP   40h

Restore the Backup

Restoring the backup is also quite straight forward.
First, display existing backups.

vraccoon@ubu:~$ velero backup get
NAME              STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
my-mysql-backup   Completed   0        0          2020-09-06 08:33:17 +0000 UTC   29d       default            <none>

Since there is only one backup, there is not much we can choose from 😉

vraccoon@ubu:~$ velero restore create mysql-restore --from-backup my-mysql-backup
Restore request "mysql-restore" submitted successfully.

Once again, we can check the details of the process.

vraccoon@ubu:~$ velero restore describe mysql-restore
Name:         mysql-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Backup:  my-mysql-backup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores (specify --details for more information):
  New:  1

It should complete within a few seconds, leaving us with restored application.

vraccoon@ubu:~/example-app$ kubectl get deploy,pods,pv,pvc,svc
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/d-mysql   1/1     1            1           6m27s

NAME                           READY   STATUS    RESTARTS   AGE
pod/d-mysql-64cdbd97c4-nq9lx   1/1     Running   0          6m27s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS                  REASON   AGE
persistentvolume/pvc-5e7229e1-8fc8-4e11-9cab-4e5d52f8139e   5Gi        RWO            Delete           Bound    default/pvc-mysql   vsan-default-storage-policy            6m25s

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/pvc-mysql   Bound    pvc-5e7229e1-8fc8-4e11-9cab-4e5d52f8139e   5Gi        RWO            vsan-default-storage-policy   6m27s

NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
service/kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP          40h
service/supervisor   ClusterIP      None            <none>          6443/TCP         40h
service/svc-mysql    LoadBalancer   10.96.156.201   172.31.60.195   3306:31745/TCP   6m27s

Looks good to me so far. Let’s confirm that the mysql tables are still there.

vraccoon@ubu:~$mycli -h 172.31.60.195 -u root -p Password123! -D TestDB -e 'SELECT * FROM TestTable'
id      firstname      lastname        reg_date
1       Aulus          Augerius        2020-09-06 08:29:21
2       Numerius       Negidius        2020-09-06 08:29:21
3       John           Doe             2020-09-06 08:29:23

Great, we have successfully restored the application.

Conclusion

The example demonstrated by me was a very simple scenario to show how it works in general. Though, Velero is a powerful tool to backup your K8s applications. It provides a lot of other features like backup schedules, auto retire backups, restore individual files rather than whole PVs, helps to migrate between Clusters,… and so much more. It’s really worth checking it out.

Leave a Reply

Your email address will not be published.