Tanzu with HashiCorp Vault Part 1
Introduction
If you are working with cloud-native technologies, especially Kubernetes, you will inevitably encounter the topic of secret management. I wanted to check HashiCorp Vault for a while now, and finally, I’ve found the time to do so. At the same time, I take the opportunity to create a series of blog posts about it.
- Part One will provide an overview of HashiCorp Vault, its terminology, and instructions on how to install it in Kubernetes using HELM.
- Part Two will focus on the HashiCorp Vault Secret Operator. I will demonstrate how to create a static secret in Vault and bring it into Kubernetes.
- Part three will cover Vault being used as a Public Key Infrastructure (PKI) as Intermediate CA, providing certificates in Kubernetes using cert-manager with the Vault plugin
What is Vault and how can it help?
HashiCorp Vault is a leading product in the field of secret management. It can do both, static secrets, which are stored and retrieved, and dynamic secrets, which are generated on-demand. Vault provides fine-grained access control for these secrets. Additionally, Vault ensures that data is encrypted (at rest and in flight), offers extensive audit logging, and – of course – supports integration with public cloud providers.
Vault Terminology
Authentication Methods
Authentication Methods are used to authenticate users … obvious huh? 😉
You can think of them as plugins that offer different authentication options such as Kubernetes, LDAP, GitHub, Azure, …
Each method has its own set of configuration options. For example, the GitHub method requires an organization and org ID, while the Kubernetes method requires a Kubernetes host and CA certificate.
You can set up every authentication method multiple times, as you might have multiple K8s Cluster / LDAP directories,… to authenticate against.
Each authentication method is assigned a unique path, which is later used as a mount path. This path acts as a configuration identifier for the authentication method.
More information: https://developer.hashicorp.com/vault/docs/auth
Roles
Roles are configured under an authentication method and connect user entities with policies. The options for configuring roles depend on the chosen authentication method. E.g. for Kubernetes, you need to define serviceAccount(s) and their corresponding namespace(s). For LDAP, you would configure names/groups. And so on…
Secret Engines
Secret Engines are responsible for storing sensitive data in Vault. Sensitive data can take different forms, ranging from simple tokens to key-value pairs (such as credentials) up to much more complex constructs. Similar to authentication Methods, there are different types of secret engines available to handle these various kinds of secrets appropriately.
When creating a Secret Engine, you specify a unique path (also similar to authentication methods). Secrets will be organized under that path, very similar to a file system structure. This path is later used for assigning policies.
More information: https://developer.hashicorp.com/vault/docs/secrets
Policies
Policies define access to specific secrets in Vault. Since “everything in Vault is path based”, it’s easy to define the scope of each policy. A simple example is a policy, that defines capabilities (HTTP verbs) for a specific path, but there are also option for more advanced parameters. Additionally, paths can contain wildecards.
A policy gets assigned to roles (within an authentication method). A policy can be assigned to multiple roles, and roles can have multiple policies assigned.
Policies are deny by default. Thus, no policy –> no access 😉
More Information: https://developer.hashicorp.com/vault/docs/concepts/policies
Setup Vault with HELM
There are several options for deploying Vault. In production it is likely that you have a high available cluster (consisting of three nodes), usually based of VMs.
Although you could run a production cluster on K8s, there are a lot more caveats you need to be aware of. At the end of the day, Vault will become one of the most critical components of your infrastructure. Just imagine what happens, if you can’t access any of your keys/tokens/secrets anymore.
For my demo scenario, I’m going to deploy Vault as a single instance on Kubernetes, using the official HELM Chart.
Prerequisites
There are only a few things I’ve done, that are not there by default:
- Installed Tanzu PackageRepo
- Installed Cert-Manager Package (but haven’t done anything here)
- Installed Contour Package (for use with LoadBalancer)
- Installed External-DNS Package (to update my Windows DNS Server)
Installation
First, we add the hashicorp HELM repository
> helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp" has been added to your repositories
Next, we grab the default values file for out vault installation:
helm show values hashicorp/vault > vault-values.yaml
There are a few values, we need to modify:
psp.enable=true # Especially on vSphere with Tanzu, PSP are still there ingress.enabled=true # this is not neccessary, but I'd like to access the GUI later ingress.hosts[0].host=vault.vraccoon.lab # The FQDN I want to access the GUI later ingress.tls[0].secret-Name=tls-vault # the secret we prepared to be used for tls ingress.tls[0].hosts[0]=vault.vraccoon.lab # the hostname for tls dataStorage.storageClass=sp-tanzu # the storageClass to be used as dataStorage auditStorage.storageClass=sp-tanzu # the storageClass to be used as auditStorage ui.enabled=true # Although, not neccessary, but I want to enable the UI
Before running the HELM installer, we need to create the Namespace and the tls-secret in it:
❯ kubectl create namespace vault namespace/vault created ❯ kubectl -n vault create secret tls tls-vault --cert=./vault.crt --key=./vault.key secret/tls-vault created
Now we can run the HELM installer
❯ helm -n vault install vault hashicorp/vault --version 0.25.0 --values vault-values.yaml W0804 14:04:35.938523 17783 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ W0804 14:04:35.940479 17783 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ W0804 14:04:36.022233 17783 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ W0804 14:04:36.023277 17783 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME: vault LAST DEPLOYED: Fri Aug 04 14:04:35 2023 NAMESPACE: vault STATUS: deployed REVISION: 1 NOTES: Thank you for installing HashiCorp Vault! Now that you have deployed Vault, you should look over the docs on using Vault with Kubernetes available here: https://www.vaultproject.io/docs/ Your release is named vault. To learn more about the release, try: $ helm status vault $ helm get manifest vault
Let’s check if the installation is ready:
kubectl -n vault get pods NAME READY STATUS RESTARTS AGE vault-0 0/1 Running 0 4m8s vault-agent-injector-5db56885fb-gvhxp 1/1 Running 0 4m9s
Note: Its normal at this point, that the vault-0 Pod is Running, but not Ready. It’ll stay like this until the vault gets unsealed, which we do next.
Therefore, we need to execute some commands within that Pod:
❯ kubectl -n vault exec -it vault-0 -- vault operator init Unseal Key 1: ZaA5U1HKvyjHkWHGTJQk/smscKfkPtRKNyxTybvrq75w Unseal Key 2: ySLqaEFeGojcNHybEDhOGqXSEKOyuchFFXHcRKZ6p5V6 Unseal Key 3: lZ4a+5+tnkgrAUq8WRshcLjvnJDUh6cohFngMiBr+6TA Unseal Key 4: mn3n5rbPhMJCokyuO+bbbxTS2GrDib5WvqZp4d+XmKLP Unseal Key 5: fK+dBhBQ1SOB9H3lqFcCxg+u4OvWZ2Y5If4XoX/TShJ9 Initial Root Token: hvs.xTu3ObTcYZDMGRDiAfq08nKc Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
Be careful with those unseal and root keys and store them somewhere safe (and don’t post them on a random internet blog 😉 )
Next, unseal the Vault – we need 3 out of the 5 unseal keys to do this:
❯ kubectl -n vault exec -it vault-0 -- vault operator unseal ZaA5U1HKvyjHkWHGTJQk/smscKfkPtRKNyxTybvrq75w Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce b844c084-2293-6dcf-e9e1-db14944049e9 Version 1.14.0 Build Date 2023-08-05T11:40:23Z Storage Type file HA Enabled false ❯ kubectl -n vault exec -it vault-0 -- vault operator unseal mn3n5rbPhMJCokyuO+bbbxTS2GrDib5WvqZp4d+XmKLP Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce b844c084-2293-6dcf-e9e1-db14944049e9 Version 1.14.0 Build Date 2023-08-05T11:40:23Z Storage Type file HA Enabled false ❯ kubectl -n vault exec -it vault-0 -- vault operator unseal fK+dBhBQ1SOB9H3lqFcCxg+u4OvWZ2Y5If4XoX/TShJ9 Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.14.0 Build Date 2023-08-05T11:40:23Z Storage Type file Cluster Name vault-cluster-04ffa0c5 Cluster ID b43d107c-6331-0cf0-b4e6-f15f042cec71 HA Enabled false
Note, that after the third command, the value for Sealed switched to false. The Vault is now Ready to use.
Let’s check if the Pod is READY:
❯ kubectl -n vault get pods NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 10m vault-agent-injector-5db56885fb-gvhxp 1/1 Running 0 10m
Next
Installing Secret Operator and retrieve a secret from Vault.
