Tanzu with HashiCorp Vault Part 3
This is the third and last part of my multi-post series about Hashicorp Vault.
- Part One provided an overview of HashiCorp Vault, its terminology, and instructions on how to install it in Kubernetes using HELM
- Part Two focused on the HashiCorp Vault Secret Operator. I demonstrated static secret where created in Vault and brought them into Kubernetes
- Part three will cover Vault being used as a Public Key Infrastructure (PKI) as Intermediate CA, providing certificates in Kubernetes using cert-manager with the Vault plugin
Why Vault as PKI?
In the previous two posts, the primary focus was on utilizing Vault as a Secret Management solution. However, Vault can do much more, for example acting as Certificate Authority.
When considering Kubernetes, certificates are managed as secrets of the type “tls”. This is not significantly different from managing typical k8s secrets. TLS secrets essentially consist of the certificate and private key encoded as base64 strings.
The actual challenge is to get your CSR automatically signed.
This is were Cert-Manager come into place. It introduces the concept of issuers (namespaced) or clusterissuers (cluster-scoped). These issuers can sign certificates. To sign them, the issuers need to have access to some sort of (sub)-CA.
The simplest way is to have the CA stored within the K8s Cluster (in form of a tls-secret). Which is then used to sign the certificates. Thinking about having a sub-ca’s private key stored in an K8s cluster as almost plain text feels a bit unsettling … 😉
A much more common approach is to create a (cluster)issuer which makes use of ACME protocol (e.g. Let’s Encrypt). As easy as this sounds at first, as complicated it could get. Most of the customers I had to not have an ACME speaking CA on prem. And if you want to use a public offering, you need to make the DNS/HTTP01 challenge work, so that public CA must query your internal domains somehow – possible, but not trivial.
This is why I want to use Vault as our CA. That way, we are tackling those concerns – Vault can run on prem, serve certificate requests and store them safely.
How does it integrate with vSphere with Tanzu?
Actually, I wanted to use the cert-manager that comes with the Tanzu Packages. Sadly, even its most recent version (1.7.2) has been EOL since about a year (check here Cert-Manager Releases) . Thus, I had to install it via HELM Chart.
Installation
Vault
I’ve set up a fresh Vault installation, like from the end of the first post (Vault running and unsealed). Simply to not get confused with other objects, that are already in place.
Vault Configuration
Authentication Method
Very similar to Part 2, we will create an authentication method based on Kubernetes, create a role and assign ServiceAccound and vault policy.
/ $ vault auth enable -path k8s-pki kubernetes Success! Enabled kubernetes auth method at: k8s-pki/ / $ vault write auth/k8s-pki/config kubernetes_host="https://172.31.192.7:6443" Success! Data written to: auth/k8s-pki/config / $ vault write auth/k8s-pki/role/r-k8s-pki bound_service_account_names=sa-issuer bound_service_account_namespaces=ns-pki policies=p-pki Success! Data written to: auth/k8s-pki/role/r-k8s-pki
Secret Engine
Again, like in Part 2, we will create a secret engine. But this time of type PKI.
/ $ vault secrets enable -path pki-vraccoon pki Success! Enabled the pki secrets engine at: pki-vraccoon/
Next we will change the maximum lease time
/ $ vault secrets tune -max-lease-ttl=43800h pki-vraccoon Success! Tuned the secrets engine at: pki-vraccoon/
Next, we will create a CSR for an intermediate CA. It is also possible to create a new Root CA. But I want the Vault-CA to be a subordinate of my actual root-ca.
/ $ vault write pki-vraccoon/intermediate/generate/internal common_name="vRaccoon Vault SubCA Demo" ttl=43800h Key Value --- ----- csr -----BEGIN CERTIFICATE REQUEST----- MIICaTCCAVECAQAwJDEiMCAGA1UEAxMZdlJhY2Nvb24gVmF1bHQgU3ViQ0EgRGVt bzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANw5VboxMUgx1KH2pAC7 jvdWMEeJ/6s6/INsW4yv8gjGwJCVxC6YhfaBYA4oTwR8eKC2uSQffyYUXugZTWDt BLbGrsWA+DkB3Y6Vq8nZX0S1bFhe7HOuqLGw8kkLGKIt2nJEzWqRbbLoe7vNflc7 +yqKFcvhmIrdZNBBOMjDCkkLrDKmM/yicoVgKfqDtIVqjR7PH+hgZjUYIBYaojkg f2+UN07fZ5xYilQoEgvDcAGdLDEH4YJTR629+A1jB7DeM6/bSsjZlMxoLWuOeCdo fqDpi5Od74o/458980hu+p01lRrDRwYHRzcYi6Z2mTHI4b70/Qwcmg+PHBmxjkjB kQsCAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQC1M18w5+CUYp6R8WS37EhRuxq0 gcKaeS94g1EfE+mpdokK9okrvnue7DXWgFgJfdOGfyu0GzHag/xitsxnGaiS0Bq4 hwY38IbfZX8lfLLy9Gk0NSxD7oyaEPu0r3t/Ti0mO+vDWm/YySjUzdWOGvvQYGpg UCpspuxOeFNaOvMF0J5AW2h7HH7MVM5DWEkUO8zkfcKfP09JdYwI8S15jaIYAwpS ZiaQh4N2NNOslvOhUv6imGv7/NxSYnQsyzUfD/RB/aEI1cq/kEZ/b+aUyY9yfjxs LAP/XQ3WT4XLloIHIuUrrk3vyfOQYhCUIErYnVQE3YlJx/imcNmW70k87MOU -----END CERTIFICATE REQUEST----- key_id e64d1762-049f-fd06-c4d7-5ba5a3f9392d
I will now take this CSR and get it signed by my root CA. Afterwards I’ll save he complete chain (intermediate ca cert –> root ca cert) as subca.crt.
/ $ cat /tmp/subca.crt -----BEGIN CERTIFICATE----- MIIFCjCCA/KgAwIBAgITXwAAAGLO1rOgeehuAQAAAAAAYjANBgkqhkiG9w0BAQsF <subca cert shortened for brevity> ofE0Tixcxldxmq8rbi0Phws8uV8MrPmzHagp0AxNUMdDtN1b38pzyMmHSgsvRQ== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDazCCAlOgAwIBAgIQcpCYdK0x2ZhC/5D79vIczjANBgkqhkiG9w0BAQsFADBI <root ca cert shortened for brevity> Jj4MJODqVnu3FZOCF8X/ -----END CERTIFICATE-----
Now we can import the SubCA
/ $ vault write pki-vraccoon/intermediate/set-signed certificate=@/tmp/subca.crt WARNING! The following warnings were returned from Vault: * This mount hasn't configured any authority information access (AIA) fields; this may make it harder for systems to find missing certificates in the chain or to validate revocation status of certificates. Consider updating /config/urls or the newly generated issuer with this information. Key Value --- ----- existing_issuers <nil> existing_keys <nil> imported_issuers [8ca0137e-a51f-b34a-d907-5f67557b8bb6] imported_keys <nil> mapping map[8ca0137e-a51f-b34a-d907-5f67557b8bb6:]
Next, set the issuing and revoke url
~ $ vault write pki-vraccoon/config/urls issuing_certificates="http://vault.vault.svc.cluster.local:8200/v1/pki_int/ca" crl_distribution_p oints="http://vault.vault.svc.cluster.local:8200/v1/pki_int/crl" Key Value --- ----- crl_distribution_points [http://vault.vault.svc.cluster.local:8200/v1/pki_int/crl] enable_templating false issuing_certificates [http://vault.vault.svc.cluster.local:8200/v1/pki_int/ca] ocsp_servers []
Next we can create the PKI role. Don’t get this confused with the role in the authentication methods. This PKI role defines what sort of csr can be signed. For example it defines what issuer to be used (in case you got multiple), what domains are allowed, key usage, etc.
~ $ vault write pki-vraccoon/roles/vraccoon.lab allowed_domains=vraccoon.lab allow_subdomains=true max_ttl=72h WARNING! The following warnings were returned from Vault: * Issuing Certificate was set to default, but no default issuing certificate (configurable at /config/issuers) is currently set Key Value --- ----- allow_any_name false allow_bare_domains false allow_glob_domains false allow_ip_sans true allow_localhost true allow_subdomains true allow_token_displayname false allow_wildcard_certificates true allowed_domains [vraccoon.lab] allowed_domains_template false allowed_other_sans [] allowed_serial_numbers [] allowed_uri_sans [] allowed_uri_sans_template false allowed_user_ids [] basic_constraints_valid_for_non_ca false client_flag true cn_validations [email hostname] code_signing_flag false country [] email_protection_flag false enforce_hostnames true ext_key_usage [] ext_key_usage_oids [] generate_lease false issuer_ref default key_bits 2048 key_type rsa key_usage [DigitalSignature KeyAgreement KeyEncipherment] locality [] max_ttl 72h no_store false not_after n/a not_before_duration 30s organization [] ou [] policy_identifiers [] postal_code [] province [] require_cn true server_flag true signature_bits 256 street_address [] ttl 0s use_csr_common_name true use_csr_sans true use_pss false
Policy
Last, we need to configure the Vault policy, which allows the user from the authentication method (in our case K8s Service Account sa-issuer) to access the paths of our PKI Secret Engine. More specific the path of our PKI Role within that Secret Engine.
vault policy write p-pki - <<EOF
path "pki-vraccoon*" { capabilities = ["read", "list"] }
path "pki-vraccoon/sign/r-vraccoon.lab" { capabilities = ["create", "update"] }
path "pki-vraccoon/issue/r-vraccoon.lab" { capabilities = ["create"] }
EOF
K8s Configuration
With the Vault config beeing finished, we can continue to configure K8s.
Cert-Manager
Installing cert-manager via HELM is pretty straight forward. Just adding the repo and install the Chart. No big customisations needed. We only need to check the flag for installing CRDs (like “issuer”).
❯ helm repo add jetstack https://charts.jetstack.io "jetstack" has been added to your repositories ❯ helm install cert-manager --namespace cert-manager jetstack/cert-manager --create-namespace --set installCRDs=true NAME: cert-manager LAST DEPLOYED: Mon Sep 11 14:53:29 2023 NAMESPACE: cert-manager STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: cert-manager v1.12.2 has been deployed successfully! In order to begin issuing certificates, you will need to set up a ClusterIssuer or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer). More information on the different types of issuers and how to configure them can be found in our documentation: https://cert-manager.io/docs/configuration/ For information on how to configure cert-manager to automatically provision Certificates for Ingress resources, take a look at the `ingress-shim` documentation: https://cert-manager.io/docs/usage/ingress/
Issuer
The issuer is the connection from cert-manager to the signing authority. In contrast to the clusterissuer, the issuer is an namespaced object. Thus we need to create the namespace first. Additionally, we need to create the servicAccount thats going to be used to connect to Vault.
❯ kubectl create namespace ns-pki namespace/ns-pki created ❯ kubectl -n ns-pki create serviceaccount sa-issuer serviceaccount/sa-issuer created
Now we can create the actual issuer object:
❯ kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: vault-issuer
namespace: ns-pki
spec:
vault:
auth:
kubernetes:
role: r-k8s-pki
mountPath: /v1/auth/k8s-pki
serviceAccountRef:
name: sa-issuer
server: http://vault.vault.svc.cluster.local:8200
path: pki-vraccoon/sign/r-vraccoon.lab
EOF
issuer.cert-manager.io/vault-issuer created
The Issuer is created, and we can check its status:
❯ kubectl -n ns-pki get issuers -o wide NAME READY STATUS AGE vault-issuer False Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: while requesting a token for the service account ns-pki/sa-issuer: serviceaccounts "sa-issuer" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "serviceaccounts/token" in API group "" in the namespace "ns-pki" 24s
As you can see, the issuer failed.
The reason is, that the cert-manager serviceAccount is trying to request a token for service account sa-issuer (which we granted permission in Vault authentication method).
But the cert-manager serviceAccount does not have the permission to get a token. So we need to create a rolebinding for it.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: r-vault-issuer
namespace: ns-pki
rules:
- apiGroups: ['']
resources: ['serviceaccounts/token']
resourceNames: ['sa-issuer']
verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rb-vault-issuer
namespace: ns-pki
subjects:
- kind: ServiceAccount
name: cert-manager
namespace: cert-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: r-vault-issuer
The role above grants permissions to create serviceaccount tokens. And it limits it to service account sa-issuer. Since its a role which is namespaced, its limited to the sa-issuer in the namespace ns-pki. This role then is assigned to serviceAccount cert-manager in namespace cert-manager.
Let’s apply this and check the issuer again.
❯ kubectl create -f rolebinding.yaml role.rbac.authorization.k8s.io/r-vault-issuer created rolebinding.rbac.authorization.k8s.io/rb-vault-issuer created ❯ k get issuer vault-issuer -o wide NAME READY STATUS AGE vault-issuer True Vault verified 31m
As you can see, the issuer is now ready and Vault verified.
Btw – secretless authentication is the preferred method since Kubernetes 1.24. And this is only supported by cert-manager v1.12 onwards.
Certificate Request
Finally, we can request a certificate. Following an example. As always, there is much more you could configure.
❯ kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cert-demo
namespace: ns-pki
spec:
secretName: cert-demo.vraccoon.lab
issuerRef:
name: vault-issuer
commonName: cert-demo.vraccoon.lab
dnsNames:
- cert-demo.vraccoon.lab
EOF
certificate.cert-manager.io/cert-demo created
Let’s check the status of the certificate:
❯ kubectl -n ns-pki get certificate -o wide NAME READY SECRET ISSUER STATUS AGE cert-demo True cert-demo.vraccoon.lab vault-issuer Certificate is up to date and has not expired 37s
This looks good. We can also check the corresponding secret:
k -n ns-pki describe secret cert-demo.vraccoon.lab
Name: cert-demo.vraccoon.lab
Namespace: ns-pki
Labels: controller.cert-manager.io/fao=true
Annotations: cert-manager.io/alt-names: cert-demo.vraccoon.lab
cert-manager.io/certificate-name: cert-demo
cert-manager.io/common-name: cert-demo.vraccoon.lab
cert-manager.io/ip-sans:
cert-manager.io/issuer-group:
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: vault-issuer
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
ca.crt: 1245 bytes
tls.crt: 3273 bytes
tls.key: 1679 bytes
It’s looking good too.
Final words
This concludes my Vault Series. I agree, that my demonstrated scenarios where quite detached from real world use cases. But my goal was to learn and show the basics and create an understanding of it.
In a real world, cert-manager would probably watch ingress objects beeing created and automatically create certifcate objects based on the fqdn listed in the ingress object.
Vault would automatically rotate certificates as needed and update the secrets in K8s seemless.
If you think this further, DNS entries would most like be created automatically based on the ingress fqdn too. This could be performed by external-dns, NSX ALB, or similar.
I hope this short trip into the Vault helped!
