Tanzu with HashiCorp Vault Part 3

This is the third and last part of my multi-post series about Hashicorp Vault.

  • Part One provided an overview of HashiCorp Vault, its terminology, and instructions on how to install it in Kubernetes using HELM
  • Part Two focused on the HashiCorp Vault Secret Operator. I demonstrated static secret where created in Vault and brought them into Kubernetes
  • Part three will cover Vault being used as a Public Key Infrastructure (PKI) as Intermediate CA, providing certificates in Kubernetes using cert-manager with the Vault plugin

Why Vault as PKI?

In the previous two posts, the primary focus was on utilizing Vault as a Secret Management solution. However, Vault can do much more, for example acting as Certificate Authority.

When considering Kubernetes, certificates are managed as secrets of the type “tls”. This is not significantly different from managing typical k8s secrets. TLS secrets essentially consist of the certificate and private key encoded as base64 strings.

The actual challenge is to get your CSR automatically signed.
This is were Cert-Manager come into place. It introduces the concept of issuers (namespaced) or clusterissuers (cluster-scoped). These issuers can sign certificates. To sign them, the issuers need to have access to some sort of (sub)-CA.

The simplest way is to have the CA stored within the K8s Cluster (in form of a tls-secret). Which is then used to sign the certificates. Thinking about having a sub-ca’s private key stored in an K8s cluster as almost plain text feels a bit unsettling … 😉

A much more common approach is to create a (cluster)issuer which makes use of ACME protocol (e.g. Let’s Encrypt). As easy as this sounds at first, as complicated it could get. Most of the customers I had to not have an ACME speaking CA on prem. And if you want to use a public offering, you need to make the DNS/HTTP01 challenge work, so that public CA must query your internal domains somehow – possible, but not trivial.

This is why I want to use Vault as our CA. That way, we are tackling those concerns – Vault can run on prem, serve certificate requests and store them safely.

How does it integrate with vSphere with Tanzu?

Actually, I wanted to use the cert-manager that comes with the Tanzu Packages. Sadly, even its most recent version (1.7.2) has been EOL since about a year (check here Cert-Manager Releases) . Thus, I had to install it via HELM Chart.



I’ve set up a fresh Vault installation, like from the end of the first post (Vault running and unsealed). Simply to not get confused with other objects, that are already in place.

Vault Configuration

Authentication Method

Very similar to Part 2, we will create an authentication method based on Kubernetes, create a role and assign ServiceAccound and vault policy.

/ $ vault auth enable -path k8s-pki kubernetes
Success! Enabled kubernetes auth method at: k8s-pki/

/ $ vault write auth/k8s-pki/config kubernetes_host=""
Success! Data written to: auth/k8s-pki/config

/ $ vault write auth/k8s-pki/role/r-k8s-pki bound_service_account_names=sa-issuer bound_service_account_namespaces=ns-pki policies=p-pki
Success! Data written to: auth/k8s-pki/role/r-k8s-pki

Secret Engine

Again, like in Part 2, we will create a secret engine. But this time of type PKI.

/ $ vault secrets enable -path pki-vraccoon pki
Success! Enabled the pki secrets engine at: pki-vraccoon/

Next we will change the maximum lease time

/ $ vault secrets tune -max-lease-ttl=43800h pki-vraccoon
Success! Tuned the secrets engine at: pki-vraccoon/

Next, we will create a CSR for an intermediate CA. It is also possible to create a new Root CA. But I want the Vault-CA to be a subordinate of my actual root-ca.

/ $ vault write pki-vraccoon/intermediate/generate/internal common_name="vRaccoon Vault SubCA Demo" ttl=43800h

Key       Value
---       -----
key_id    e64d1762-049f-fd06-c4d7-5ba5a3f9392d

I will now take this CSR and get it signed by my root CA. Afterwards I’ll save he complete chain (intermediate ca cert –> root ca cert) as subca.crt.

/ $ cat /tmp/subca.crt 
<subca cert shortened for brevity>
<root ca cert shortened for brevity>

Now we can import the SubCA

/ $  vault write pki-vraccoon/intermediate/set-signed certificate=@/tmp/subca.crt
WARNING! The following warnings were returned from Vault:

  * This mount hasn't configured any authority information access (AIA)
  fields; this may make it harder for systems to find missing certificates
  in the chain or to validate revocation status of certificates. Consider
  updating /config/urls or the newly generated issuer with this information.

Key                 Value
---                 -----
existing_issuers    <nil>
existing_keys       <nil>
imported_issuers    [8ca0137e-a51f-b34a-d907-5f67557b8bb6]
imported_keys       <nil>
mapping             map[8ca0137e-a51f-b34a-d907-5f67557b8bb6:]

Next, set the issuing and revoke url

~ $ vault write pki-vraccoon/config/urls issuing_certificates="http://vault.vault.svc.cluster.local:8200/v1/pki_int/ca" crl_distribution_p
Key                        Value
---                        -----
crl_distribution_points    [http://vault.vault.svc.cluster.local:8200/v1/pki_int/crl]
enable_templating          false
issuing_certificates       [http://vault.vault.svc.cluster.local:8200/v1/pki_int/ca]
ocsp_servers               []

Next we can create the PKI role. Don’t get this confused with the role in the authentication methods. This PKI role defines what sort of csr can be signed. For example it defines what issuer to be used (in case you got multiple), what domains are allowed, key usage, etc.

~ $ vault write pki-vraccoon/roles/vraccoon.lab allowed_domains=vraccoon.lab allow_subdomains=true max_ttl=72h
WARNING! The following warnings were returned from Vault:

  * Issuing Certificate was set to default, but no default issuing certificate
  (configurable at /config/issuers) is currently set

Key                                   Value
---                                   -----
allow_any_name                        false
allow_bare_domains                    false
allow_glob_domains                    false
allow_ip_sans                         true
allow_localhost                       true
allow_subdomains                      true
allow_token_displayname               false
allow_wildcard_certificates           true
allowed_domains                       [vraccoon.lab]
allowed_domains_template              false
allowed_other_sans                    []
allowed_serial_numbers                []
allowed_uri_sans                      []
allowed_uri_sans_template             false
allowed_user_ids                      []
basic_constraints_valid_for_non_ca    false
client_flag                           true
cn_validations                        [email hostname]
code_signing_flag                     false
country                               []
email_protection_flag                 false
enforce_hostnames                     true
ext_key_usage                         []
ext_key_usage_oids                    []
generate_lease                        false
issuer_ref                            default
key_bits                              2048
key_type                              rsa
key_usage                             [DigitalSignature KeyAgreement KeyEncipherment]
locality                              []
max_ttl                               72h
no_store                              false
not_after                             n/a
not_before_duration                   30s
organization                          []
ou                                    []
policy_identifiers                    []
postal_code                           []
province                              []
require_cn                            true
server_flag                           true
signature_bits                        256
street_address                        []
ttl                                   0s
use_csr_common_name                   true
use_csr_sans                          true
use_pss                               false


Last, we need to configure the Vault policy, which allows the user from the authentication method (in our case K8s Service Account sa-issuer) to access the paths of our PKI Secret Engine. More specific the path of our PKI Role within that Secret Engine.

vault policy write p-pki - <<EOF
path "pki-vraccoon*"                       { capabilities = ["read", "list"] }
path "pki-vraccoon/sign/r-vraccoon.lab"    { capabilities = ["create", "update"] }
path "pki-vraccoon/issue/r-vraccoon.lab"   { capabilities = ["create"] }

K8s Configuration

With the Vault config beeing finished, we can continue to configure K8s.


Installing cert-manager via HELM is pretty straight forward. Just adding the repo and install the Chart. No big customisations needed. We only need to check the flag for installing CRDs (like “issuer”).

❯ helm repo add jetstack https://charts.jetstack.io                                                                                                                                                                                                                                                                                 
"jetstack" has been added to your repositories

❯ helm install cert-manager --namespace cert-manager jetstack/cert-manager --create-namespace --set installCRDs=true

NAME: cert-manager
LAST DEPLOYED: Mon Sep 11 14:53:29 2023
NAMESPACE: cert-manager
STATUS: deployed
cert-manager v1.12.2 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:


For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`



The issuer is the connection from cert-manager to the signing authority. In contrast to the clusterissuer, the issuer is an namespaced object. Thus we need to create the namespace first. Additionally, we need to create the servicAccount thats going to be used to connect to Vault.

❯ kubectl create namespace ns-pki
namespace/ns-pki created

❯ kubectl -n ns-pki create serviceaccount sa-issuer
serviceaccount/sa-issuer created

Now we can create the actual issuer object:

❯ kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
  name: vault-issuer
  namespace: ns-pki
        role: r-k8s-pki
        mountPath: /v1/auth/k8s-pki
          name: sa-issuer
    server: http://vault.vault.svc.cluster.local:8200
    path: pki-vraccoon/sign/r-vraccoon.lab

issuer.cert-manager.io/vault-issuer created

The Issuer is created, and we can check its status:

❯ kubectl -n ns-pki get issuers -o wide
NAME           READY   STATUS                                                                                                                                                                                                                                                                                                                                                      AGE
vault-issuer   False   Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: while requesting a token for the service account ns-pki/sa-issuer: serviceaccounts "sa-issuer" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "serviceaccounts/token" in API group "" in the namespace "ns-pki"   24s

As you can see, the issuer failed.
The reason is, that the cert-manager serviceAccount is trying to request a token for service account sa-issuer (which we granted permission in Vault authentication method).
But the cert-manager serviceAccount does not have the permission to get a token. So we need to create a rolebinding for it.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
  name: r-vault-issuer
  namespace: ns-pki
  - apiGroups: ['']
    resources: ['serviceaccounts/token']
    resourceNames: ['sa-issuer']
    verbs: ['create']
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
  name: rb-vault-issuer
  namespace:  ns-pki
  - kind: ServiceAccount
    name: cert-manager
    namespace: cert-manager
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: r-vault-issuer

The role above grants permissions to create serviceaccount tokens. And it limits it to service account sa-issuer. Since its a role which is namespaced, its limited to the sa-issuer in the namespace ns-pki. This role then is assigned to serviceAccount cert-manager in namespace cert-manager.
Let’s apply this and check the issuer again.

❯ kubectl create -f rolebinding.yaml
role.rbac.authorization.k8s.io/r-vault-issuer created
rolebinding.rbac.authorization.k8s.io/rb-vault-issuer created

❯ k get issuer vault-issuer -o wide
NAME           READY   STATUS           AGE
vault-issuer   True    Vault verified   31m

As you can see, the issuer is now ready and Vault verified.
Btw – secretless authentication is the preferred method since Kubernetes 1.24. And this is only supported by cert-manager v1.12 onwards.

Certificate Request

Finally, we can request a certificate. Following an example. As always, there is much more you could configure.

❯ kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
  name: cert-demo
  namespace: ns-pki
  secretName: cert-demo.vraccoon.lab
    name: vault-issuer
  commonName: cert-demo.vraccoon.lab
  - cert-demo.vraccoon.lab

certificate.cert-manager.io/cert-demo created

Let’s check the status of the certificate:

❯ kubectl -n ns-pki get certificate -o wide
NAME        READY   SECRET                    ISSUER         STATUS                                          AGE
cert-demo   True    cert-demo.vraccoon.lab    vault-issuer   Certificate is up to date and has not expired   37s

This looks good. We can also check the corresponding secret:

k -n ns-pki describe secret cert-demo.vraccoon.lab
Name:         cert-demo.vraccoon.lab
Namespace:    ns-pki
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: cert-demo.vraccoon.lab
              cert-manager.io/certificate-name: cert-demo
              cert-manager.io/common-name: cert-demo.vraccoon.lab
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: vault-issuer

Type:  kubernetes.io/tls

ca.crt:   1245 bytes
tls.crt:  3273 bytes
tls.key:  1679 bytes

It’s looking good too.

Final words

This concludes my Vault Series. I agree, that my demonstrated scenarios where quite detached from real world use cases. But my goal was to learn and show the basics and create an understanding of it.

In a real world, cert-manager would probably watch ingress objects beeing created and automatically create certifcate objects based on the fqdn listed in the ingress object.
Vault would automatically rotate certificates as needed and update the secrets in K8s seemless.
If you think this further, DNS entries would most like be created automatically based on the ingress fqdn too. This could be performed by external-dns, NSX ALB, or similar.

I hope this short trip into the Vault helped!

Leave a Reply

Your email address will not be published. Required fields are marked *