SSH (root) into the nested Tanzu Kubernetes Clusters

If you are running vSphere 7 with Kubernetes, you basically have two different kinds of Kubernetes Clusters – the first one is the Supervisor Cluster, which (as the name suggests) kind of controls everything.
The second kind is the Tanzu Kubernetes Cluster, which is a nested Cluster provisioned by the Supervisor Cluster through ClusterAPI.

The Tanzu Kubernetes Cluster consists of VMs. And the day might come, where you have to connect to these VMs (e.g. for troubleshooting).
Authenticating against the Kubernetes service inside these VMs is super easy (using kubectl vSphere login […] ). But if you want to SSH into these VMs, it gets a bit tricky. Not complicated though, just not obvious.

My Environment

I’ve a Supervisor Cluster called “COMP”.
In there is a Namespace called “prod”.
In this Namespace is a Tanzu Kubernetes cluster called “tkg-prod”.
This Tanzu Kubernetes Cluster consists of three Masters and three Workers.

Let’s take a look at this from the CLI. I’ve logged in with user “administrator@vsphere.local” to the Namespace “prod”.

vraccoon@ubu:~$ kubectl get tanzukubernetesclusters.run.tanzu.vmware.com
NAME       CONTROL PLANE   WORKER   DISTRIBUTION                     AGE    PHASE
tkg-prod   3               3        v1.17.8+vmware.1-tkg.1.5417466   122m   running

vraccoon@ubu:~$ kubectl -n prod get virtualmachines.vmoperator.vmware.com
NAME                                      AGE
tkg-prod-control-plane-2ng7d              122m
tkg-prod-control-plane-cbjb6              109m
tkg-prod-control-plane-ggz5x              109m
tkg-prod-workers-bfmhv-66bdd9bb44-drpp5   109m
tkg-prod-workers-bfmhv-66bdd9bb44-xzwpg   109m
tkg-prod-workers-bfmhv-66bdd9bb44-zl4st   109m

Dig a bit deeper on the VM Details:

vraccoon@ubu:~$ kubectl get virtualmachines.vmoperator.vmware.com -o json | jq -r '[.items[] | {namespace:.metadata.namespace, name:.metadata.name, internalIP: .status.vmIp}]'
[
  {
    "namespace": "prod",
    "name": "tkg-prod-control-plane-2ng7d",
    "internalIP": "10.244.1.2"
  },
  {
    "namespace": "prod",
    "name": "tkg-prod-control-plane-cbjb6",
    "internalIP": "10.244.1.4"
  },
  {
    "namespace": "prod",
    "name": "tkg-prod-control-plane-ggz5x",
    "internalIP": "10.244.1.3"
  },
  {
    "namespace": "prod",
    "name": "tkg-prod-workers-bfmhv-66bdd9bb44-drpp5",
    "internalIP": "10.244.1.5"
  },
  {
    "namespace": "prod",
    "name": "tkg-prod-workers-bfmhv-66bdd9bb44-xzwpg",
    "internalIP": "10.244.1.7"
  },
  {
    "namespace": "prod",
    "name": "tkg-prod-workers-bfmhv-66bdd9bb44-zl4st",
    "internalIP": "10.244.1.6"
  }
]

The Challenges

There are two obstacles we have to overcome here.

  1. Get Login Credentials
  2. Actually reach the VMs

Get Login Credentials

The login Credentials are actually saved as Secrets in the Namespace of the nested Tanzu Kubernetes Cluster.
They are called like “<clustername>-ssh” and “<clustername>-ssh-password“.

vraccoon@ubu:~$ kubectl get secrets
NAME                         TYPE                                  DATA   AGE
default-token-bgswg          kubernetes.io/service-account-token   3      127m
tkg-prod-ca                  Opaque                                2      122m
tkg-prod-ccm-token-b55kd     kubernetes.io/service-account-token   3      110m
tkg-prod-encryption          Opaque                                1      123m
tkg-prod-etcd                Opaque                                2      122m
tkg-prod-kubeconfig          Opaque                                1      122m
tkg-prod-proxy               Opaque                                2      122m
tkg-prod-pvcsi-token-pcwkv   kubernetes.io/service-account-token   3      110m
tkg-prod-sa                  Opaque                                2      122m
tkg-prod-ssh                 kubernetes.io/ssh-auth                1      123m
tkg-prod-ssh-password        Opaque                                1      123m

In my case line 13 contains the ssh private key and line 14 the user password. The user in both cases is “vmware-system-user“. It’s up to you, which one you use. Just note, that the content of these Secrets is base64 encoded (as usual with Kubernetes).

Extracting the password:

vraccoon@ubu:~/tmp$ kubectl -n prod get secret tkg-prod-ssh-password -o jsonpath="{@.data.ssh-passwordkey}" | base64 -d
GfrqXuFO845RGdIVszqTmPUKHQQG/8IQRCMcUzMtuI8=

Extracting the SSH-Key and write it into a file:

vraccoon@ubu:~/tmp$ kubectl -n prod get secret tkg-prod-ssh -o jsonpath="{@.data.ssh-privatekey}" | base64 -d > ssh_tkg-prod.priv

vraccoon@ubu:~/tmp$ cat ssh_tkg-prod.priv
-----BEGIN RSA PRIVATE KEY-----
<output omitted>
-----END RSA PRIVATE KEY-----

Building a JumpBox (quick’n’dirty)

Our second challenge is, that these VMs only have internal IP which you don’t reach from outside. In theory you could mess with your routing, but I would not recommend this.
Instead, we follow the VMware recommended way an build a JumpBox within that namespace, so we can reach the internal IP of our Tanzu Kubernetes Cluster VMs.

Create a Pod, that runs SSH and a shell:

vraccoon@ubu:~$ kubectl run ssh-jumpbox --image=kubernetesio/sshd-jumpserver
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/ssh-jumpbox created

Jump into that Container and ssh into the Tanzu Kubernetes Cluster VM, using the password and IP we have gathered earlier:

vraccoon@ubu:~$  kubectl exec -it ssh-jumpbox-6f6ddf58d-269kh bash

[root@ssh-jumpbox-6f6ddf58d-269kh /]# ssh vmware-system-user@10.244.1.2
The authenticity of host '10.244.1.2 (10.244.1.2)' can't be established.
ECDSA key fingerprint is SHA256:/JsP9yQ0WBj3d3ITWqoN/vq65r1MZv6Hf8+XerbumSI.
ECDSA key fingerprint is MD5:a8:5d:8d:c3:18:ad:fb:70:24:fd:b4:21:1b:eb:d0:b3.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.244.1.2' (ECDSA) to the list of known hosts.
Welcome to Photon 3.0 (\m) - Kernel \r (\l)
vmware-system-user@10.244.1.2's password:

Last login: Sun Jul 19 15:26:37 2020 from 172.31.60.238
 15:32:16 up  2:17,  0 users,  load average: 0.56, 0.27, 0.28
tdnf update info not available yet!

vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$ 

As you can tell by the hostname, we are connected to the first Master VMs of our Tanzu Kubernetes Cluster.

Building a JumpBox (professional)

Although the way shown before works, it’s not a nice way. I’ve decided to show that way first, to give a better understanding of how it works.
If you wanna do it a bit more professional, you would use the ssh-private key instead of the password. If you want to use it in a pod, you would either have to copy it (dirty again ^^) or mount the secret into the pod. VMware provides a small manifest with a jumpbox-container which does exactly this:

apiVersion: v1
kind: Pod
metadata:
  name: jumpbox
  namespace: YOUR-NAMESPACE
spec:
  containers:
  - image: "photon:3.0"
    name: jumpbox
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "yum install -y openssh-server; mkdir /root/.ssh; cp /root/ssh/ssh-privatekey /root/.ssh/id_rsa; chmod 600 /root/.ssh/id_rsa; while true; do sleep 30; done;" ]
    volumeMounts:
      - mountPath: "/root/ssh"
        name: ssh-key
        readOnly: true
  volumes:
    - name: ssh-key
      secret:
        secretName: YOUR-CLUSTER-NAME-ssh

This Pod uses the photon image (what else would vmware use ? =D ), mounts the Tanzu Kubernetes Cluster ssh-private-key-secret, copy it’s content to a local directory and install the openssh-server.
All you have to do is modify the following:
Line 5: The Namespace where the Tanzu Kubernetes Cluster sits (in our case “prod“)
Line 19: the name of the ssh-private-key secret (in our case “tkg-prod-ssh

After modifying the manifest, you can run the pod:

vraccoon@ubu:~/tmp$ kubectl create -f jumpbox.yaml
pod/jumpbox created

Instead of first jumping into the pod and then jump again into the Tanzu Kubernetes Cluster VM, you can combine both steps with the following command:

kubectl -n <namespace> exec -it jumpbox /usr/bin/ssh vmware-system-user@<VM IP>

vraccoon@ubu:~/tmp$ kubectl -n prod exec -it jumpbox /usr/bin/ssh vmware-system-user@10.244.1.2

The authenticity of host '10.244.1.2 (10.244.1.2)' can't be established.
ECDSA key fingerprint is SHA256:/JsP9yQ0WBj3d3ITWqoN/vq65r1MZv6Hf8+XerbumSI.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.244.1.2' (ECDSA) to the list of known hosts.
Welcome to Photon 3.0 (\m) - Kernel \r (\l)
Last login: Sun Jul 19 15:32:16 2020 from 172.31.60.238
 15:49:14 up  2:34,  0 users,  load average: 0.09, 0.27, 0.28
tdnf update info not available yet!

vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$

As you can see, we are again connected to our first Master VM.

Getting Root Access

The user “vmware-system-user” is allowed to escalate privileges:

vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$ sudo su -
root@tkg-prod-control-plane-2ng7d [ ~ ]#

Leave a Reply

Your email address will not be published.