SSH (root) into the nested Tanzu Kubernetes Clusters
If you are running vSphere 7 with Kubernetes, you basically have two different kinds of Kubernetes Clusters – the first one is the Supervisor Cluster, which (as the name suggests) kind of controls everything.
The second kind is the Tanzu Kubernetes Cluster, which is a nested Cluster provisioned by the Supervisor Cluster through ClusterAPI.
The Tanzu Kubernetes Cluster consists of VMs. And the day might come, where you have to connect to these VMs (e.g. for troubleshooting).
Authenticating against the Kubernetes service inside these VMs is super easy (using kubectl vSphere login […] ). But if you want to SSH into these VMs, it gets a bit tricky. Not complicated though, just not obvious.
My Environment
I’ve a Supervisor Cluster called “COMP”.
In there is a Namespace called “prod”.
In this Namespace is a Tanzu Kubernetes cluster called “tkg-prod”.
This Tanzu Kubernetes Cluster consists of three Masters and three Workers.
Let’s take a look at this from the CLI. I’ve logged in with user “administrator@vsphere.local” to the Namespace “prod”.
vraccoon@ubu:~$ kubectl get tanzukubernetesclusters.run.tanzu.vmware.com NAME CONTROL PLANE WORKER DISTRIBUTION AGE PHASE tkg-prod 3 3 v1.17.8+vmware.1-tkg.1.5417466 122m running vraccoon@ubu:~$ kubectl -n prod get virtualmachines.vmoperator.vmware.com NAME AGE tkg-prod-control-plane-2ng7d 122m tkg-prod-control-plane-cbjb6 109m tkg-prod-control-plane-ggz5x 109m tkg-prod-workers-bfmhv-66bdd9bb44-drpp5 109m tkg-prod-workers-bfmhv-66bdd9bb44-xzwpg 109m tkg-prod-workers-bfmhv-66bdd9bb44-zl4st 109m
Dig a bit deeper on the VM Details:
vraccoon@ubu:~$ kubectl get virtualmachines.vmoperator.vmware.com -o json | jq -r '[.items[] | {namespace:.metadata.namespace, name:.metadata.name, internalIP: .status.vmIp}]'
[
{
"namespace": "prod",
"name": "tkg-prod-control-plane-2ng7d",
"internalIP": "10.244.1.2"
},
{
"namespace": "prod",
"name": "tkg-prod-control-plane-cbjb6",
"internalIP": "10.244.1.4"
},
{
"namespace": "prod",
"name": "tkg-prod-control-plane-ggz5x",
"internalIP": "10.244.1.3"
},
{
"namespace": "prod",
"name": "tkg-prod-workers-bfmhv-66bdd9bb44-drpp5",
"internalIP": "10.244.1.5"
},
{
"namespace": "prod",
"name": "tkg-prod-workers-bfmhv-66bdd9bb44-xzwpg",
"internalIP": "10.244.1.7"
},
{
"namespace": "prod",
"name": "tkg-prod-workers-bfmhv-66bdd9bb44-zl4st",
"internalIP": "10.244.1.6"
}
]
The Challenges
There are two obstacles we have to overcome here.
- Get Login Credentials
- Actually reach the VMs
Get Login Credentials
The login Credentials are actually saved as Secrets in the Namespace of the nested Tanzu Kubernetes Cluster.
They are called like “<clustername>-ssh” and “<clustername>-ssh-password“.
vraccoon@ubu:~$ kubectl get secrets NAME TYPE DATA AGE default-token-bgswg kubernetes.io/service-account-token 3 127m tkg-prod-ca Opaque 2 122m tkg-prod-ccm-token-b55kd kubernetes.io/service-account-token 3 110m tkg-prod-encryption Opaque 1 123m tkg-prod-etcd Opaque 2 122m tkg-prod-kubeconfig Opaque 1 122m tkg-prod-proxy Opaque 2 122m tkg-prod-pvcsi-token-pcwkv kubernetes.io/service-account-token 3 110m tkg-prod-sa Opaque 2 122m tkg-prod-ssh kubernetes.io/ssh-auth 1 123m tkg-prod-ssh-password Opaque 1 123m
In my case line 13 contains the ssh private key and line 14 the user password. The user in both cases is “vmware-system-user“. It’s up to you, which one you use. Just note, that the content of these Secrets is base64 encoded (as usual with Kubernetes).
Extracting the password:
vraccoon@ubu:~/tmp$ kubectl -n prod get secret tkg-prod-ssh-password -o jsonpath="{@.data.ssh-passwordkey}" | base64 -d
GfrqXuFO845RGdIVszqTmPUKHQQG/8IQRCMcUzMtuI8=
Extracting the SSH-Key and write it into a file:
vraccoon@ubu:~/tmp$ kubectl -n prod get secret tkg-prod-ssh -o jsonpath="{@.data.ssh-privatekey}" | base64 -d > ssh_tkg-prod.priv
vraccoon@ubu:~/tmp$ cat ssh_tkg-prod.priv
-----BEGIN RSA PRIVATE KEY-----
<output omitted>
-----END RSA PRIVATE KEY-----
Building a JumpBox (quick’n’dirty)
Our second challenge is, that these VMs only have internal IP which you don’t reach from outside. In theory you could mess with your routing, but I would not recommend this.
Instead, we follow the VMware recommended way an build a JumpBox within that namespace, so we can reach the internal IP of our Tanzu Kubernetes Cluster VMs.
Create a Pod, that runs SSH and a shell:
vraccoon@ubu:~$ kubectl run ssh-jumpbox --image=kubernetesio/sshd-jumpserver kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/ssh-jumpbox created
Jump into that Container and ssh into the Tanzu Kubernetes Cluster VM, using the password and IP we have gathered earlier:
vraccoon@ubu:~$ kubectl exec -it ssh-jumpbox-6f6ddf58d-269kh bash [root@ssh-jumpbox-6f6ddf58d-269kh /]# ssh vmware-system-user@10.244.1.2 The authenticity of host '10.244.1.2 (10.244.1.2)' can't be established. ECDSA key fingerprint is SHA256:/JsP9yQ0WBj3d3ITWqoN/vq65r1MZv6Hf8+XerbumSI. ECDSA key fingerprint is MD5:a8:5d:8d:c3:18:ad:fb:70:24:fd:b4:21:1b:eb:d0:b3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.244.1.2' (ECDSA) to the list of known hosts. Welcome to Photon 3.0 (\m) - Kernel \r (\l) vmware-system-user@10.244.1.2's password: Last login: Sun Jul 19 15:26:37 2020 from 172.31.60.238 15:32:16 up 2:17, 0 users, load average: 0.56, 0.27, 0.28 tdnf update info not available yet! vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$
As you can tell by the hostname, we are connected to the first Master VMs of our Tanzu Kubernetes Cluster.
Building a JumpBox (professional)
Although the way shown before works, it’s not a nice way. I’ve decided to show that way first, to give a better understanding of how it works.
If you wanna do it a bit more professional, you would use the ssh-private key instead of the password. If you want to use it in a pod, you would either have to copy it (dirty again ^^) or mount the secret into the pod. VMware provides a small manifest with a jumpbox-container which does exactly this:
apiVersion: v1
kind: Pod
metadata:
name: jumpbox
namespace: YOUR-NAMESPACE
spec:
containers:
- image: "photon:3.0"
name: jumpbox
command: [ "/bin/bash", "-c", "--" ]
args: [ "yum install -y openssh-server; mkdir /root/.ssh; cp /root/ssh/ssh-privatekey /root/.ssh/id_rsa; chmod 600 /root/.ssh/id_rsa; while true; do sleep 30; done;" ]
volumeMounts:
- mountPath: "/root/ssh"
name: ssh-key
readOnly: true
volumes:
- name: ssh-key
secret:
secretName: YOUR-CLUSTER-NAME-ssh
This Pod uses the photon image (what else would vmware use ? =D ), mounts the Tanzu Kubernetes Cluster ssh-private-key-secret, copy it’s content to a local directory and install the openssh-server.
All you have to do is modify the following:
Line 5: The Namespace where the Tanzu Kubernetes Cluster sits (in our case “prod“)
Line 19: the name of the ssh-private-key secret (in our case “tkg-prod-ssh“
After modifying the manifest, you can run the pod:
vraccoon@ubu:~/tmp$ kubectl create -f jumpbox.yaml pod/jumpbox created
Instead of first jumping into the pod and then jump again into the Tanzu Kubernetes Cluster VM, you can combine both steps with the following command:
kubectl -n <namespace> exec -it jumpbox /usr/bin/ssh vmware-system-user@<VM IP>
vraccoon@ubu:~/tmp$ kubectl -n prod exec -it jumpbox /usr/bin/ssh vmware-system-user@10.244.1.2 The authenticity of host '10.244.1.2 (10.244.1.2)' can't be established. ECDSA key fingerprint is SHA256:/JsP9yQ0WBj3d3ITWqoN/vq65r1MZv6Hf8+XerbumSI. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.244.1.2' (ECDSA) to the list of known hosts. Welcome to Photon 3.0 (\m) - Kernel \r (\l) Last login: Sun Jul 19 15:32:16 2020 from 172.31.60.238 15:49:14 up 2:34, 0 users, load average: 0.09, 0.27, 0.28 tdnf update info not available yet! vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$
As you can see, we are again connected to our first Master VM.
Getting Root Access
The user “vmware-system-user” is allowed to escalate privileges:
vmware-system-user@tkg-prod-control-plane-2ng7d [ ~ ]$ sudo su - root@tkg-prod-control-plane-2ng7d [ ~ ]#
