Tanzu – LoadBalancer IP from specific IP Pool

Note: When I’m talking about Ingress in this article, I’m not referring to the K8s term, which is L7 LoadBalancing. Instead I’m referring to the NSX-T / vSphere with Tanzu term, which defines an entry area for your traffic e.g. in form of LoadBalancers.

Sometimes you got special services in your K8s clusters, that requires specials IPs to be exposed, that are not following your global settings.
With vSphere with Tanzu + NSX-T you got different options to do that.
During the installation of your Supervisor Cluster, you set two global pools – Ingress and Egress. Ingress is the pool, that is globally used for your L4 LoadBalancers until you specify something else.

If you create a new (Supervisor) Namespace, you got the option to “Override cluster network settings”.

With that, you can (next to other parameters) change the Ingress pool for this namespace. In that case, every LoadBalancer IP get assigned from that pool (no matter if for VMService, your K8s Endpoints or K8s-Service of type LoadBalancer).
But what, if you are good with the global pool and only have one (or a few) services, that are supposed to be served by another IP pool? Building an entire (Supervisor) Namespace for it seems a bit much (and would waste even more of your special IPs).

My Environment

  • NSX-T: 3.2.0.1.0.19232396
  • vSphere: 7.0U3g (Build 20150588)
  • vSphere Namespaces: 0.0.17-20026652
  • SupervisorCluster: v1.22.6+vmware.1-vsc0.0.17-20026652
  • Global Ingress: 172.31.250.0/24
  • Global Egress: 172.31.251.0/24
  • Demo Ingress: 172.31.252.0/24

HowTo

NSX-T IP Pool

First I create a new IP Pool in NSX-T with my to be assigned IPs.
Within NSX-T –> Networking (1) –> IP Address Pools (2) –> ADD IP ADDRESS POOL (3)

Give it a name (I call it “ipp-k8s-demo“) and click SET (4) below Subnets.

Click ADD SUBNET (1) –> IP Range (2) and fill out IP Range (3) and CIDR (4). Then click ADD to finish, followed by APPLY and SAVE.

Supervisor Namespace

Next, create a Namespace without any override, thus it’s using the global network settings. I call it “sns-ip-demo”.

Since I’m going to demo it with a vSphere Pod, I only need to grant permissions to a user. No Storage or VMClass needed.

Create Pod and Service

First, login and change into the context sns-ip-demo (this context isn’t special to any other context, it is just where I want to place my demo).

╭─vraccoon@home ~
╰─➤  kubectl vsphere login --server 172.31.250.7 -u administrator@vsphere.local  --insecure-skip-tls-verify                                           1 ↵
Welcome to Photon 3.0 (\m) - Kernel \r (\l)

Logged in successfully.

You have access to the following contexts:
   172.31.250.7
   sns-backend
   sns-ip-demo

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`
╭─vraccoon@home ~
╰─➤  k config use-context sns-ip-demo
Switched to context "sns-ip-demo".

Next, I’ll create a Pod with a webserver in it (the webserver doesn’t matter, it’s just for demo purposes). I like to take the tutum/hello-world. It’s super old, but also super lightweight.

╭─vraccoon@home ~
╰─➤  kubectl run pod-ip-demo --image tutum/hello-world
pod/pod-ip-demo created
╭─vraccoon@home ~
╰─➤  kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
pod-ip-demo   1/1     Running   0          12s

Next, we’ll create the server. We are not doing this quick and dirty, like the pod. But we are taking a shortcut 🙂

╭─vraccoon@home ~
╰─➤  kubectl expose pod pod-ip-demo --type LoadBalancer --port 80 --name svc-ip-demo --dry-run=client -o yaml > svc-ip-demo.yaml

This command basically describes a service of type LoadBalancer, that is listening on port 80 and is called svc-ip-demo. But instead of actually creating it, we are only doing a dry-run and redirect it into a yaml file, where we gonna add something. The YAML Files looks as follows:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    run: pod-ip-demo
  name: svc-ip-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: pod-ip-demo
  type: LoadBalancer
  loadBalancerIP: ipp-k8s-demo
status:
  loadBalancer: {}

I’ve added line 16. Usually you would use the spec.loadBalancerIP key to specify an IP address. But with our NSX-T setup we can also enter the IP pool name, we created earlier.
Next, create the service and verify.

╭─vraccoon@home ~
╰─➤  k create -f svc-ip-demo.yaml
service/svc-ip-demo created
╭─vraccoon@home ~
╰─➤  k get svc
NAME          TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
svc-ip-demo   LoadBalancer   10.96.0.215   <pending>     80:31732/TCP   96s

Troubleshoot

You might have noticed, for some reason the service’ EXTERNAL IP stays at <pending>.
Let’s check the ncp logs for our ip pool.
Line 4 gives us the hint – we have no access to the IP pool.

╭─vraccoon@home ~
╰─➤  kubectl -n vmware-system-nsx logs nsx-ncp-7fd86bcfb5-w4446 | grep ipp-k8s-demo
[ncp GreenThread-72 E errorCode="NCP00096"] nsx_ujo.ncp.nsx.policy.nsxapi Failed to find valid ip pool ipp-k8s-demo
[ncp GreenThread-72 W] nsx_ujo.ncp.k8s.service_lb_controller Failed to process L4 Lb service bdad0ffd-cf8a-4cb7-9f59-4921d8602843: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328 has no permission to access LB_POOL: ipp-k8s-demo
ValueError: 'ipp-k8s-demo' does not appear to be an IPv4 or IPv6 address
nsx_ujo.common.exceptions.PoolAccessDenied: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328 has no permission to access LB_POOL: ipp-k8s-demo
ValueError: 'ipp-k8s-demo' does not appear to be an IPv4 or IPv6 address
nsx_ujo.common.exceptions.PoolAccessDenied: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328 has no permission to access LB_POOL: ipp-k8s-demo
ValueError: 'ipp-k8s-demo' does not appear to be an IPv4 or IPv6 address
nsx_ujo.common.exceptions.PoolAccessDenied: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328 has no permission to access LB_POOL: ipp-k8s-demo
[ncp GreenThread-150 E errorCode="NCP00096"] nsx_ujo.ncp.nsx.policy.nsxapi REPEATS: 1 repeats in 3 sec: Failed to find valid ip pool ipp-k8s-demo
[ncp GreenThread-150 W] nsx_ujo.ncp.k8s.service_lb_controller REPEATS: 1 repeats in 3 sec: Failed to process L4 Lb service bdad0ffd-cf8a-4cb7-9f59-4921d8602843: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328 has no permission to access LB_POOL: ipp-k8s-demo

NSX-T permissions

To fix this, we need to take a note of our “name”, which starts with cluster:domain-… and go back to our IP pool in NSX-T.
Select the Edit on the IP pool. We need to add the following tag:
Scope: ncp/owner
Tag: cluster:domain-c1006:b1476f50-652f-4237-ac88-8514054d5328  (<– the name we captured before)

Verify

We could wait and it should eventually assign an IP, as it keeps trying to access the pool. But since I’m impatient, I’ll delete and recreate the service

╭─vraccoon@home ~
╰─➤  kubectl delete -f svc-ip-demo.yaml
service "svc-ip-demo" deleted
╭─vraccoon@home ~
╰─➤  kubectl create -f svc-ip-demo.yaml
service/svc-ip-demo created
╭─vraccoon@home ~
╰─➤  k get svc
NAME          TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
svc-ip-demo   LoadBalancer   10.96.0.126   172.31.252.1   80:32266/TCP   3s

And finally do a curl on the IP to check whether it’s working as expected:

╭─vraccoon@home ~
╰─➤  curl 172.31.252.1
<html>
<head>
        <title>Hello world!</title>
        <link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
        <style>
        body {
                background-color: white;
                text-align: center;
                padding: 50px;
                font-family: "Open Sans","Helvetica Neue",Helvetica,Arial,sans-serif;
        }

        #logo {
                margin-bottom: 40px;
        }
        </style>
</head>
<body>
        <img id="logo" src="logo.png" />
        <h1>Hello world!</h1>
                                <h3>Links found</h3>
                                        <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br />
                                </body>
</html>

Leave a Reply

Your email address will not be published.