Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking
This Post is part of a series:
Deploy vSphere with Kubernetes (Project Pacific) – Part 1: Inventory and Design
–> Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking
Deploy vSphere with Kubernetes (Project Pacific) – Part 3: Prepare base storage (FreeNAS)
Deploy vSphere with Kubernetes (Project Pacific) – Part 4: Install NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 5: Configure NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 6: Some basic testing
Deploy vSphere with Kubernetes (Project Pacific) – Part 7: Enable Workload Management
Deploy vSphere with Kubernetes (Project Pacific) – Part 8: Deploy first Workloads
In the first part, we went through the different deployment models, the hardware I’m using and decided on a rough design.
Part 2 is all about the Network. I will show you how I’ve configured my physical Switch and how the ESXi Hosts are connected to the vDS.
Configure Networking
Please note that the Networks Overlay-ESXi, Overlay-Edge, and NSX-Uplink are propagated to the NSX, but do not have their own PortGroups on the vDS.
Overlay-ESXi is managed by NSX, for Overlay-EDGE and NSX-Uplink, we are going to connect the EDGE Appliance to the Trunk Portgroup (in order to be able to manage the VLAN Tags from NSX).
Required Networks
| Network | Subnet | VLAN ID | vDS Portgroup | Description |
|---|---|---|---|---|
| Management | 172.31.1.0/24 | 1 (untagged) | vDS-Collapsed-MGMT | Management Network for vmk, vCenter, NSX Manager, EDGEs,… |
| vMotion | 172.31.2.0/24 | 2 | vDS-Collapsed-vMotion | Shared vMotion network accross Clusters |
| NFS | 172.31.3.0/24 | 3 | vDS-Collapsed-NFS | Freenas, vmk for NFS |
| VSAN | 172.31.4.0/24 | 4 | vDS-Collapsed-VSAN | Management Cluster, VSAN vmk |
| Overlay-ESXi | 172.31.50.0/24 | 50 | – | Overlay Network for ESXi; no Portgroup needed, as it is going to be managed by NSX |
| Overlay-EDGE | 172.31.51.0/24 | 51 | – | Overlay Network for EDGEs; no Portgroup needed, as it is going to be managed by NSX |
| NSX Uplink | 172.31.60.0/24 | 60 | – | Subnet for Northbound Connectivity from T0 Routers |
| – | – | vDS-Collapsed-Trunk | Portgroup with all VLANs, used for NSX Edges |
Switch Config
Following the Switch-interface configurations. Nothing special, just setting them up as trunk ports, allowing all VLANs and enabling STP Portfast.
cisco-sg300#show run interface GE 1-10 interface GigabitEthernet1 description CESXi1-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet2 description CESXi1-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet3 description CESXi2-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet4 description CESXi2-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet5 description CESXi3-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet6 description CESXi3-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description "switch " !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet7 description MESXi1-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet8 description MESXi2-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet9 description MESXi3-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description "switch " !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet10 description UPLINK service-acl output ACL-Restrict-Internet-from-lab default-action deny-any switchport mode trunk switchport trunk allowed vlan 150 !
Following, the VLANs created. (You can ignore Backend, its just the Uplink from my Lab to my Home Network 😉 )
cisco-sg300#sh vlan Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN Vlan Name Tagged Ports UnTagged Ports Created by ---- ----------------- ------------------ ------------------ ---------------- 1 MGMT gi1-9,Po1-8 DV 2 vMotion gi1-9 S 3 NFS gi1-9 S 4 VSAN gi1-9 S 50 Overlay-ESXi gi1-9 S 51 Overlay-EDGE gi1-9 S 60 NSX-UPLINK gi1-9 S 150 BACKEND gi1-10 S
I’ve created an SVI for each VLAN on the Switch, using the first IP in every subnet. They also serve als default Gateways in their respective subnets, since the Cisco SG350 is handling all the non-nsx routing.
cisco-sg300#show ip int
IP Address I/F I/F Status Type Directed Prec Redirect Status
admin/oper Broadcast
------------------ --------- ---------- ------- --------- ---- -------- ------
172.16.0.3/29 vlan 150 UP/UP Static disable No enable Valid
172.31.1.1/24 vlan 1 UP/UP Static disable No enable Valid
172.31.2.1/24 vlan 2 UP/UP Static disable No enable Valid
172.31.3.1/24 vlan 3 UP/UP Static disable No enable Valid
172.31.4.1/24 vlan 4 UP/UP Static disable No enable Valid
172.31.50.1/24 vlan 50 UP/UP Static disable No enable Valid
172.31.51.1/24 vlan 51 UP/UP Static disable No enable Valid
172.31.60.1/24 vlan 60 UP/UP Static disable No enable Valid
Last, but probably most forgotten – enabling Jumbo Frames:
cisco-sg300#configure t cisco-sg300(config)#port jumbo-frame This setting will take effect only after copying running configuration to startup configuration and resetting the device
ESXi Host Network Config
Following, a little sketch on how the ESXi Host VMKs are connected to the vDS Switch.
While configuring the vDS Switch (and migrating / creating the VMKs), there is only one important thing: Configure MTU of at least 1600 on the vDS.
