Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking

This Post is part of a series:

Deploy vSphere with Kubernetes (Project Pacific) – Part 1: Inventory and Design
–> Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking
Deploy vSphere with Kubernetes (Project Pacific) – Part 3: Prepare base storage (FreeNAS)
Deploy vSphere with Kubernetes (Project Pacific) – Part 4: Install NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 5: Configure NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 6: Some basic testing
Deploy vSphere with Kubernetes (Project Pacific) – Part 7: Enable Workload Management
Deploy vSphere with Kubernetes (Project Pacific) – Part 8: Deploy first Workloads

In the first part, we went through the different deployment models, the hardware I’m using and decided on a rough design.
Part 2 is all about the Network. I will show you how I’ve configured my physical Switch and how the ESXi Hosts are connected to the vDS.

Configure Networking

Please note that the Networks Overlay-ESXi, Overlay-Edge, and NSX-Uplink are propagated to the NSX, but do not have their own PortGroups on the vDS.
Overlay-ESXi is managed by NSX, for Overlay-EDGE and NSX-Uplink, we are going to connect the EDGE Appliance to the Trunk Portgroup (in order to be able to manage the VLAN Tags from NSX).

Required Networks

NetworkSubnetVLAN IDvDS PortgroupDescription
Management172.31.1.0/241 (untagged)vDS-Collapsed-MGMTManagement Network for vmk, vCenter, NSX Manager, EDGEs,…
vMotion172.31.2.0/242vDS-Collapsed-vMotionShared vMotion network accross Clusters
NFS172.31.3.0/243vDS-Collapsed-NFSFreenas, vmk for NFS
VSAN172.31.4.0/244vDS-Collapsed-VSANManagement Cluster, VSAN vmk
Overlay-ESXi172.31.50.0/2450Overlay Network for ESXi; no Portgroup needed, as it is going to be managed by NSX
Overlay-EDGE172.31.51.0/2451Overlay Network for EDGEs; no Portgroup needed, as it is going to be managed by NSX
NSX Uplink172.31.60.0/2460Subnet for Northbound Connectivity from T0 Routers
vDS-Collapsed-TrunkPortgroup with all VLANs, used for NSX Edges

Switch Config

Following the Switch-interface configurations. Nothing special, just setting them up as trunk ports, allowing all VLANs and enabling STP Portfast.

cisco-sg300#show run interface GE 1-10

interface GigabitEthernet1
 description CESXi1-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet2
 description CESXi1-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet3
 description CESXi2-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet4
 description CESXi2-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet5
 description CESXi3-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet6
 description CESXi3-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description "switch "
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet7
 description MESXi1-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet8
 description MESXi2-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet9
 description MESXi3-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description "switch  "
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet10
 description UPLINK
 service-acl output ACL-Restrict-Internet-from-lab default-action deny-any
 switchport mode trunk
 switchport trunk allowed vlan 150
!

Following, the VLANs created. (You can ignore Backend, its just the Uplink from my Lab to my Home Network 😉 )

cisco-sg300#sh vlan
Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN

Vlan       Name           Tagged Ports      UnTagged Ports      Created by
---- ----------------- ------------------ ------------------ ----------------
 1         MGMT                              gi1-9,Po1-8            DV
 2        vMotion            gi1-9                                  S
 3          NFS              gi1-9                                  S
 4         VSAN              gi1-9                                  S
 50    Overlay-ESXi          gi1-9                                  S
 51    Overlay-EDGE          gi1-9                                  S
 60     NSX-UPLINK           gi1-9                                  S
150       BACKEND            gi1-10                                 S

I’ve created an SVI for each VLAN on the Switch, using the first IP in every subnet. They also serve als default Gateways in their respective subnets, since the Cisco SG350 is handling all the non-nsx routing.

cisco-sg300#show ip int

    IP Address        I/F    I/F Status  Type   Directed  Prec Redirect Status
                             admin/oper         Broadcast
------------------ --------- ---------- ------- --------- ---- -------- ------
172.16.0.3/29      vlan 150  UP/UP      Static  disable   No   enable   Valid
172.31.1.1/24      vlan 1    UP/UP      Static  disable   No   enable   Valid
172.31.2.1/24      vlan 2    UP/UP      Static  disable   No   enable   Valid
172.31.3.1/24      vlan 3    UP/UP      Static  disable   No   enable   Valid
172.31.4.1/24      vlan 4    UP/UP      Static  disable   No   enable   Valid
172.31.50.1/24     vlan 50   UP/UP      Static  disable   No   enable   Valid
172.31.51.1/24     vlan 51   UP/UP      Static  disable   No   enable   Valid
172.31.60.1/24     vlan 60   UP/UP      Static  disable   No   enable   Valid

Last, but probably most forgotten – enabling Jumbo Frames:

cisco-sg300#configure t
cisco-sg300(config)#port jumbo-frame
This setting will take effect only after copying running configuration to startup configuration and resetting the device

ESXi Host Network Config

Following, a little sketch on how the ESXi Host VMKs are connected to the vDS Switch.

While configuring the vDS Switch (and migrating / creating the VMKs), there is only one important thing: Configure MTU of at least 1600 on the vDS.

Next – Prepare base storage (FreeNAS)

Leave a Reply

Your email address will not be published. Required fields are marked *