Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking

This Post is part of a series:

Deploy vSphere with Kubernetes (Project Pacific) – Part 1: Inventory and Design
–> Deploy vSphere with Kubernetes (Project Pacific) – Part 2: Prepare base networking
Deploy vSphere with Kubernetes (Project Pacific) – Part 3: Prepare base storage (FreeNAS)
Deploy vSphere with Kubernetes (Project Pacific) – Part 4: Install NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 5: Configure NSX-T
Deploy vSphere with Kubernetes (Project Pacific) – Part 6: Some basic testing
Deploy vSphere with Kubernetes (Project Pacific) – Part 7: Enable Workload Management
Deploy vSphere with Kubernetes (Project Pacific) – Part 8: Deploy first Workloads

In the first part, we went through the different deployment models, the hardware I’m using and decided on a rough design.
Part 2 is all about the Network. I will show you how I’ve configured my physical Switch and how the ESXi Hosts are connected to the vDS.

Configure Networking

Please note that the Networks Overlay-ESXi, Overlay-Edge, and NSX-Uplink are propagated to the NSX, but do not have their own PortGroups on the vDS.
Overlay-ESXi is managed by NSX, for Overlay-EDGE and NSX-Uplink, we are going to connect the EDGE Appliance to the Trunk Portgroup (in order to be able to manage the VLAN Tags from NSX).

Required Networks

NetworkSubnetVLAN IDvDS PortgroupDescription
Management172.31.1.0/241 (untagged)vDS-Collapsed-MGMTManagement Network for vmk, vCenter, NSX Manager, EDGEs,…
vMotion172.31.2.0/242vDS-Collapsed-vMotionShared vMotion network accross Clusters
NFS172.31.3.0/243vDS-Collapsed-NFSFreenas, vmk for NFS
VSAN172.31.4.0/244vDS-Collapsed-VSANManagement Cluster, VSAN vmk
Overlay-ESXi172.31.50.0/2450Overlay Network for ESXi; no Portgroup needed, as it is going to be managed by NSX
Overlay-EDGE172.31.51.0/2451Overlay Network for EDGEs; no Portgroup needed, as it is going to be managed by NSX
NSX Uplink172.31.60.0/2460Subnet for Northbound Connectivity from T0 Routers
vDS-Collapsed-TrunkPortgroup with all VLANs, used for NSX Edges

Switch Config

Following the Switch-interface configurations. Nothing special, just setting them up as trunk ports, allowing all VLANs and enabling STP Portfast.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cisco-sg300#show run interface GE 1-10
interface GigabitEthernet1
description CESXi1-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet2
description CESXi1-vmnic1
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet3
description CESXi2-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet4
description CESXi2-vmnic1
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet5
description CESXi3-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet6
description CESXi3-vmnic1
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description "switch "
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet7
description MESXi1-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet8
description MESXi2-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description switch
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet9
description MESXi3-vmnic0
spanning-tree portfast
spanning-tree link-type point-to-point
switchport mode trunk
macro description "switch "
!next command is internal.
macro auto smartport dynamic_type switch
!
interface GigabitEthernet10
description UPLINK
service-acl output ACL-Restrict-Internet-from-lab default-action deny-any
switchport mode trunk
switchport trunk allowed vlan 150
!
cisco-sg300#show run interface GE 1-10 interface GigabitEthernet1 description CESXi1-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet2 description CESXi1-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet3 description CESXi2-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet4 description CESXi2-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet5 description CESXi3-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet6 description CESXi3-vmnic1 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description "switch " !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet7 description MESXi1-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet8 description MESXi2-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description switch !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet9 description MESXi3-vmnic0 spanning-tree portfast spanning-tree link-type point-to-point switchport mode trunk macro description "switch " !next command is internal. macro auto smartport dynamic_type switch ! interface GigabitEthernet10 description UPLINK service-acl output ACL-Restrict-Internet-from-lab default-action deny-any switchport mode trunk switchport trunk allowed vlan 150 !
cisco-sg300#show run interface GE 1-10

interface GigabitEthernet1
 description CESXi1-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet2
 description CESXi1-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet3
 description CESXi2-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet4
 description CESXi2-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet5
 description CESXi3-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet6
 description CESXi3-vmnic1
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description "switch "
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet7
 description MESXi1-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet8
 description MESXi2-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet9
 description MESXi3-vmnic0
 spanning-tree portfast
 spanning-tree link-type point-to-point
 switchport mode trunk
 macro description "switch  "
 !next command is internal.
 macro auto smartport dynamic_type switch
!
interface GigabitEthernet10
 description UPLINK
 service-acl output ACL-Restrict-Internet-from-lab default-action deny-any
 switchport mode trunk
 switchport trunk allowed vlan 150
!

Following, the VLANs created. (You can ignore Backend, its just the Uplink from my Lab to my Home Network 😉 )

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cisco-sg300#sh vlan
Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN
Vlan Name Tagged Ports UnTagged Ports Created by
---- ----------------- ------------------ ------------------ ----------------
1 MGMT gi1-9,Po1-8 DV
2 vMotion gi1-9 S
3 NFS gi1-9 S
4 VSAN gi1-9 S
50 Overlay-ESXi gi1-9 S
51 Overlay-EDGE gi1-9 S
60 NSX-UPLINK gi1-9 S
150 BACKEND gi1-10 S
cisco-sg300#sh vlan Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN Vlan Name Tagged Ports UnTagged Ports Created by ---- ----------------- ------------------ ------------------ ---------------- 1 MGMT gi1-9,Po1-8 DV 2 vMotion gi1-9 S 3 NFS gi1-9 S 4 VSAN gi1-9 S 50 Overlay-ESXi gi1-9 S 51 Overlay-EDGE gi1-9 S 60 NSX-UPLINK gi1-9 S 150 BACKEND gi1-10 S
cisco-sg300#sh vlan
Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN

Vlan       Name           Tagged Ports      UnTagged Ports      Created by
---- ----------------- ------------------ ------------------ ----------------
 1         MGMT                              gi1-9,Po1-8            DV
 2        vMotion            gi1-9                                  S
 3          NFS              gi1-9                                  S
 4         VSAN              gi1-9                                  S
 50    Overlay-ESXi          gi1-9                                  S
 51    Overlay-EDGE          gi1-9                                  S
 60     NSX-UPLINK           gi1-9                                  S
150       BACKEND            gi1-10                                 S

I’ve created an SVI for each VLAN on the Switch, using the first IP in every subnet. They also serve als default Gateways in their respective subnets, since the Cisco SG350 is handling all the non-nsx routing.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cisco-sg300#show ip int
IP Address I/F I/F Status Type Directed Prec Redirect Status
admin/oper Broadcast
------------------ --------- ---------- ------- --------- ---- -------- ------
172.16.0.3/29 vlan 150 UP/UP Static disable No enable Valid
172.31.1.1/24 vlan 1 UP/UP Static disable No enable Valid
172.31.2.1/24 vlan 2 UP/UP Static disable No enable Valid
172.31.3.1/24 vlan 3 UP/UP Static disable No enable Valid
172.31.4.1/24 vlan 4 UP/UP Static disable No enable Valid
172.31.50.1/24 vlan 50 UP/UP Static disable No enable Valid
172.31.51.1/24 vlan 51 UP/UP Static disable No enable Valid
172.31.60.1/24 vlan 60 UP/UP Static disable No enable Valid
cisco-sg300#show ip int IP Address I/F I/F Status Type Directed Prec Redirect Status admin/oper Broadcast ------------------ --------- ---------- ------- --------- ---- -------- ------ 172.16.0.3/29 vlan 150 UP/UP Static disable No enable Valid 172.31.1.1/24 vlan 1 UP/UP Static disable No enable Valid 172.31.2.1/24 vlan 2 UP/UP Static disable No enable Valid 172.31.3.1/24 vlan 3 UP/UP Static disable No enable Valid 172.31.4.1/24 vlan 4 UP/UP Static disable No enable Valid 172.31.50.1/24 vlan 50 UP/UP Static disable No enable Valid 172.31.51.1/24 vlan 51 UP/UP Static disable No enable Valid 172.31.60.1/24 vlan 60 UP/UP Static disable No enable Valid
cisco-sg300#show ip int

    IP Address        I/F    I/F Status  Type   Directed  Prec Redirect Status
                             admin/oper         Broadcast
------------------ --------- ---------- ------- --------- ---- -------- ------
172.16.0.3/29      vlan 150  UP/UP      Static  disable   No   enable   Valid
172.31.1.1/24      vlan 1    UP/UP      Static  disable   No   enable   Valid
172.31.2.1/24      vlan 2    UP/UP      Static  disable   No   enable   Valid
172.31.3.1/24      vlan 3    UP/UP      Static  disable   No   enable   Valid
172.31.4.1/24      vlan 4    UP/UP      Static  disable   No   enable   Valid
172.31.50.1/24     vlan 50   UP/UP      Static  disable   No   enable   Valid
172.31.51.1/24     vlan 51   UP/UP      Static  disable   No   enable   Valid
172.31.60.1/24     vlan 60   UP/UP      Static  disable   No   enable   Valid

Last, but probably most forgotten – enabling Jumbo Frames:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cisco-sg300#configure t
cisco-sg300(config)#port jumbo-frame
This setting will take effect only after copying running configuration to startup configuration and resetting the device
cisco-sg300#configure t cisco-sg300(config)#port jumbo-frame This setting will take effect only after copying running configuration to startup configuration and resetting the device
cisco-sg300#configure t
cisco-sg300(config)#port jumbo-frame
This setting will take effect only after copying running configuration to startup configuration and resetting the device

ESXi Host Network Config

Following, a little sketch on how the ESXi Host VMKs are connected to the vDS Switch.

While configuring the vDS Switch (and migrating / creating the VMKs), there is only one important thing: Configure MTU of at least 1600 on the vDS.

Next – Prepare base storage (FreeNAS)

Leave a Reply

Your email address will not be published. Required fields are marked *