Deploy vSphere with Kubernetes (Project Pacific) – Part 3: Prepare base storage (FreeNAS)

This Post is part of a series:

In Part 2, we have configured the base networking. Part 3 takes care of the storage.
Initially, I was thinking about using VSAN for the Workload Cluster, as this is the recommended solution by VMware. Sadly, I neither have a fourth Hades Canyon nor is their hardware config suitable for being a VSAN Cluster (the past use cases never required that).
My next thought was to use another cool new Feature – VSAN File Services (since I’m using VSAN in the MGMT Cluster). Sadly, this caused a lot of weird issues. For example, the NSX-T Edges had trouble starting some services, while the SupervisorControlPlane VMs (SCP) couldn’t get configured properly. I really don’t know how this relates to each other, but after migrating them to the FreeNAS NFS, everything was working fine. To be clear, that was only the case on the VSAN NFS Datastores, not on the “native” VSAN Datastore.
Eventually, I need to dig deeper into this, but that wasn’t my goal here 😉

So I ended up, installing a FreeNAS VM in the Workload Cluster. Not optimal, but ok for a Lab environment.

In this part, we are going to:

Install and Configure FreeNAS
Mount the NFS Datastore
Tag the NFS Datastore and create a Storage Policy

FreeNAS

Install FreeNAS

First, download the FreeNAS ISO from https://www.freenas.org/download/

Then create a new VM.
Configure it as following:

Guest OS Family: Other
Guest OS Version: FreeBSD 12 or later version (64-bit)

For the Hardware, configure (at least) the following:

2x CPU
8 GB Memory
16 GB Disk (for OS)
750 GB Disk (for NFS Share, thin provisioned, might also be smaller)
Select Network vDS-Collapsed-NFS
Mount the FreeNAS ISO

Next, boot the VM, make sure its starting from CD (should be default).
In the Installer Screen choose 1 Install/Upgrade

Select the 16GB disk for the OS installation

Confirm the warning and proceed (its just about erasing data … =) )

Enter Password and continue

Select Boot via BIOS

When the installer finishes, select Reboot System

Configure FreeNAS

After FreeNAS has booted, we need to configure IP Settings on the Console.

Configure Network Interfaces
IP: 172.31.3.131/24

Configure Default Router
IPv4 Default Route: 172.31.3.1

Configure DNS
DNS Domain: cloudborn.lab
DNS Nameserver1: 172.31.1.10

From now on, you should be able to reach the Web GUI.
If so, after logging in, navigate to Storage –> Pools –> Click Add

Select Create new pool and click CREATE POOL

Next, name your pool (“pool-1” in my case), add the available disk to the pool and click CREATE

Next navigate to Sharing –> Unix (NFS) Shares –> Click ADD

Configure the following:
Drill down to the pool-1 directory
Select All dirs

Below ADVANCED MODE
Maproot User: root
Maproot Group: nogroup

Click SAVE.

Add the Datastore to the COMP Cluster

Simply add a new Datastore with the following Parameters:
Type: NFS
NFS Version: 3
Datastore Name: DS-NFS-C1
Folder: /mnt/pool-1
Server: 172.31.3.131
Hosts: All hosts in the COMP Cluster

Tag & Storage Policy

The Supervisor Cluster itself and also everything sitting inside the Namespaces need to have storage assigned (Captain Obvious!).
But you will never assign a Datastore directly to any of the components. Instead, Storage Policies are the way to go.
In the real world, it makes sense to have multiple Storage Policies for SCP VMs, container images, ephemeral disks, PVs,…
Depending on compliance/governance, you might want to have dedicated Storage Policies for each Namespace (=Tenant) or simply provide storage with different performance profiles.

For now, I will only create one Datastore and simply use it for everything.