Create VMFS Datastore on Boot USB Device
For my Home Lab, I wanted to use my internal Hard-drives for vSAN. Thus, I had to install ESXi on an USB Drive. Since I used 64 GB USB Drives, and ESXi requires only 10-15 GB, I wanted to make the remaining space usable as VMFS Datastores. Not to run VMs on it, but for syslog, maybe ISOs, …
This turned out to be a bit more tricky than expected. There are a bunch of great articles out there, that show how to create VMFS Datastores on empty USB drives. But I haven’t found an article on how to install a VMFS datastore next to a ESXi installation on the same USB device.
So, in order to make the remaining space on my USB drive usable as VMFS Datastore, I had to perform the following steps:
1. First stop (and disable) the usbarbitrator service
(Note – this will prevent all USB devices from being passed-through to any VMs)
[root@esxi3:~] /etc/init.d/usbarbitrator stop [root@esxi3:~] chkconfig usbarbitrator off
2. Next, let’s have a look at the current partition table of our USB device, found in /dev/disks/:
[root@esxi3:~] ls -aslh
total 2505694880
0 drwxr-xr-x 2 root root 512 Aug 1 10:31 .
0 drwxr-xr-x 19 root root 512 Aug 1 10:31 ..
60063744 -rw------- 1 root root 57.3G Aug 1 10:31 mpx.vmhba32:C0:T0:L0
4064 -rw------- 1 root root 4.0M Aug 1 10:31 mpx.vmhba32:C0:T0:L0:1
255984 -rw------- 1 root root 250.0M Aug 1 10:31 mpx.vmhba32:C0:T0:L0:5
255984 -rw------- 1 root root 250.0M Aug 1 10:31 mpx.vmhba32:C0:T0:L0:6
112624 -rw------- 1 root root 110.0M Aug 1 10:31 mpx.vmhba32:C0:T0:L0:7
292848 -rw------- 1 root root 286.0M Aug 1 10:31 mpx.vmhba32:C0:T0:L0:8
2789376 -rw------- 1 root root 2.7G Aug 1 10:31 mpx.vmhba32:C0:T0:L0:9
[remaining output omitted]
As you can see, “mpx.vmhba32:C0:T0:L0” is my USB drive. Though, the output is already showing the partitions (lines with “:<number>” at the end) we need a bit more detailed information.
3. List partitions using partedUtil
[root@esxi3:~] partedUtil getptbl /dev/disks/mpx.vmhba32\:C0\:T0\:L0 gpt 7477 255 63 120127488 1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128 5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0 6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0 7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0 8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0 9 1843200 7421951 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
On a high level, the output means the following:
<PartitionNumber> – <StartingSector< – >EndingSector> – <Type(vmfs/vsan/…)> – <Attribute>
If you are interested in further details, check out https://kb.vmware.com/s/article/1036609
From line 9, we know already the last used sector, which is 7421951. That means, our new partition can’t start before sector 7421952.
4. Let’s find out, which is the last usable Sector:
[root@esxi3:~] partedUtil getUsableSectors /dev/disks/mpx.vmhba32\:C0\:T0\:L0 34 120127454
120127454 is the last sector of the device. In order to keep this command working, even after our modifications, we have to substract 1 sector.
Now, we know that our partition can span across sectors 7421952 to 120127453.
5. Next, we need to find out what type we need for our VMFS datastore:
[root@esxi3:~] partedUtil showGuids Partition Type GUID vmfs AA31E02A400F11DB9590000C2911D1B8 vmkDiagnostic 9D27538040AD11DBBF97000C2911D1B8 vsan 381CFCCC728811E092EE000C2911D0B2 virsto 77719A0CA4A011E3A47E000C29745A24 VMware Reserved 9198EFFC31C011DB8F78000C2911D1B8 Basic Data EBD0A0A2B9E5443387C068B6B72699C7 Linux Swap 0657FD6DA4AB43C484E50933C84B4F4F Linux Lvm E6D6D379F50744C2A23C238F2A3DF928 Linux Raid A19D880F05FC4D3BA006743F0F84911E Efi System C12A7328F81F11D2BA4B00A0C93EC93B Microsoft Reserved E3C9E3160B5C4DB8817DF92DF00215AE Unused Entry 00000000000000000000000000000000
In theory, we now have all required information:
Partition Number: 10 (just the next usable)
Start Sector: 7421952
End Sector: 120127453
Type: AA31E02A400F11DB9590000C2911D1B8
Attribute: 0
6. Next, construct the command to create the new partition.
Important – you have to enter the information for all partitions, as the command overwrites the partition table.
[root@esxi3:~] partedUtil setptbl /vmfs/devices/disks/mpx.vmhba32\:C0\:T0\:L0 gpt "1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B 128" "5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 0" "6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 0" "7 1 032224 1257471 9D27538040AD11DBBF97000C2911D1B8 0" "8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 0" "9 1843200 7421951 9D27538040AD11DBBF97000C2911D1B8 0" "10 7421952 120127453 AA31E02A400F11DB9590000C2911D1B8 0" [output omitted] Error: Read-only file system during write on /dev/disks/mpx.vmhba32:C0:T0:L0 SetPtableGpt: Unable to commit to disk
To find out, what went wrong, took me quite some time. But eventually, I stumbled across this thread in the VMware Forum:
https://communities.vmware.com/message/2472738
It outlines, that the issue is caused by the vmkdiagnostics partitions (7 and 9 in our example), which were automatically created during the installation and are used to store coredump information.
7. So, let’s check the partitions being used for coredump:
root@esxi3:~] esxcli system coredump partition list Name Path Active Configured ---------------------- ------------------------------------------ ------ ---------- mpx.vmhba32:C0:T0:L0:7 /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7 false false mpx.vmhba32:C0:T0:L0:9 /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:9 true true
They are indeed.
8. Disable Coredump
esxcli system coredump partition set --enable false
9. Try again to create the new partition table
[root@esxi3:~] partedUtil setptbl /vmfs/devices/disks/mpx.vmhba32\:C0\:T0\:L0 gpt "1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B 128" "5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 0" "6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 0" "7 1 032224 1257471 9D27538040AD11DBBF97000C2911D1B8 0" "8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 0" "9 1843200 7421951 9D27538040AD11DBBF97000C2911D1B8 0" "10 7421952 120127453 AA31E02A400F11DB9590000C2911D1B8 0" gpt 0 0 0 0 1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B 128 5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 0 6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 0 7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 0 8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 0 9 1843200 7421951 9D27538040AD11DBBF97000C2911D1B8 0 10 7421952 120127453 AA31E02A400F11DB9590000C2911D1B8 0
Now it worked.
10. Last thing to do is to create the VMFS6 Datastore
[root@esxi3:~] vmkfstools -C vmfs6 -S ESXi3-LocalUSB /dev/disks/mpx.vmhba32\:C0\:T0\:L0\:10 create fs deviceName:'/dev/disks/mpx.vmhba32:C0:T0:L0:10', fsShortName:'vmfs6', fsName:'ESXi3-LocalUSB' deviceFullPath:/dev/disks/mpx.vmhba32:C0:T0:L0:10 deviceFile:mpx.vmhba32:C0:T0:L0:10 ATS on device /dev/disks/mpx.vmhba32:C0:T0:L0:10: not supported . Checking if remote hosts are using this device as a valid file system. This may take a few seconds... Creating vmfs6 file system on "mpx.vmhba32:C0:T0:L0:10" with blockSize 1048576, unmapGranularity 1048576, unmapPriority default and volume label "ESXi3-LocalUSB". Successfully created new volume: 5d42cb1c-a9f53c50-4bca-54b20310b9e1
The newly created Datastore is now visible in vSphere GUI.
