iSCSI NETWORK BOOT
With the NUC Skull Canyon it is possible to boot an operating system from an iSCSI LUN. Whereas with other operating systems (e.g. Windows), certain steps are required to install a software iSCSI initiator inside the operating system and then configure the iSCSI initiator to boot the operating system from an iSCSI LUN, VMware ESXi is automatically enabled for booting from an iSCSI LUN as long as the iSCSI LUN is mounted before ESXi installer is loaded. If the iSCSI LUN is mounted properly, the ESXi installer will detect the mounted iSCSI LUN and will offer it as one of the available targets for installing ESXi. Therefore, to boot ESXi from an iSCSI LUN, nothing needs to be configured in ESXi itself besides selecting the iSCSI LUN as the destination to which ESXi should be installed (not unlike selecting any other drive visible by the ESXi installer).
The reason iSCSI Network Boot is a valuable option to boot ESXi on a NUC Skull Canyon is that the NUC is sold without any storage installed. The two alternatives to booting ESXi from an iSCSI LUN is to purchase a local SSD storage for the NUC for holding the ESXi image or to use a USB flash drive for using it as a boot device for ESXi. In my opinion, the first alternative is too expensive, and the second is not graceful, if cheap. Additionally, booting from an iSCSI target affords the following benefits:
- Different ESXi versions can be easily switched on the same NUC Skull Canyon by rebooting the NUC into BIOS and selecting a different order of pre-configured iSCSI LUNs where various ESXi versions have been installed.
Note: Each ESXi image should be hosted on an iSCSI LUN dedicated to that ESXi image, but multiple iSCSI LUNs can be pre-configured in the NUC Skull Canyon BIOS iSCSI Configuration utility. - The iSCSI LUN snapshot capability of the iSCSI-based SAN can be utilized for an easy roll-back to a previous state if the changes you are making in the ESXi configuration go awry and need to be quickly restored to the previous state. Additionally, iSCSI LUN snapshots could be used for rolling back to the previous ESXi version if the upgrade to a later ESXi version causes an issue. Of course, you can use the built-in ESXi feature to roll back to the previous version as well, but at least, you will have several options for rolling back after a bad upgrade.
- You will not have to keep multiple USB3 flash drives in your drawer with different ESXi versions if you want to switch back and forth between ESXi versions. So, ESXi network boot will save you a few bucks.
- The ESXi image installed on an iSCSI LUN resides on a RAID-protected volume, which means that the ESXi configuration on which you may have worked for a while is much safer than if the ESXi image were installed on a USB flash drive.
CONFIGURING iSCSI NETWORK BOOT ON THE SKULL CANYON NUC
1. During POST, press F2, and once in Skull Canyon NUC BIOS, navigate to Advanced > Boot, select the Boot Configuration tab and under the Boot Devices section enable “Internal UEFI Shell”. Then press F10 to Save and Exit.
2. During POST, press F2, and once in BIOS, navigate to Home > Advanced > Boot, select the Boot Priority tab and drag UEFI:Built-in EFI Shell to the very top to make it the primary boot option. Press F10 to Save and Exit.
3. During POST, press F2 and once in BIOS, navigate to Home > Advanced > Devices select Add-In Config, and click “iSCSI Configuration”.
4. Select the MAC address of the NIC to be used for iSCSI Network Boot.
Note: I would advise against using the secondary NIC for booting ESXi due to the secondary NIC detection issues described here.
– Configure iSCSI Initiator Name (e.g.: iqn.2004-10.com.telecastle:nuc1)
– Configure iSCSI Attempt (#1, #2, etc.)
Note: Do not forget to navigate to the Save Changes menu item under each attempt and save changes. Pressing F10 to save changes doesn’t seem to work in iSCSI Configuration, but do it anyway.
5. Exit out of iSCSI Configuration utility to BIOS
6. Back in BIOS, navigate to Home > Advanced > Devices, select the Add-In Config tab and click on “Intel(R) Ethernet Connection (H)”. In the next screen, under NIC Configuration you can configure Wake On LAN and Link Speed/Duplex. Save and exit back to BIOS.
7. In BIOS, navigate to Advanced > Devices > Onboard Devices and disable Thunderbolt Controller.
Note: This is a temporary step to allow the ESXi installer to succeed. The Thunderbolt Controller can be re-enabled in BIOS once ESXi is installed.
8. Press F10 to Save and Exit
9. Insert a USB drive with ESXi 5.5 Ub3 or 6.0U2 Installation image as the Skull Canyon NUC is rebooting.
Note: An ESXi installation image can be transferred to a USB flash drive using utilities like UNetbooten or Rufus. You can use a USB drive as small as 1 GB for a bootable ESXi installation media.
10. The Skull Cayon NUC should boot into EFI Shell. At the top of the screen, you will see the Device mapping table with the device names that have been enumerated by EFI Shell.
– USB drive will be enumerated as fs0 and blk0 (two aliases of the same USB device)
– iSCSI LUN will be enumerated as blk1
11. Issue the following command to mount the USB drive:
mount fs0
12. Switch to the USB drive:
fs0:
13. Make sure you can access the file system on the USB drive:
dir
14. Issue the following command to mount the iSCSI LUN:
mount blk1
15. Switch to the iSCSI LUN:
blk1:
16. Make sure you can access the file system on the iSCSI LUN:
Note: If the iSCSI LUN has not yet been formatted, you won’t be able to access the file system on the LUN, which is fine.
17. Switch to the USB drive:
fs0:
18. Change directory to to /efi/boot
cd efi/boot
19. Start the ESXi installer
bootx64.efi
20. Let ESXi installer load its Welcome to the VMware ESXi <version_number> screen, where you are prompted to either Continue or Cancel. Press Enter to continue.
21. On the End User License Agreement screen, press F11 to Accept and Continue.
22. The ESXi installer will scan available devices and will display the USB device with the ESXi installer (which can be used to install ESXi because the ESXi installer is now fully loaded to RAM). Additionally, there will be one or more drives displayed whose UDisk is “iSCSI Storage”. These are iSCSI LUNs mounted from EFI Shell.
Note 1: If you have more than one iSCSI LUNs visible here, make sure you select the one where you want to install this version of ESXi. If in doubt, highlight the iSCSI LUN and press F1 for details. Once ESXi installer gathers the details about this iSCSI LUN, you will see the LUN ID and whether or not ESXi is already installed on that volume, including the ESXi version.
Note 2: You can have several iSCSI LUNs configured for different ESXi versions, and you can load different ESXi versions via iSCSI Boot by enabling only the iSCSI Attempt that corresponds to the LUN with that ESXi version by navigating in BIOS to Advanced > Devices > Add-In Config > iSCSI Configuration
23. Once you select the desired iSCSI LUN to install ESXi on, complete ESXi installation as usual.
24. At the end of ESXi installation, you will be prompted to reboot. Once you confirm the reboot, the NUC will reboot.
25. As the NUC reboots, press F2 during POST to enter BIOS and navigate to Advanced > Boot > Boot Configuration > Boot Priority. Make sure that the following boot devices are listed in the following order in the Boot Drive Order section lists the following devices in this order:
UEFI : LAN : SCSI Disk Device : Part 0 : OS Bootloader
UEFI : LAN : SCSI Disk Device : Part 1 : OS Bootloader
UEFI : LAN : SCSI Disk Device : Part 4 : OS Bootloader
UEFI : LAN : SCSI Disk Device : Part 5 : OS Bootloader
UEFI : LAN : SCSI Disk Device : Part 7 : OS Bootloader
UEFI : Built-in EFI Shell
UEFI : LAN : IP4 Intel(R) Ethernet Connection (H) I219-LM (this is PXE boot via IPv4)
UEFI : LAN : IP6 Intel(R) Ethernet Connection (H) I219-LM (this is PXE boot via IPv6)
UEFI: USB: <usb_device_name>
Note: If you want to make a bootable USB device to have priority over iSCSI Boot, navigate to Advanced > Boot > Boot Configuration and select the checkbox next to “Boot USB Devices First”.
26. Navigate to Advanced > Onboard Devices and re-enable the Thunderbolt Controller
27. Press F10 to Save and Exit.
28. !!!Important!!! – As NUC is rebooting, remove the USB drive that you used for installing ESXi.
29. Once NUC completes POST, it should boot ESXi from the iSCSI LUN.
30. Proceed to configure ESXi as you normally would.
TROUBLESHOOTING NUC SKULL CANYON’S ISCSI NETWORK BOOT
It’s a good idea to keep the UEFI Boot option enabled in Skull Canyon NUC’s BIOS and have the EFI device placed behind iSCSI devices in the Boot Priority window. This will allow you to troubleshoot iSCSI boot per instructions below. If you do this, when iSCSI boot fails, the NUC will boot into EFI Shell.
Issue #1: NUC boots into EFI Shell, an you see the following message:
map: Cannot find required map name
This means that the NUC’s iSCSI initiator was not able to connect to the iSCSI LUN.
Solutions to Issue #1: Troubleshoot network connectivity to the iSCSI LUN:
a. Make sure that you have specified correct names for the iSCSI target and the correct LUN ID as well as correct IP address for the iSCSI SAN in the iSCSI configuration utility in the Skull Canyon NUC’s BIOS.
b. If you are using a Cisco Ethernet switch, and the Ethernet port of the switch is configured in trunk mode (802.1Q VLAN tags), Spanning Tree Protocol (STP) goes over its states when the NUC turns on, and the STP forwarding state is reached too late, so when the NUC tries to boot from the iSCSI LUN, the Ethernet switch port is not yet in the forwarding state. This issue can be mitigated by adjusting the following settings in the iSCSI Configuration utility in the NUC’s BIOS:
— Connection Retry Count [1] (default 0)
— Connection Establishing Timeout 10000 (default 1000)
Note 1: The only STP mode that worked was “spanning-tree portfast edge trunk” when the switch port is configured for 802.1Q trunk.
One reason to configure the Cisco switch port for 802.1Q trunking is if you boot up ESXi from the same GigabitEthernet port that will be used in ESXi by various VMs for VM data traffic and your vSwitches in ESXi are configured with port groups assigned to different VLANs.
If you do not use VLANs for guest VM traffic in ESXi, or if you boot ESXi from a secondary NIC (used exclusively for iSCSI traffic), you can configure the Cisco switch port as an access port (instead of trunk port), and then you will need to configure the command “spanning-tree portfast edge” under the switch port. This will allow NUC to boot ESXi from an iSCSI LUN without any issues.
Issue #2: NUC boots into EFI Shell and displays the Device Mapping Table.
This occurs after the NUC failed to boot from the iSCSI LUN (as in Issue #1) and you have successfully troubleshot and corrected the issue that caused it. The reason the NUC booted into EFI Shell this time is because after the previous unsuccessful iSCSI Boot, the boot order in the Boot Priority window in the Skull Canyon NUC’s BIOS has changed, and the iSCSI LUNs are listed after the EFI Shell boot options in that menu.
Solution to Issue #2: Enter the NUC’s BIOS (by pressing F2 at POST) and change the boot order in the Boot Priority window (see Step #25 above), then select F10 (Save and Exit), and when the NUC restarts, it should boot ESXi from the highest-priority iSCSI target reachable by the NUC.
telecastle 
One thought on “Building a Modern SOHO Virtualization Lab – Part 4”