GOING BEYOND SIX-VM-PER-HOST LIMIT
In the first blog post of this series, I described the progression of my VMware ESXi virtualization lab, as I was able to scale the number of concurrently running VMs from four to 12 VMs running on two 2012 2.6 GHz quad-core Mac Minis.
In early of 2016, I thought that I had solved the VM scaling problem in my lab by having purchased an iSCSI NAS capable of read-write SSD caching, but just a few days later I learned that I was wrong again. As I attempted to add even more VMs to my lab, I realized that I had a new bottleneck of 12 VMs. It turned out that I could not run more than 6 VMs on each Mac Mini, as powering up a 7th VM on either Mac Mini resulted in the same old issue manifesting itself in all VMs on the Mac Mini becoming unresponsive and eventually crashing. To my chagrin, I realized that I had not yet found the final solution to my problem.
2016 NUC SKULL CANYON AS A VMWARE ESXI HOST
In March of 2016, I started looking for a replacement platform for my Late 2012 Mac Minis because by then I had a hunch that my new bottleneck was no longer storage related but rather was caused by the 16 GB RAM limitation in the Late 2012 Mac Mini. After a few minutes of research, I stumbled upon several web sites announcing that Intel was about to release its first quad-core i7 mini computer named NUC Skull Canyon. I decided to order a NUC Skull Canyon, and below is a summary of my two-week-long testing of the NUC Skull Canyon as an ESXi host:
1. ESXi versions : Standard images of both ESXi 5.5 U3b and ESXi 6.0 U2 work with the 2016 NUC Skull Canyon without any additional VIBs. Wi-Fi (and probably Bluetooth) do not work in ESXi, which is true for any hardware platform running ESXi, but these devices can be used for DirectPath I/O pass-through to the VMs that support them.
2. Primary on-board NIC (Intel I219-LM): I have not being able to configure Jumbo Frames on the vSwitch connected to this NIC via the GUI (EHC and vSphere Client) as well as via ESX CLI. My current lab setup is such that I use the primary NIC for the VM traffic and use the secondary NIC for the iSCSI storage traffic to the iSCSI SAN; therefore, Jumbo Frames on the primary on-board NIC are not a requirement in my setup.
3. iSCSI Boot: iSCSI Boot works well with the NUC Skull Canyon. I created dedicated LUNs to host different ESXi images on a QNAP iSCSI target, and I can now easily switch among several ESXi versions on the NUC Skull Canyon without having to have a drawer full of USB flash drives for each ESXi image.
Note: To be sure, configuring iSCSI boot for ESXi is a finicky process, and it requires patience, accuracy, and deliberation in the configuration process. It may also be necessary to change the switch port configuration, especially with 802.1Q VLAN tagging switch ports. The benefits of using iSCSI Boot with ESXi is the fact that the Skull Canyon NUC could be purchased without SSDs, and a USB flash drive would not have to be plugged in to the NUC Skull Canyon’s USB port for ESXi to boot from. This is an ultimate solution for deploying thin ESXi hosts in a lab environment.
4. Secondary NIC: I’ve had a success with the Apple Thunderbolt to Ethernet adapter connected to the NUC Skull Canyon’s Thunderbolt 3 interface via the Startech Thunderbolt 3 to Thunderbolt adapter (released in late May, 2016) as well as via the Apple Thunderbolt 3 to Thunderbolt adapter (released in late October, 2016). There is a caveat that the Apple Thunderbolt to Ethernet adapter is not detected by the NUC Skull Canyon upon a cold boot, but it’s easy to remediate this issue (see my second post on the NUC Skull Canyon for the details). There are also a few Power settings in the NUC Skull Canyon’ BIOS (covered in my second post) that need to be modified for this secondary NIC to work properly in various reboot and power off scenarios.
Note: William Lam and others have had success compiling drivers for USB3 to GigabitEthernet adapters for use as a secondary (or perhaps even tertiary NIC), but I decided to stay away from that route and continue to use the Apple Thunderbolt to Ethernet adapter, which has been solid and dependable in my lab environment since 2012.
5. Chassis fan speed: With the default Cooling settings in the NUC Skull Canyon BIOS, the chassis fan is too loud for my taste. However, with a simple modification of the Cooling settings in NUC Skull Canyon’s BIOS, the NUC Skull Canyon makes very little noise in my lab environment as long as CPU utilization stays below 30%.
Note: A fanless case for the NUC Skull Canyon made by Akasa will most likely solve the not so great build quality of the factory case and will for sure solve the fan noise issues when this case is released (hopefully as soon as summer of 2016). However, this case will most likely cost in the neighborhood of $200 (USD).
6. Viability of NUC Skull Canyon as ESXi Host: I’ve decided that the NUC Skull Canyon is fit to replace the 2012 quad-core i7 Mac Minis in my lab. In fact, I’ve just decommissioned one 2012 Mac Mini and replaced it with the 2016 NUC Skull Canyon. Compared to the 2012 Mac Mini, the 2016 NUC Skull Canyon can run at least twice as many VMs (I’ve tested 12 concurrently running VMs so far) mostly due to the NUC’s ability to support up to 32 GB of RAM. Unfortunately, the build and the feel of the 2016 NUC Skull Canyon is significantly inferior to the those of the 2012 Mac Mini, so it’s been rather hard for me to bid farewell to my Mac Minis. On the other hand, I’m very pleased with the reliability of my lab and the responsiveness of all 12 VMs running on the 2016 NUC Skull Canyon.
7. Total number of VMs per NUC Skull Canyon: So far I have not tried to run more than 12 VMs on one NUC Skull Canyon, and I think for now I will stick with a maximum of 12 VMs per NUC due to my desire to have nearly-silent ESXi hosts. The Cooling settings that I modified in the NUC Skull Canyon’s BIOS allow the NUC to be nearly silent when the CPU utilization stays under 30%, which is the level of the NUC CPU utilization with 12 VMs running concurrently. Adding a few more VMs to the NUC would most definitely result in the average CPU utilization exceeding 30%, which would lead to higher fan speeds. Increased fan speeds would make the NUC Skull Canyon too loud for my taste, so I will stick with 12 VMs per NUC for now.
2016 NUC SKULL CANYON VS LATE 2012 MAC MINI AS ESXI HOST
The biggest improvement between the 2016 NUC Skull Canyon and the Late 2012 Mac Mini that I have discovered in my SOHO lab is the fact that I can run twice as many VMs on one 2016 NUC Skull Canyon as I can on one Late 2012 Mac Mini. Specifically, I am now running 12 Cisco Unified Communications server VMs on one 2016 NUC Skull Canyon equipped with 32 GB of RAM, and the performance of all 12 VMs is exceptional. With the 2016 NUC Skull Canyon, I can now increase the number of VMs in my lab to a few dozen VMs by adding just one or two more 2016 NUC Skull Canyon machines. This, of course, requires a storage solution that can yield enough IOPS to support I/O from dozens of VMs running concurrently.
After years of using the Late 2012 Mac Mini (2.6 GHz quad-core i7 CPU), and a few weeks of testing the 2016 NUC Skull Canyon, I have come to the conclusion that the 2016 NUC Skull Canyon outperforms the Late 2012 Mac Mini so dramatically mostly due to the amount of RAM that the 2016 NUC Skull Canyon supports (up to 32 GB), whereas the Late 2012 Mac Mini is limited to 16 GB of RAM. As for the difference in CPU, even though the Mac Mini’s quad-core 2.6 GHz i7 CPU is several generations older than the NUC’s quad-core 2.6 GHz i7 CPU, when it comes to using both systems as VMware ESXi hosts, the difference in CPU performance is negligible, and both systems are quite powerful as ESXi hosts for their form factor.
Some Mac Mini owners are hoping that Apple will upgrade the memory reference code in the Late 2012 Mac Mini platform that would enable support of up to 32 GB of RAM in Late 2012 Mac Minis, especially because the CPU used in these machines supports up to 32 GB of RAM. However, chances are slim to none that Apple would ever do this, and so the Late 2012 Mac Minis will eventually have to be replaced by another platform in some SOHO lab environments that require a higher density of VMs per ESXi host. Therefore, the 2016 NUC Skull Canyon could be a viable replacement for the 2012 Mac Mini when used as an ESXi host.
In my SOHO lab, I have tested the 2016 NUC Skull Canyon with standard images of VMware ESXi 6.0 U2 and VMware ESXi 5.5 U3b, and both of these ESXi versions ran on the 2016 NUC Skull Canyon without any additional device drivers.
It’s important to note that running ESXi on the 2016 NUC Skull Canyon (as well as on the Mac Mini) is not supported by VMware and should only be done in lab environments. This post, therefore, is not an endorsement of using the 2016 NUC Skull Canyon as an ESXi host in production environments.
telecastle 