Tag Archives: howto

How to run nested ESXi 7 on QNAP’s Virtualization Station

**Important update at the end that should be read prior to wasting your time.

This weekend I found myself in need of an additional ESXi host so instead of acquiring new hardware I figured I might as well run it nested on my beefy QNAP TVS-h1288X with its Xeon CPU and 72GB of RAM. I already use the QEMU-based Virtualization Station (VS) for hosting my primary domain controller and it’s my go-to host for spinning up my ONTAP simulators so I figured nesting an ESXi VM shouldn’t be that difficult. What I hadn’t taken into account however is the fact that VMware has deprecated the VMKlinux Driver Stack, removing support for all of the NICs VS makes available to you in the GUI while provisioning new virtual machines. At first I researched injecting drivers or rolling my own installation ISO but these seemed overly complicated and somewhat outdated in their documentation. Instead I decided to get inside of VS and see if I could do something from that angle, it was after all simply their own version of QEMU.

I started the installation process, but it wasn’t long before I received this error message:

ESXi 7 No Network Adapters error message

I shut down the VM, and changed the NIC type over and over eventually exhausting the five possibilities presented in the VS GUI:

Not even the trusty old e1000 NIC, listed as Intel Gigabit Ethernet above worked…Over to the CLI I went. Some Googling around on the subject lead me to believe there was a command that would produce a list of supported virtualized devices, but the commands I was finding were for native KVM/QEMU installs and not intended for VS so I poked around and came across the qemu-system-x86_64 command, and when I ran it with the parameters -device help and it produced the following, abbreviated list:

./qemu-system-x86_64 -device help
[VL] This is a NROMAL VM
Controller/Bridge/Hub devices:
name "i82801b11-bridge", bus PCI
................<SNIP>
Network devices:
name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
................<SNIP>
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"
................<SNIP>

That last line is exactly what I was looking for, this lead me to believe that QEMU should be able to support the VMXNET3 network device so I cd’d over to the .qpkg/QKVM/usr/etc/libvirt/qemu directory and opened up the XML file associated with my ESXi VM and changed the following sections:

<interface type='bridge'>
      <mac address='00:50:56:af:30:fe'/>
      <source bridge='qvs0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

to:

<interface type='bridge'>
      <mac address='00:50:56:af:30:fe'/>
      <source bridge='qvs0'/>
      <model type='vmxnet3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

I saved the file and for good measure I also restarted VS. I booted the VM, and I received the same error message as above. This time I cc’d over to .qpkg/QKVM/var/run/libvirt/qemu and had a look at the XML file that represented the running config of the VM, and the NIC was still set to e1000. It took me a bit of hacking around to determine that in order to make this change persistent, I needed to edit the XML file using:

virsh edit 122d6cbc-b47c-4c18-b783-697397be149b

That last string of text being the UUID of the VM in question. If you’re unsure of what the UUID is of a given VM, simply grep “qvs:name” from all the XML files in the .qpkg/QKVM/usr/etc/libvirt/qemu directory. I made the same change as I had previously, exited the editor and booted the VM once again…This time, success! My ESXi 7.0u2 host booted fine and didn’t complain about the network. I went through the configuration and it is now up and running fine. The GUI still lists the NIC as Intel Gigabit Ethernet.

I’m reluctant to make any changes to the VM using the GUI at this time for fear of the NIC information changing, but I’m okay not using the GUI if it means being able to nest ESXi 7 on Virtualization Station for testing purposes.

**Update: While the ESXi 7.0u2 VM would boot find, I have been unable to actually add it to my vCenter server. I tried running the VM on my physical ESXi host and was able to add it to vCenter, then I powered down the ESXi VM and imported it into VS. The import worked, but then it showed as disconnected from vCenter. Next I tried importing vCenter into the virtualized ESXi host, but that won’t boot as VS isn’t presenting the VT-x flag even though I have CPU passthrough enabled. I’m still going to try and get this going, but won’t have time to devote to troubleshooting VS for a couple of days.

macOS How-To Guide: Installing vCenter Server Appliance from ISO directly

Ever since macOS started enforcing code signing there’s been the occasional hoop to jump through to get non-App Store software to run. Typically it’s as easy as right-clicking on the binary and choosing Open, which is all well and good until that application needs to launch a subsequent one. Recently I downloaded the ISO for vCenter Server Appliance and double-clicked on it to mount it. I then navigated to:

/Volumes/VMware VCSA/vcsa-ui-installer/mac

Once here, I double-clicked Installer[.app] and of course got the following:

I dutifully two-finger-clicked (ie: right-click) and chose Open and then Open again and proceeded with the initial vCenter configuration. Not too far into the process, Installer wanted to call ovftool but since this was a direct launch, I received a message similar to the previous one:

At this point, I couldn’t find a way to work around this security control directly so I decided to do some research. As with most things VMware-related I came across a post by William Lam, specifically one entitled How to exclude VCSA UI/CLI Installer from MacOS Catalina Security Gatekeeper?. This post lead me to this command:

sudo xattr -r -d com.apple.quarantine <directory of ISO contents>

but that assumes you’ve copied the ISO to another drive and not running it directly as if it were a read-only file system mounted much like the DVD media it represents. I copied all ~8GB of the ISO to my local SSD issued the command above and sure enough, it was going to work. I wasn’t happy with this however and was determined to run this installer from the ISO as intended. The root of the problem is that when you mount the ISO, either by double clicking on it in Finder or issuing an hdiutil mount image.iso, it mounts the file system with the quarantine option:

I did some quick research on how to mount an ISO without this option using any of mount, hdiutil, diskutil or Disk Utility[.app] to no avail. I did notice however that after I unmounted the ISO using diskutil unmount /dev/disk3 that the image remained in the sidebar albeit greyed out:

I decided to right-click on it here and choose Mount:

A quick check over in Terminal[.app] and voilà, no quarantine!

At this point I navigated to the Installer once again and was able to run through to completion without any security notifications. I’m not sure if this is a well known workaround but I didn’t find any reference to it online so hopefully someone will benefit from it.

TL;DR: Instructions for running installers directly from read-only mounted file systems on macOS:

  • Double-click the image so that it mounts normally with DiskImageMounter.app
  • Issue the following CLI command to unmount it: diskutil unmount /dev/diskX
  • Launch Disk Utility.app
  • Right-click the image name in the side bar and choose Mount
  • Your installer can be run directly from the r/o filesystem

Installing the ONTAP 9.7 simulator in Fusion 12.1

At the time of writing this 9.8 is available but I’m specifically writing this for someone who is trying to install 9.7 and having problems. Before I get into the actual simulator installation we need to come cover some stuff around VMware Fusion first.

With regards to networking, VMware Fusion can provide three different interface types, they are as follows:

  1. Bridged – this type puts the interface directly on the same LAN as your Mac, this is great if you want the VM to appear as though it’s on the network that your Mac is using.
  2. Host-only – this is a completely isolated network, the only hosts that can access it are those on your Mac configured with this type of interface. There is no external access with this type.
  3. NAT – this is similar to number two, but allows the host with this type to reach out of the Mac, such as for Internet access.

    If you want more details on this please go read this KB.

By default, the simulator has four network interface; the first two, e0a/e0b are for the ClusterNet network, the back-end network used by cluster nodes to communicate with each other, and should be of type host-only. The second two, e0c/e0d are for client access and management access, these are of type NAT but can also be set to bridged. If you use Nat, then VMware will assign IP addresses via DHCP based on the configuration of the VMNET8 interface settings; to view this cat the file located here:

/Library/Preferences/VMware\ Fusion/vmnet8/dhcpd.conf

Mine looks like this:

allow unknown-clients;
default-lease-time 1800;                # default is 30 minutes
max-lease-time 7200;                    # default is 2 hours

subnet 172.16.133.0 netmask 255.255.255.0 {
	range 172.16.133.128 172.16.133.254;
	option broadcast-address 172.16.133.255;
	option domain-name-servers 172.16.133.2;
	option domain-name localdomain;
	default-lease-time 1800;                # default is 30 minutes
	max-lease-time 7200;                    # default is 2 hours
	option netbios-name-servers 172.16.133.2;
	option routers 172.16.133.2;
}
host vmnet8 {
	hardware ethernet 00:50:56:C0:00:08;
	fixed-address 172.16.133.1;
	option domain-name-servers 0.0.0.0;
	option domain-name "";
	option routers 0.0.0.0;
}

What this means is that any interface set to NAT in my instance of Fusion will receive DHCP addresses in the subnet 172.16.133.0/24, but the DHCP pool itself is only 172.16.133.[128-254]. The subnet mask will still be 255.255.255.0 (ie: /24) and the default gateway is 172.16.133.2 as that is the internal interface of the virtual router created to do the NAT; .1 is held by the “external” interface which you can view by issuing an ifconfig vmnet8 at the command prompt. Note, this interface is created when Fusion is launched and torn down when you quit. If you set the interface type to bridged, those interfaces will get DHCP addresses from the same LAN that the Mac is connected to.

On to the actual installation…

First thing you need to do is download the OVA from NetApp:

  1. Go to https://support.netapp.com/
  2. login (yes, required)
  3. At the top click Downloads → Product Evaluation
  4. Click “Data ONTAP™ Simulator
  5. Agree to the terms
  6. Download the OVA and license keys for the version you’re looking for.

Now that you have the OVA, you’re ready to import it into Fusion. Launch Fusion, then click the + sign and choose Import:

import

Browse for and open the downloaded OVA:

choose file
open

Now click continue:

continue

Give the folder you’re going to store it in a name and click save, I like to name it after the node:

ONTAP 9.7, Node 1

Fusion will import the OVA and present you with the settings. You can modify them if you want, but for now I’m going to leave them as default. Click Finish:

finish

You’ll likely be asked if you’d like to upgrade the VM version, don’t bother:

At this point the vSIM will boot for its first time, I believe the official instructions tell you to hit CTRL-C, halt the boot and call for the maintenance menu then issue an option 4, but if this is the first node you do not have to do that. The root aggregate is automatically created:

First boot with aggr0 creation

Now you can open a browser and point it at the IP address listed on your screen, in my case it will be https://172.16.133.132/, but it may be different for you. You will get a certificate error, but bypass that to access the GUI to finish the configuration. IF you do not get the following screen or get no site at all, there’s something else wrong. Also, hover your mouse over the node in the Health card, if the serial number doesn’t appear, refresh the web page, otherwise configuration will fail:

No node serial

It should look like this:

With node serial

Now enter all the required information, since the IP addresses are being statically assigned, I’m choosing ones outside of the DHCP range, as should you:

Cluster name and admin password
Networking information

I don’t check the “single-node” box, it will still work as a single node if you don’t but if you do, it removes the ClusterNet interfaces completely. I like having those interfaces for experimentation and teaching purposes; also it keeps the door open to adding a second node, which I will cover in a follow up post if there is anyone interested. Now click Submit:

other info

At this point I like to start pinging either the cluster IP I specified or the node IP so I can see when the cluster gets configured since the browser doesn’t always refresh to the new IP address:

ping

Once ping starts responding, go ahead and visit the new IP address via your browser:

Now the person I wrote this blog entry for isn’t getting the GUI above, but instead the GUI for the out-of-band interface for a UCS server, so the IP space their vmnet8 is using collides with production IP space. This can be verified at this point by disconnecting any Ethernet connections and turning off WiFi, once that is done, reload the browser and the IP conflict should be resolved until you’re connected once again. To resolve it permanently, that person will need to edit the dhcpd.conf file for vmnet8 mentioned above, using a subnet known to not conflict. Here’s an example, alternative dhcpd.conf:

allow unknown-clients;
default-lease-time 1800;                # default is 30 minutes
max-lease-time 7200;                    # default is 2 hours

subnet 10.0.0.0 netmask 255.255.255.0 {
	range 10.0.0.128 10.0.0.254;
	option broadcast-address 10.0.0.255;
	option domain-name-servers 10.0.0.2;
	option domain-name localdomain;
	default-lease-time 1800;                # default is 30 minutes
	max-lease-time 7200;                    # default is 2 hours
	option netbios-name-servers 10.0.0.2;
	option routers 10.0.0.2;
}
host vmnet8 {
	hardware ethernet 00:50:56:C0:00:08;
	fixed-address 10.0.0.1;
	option domain-name-servers 0.0.0.0;
	option domain-name "";
	option routers 0.0.0.0;
}

This changes the subnet in use to 10.0.0.0/24 with the DHCP range being 10.0.0.[128-254] and the default gateway of VMs using it to 10.0.0.2.

This is where I’m going to end this post for now as the simulator is now accessible via HTTPS and SSH and ONTAP is ready to be configured. You will still need to assign disks, create a local storage tier (aggregate) as well as an SVM with volume(s) for data among other things. The intent of this post was to get this far, not to teach ONTAP. If you’d like to see a post around either adding a second node to the cluster or configuring ONTAP on the first one, please leave a comment and I’ll try and get around to it.