Booting libvirt VMs from iSCSI Block Devices

A little bit of background for my previous article: I am working on booting all of my libvirt virtual machines off ZVOLs exposed via iSCSI from a single, FreeBSD-based ZFS pool.

The FreeBSD handbook does a fine job of explaining how to set up iSCSI targets using ctld, so this article is going to focus on the inadequately documented business of mounting iSCSI block devices on linux, and installing operating systems on them using libvirt.

Objective

Install windows on virtual machine sevenfoldgates.anserinae.net whose primary disk is a 100G iSCSI LUN residing on highcharity.anserinae.net.

All commands are executed as root.

Create a storage pool backed by iSCSI

# virsh pool-define-as sevenfoldgates-1 iscsi \
        --source-host highcharity.anserinae.net \
        --source-dev iqn.2016-04.net.anserinae:sevenfoldgates-1 \
        --target /dev
Pool sevenfoldgates-1 defined

Build and confirm the pool:

# virsh pool-build sevenfoldgates-1
Pool sevenfoldgates-1 built
# virsh pool-start sevenfoldgates-1
# virsh pool-list --all
Name               State        Autostart
-----------------------------------------
sevenfoldgates-1   active       yes

sevenfoldgates-1 is a 100G zvol, and now shows as device /dev/sde on our system:

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  3.7T  0 disk
└─sda1   8:1    0  3.7T  0 part
sdb      8:16   0  3.7T  0 disk
└─sdb1   8:17   0  3.7T  0 part
sdc      8:32   0  3.7T  0 disk
sdd      8:48   0  3.7T  0 disk
└─sdd1   8:49   0  3.7T  0 part
*sde     8:64   1  100G  0 disk

Under normal circumstances, we would create volumes on our shiny new block device using virsh vol-create, and use the resulting volume to boot our virtual machine. Unfortunately, like so much of libvirt, iSCSI-based storage pools maddeningly deviate from what is expected:

# virsh vol-create-as sevenfoldgates-1 sevenfoldgates-1_1 100G
error: Failed to create vol sevenfoldgates-1_1
error: this function is not supported by the connection driver: storage pool does not support volume creation

It turns out that libvirt automatically maps individual LUNs to volumes, with no need for the intermediate volume creation step. Thanks, I guess?

# virsh vol-list sevenfoldgates-1
 Name                 Path
------------------------------------------------------------------------------
 unit:0:0:0           /dev/sde

Install the VM

It's almost all over now except for the screaming (yes, I still use Windows 7):

virt-install \
--name sevenfoldgates.anserinae.net \
--ram 32768 \
--disk vol=sevenfoldgates-1/unit:0:0:0 \
--vcpus 2 \
--network bridge=vmbr0 \
--graphics vnc,listen=0.0.0.0,port=5903 \
--console pty,target_type=serial \
--cdrom /var/lib/libvirt/images/Windows-7-SP1.iso

Note the structure of the --disk option: it tells virt-install to install the VM on volume unit:0:0:0 of our shiny new sevenfoldgates-1 storage pool.

VNC into the host on port 5903 and we see this, indicating that our virtual machine has successfully booted off our iSCSI LUN:

Conclusion

As always, I am amazed by how much effort goes into building complex functionality into Linux, and how little effort is expended on documenting said functionality. Most of the information on this post comes from this PDF, found on the Dell website via the 8th or 9th page of Google search results for 'libvirt iscsi boot'.

If you ever read this, thanks for the pointers Jose De La Rosa!