Tag Archives: virt-install

virt-install + nbdkit live install

This seems to be completely undocumented which is why I’m writing this … It is possible to boot a Linux guest (Fedora in this case) from a live CD on a website without downloading it. I’m using our favourite flexible NBD server, nbdkit and virt-install.

First of all we’ll run nbdkit and attach it to the Fedora 29 live workstation ISO. To make this work more efficiently I’m going to place a couple of filters on top — one is the readahead (prefetch) filter recently added to nbdkit 1.12, and the other is the cache filter. In combination these filters should reduce the load on the website and improve local performance.

$ rm /tmp/socket
$ nbdkit -f -U /tmp/socket --filter=readahead --filter=cache \
    curl https://download.fedoraproject.org/pub/fedora/linux/releases/29/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-29-1.2.iso

I actually replaced that URL with a UK-based mirror to make the process a little faster.

Now comes the undocumented virt-install command:

$ virt-install --name test --ram 2048 \
    --disk /var/tmp/disk.img,size=10 
    --disk device=cdrom,source_protocol=nbd,source_host_transport=unix,source_host_socket=/tmp/socket \
    --os-variant fedora29

After a bit of grinding that should boot into Fedora 29, and you never (not explicitly at least) had to download the ISO.


To be fair qemu does also have a curl driver which virt-install could use, but nbdkit is better with the filters and plugins system giving you ultimate flexibility — check out my video about it.


Filed under Uncategorized

virt-builder Debian 8 (Jessie) image

Debian 8 was released a couple of days ago, and you can now install it through virt-builder.

Use --notes to read the release notes:

$ virt-builder debian-8 --notes

To build an image:

$ virt-builder debian-8 \
    --firstboot-command "dpkg-reconfigure openssh-server"

To boot it under libvirt:

$ virt-install --import \
  --name debian-8 --ram 2048 \
  --disk path=debian-8.img,format=raw --os-variant=debianwheezy

(At some point --os-variant=debianjessie will work, but virt-install doesn’t support it yet)

Update: This is how I ended up running Debian 8:

$ virt-builder debian-8 \
    --size=30G \
    --root-password PASSWORD \
    --edit '/etc/apt/sources.list: s/wheezy/jessie/g' \
    --run-command '
      apt-get -y install debian-keyring debian-archive-keyring
      apt-key update
    ' \
    --install emacs,nfs-common,sudo \
    --edit '/etc/ssh/sshd_config:
              s/^#PermitEmptyPasswords no/PermitEmptyPasswords yes/' \
    --firstboot FIRSTBOOT.sh
    --run-command 'update-rc.d virt-sysprep-firstboot defaults' \
    --run-command 'killall dbus-daemon cgmanager ||:'

Leave a comment

Filed under Uncategorized

virt-builder: Fedora 21 ppc64 and ppc64le images

virt-builder now has Fedora 21 ppc64 and ppc64le images available, and you can run these under emulation on an x86-64 host. Here’s how to do it:

$ virt-builder --arch ppc64 fedora-21 \
    -o fedora-21-ppc64.img


$ virt-builder --arch ppc64le fedora-21 \
    -o fedora-21-ppc64le.img

To boot them:

$ qemu-system-ppc64 -M pseries -cpu POWER8 -m 4096 \
    -drive file=fedora-21-ppc64[le].img \
    -serial stdio

Oddly the boot messages will appear on the GUI, but the login prompt will only appear on the serial console. (Fixed)

Libvirt also has support, so with a sufficiently new version of the toolchain you can also use:

$ virt-install --import --name=guestname \
    --ram=4096 --vcpus=1 \
    --os-type=linux --os-variant=fedora21 \
    --arch=ppc64[le] --machine pseries \
$ virsh start guestname

It’s quite fun to play with Big Iron, even in an emulator that runs at about 1/1000th the speed of the real thing. I know a lot about this, because we have POWER8 machines at Red Hat, and they really are the fastest computers alive, by a significant multiple. Of course, they also cost a fortune and use huge amounts of power.

Some random observations:

  1. The virt-builder --size parameter cannot resize the ppc64 guest filesystem correctly, because Anaconda uses an extended partition. Workaround is to either add a second disk or to create another extended partition in the extra space. (Fixed)
  2. The disks are ibmvscsi model (not virtio or ide). This is the default, but something to think about if you edit or create the libvirt XML manually.
  3. Somehow the same CPU/machine model works for both Big Endian and Little Endian guests. It must somehow auto-detect the guest type, but I couldn’t work out how that works. Anyway, it just works by magic. it’s done by the kernel
  4. libguestfs inspection is broken for ppc64le
  5. Because TCG (qemu software emulation) is single threaded, only use a single vCPU. If you use more, it’ll actually slow the thing down.

Thanks: Maros Zatko for working out the virt-install command line and implementing the virt-builder script to build the images.

Leave a comment

Filed under Uncategorized

Tip: Create a libvirt guest using virt-builder

virt-builder and virt-install together make it easy to create libvirt guests in just a few minutes.

Before you start, read the installation notes for the OS you’re about to install. They may contain essential tips for a successful installation:

$ virt-builder --notes ubuntu-13.10

Note I’m running this as root, but that’s only necessary because I want to write the guest to a logical volume. Normally you don’t (and shouldn’t) run virt-builder as root.

# lvcreate -L 50G -n builder-ubuntu1310 /dev/vg_data
# virt-builder ubuntu-13.10 \
    --output /dev/vg_data/builder-ubuntu1310 \
    --hostname builder-ubuntu1310 \
    --install build-essential,debhelper
[   0.0] Downloading: http://libguestfs.org/download/builder/ubuntu-13.10.xz
[   1.0] Uncompressing: http://libguestfs.org/download/builder/ubuntu-13.10.xz
[  23.0] Running virt-resize to expand the disk
[  99.0] Opening the new disk
[ 103.0] Setting a random seed
[ 103.0] Setting the hostname: builder-ubuntu1310
[ 103.0] Random root password: mOQ8YdwKuxkMlpOU [did you mean to use --root-password?]
[ 103.0] Installing packages: build-essential debhelper
[ 423.0] Finishing off
Output: /dev/vg_data/builder-ubuntu1310
Total usable space: 48.2G
Free space: 47.1G (97%)
# virt-install --import \
    --name builder-ubuntu1310 --ram 4096 \
    --disk path=/dev/vg_data/builder-ubuntu1310,format=raw

(Yes, I forgot to use the --root-password option …)


Filed under Uncategorized

virt-tools.org: Install a VM using the command line

Read our guide to installing virtual machines using the command line tools.

Leave a comment

Filed under Uncategorized

Prebuilt distributions part 3

[This is the final article in a three part series. Read part 1 and part 2.]

In this part I was going to discuss the different installation options, and I’ll give a quick introduction to each at the end. But you know? What I really need is this tool:

| Rich's Amazing Bug Environment Reproducer Tool!        |
| Bugzilla number: # [545116] [ Go ]                     |
|                                                        |
| Preselecting Fedora Rawhide image.  If this is wrong,  |
| choose the right image from the list below:            |
|                                                        |
| [X] Fedora Rawhide (x86-64)                            |
| [ ] Fedora 12                                          |
| [ ] Fedora 11                                          |
| [ ] Ubuntu 9.11                                        |
| [ ] Ubuntu 9.04                                        |
| [ ] Debian 6.0                                         |
| [ ] Debian 5.0    [Download more distributions]        |
|                                                        |
| [X] x86-64   [ ] i386                                  |
|                                                        |
| [Create  virtual machine now]                          |

The Create virtual machine now button must take no more than 60 seconds to deliver the virtual machine, plus maybe another 60 seconds to boot it and give me the login prompt. I can then go in, configure it and reproduce the bug, and snapshot it, so I can go back and test fixes later. (I’d like the snapshot workflow to go through the user interface too, and ponies).

The only approach that doesn’t take ages to install is to have the distributions prebuilt as small disk images locally (as described already in part 2).

Note stuff I don’t want to have to specify: storage, download URLs, VM names, etc etc.

Why doesn’t this exist already, or perhaps it does?

I’d also like to suggest that the resizable disk image should become yet another distribution format that Linux distros provide. Many do already, albeit buried inside a live CD.

As promised, my quick summary of installation methods:


virt-install is a great command line tool for installing new VMs. I have these two commands pretty much permanently in my command history:

# lvcreate -n F13Rawhidex64 -L 10G vg_trick
# virt-install -v -n F13Rawhidex64  --accelerate \
    -r 512 -f /dev/vg_trick/F13Rawhidex64 \
    --cdrom Fedora-12-x86_64-DVD.iso

The commands above lead to a manual installation, but you can also specify a kickstart file, although the syntax for doing this is not obvious and not documented in the manual page.

Nevertheless, even an install from a CD-ROM on the local drive (as in the example above) is a slow process, taking anything up to an hour.


Kickstart is the method used to automate Anaconda installs, ie. installs of all Fedora, Red Hat and derived distributions. It’s a little unwieldy because you have to write a separate kickstart file, then host that on a web server (so the VM can find it), then add a special magic kernel parameter when booting the VM: ks=http://local.example.com/kickstart.ks

I found in the past that it’s better to keep a collection of kickstart files around and iterate them slowly so you can find out what works and what doesn’t.

Also, kickstarts are automatic, but just as slow as manual installation.


Debian’s debootstrap (and the Fedora equivalent I wrote called febootstrap) is, as far as I know, the standard way to install a new Debian instance from an existing Debian instance. Unlike our VM installs above, debootstrap creates a “chroot” directory, and what you do with that directory is then up to you. Usually you would debootstrap into some mounted device. debootstrap is very simple to use and driven completely from the command line.

It’s not particularly fast. Because it downloads, unpacks and installs each package, it can take an hour or more to run.


This is Ubuntu’s wrapper around debootstrap and libvirt/qemu.


Cobbler is a network install server which lets you pxeboot and configure a network of machines (and virtual machines) from a central server. It has lots of wonderful features — go and read the web site.

Leave a comment

Filed under Uncategorized

Prebuilt distributions part 2

In part 1 I discussed how these days Linux Live CDs usually come with a prebuilt disk image of the distro which is simply copied over to the hard disk during installation. (The “old” method was to rpm/dpkg-install the packages which is much more time-consuming). However my first test wasn’t very successful because I was using the “cp” command to copy files.

Anaconda (the Fedora installer) is smarter than this. It “dd”s the prebuilt disk image to the hard disk and then uses an ext2/3/4 utility called resize2fs to expand it to the correct size.

I changed the previous guestfish script to take this approach.

The new/Anaconda approach is much faster. Our total time is down from over 18 minutes to 2½ minutes (approximately 2 minutes for the “dd”, 2 seconds for the resize2fs, and the rest of the time taken doing the partitioning and LVM creation).

Unfortunately we have to leave Ubuntu behind at this point. Ubuntu ships with a squashfs, and I’m not aware of any way to turn this into an ext3 partition efficiently (except to use “cp” which we showed in part 1 was very slow). The new script only works with Fedora Live CD ISOs.

The new script is after the cut.

Continue reading

1 Comment

Filed under Uncategorized

Prebuilt distributions part 1

Previously I took a look at unpacking Fedora and Ubuntu live CDs to find out what’s inside them and to ask the question can we use the prebuilt filesystem image that these live CDs contain to quickly create a Fedora or Ubuntu “all-defaults” virtual machine?

This is my first attempt, and it’s not successful, but it does demonstrate a large and interesting guestfish script doing a non-trivial amount of work.

This script:

  1. mounts the prebuilt filesystem from either a Fedora or Ubuntu live CD
  2. creates a disk image with a 200 MB /boot partition and a single / (root) logical volume covering the remainder of the disk
  3. uses the cp -a command to recursively copy the prebuilt filesystem to the disk

Where it fails is that “cp” isn’t very fast. On my local machine it took 18 minutes to copy all the files across, which means this isn’t a practical “instant install” method. (I didn’t in the end try to boot the final disk image).

In part 2 this week, I’ll look at the approach that anaconda takes: It dd’s the disk image and then runs resize2fs on it to expand it into the available space.

In part 3 I’ll compare this approach to others: virt-install, manual installation, kickstart, cobbler, debootstrap and ubuntu-vm-builder.

The script itself follows after the cut:

Continue reading


Filed under Uncategorized

Looking closer at Fedora, Ubuntu live CDs

Previously I’ve shown you can use guestfish to unpack a Fedora live CD.

I’m interested in whether we can use the contents of these live CDs to mass-install operating systems using libguestfs.

If you imagine that you go through an “all defaults” install of say Fedora or Ubuntu to a new virtual machine, then when you end up with is an identical disk image containing 1-2 GB of default packages and a lot of empty space. Two people asked to go through the same all-defaults install of the same distro would end up with roughly the same content. The details on the disk would be slightly different because some parts of the disk partitioning and ext3 superblocks change slightly depending on the overall disk size. But really those things can be fixed up afterwards using a little repartitioning, lvresize and resize2fs.

Let’s look inside an Ubuntu live CD:

$ guestfish --ro -a ubuntu-9.10-desktop-amd64.iso

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> run
><fs> list-devices
><fs> file /dev/sda
ISO 9660 CD-ROM filesystem data 'Ubuntu 9.10 amd64
><fs> mkmountpoint /t1
><fs> mount /dev/sda /t1
><fs> ll /t1/casper
total 691049
dr-xr-xr-x  2 root root      2048 Oct 27 14:31 .
dr-xr-xr-x 10 root root      2048 Oct 27 14:31 ..
-r--r--r--  2 root root     37288 Oct 27 14:19 filesystem.manifest
-r--r--r--  2 root root     35354 Oct 27 14:16 filesystem.manifest-desktop
-r--r--r--  2 root root 697778176 Oct 27 14:25 filesystem.squashfs
-r--r--r--  2 root root   5836401 Oct 27 14:20 initrd.lz
-r--r--r--  2 root root   3941696 Oct 16 12:12 vmlinuz
><fs> mkmountpoint /t2
><fs> mount-loop /t1/casper/filesystem.squashfs /t2
><fs> cat /t2/etc/debian_version

The file /casper/filesystem.squashfs seems to be a complete Ubuntu installation, and if I’m understanding this correctly the Ubuntu installer will copy this to the newly created filesystem directly. That will be the new Ubuntu installation, plus or minus some config file changes and some extra packages downloaded afterwards from the net.

Here’s the same examination of the Fedora 12 Live CD:

$ guestfish --ro -a Fedora-12-x86_64-Live.iso

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> run
><fs> list-devices
><fs> file /dev/sda
ISO 9660 CD-ROM filesystem data 'Fedora-12-x86_64-Live
><fs> mkmountpoint /t1
><fs> mount /dev/sda /t1
><fs> ls /t1
><fs> ll /t1/LiveOS
total 655291
dr-xr-xr-x 2 root root      2048 Nov  9 14:45 .
dr-xr-xr-x 5 root root      2048 Nov  9 14:44 ..
-r-xr-xr-x 1 root root     23040 Nov  9 14:44 livecd-iso-to-disk
-r-xr-xr-x 1 root root      8192 Nov  9 14:45 osmin.img
-r-xr-xr-x 1 root root 670982144 Nov  9 14:48 squashfs.img
><fs> mkmountpoint /t2
><fs> mount-loop /t1/LiveOS/squashfs.img /t2
><fs> ls /t2
><fs> ls /t2/LiveOS/
><fs> mkmountpoint /t3
><fs> mount-loop /t2/LiveOS/ext3fs.img /t3
><fs> cat /t3/etc/redhat-release
Fedora release 12 (Constantine)

Again I hope I assume correctly that the installer copies ext3fs.img to the hard disk when installing Fedora 12.

So my vague plan, assuming anything I’ve written above is correct, is to take these pre-made filesystem images and allow people to quickly install specific operating system images from a simple tool:

$ virt-press Fedora-12 F12

which would stamp out a Fedora 12 VM in a few seconds and register it with libvirt as “F12”.

I’m not sure this is possible yet …


Filed under Uncategorized