Setting up a serial console in qemu and libvirt

I always forget how to do this, but in fact it’s quite simple.

First ensure your libvirt XML contains a fragment like this (my guest, installed using virt-install, already had this).

<serial type='pty'>
  <target port='0'/>
</serial>
<console type='pty'>
  <target type='serial' port='0'/>
</console>

Second, edit /boot/grub/grub.conf inside the guest, adding the console=ttyS0 element to the kernel command line:

# virt-edit Guestname /boot/grub/grub.conf
...
title Fedora (2.6.38.6-26.rc1.fc15.x86_64)
	root (hd0,0)
	kernel /vmlinuz ro [...] console=ttyS0
...

You don’t need to set the speed. I believe it defaults to 115200 8N1, but I don’t think that qemu serial ports have a “speed” as such, since the hardware is emulated.

Third, start the guest and dump out the running XML:

# virsh start Guestname && virsh dumpxml Guestname
...
    <console type='pty' tty='/dev/pts/8'>

Notice the randomly assigned pty on the host side (/dev/pts/8). Connect to that with Minicom[1], and you should see boot messages and/or a login prompt.

[1] Is there something better than minicom? It’s a horrible program, always has been.

Tip: replace text strings in a file using guestfish

virt-edit has a handy -e option that lets you do replacements on files. For example this wipes out your root password:

virt-edit domname /etc/passwd -e 's/^root:.*?:/root::/'

How can you do the same thing from guestfish or the libguestfs API?

There’s no support for this operation directly in the API, but you can download the file, use sed/perl/whatever on it locally, and upload it, and that is essentially the same thing that virt-edit is doing.

Here’s how to do that easily in guestfish:

$ guestfish --rw -i -d domname
><fs> download /etc/passwd /tmp/passwd
><fs> ! sed 's/^root:[^:]\+:/root::/' /tmp/passwd > /tmp/passwd.new
><fs> upload /tmp/passwd.new /etc/passwd

In guestfish, ! before a command runs a local command.

New (Year’s) libguestfs tools: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out

One aim with libguestfs development is to make easy and common file operations easy. Although you can already upload and download files into virtual machines using guestfish commands, is there a way to make this common operation easier to discover?

One way is to add more virt commands, which I’ve found that users have least difficulty discovering because they are on the website, autocompleted when you hit virt-[tab], and listed as separated manual pages.

So today I added four more commands for uploading and downloading: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out.

The way you use them is very simple:

$ mkdir homes
$ virt-copy-out -d Fedora14 /home homes/

$ virt-tar-out -d Fedora14 /home - | \
    gzip --best > homes.tar.gz

These commands are just small shell script wrappers around guestfish, but I hope they make common things a little bit easier.

You can get these new commands from Fedora Rawhide, or as binaries for Debian or Ubuntu.

Tip: Uploading and downloading

Talked to someone today who was confused by how to get files in and out of a virtual machine disk image. Libguestfs, guestfish, virt-cat, virt-edit, virt-ls and virt-tar offer several ways to do this, and in this article I’m going to summarize all these different ways.

First of all, answer these questions:

1. Upload (ie. from your host into the disk image) or
   download (disk image down to host)?
2. Single file or lots of files?
3. Have you got / do you want a tarball, or a directory containing the individual files?
4. Do you want to download the files or just get a list of files?

I should note as usual that uploading or modifying a guest is only safe if the guest is shut off. If you try to modify a live guest you’ll get disk corruption, so watch out.

Downloading a single file

Easiest way is with virt-cat:

virt-cat GuestName /path/to/file > file

An alternative is to use guestfish download:

guestfish -i --ro -d GuestName \
    download /path/to/file ./localfile

Uploading a single file

If the file exists and it’s a text file that you want to edit, the easiest way is to use virt-edit:

virt-edit GuestName /path/to/file

To upload a single file in general, use guestfish upload:

guestfish -i -d GuestName \
    upload ./localfile /path/to/file

Downloading multiple files

Easiest way is probably to use the copy-out command:

guestfish -i --ro -d GuestName \
    copy-out /directory ./localdir

Uploading multiple files

Use guestfish copy-in command:

guestfish -i -d GuestName <<EOF
  mkdir /directory
  copy-in ./localdir /directory
EOF

If you want to quickly upload lots of files, the fastest method is to prepare an ISO (mkisofs) or squashfs containing the files and attach it as an extra disk. You can then copy files quickly to where you need them, eg:

guestfish -i -d GuestName -a my.iso <<EOF
  mount-ro /dev/sdb /mnt
  cp-a /mnt/files /someplace
EOF

Downloading or uploading a tarball

There is a high level tool called virt-tar which automates this. You can also do it semi-manually using the guestfish commands tar-in, tar-out, tgz-in, tgz-out (and more, see the guestfish documentation).

Listing files

Finally to list out files, use the high level virt-ls tool, or the low level guestfish commands ls (flat list of a single directory) or find0 (recursive listing of subdirectories).

Tip: Run Xen as a KVM guest

(Thanks to Dan for pointing out that this is possible)

You can run legacy Xen hypervisor as a KVM guest, which is useful for testing and development. Since the Xen HV won’t have access to the hardware (in particular, to hardware virt) you can only run Xen paravirt guests this way, which in reality means only Linux XenPV guests. [It is supposed to be possible, if you have AMD hardware supporting nested SVN, to get Xen fullvirt guests going, but I did not try this.]

Preparation:

1. Install a RHEL 5 guest first. This will later be changed into a “Xen host” guest.
2. You will definitely appreciate having libguestfs around since you can do virt-edit RHEL5Xen /boot/grub/grub.conf to adjust the boot configuration iteratively.

Boot the RHEL 5 guest ordinarily, and install the appropriate kernel-xen and xen packages, and edit grub.conf to enable the Xen hypervisor:

default=0
[...]
title Red Hat Enterprise Linux Server (2.6.18-194.17.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.17.1.el5 noapic
        module /vmlinuz-2.6.18-194.17.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-194.17.1.el5xen.img

I added the noapic option, and removed rhgb quiet so I could see what was going on.

Furthermore you need to switch the guest to using emulated devices (IDE, rtl8139) instead of virtio, if it’s not already. Shut down the guest and do:

# virsh edit RHEL5Xen

and change <target dev=’vda’ bus=’virtio’/> to <target dev=’sda’ bus=’ide’/> and remove any <address> element in the same <disk> block. Similarly if there is a network card, make it an emulated rtl8139 instead of virtio.

If during boot you see the error:

request_module: runaway loop modprobe binfmt-464c

then this means you’ve got the wrong kernel-xen package installed, usually that you’ve got the 32 bit kernel-xen.i686 with a 64 bit userspace.

As I said before, I really appreciated guestfish / virt-edit to let me interactively edit the grub configuration and go back and forwards between Xen and ordinary kernel until I got it right.

But finally it did boot, and it was running the Xen hypervisor and dom0:

# /etc/init.d/xend start
Starting xend:                                             [  OK  ]
# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      864     1 r-----     48.3
# uname -a
Linux rhel5xenx64.home.annexia.org 2.6.18-194.17.1.el5xen #1 SMP Mon Sep 20 07:20:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Update #1

If you have libvirt running inside this guest, it will try to register a bridged network with the address 192.168.122.x which is the exact same address as your host, and that causes confusion. Typical symptoms are that you can’t ssh into the Xen guest from the host, and that networking inside the Xen guest seems broken in strange ways. The solution is simple. In the Xen guest type:

# virsh net-edit default

and change “122″ everywhere to something else, eg. “123″.

Then restart the default network:

# virsh net-destroy default
# virsh net-start default

and everything will work again.

Update #2

Because this KVM guest is running a Xen hypervisor and guests, you need to give it a bit more memory. I bumped mine up to 2G, allowing me to comfortably install and run one (nested) guest:

# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     1224     1 r-----    103.8
RHEL5xenguest                              1      767     1 -b----     33.0

libguestfs binaries for all Linux distros

I uploaded some distro-independent Linux/x86-64 binaries of libguestfs, guestfish, guestmount and the virt-* tools. Be sure to read the README file first.

These are a little experimental and I’d welcome feedback. I got them to work fine on Debian 5.0 after upgrading glibc and Perl, but YMMV.

Update: OpenSUSE 11.3 working.

The version of KVM supplied is too old (doesn’t support virtio-serial) so I had to compile qemu from git and drop the following qemu wrapper in place:

#!/bin/sh -
qemudir=/home/rjones/d/qemu
exec $qemudir/x86_64-softmmu/qemu-system-x86_64 -L $qemudir/pc-bios "$@"

The next problem which had me confused for a very long time was that qemu kept aborting while allocating memory. After trying things like adding swap, playing with overcommit settings and so on, it turned out that SUSE uses some really silly, and very small default ulimits for virtual memory. You have to do:

ulimit -Hv unlimited
ulimit -Sv unlimited
ulimit -Hm unlimited
ulimit -Sm unlimited

Update: Ubuntu 10.04 working.

As with SUSE, the version of KVM shipped by Ubuntu is too old to support virtio-serial. All I had to do was replace it with qemu from git and the same qemu wrapper above.

After that guestfish works fine.

If you need the Perl tools, then you have to upgrade Perl to 5.12.1, install hivex, and link libpcre.so.0 -> libpcre.so.3.

Sed-like configuration file editing with virt-edit

A new option in virt-edit lets you non-interactively edit files in a virtual machine, somewhat like using sed or perl -pe. For example, change the init default level to 3:

virt-edit test.img /etc/inittab -e 's/^id:.*/id:3:initdefault:/'

Delete the root password from a VM:

virt-edit test.img /etc/passwd -e 's/^root:.*?:/root::/'

guestfish -i, that’s i for “inspector” not “interactive”

I wrote this FAQ entry today …

---

• What’s the deal with guestfish -i?
• Why does virt-cat only work on a real VM image, but virt-df works on any disk image?
• What does “no root device found in this operating system image” mean?

These questions are all related at a fundamental level which may not be immediately obvious.

At the libguestfs API and guestfish level, a “disk image” is just a pile of partitions and filesystems.

In contrast, when the virtual machine boots, it mounts those filesystems into a consistent hierarchy such as:

 /          (/dev/sda2)
 |
 +-- /boot  (/dev/sda1)
 |
 +-- /home  (/dev/vg_external/Homes)
 |
 +-- /usr   (/dev/vg_os/lv_usr)
 |
 +-- /var   (/dev/vg_os/lv_var)

(or drive letters on Windows).

The libguestfs API however knows nothing about how filesystems are mounted. In general, a disk image that we just made up might not contain an operating system or “mountpoints” at all.

Nevertheless, users expect some tools (like virt-cat) to work with VM paths:

virt-cat fedora.img /var/log/messages

How does virt-cat know that /var is a separate partition?

The trick is a large external program called virt-inspector. Virt-inspector contains all sorts of tests and heuristics, so it can take a disk image about which nothing is known beforehand and work out both what it is, and how filesystems would be mounted.

Some tools, such as virt-cat, virt-edit, virt-ls and virt-tar use the same virt-inspector code to map VM paths. Other tools, such as virt-df and virt-list-filesystems operate entirely at the raw “big pile of filesystems” level of the libguestfs API, and don’t use virt-inspector code.

guestfish is in an interesting middle ground. If you use the -a and -m command line options, then you have to tell guestfish exactly how to add disk images and where to mount partitions. This is the raw API level.

However there is also a guestfish -i (for “inspector”) mode. In this mode, guestfish runs virt-inspector, and then (if successful) virt-inspector runs guestfish a second time with the correct -a and -m options. This is much slower, but usually simpler for the user. They can just do this and it all works:

guestfish -i SomeLibvirtGuest

The error “no root device found in this operating system image” is related to this. It means virt-inspector was unable to locate an operating system within the disk image you gave it. You might see this from programs like virt-cat if you try to run them on something which is just a disk image, not a virtual machine.

Poor man’s P2V

(P2V = physical to virtual, taking a physical machine and converting it into a virtual machine)

What happens when you have an old server sitting in the corner — the hardware is flaky and you need to set up a virtual equivalent ASAP, but no one can remember how that old server is configured? People will sell you very expensive software to solve this problem for you.

But if you have some time and patience you can do P2V conversions by hand for free, and it’s not too hard. Here’s how.

First of all, grab a bootable Linux rescue CD. A Fedora CD in rescue mode will do just fine, but choose any Linux CD that you’re happy with.

Now you boot your machine from the CD, so that we are in a modern, Linux-based OS. From the Linux command line you will see the physical disks in the machine like /dev/sda or /dev/hda (check /sys/block).

All you do is take each physical disk in turn and copy it complete over to your virtualization host. Just do:

dd if=/dev/sda | ssh root@virthost 'cat > /var/lib/libvirt/images/guest.img'

That makes a block-for-block identical copy of the hard disk, and it usually takes several minutes to an hour to copy everything across, depending how fast the old server and the network is.

Now over to your virtualization host, how do you boot this?

With very recent versions of virt-install there is a virt-install --import option that you can use to import disk images directly.

Alternately, write a libvirt XML configuration file for the virtual machine. It’s usually best to start off with an existing XML configuration, so just pick another guest at random and do virsh dumpxml foo. Take that output, modify it suitably, make sure it’s full virt (“hvm”), and boot using:

virsh define guest.xml

Now at this point it may not in fact boot. You might need to edit a few things inside the virtual machine disk image, typically /etc/fstab, maybe install a new kernel, perhaps edit the network configuration.

Step forward: guestfish and virt-edit.

$ guestfish -i guest.img
$ virt-edit guest.img /etc/fstab

(Note: I am the author of virt-p2v which is currently in a big version 2.0 rewrite).

virt-edit

New in libguestfs 1.0.72 will be a tool for editing files in a virtual machine.

There’s a million uses for this. Change grub configuration:

# virt-edit mydomain /boot/grub/grub.conf

Edit the message of the day:

# virt-edit mydomain /etc/motd

Change a configuration file:

# virt-edit mydomain /etc/fstab

You get the idea …

For more complex changes, or to edit multiple files, use the libguestfs API or guestfish, but I hope this solves some of the simpler sysadmin tasks.