Masking systemd services in a guest

In the previous post I told you how to get cloud-init to work in non-cloud environments.

What if you need to disable cloud-init entirely?

With systemd services and guestfish this is easy:

$ guestfish -a disk.img -i \
     ln-sf /dev/null /etc/systemd/system/cloud-init.service

Why not use this opportunity to get rid of tmp-on-tmpfs at the same time:

$ guestfish -a disk.img -i \
     ln-sf /dev/null /etc/systemd/system/tmp.mount

Systemd’s design of mapping services to files also makes it easy to list the available services in a guest:

$ virt-ls -a /tmp/fedora-19.img -R /lib/systemd/system

Using libguestfs to find out why a Windows guest was “hanging”

While diagnosing a bug where a Windows guest hangs at boot, I used libguestfs to find out what files were being updated on the disk. Here is how.

First of all I used virt-ls to get a listing of all the files in the guest and the last time they were updated:

$ virt-ls -lR -a /path/to/winxp.img --time-relative / | \
  grep '^-' > /tmp/files

The colums in the output file look like this:

- 0777          0 12022162 12022162 12022162 /AUTOEXEC.BAT

The three numbers in columns 4, 5 and 6 (“12022162”) are the ones we are interested in. These are the time of last access, time of last modification and time of last status change, in seconds before now (because of the --time-relative flag).

So now we’re just looking for the files where column 6 is a small number. Everything that’s been touched in the last 2 minutes, for example:

$ awk '$6 < 120' < /tmp/files
- 0777       1024       40       40       40 /Documents and Settings/rjones/NTUSER.DAT.LOG
- 0777       7414       30       30       30 /WINDOWS/Prefetch/LOGON.SCR-151EFAEA.pf
- 0777    9445376       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/DataStore.edb
- 0777       8192       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/Logs/edb.chk
- 0777     131072       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/Logs/edb.log
- 0777     203243       49       49       49 /WINDOWS/WindowsUpdate.log
- 0777       1024       41       40       40 /WINDOWS/system32/config/SAM.LOG
- 0777     262144      645       42       42 /WINDOWS/system32/config/SECURITY
- 0777      20480       42       41       41 /WINDOWS/system32/config/SECURITY.LOG

Ah.

Looks to me like Windows Update is running.

We can confirm this easily:

$ virt-cat -a /path/to/winxp.img /WINDOWS/WindowsUpdate.log|tail
2012-02-27	19:17:57:718	 824	144	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:12:546	 824	144	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:39:015	 824	14c	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:49:031	 824	7b8	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:58:046	 824	14c	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:58:062	 824	748	AU	AU checked download status and it changed: Downloading is paused

Indeed soon afterwards the guest came back to life, after downloading all its Windows Updates.

New in libguestfs 1.12: use virt-ls to analyze guests

In this series of posts I’ll be looking at what’s new in the forthcoming release of libguestfs 1.12.

With the forthcoming virt-ls -lR option you will be able to extract the file metadata from a virtual machine easily. The output format is designed so that simple grep patterns can be used to detect interesting things in the output.

For example to display the names of all setuid and setgid files in the VM:

# virt-ls -lR -d guest / | grep '^- [42]'
- 4755      12544 /bin/cgexec -
- 4755      32448 /bin/fusermount -
- 4755      78648 /bin/mount -
- 4755      43160 /bin/ping -
- 4755      47888 /bin/ping6 -
- 4755      34904 /bin/su -
- 4755      50432 /bin/umount -
[...]

To display all public writable directories:

# virt-ls -lR -d guest / | grep '^d ...7'
d 1777      12288 /tmp -
d 1777       4096 /tmp/.ICE-unix -
d 1777       4096 /tmp/.X11-unix -
d 1777       4096 /var/tmp -

To display files larger than 10MB in home directories:

# virt-ls -lR -d guest /home | awk '$3 >= 10*1024*1024'

Find regular files modified in the last 24 hours:

# virt-ls -lR -d guest --time-days / |
    grep '^-' |
    awk '$6 < 1'
[...]
- 0600        138   0   0   0 /home/rjones/.Xauthority
- 0600         69   0   0   0 /root/.xauthsdYvWC
- 0444         11   0   0   0 /tmp/.X0-lock
[...]

Also filesystem comparisons are made much simpler. So to display changes in files between a snapshot and the latest version of a VM you would simply do:

# virt-ls -lR -a snapshot.img / --uids --time-t --checksum > old
# virt-ls -lR -a current.img / --uids --time-t --checksum > new
# diff -u old new | less

Tip: Using a backing file to record file and registry changes, addendum

Part 1, part 2, and part 3

Inspired by the tricky and slow method to pull out file metadata that I showed in part 2 I added some features to virt-ls to make this much easier. These features are not yet in virt-ls. You will either have to apply this patch series or wait for libguestfs ≥ 1.11.9.

With the forthcoming virt-ls -lR option you will be able to extract the file metadata from a virtual machine easily. The output format is designed so that simple grep patterns can be used to detect interesting things in the output.

For example to display the names of all setuid and setgid files in the VM:

# virt-ls -lR -d guest / | grep '^- [42]'
- 4755      12544 /bin/cgexec -
- 4755      32448 /bin/fusermount -
- 4755      78648 /bin/mount -
- 4755      43160 /bin/ping -
- 4755      47888 /bin/ping6 -
- 4755      34904 /bin/su -
- 4755      50432 /bin/umount -
[...]

To display all public writable directories:

# virt-ls -lR -d guest / | grep '^d ...7'
d 1777      12288 /tmp -
d 1777       4096 /tmp/.ICE-unix -
d 1777       4096 /tmp/.X11-unix -
d 1777       4096 /var/tmp -

To display files larger than 10MB in home directories:

# virt-ls -lR -d guest /home | awk '$3 >= 10*1024*1024'

Find regular files modified in the last 24 hours:

# virt-ls -lR -d guest --time-days / |
    grep '^-' |
    awk '$6 < 1'
[...]
- 0600        138   0   0   0 /home/rjones/.Xauthority
- 0600         69   0   0   0 /root/.xauthsdYvWC
- 0444         11   0   0   0 /tmp/.X0-lock
[...]

Also filesystem comparisons are made much simpler. So to display changes in files between a snapshot and the latest version of a VM you would simply do:

# virt-ls -lR -a snapshot.img / --uids --time-t --checksum > old
# virt-ls -lR -a current.img / --uids --time-t --checksum > new
# diff -u old new | less

Tip: Uploading and downloading

Talked to someone today who was confused by how to get files in and out of a virtual machine disk image. Libguestfs, guestfish, virt-cat, virt-edit, virt-ls and virt-tar offer several ways to do this, and in this article I’m going to summarize all these different ways.

First of all, answer these questions:

1. Upload (ie. from your host into the disk image) or
   download (disk image down to host)?
2. Single file or lots of files?
3. Have you got / do you want a tarball, or a directory containing the individual files?
4. Do you want to download the files or just get a list of files?

I should note as usual that uploading or modifying a guest is only safe if the guest is shut off. If you try to modify a live guest you’ll get disk corruption, so watch out.

Downloading a single file

Easiest way is with virt-cat:

virt-cat GuestName /path/to/file > file

An alternative is to use guestfish download:

guestfish -i --ro -d GuestName \
    download /path/to/file ./localfile

Uploading a single file

If the file exists and it’s a text file that you want to edit, the easiest way is to use virt-edit:

virt-edit GuestName /path/to/file

To upload a single file in general, use guestfish upload:

guestfish -i -d GuestName \
    upload ./localfile /path/to/file

Downloading multiple files

Easiest way is probably to use the copy-out command:

guestfish -i --ro -d GuestName \
    copy-out /directory ./localdir

Uploading multiple files

Use guestfish copy-in command:

guestfish -i -d GuestName <<EOF
  mkdir /directory
  copy-in ./localdir /directory
EOF

If you want to quickly upload lots of files, the fastest method is to prepare an ISO (mkisofs) or squashfs containing the files and attach it as an extra disk. You can then copy files quickly to where you need them, eg:

guestfish -i -d GuestName -a my.iso <<EOF
  mount-ro /dev/sdb /mnt
  cp-a /mnt/files /someplace
EOF

Downloading or uploading a tarball

There is a high level tool called virt-tar which automates this. You can also do it semi-manually using the guestfish commands tar-in, tar-out, tgz-in, tgz-out (and more, see the guestfish documentation).

Listing files

Finally to list out files, use the high level virt-ls tool, or the low level guestfish commands ls (flat list of a single directory) or find0 (recursive listing of subdirectories).

guestfish -i, that’s i for “inspector” not “interactive”

I wrote this FAQ entry today …

---

• What’s the deal with guestfish -i?
• Why does virt-cat only work on a real VM image, but virt-df works on any disk image?
• What does “no root device found in this operating system image” mean?

These questions are all related at a fundamental level which may not be immediately obvious.

At the libguestfs API and guestfish level, a “disk image” is just a pile of partitions and filesystems.

In contrast, when the virtual machine boots, it mounts those filesystems into a consistent hierarchy such as:

 /          (/dev/sda2)
 |
 +-- /boot  (/dev/sda1)
 |
 +-- /home  (/dev/vg_external/Homes)
 |
 +-- /usr   (/dev/vg_os/lv_usr)
 |
 +-- /var   (/dev/vg_os/lv_var)

(or drive letters on Windows).

The libguestfs API however knows nothing about how filesystems are mounted. In general, a disk image that we just made up might not contain an operating system or “mountpoints” at all.

Nevertheless, users expect some tools (like virt-cat) to work with VM paths:

virt-cat fedora.img /var/log/messages

How does virt-cat know that /var is a separate partition?

The trick is a large external program called virt-inspector. Virt-inspector contains all sorts of tests and heuristics, so it can take a disk image about which nothing is known beforehand and work out both what it is, and how filesystems would be mounted.

Some tools, such as virt-cat, virt-edit, virt-ls and virt-tar use the same virt-inspector code to map VM paths. Other tools, such as virt-df and virt-list-filesystems operate entirely at the raw “big pile of filesystems” level of the libguestfs API, and don’t use virt-inspector code.

guestfish is in an interesting middle ground. If you use the -a and -m command line options, then you have to tell guestfish exactly how to add disk images and where to mount partitions. This is the raw API level.

However there is also a guestfish -i (for “inspector”) mode. In this mode, guestfish runs virt-inspector, and then (if successful) virt-inspector runs guestfish a second time with the correct -a and -m options. This is much slower, but usually simpler for the user. They can just do this and it all works:

guestfish -i SomeLibvirtGuest

The error “no root device found in this operating system image” is related to this. It means virt-inspector was unable to locate an operating system within the disk image you gave it. You might see this from programs like virt-cat if you try to run them on something which is just a disk image, not a virtual machine.

New tool: virt-ls

Want to list out the files in your virtual machine? Step forward virt-ls:

# virt-ls CentOS5x32 /
.autofsck
bin
boot
dev
etc
[...]

# virt-ls -R CentOS5x32 /etc/httpd
modules
conf
conf/httpd.conf
conf/magic
run
logs
conf.d
conf.d/welcome.conf
conf.d/README
conf.d/proxy_ajp.conf

See also: virt-cat, guestfish.

Do you want to suggest an addition to the virt-* tools family? Please comment here or let me know.