Tag Archives: virt-df

Tip: find out when filesystems get full with virt-df (working version!)

Thanks go to “S” for pointing out the mistake in my last posting. It’s easy to fix though.

When virt-df is run with the –csv option, it outputs a Comma-Separated Values format spreadsheet. CSV is not totally straightforward to parse and so you should leave it up to a specialized tool. Of course I recommend my own tool csvtool which is available in Fedora, Debian and Ubuntu.

The basic plan is to use csvtool to drop the first row (the column headings row), and pull out columns 1, 2 and 6 from the rest of the output. Those columns are respectively the virtual machine name, the filesystem, and the %usage:

$ virt-df --csv | csvtool drop 1 - | csvtool col 1-2,6 -
Debian5x64,/dev/debian5x64/home,35.6%
Debian5x64,/dev/debian5x64/root,60.9%
Debian5x64,/dev/debian5x64/tmp,3.3%
Debian5x64,/dev/debian5x64/usr,40.5%
Debian5x64,/dev/debian5x64/var,15.2%
Debian5x64,/dev/vda1,20.3%
RHEL620100329n0x64,/dev/vda1,7.4%
RHEL620100329n0x64,/dev/vg_rhel620100329n0/lv_root,16.7%
Ubuntu910x64,/dev/vda1,42.5%
[...snipped...]

Adding the previous awk test gives the desired output, all filesystems with ≥ 60% usage:

$ virt-df --csv |
    csvtool drop 1 - |
    csvtool col 1-2,6 - |
    csvtool readable - |
    awk 'strtonum(substr($3,0,length($3)-1))>=60 {print}'
Debian5x64         /dev/debian5x64/root            60.9% 
F13Rawhidex64      /dev/vg_f13rawhide/lv_root      66.0% 
Windows7x32        /dev/vda2                       74.6% 
Windows7x64        /dev/vda2                       90.7% 

This can be placed in a script or cron job.

1 Comment

Filed under Uncategorized

Tip: find out when filesystems get full with virt-df

Another easy tip for the day: Want to receive a warning when one of your virtual machines’ filesystems is filling up? Use virt-df, awk, and cron like this.

Note: I’ve set this up so I can do it as non-root, but that meant I had to add myself to the ‘disk’ group first, so that virt-df can read out the contents of the disks of the virtual machines. There’s one extra thing needed which is to set the LIBVIRT_DEFAULT_URI environment variable to access the global libvirtd socket.

$ export LIBVIRT_DEFAULT_URI='qemu+unix:///system?socket=/var/run/libvirt/libvirt-sock-ro'
$ virt-df | awk 'strtonum(substr($5,0,length($5)-1))>=60 {print}'
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%

You can set the threshold to whatever you want. In the above awk code it’s set to 60%, you’d probably want it higher.

Now put those two commands into a script file somewhere, and create a cronjob. The following one runs daily:

MAILTO=you@example.com
0 1 * * * /usr/local/etc/diskcheck.sh

What do you do when a VM is running out of space? Why, you run virt-resize of course.


Ob-geekiness: The full virt-df output from one of my test hosts. This one is used for a lot of interop testing:

$ virt-df
Filesystem                           1K-blocks       Used  Available  Use%
RHEL6200910210x32:/dev/vda1             198337      36582     151515   19%
RHEL6200910210x32:/dev/vg_rhel6200910210x32/lv_root
                                       9147776    5305876    3377212   59%
RHEL6201002033x64:/dev/vda1             495844      32396     437848    7%
RHEL6201002033x64:/dev/vg_rhel6201002033x64/lv_root
                                      18102140    4097800   13084788   23%
Debian5x64:/dev/debian5x64/home        3555936    1264800    2110504   36%
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Debian5x64:/dev/debian5x64/tmp          309401      10294     283133    4%
Debian5x64:/dev/debian5x64/usr         3539776    1434740    1925224   41%
Debian5x64:/dev/debian5x64/var         1741648     264632    1388544   16%
Debian5x64:/dev/vda1                    233335      47272     173615   21%
Ubuntu910x64:/dev/vda1                 9827520    4180524    5147780   43%
RHEL6Alpha3x64:/dev/vda1                198337      22879     165218   12%
RHEL6Alpha3x64:/dev/vg_rhel6alpha3x64/lv_root
                                       8180152    4174904    3589712   52%
RHEL54Betax64:/dev/VolGroup00/LogVol00
                                      15109112    8695548    5633688   58%
RHEL54Betax64:/dev/vda1                 101086      12449      83418   13%
F10x32:/dev/VolGroup00/LogVol00        9191640    3083532    5642856   34%
F10x32:/dev/vda1                        194442      20706     163697   11%
F12x64:/dev/vda1                        198337      22782     165315   12%
F12x64:/dev/vg_f12x64/lv_root          9115576    4396520    4256004   49%
F13Rawhidex64:/dev/vda1                 198337      60031     128066   31%
F13Rawhidex64:/dev/vg_f13rawhide/lv_root
                                       9115576    6018664    2633860   67%
F12x64preview:/dev/vda1                 198337      22782     165315   12%
F12x64preview:/dev/vg_f12x64/lv_root   9115576    4781384    3871140   53%
Win2003x32:/dev/vda1                  20956760    3053092   17903668   15%
VSphere:/dev/vda1                     31447204    8159028   23288176   26%
Windows7x32:/dev/vda1                   102396      24712      77684   25%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda1                   102396      24704      77692   25%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%
CentOS5x32:/dev/VolGroup00/LogVol00    9014656    4069840    4479512   46%
CentOS5x32:/dev/vda1                    101086      36210      59657   36%

Update There is a small but important bug in this code. Please see the fixed version.

3 Comments

Filed under Uncategorized

guestfish -i, that’s i for “inspector” not “interactive”

I wrote this FAQ entry today …


  • What’s the deal with guestfish -i?
  • Why does virt-cat only work on a real VM image, but virt-df works on any disk image?
  • What does “no root device found in this operating system image” mean?

These questions are all related at a fundamental level which may not be immediately obvious.

At the libguestfs API and guestfish level, a “disk image” is just a pile of partitions and filesystems.

In contrast, when the virtual machine boots, it mounts those filesystems into a consistent hierarchy such as:

 /          (/dev/sda2)
 |
 +-- /boot  (/dev/sda1)
 |
 +-- /home  (/dev/vg_external/Homes)
 |
 +-- /usr   (/dev/vg_os/lv_usr)
 |
 +-- /var   (/dev/vg_os/lv_var)

(or drive letters on Windows).

The libguestfs API however knows nothing about how filesystems are mounted. In general, a disk image that we just made up might not contain an operating system or “mountpoints” at all.

Nevertheless, users expect some tools (like virt-cat) to work with VM paths:

virt-cat fedora.img /var/log/messages

How does virt-cat know that /var is a separate partition?

The trick is a large external program called virt-inspector. Virt-inspector contains all sorts of tests and heuristics, so it can take a disk image about which nothing is known beforehand and work out both what it is, and how filesystems would be mounted.

Some tools, such as virt-cat, virt-edit, virt-ls and virt-tar use the same virt-inspector code to map VM paths. Other tools, such as virt-df and virt-list-filesystems operate entirely at the raw “big pile of filesystems” level of the libguestfs API, and don’t use virt-inspector code.

guestfish is in an interesting middle ground. If you use the -a and -m command line options, then you have to tell guestfish exactly how to add disk images and where to mount partitions. This is the raw API level.

However there is also a guestfish -i (for “inspector”) mode. In this mode, guestfish runs virt-inspector, and then (if successful) virt-inspector runs guestfish a second time with the correct -a and -m options. This is much slower, but usually simpler for the user. They can just do this and it all works:

guestfish -i SomeLibvirtGuest

The error “no root device found in this operating system image” is related to this. It means virt-inspector was unable to locate an operating system within the disk image you gave it. You might see this from programs like virt-cat if you try to run them on something which is just a disk image, not a virtual machine.

2 Comments

Filed under Uncategorized

Tip: use SystemTap to monitor SELinux changes to files

Something unknown was changing the labels on certain devices behind my back. We couldn’t find out what it was using ordinary diagnostics, so I decided to investigate if we could do this with SystemTap. I quickly found an existing script to monitor changes in ordinary file attributes. This won’t work for SELinux labels though because those are stored in ext2/3/4 extended attributes (xattrs).

Basically I had to modify that script to monitor calls to setxattr instead.

Using the LXR I found that the call is implemented in Linux in fs/xattr.c, function vfs_setxattr. I had to modify the script to probe that kernel function, and the parameters are slightly different too.

I also had to install the correct kernel-{,PAE-}debuginfo package corresponding to my installed kernel. This is how SystemTap is able to resolve symbols in the current kernel.

/* Watch changes to xattrs on an inode.
 * http://rwmj.wordpress.com/2010/03/09/tip-use-systemtap-to-monitor-selinux-changes-to-files/
 */

probe kernel.function("vfs_setxattr")
{
  dev_nr = $dentry->d_inode->i_sb->s_dev
  inode_nr = $dentry->d_inode->i_ino

  if (inode_nr == $1)
    printf ("%s(%d) %s 0x%x/%u %d %s %s\n",
      execname(), pid(), probefunc(), dev_nr, inode_nr, uid(),
      kernel_string ($name), kernel_string_n ($value, $size))
}

Then run it with:

# stap -v /tmp/inodewatchxattr.stp inodenum

The bug turned out to be udevd, which I don’t think anyone was expecting …

libvirtd(4338) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:svirt_image_t:s0:c177,c272
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0

All in all, I’m impressed with SystemTap. It’s a simple, strongly-typed, sane programming language with type inference. Thankfully Python was not an influence on it.

1 Comment

Filed under Uncategorized

Ubuntu 9.10 packages for libguestfs

By popular demand, I’ve built Ubuntu packages for the latest libguestfs, guestfish, virt-inspector, virt-cat and virt-df:

http://www.annexia.org/tmp/debian/

These are experimental. If they break, you get to keep both pieces.

One point in particular is there is no Perl Sys::Virt package in Ubuntu, which means you have to supply this yourself, else virt-inspector etc. won’t work.

2 Comments

Filed under Uncategorized

virt-df on Xen

Virt-df works on Xen so you can display free disk space in a Xen VM.

# virt-df -h 2>&1 | grep -v '/dev/kqemu'
Domain-0 seems to have no disk devices
Filesystem                                Size       Used  Available  Use%
RHEL39FV64:/dev/hda1                     98.7M      13.5M      80.1M 18.8%
RHEL39FV64:/dev/hda2                      6.8G       1.8G       4.7G 31.0%
RHEL39FV64:/dev/hdb1                     30.4M       1.0M      27.8M  8.5%
RHEL39FV32:/dev/hda1                     98.7M      14.5M      79.1M 19.9%
RHEL39FV32:/dev/hda2                      6.8G       1.5G       4.9G 27.8%
RHEL39FV32:/dev/hdb1                     30.4M       1.0M      27.8M  8.5%
RHEL53PV64:/dev/VolGroup00/LogVol00       6.6G       2.1G       4.2G 37.3%
RHEL53PV64:/dev/hda1                     98.7M      13.1M      80.5M 18.5%
RHEL48PV64:/dev/VolGroup00/LogVol00       6.7G       2.1G       4.3G 36.6%
RHEL48PV64:/dev/hda1                     98.7M       9.2M      84.4M 14.5%
RHEL52PV32:/dev/VolGroup00/LogVol00       6.6G       2.5G       3.7G 43.5%
RHEL52PV32:/dev/hda1                     98.7M      12.7M      80.9M 18.0%
RHEL53PV32:/dev/VolGroup00/LogVol00       6.6G       1.9G       4.4G 34.1%
RHEL53PV32:/dev/hda1                     98.7M      12.8M      80.8M 18.2%
RHEL52FV32:/dev/VolGroup00/LogVol00       6.7G       2.6G       3.8G 43.5%
RHEL52FV32:/dev/hda1                     98.7M      14.5M      79.1M 19.8%
RHEL52FV32:/dev/hdb1                     30.3M       1.4M      27.4M  9.7%
RHEL48PV32:/dev/VolGroup00/LogVol00       6.7G       1.8G       4.6G 31.7%
RHEL48PV32:/dev/hda1                     98.7M       8.6M      85.0M 13.9%
RHEL47FV32:/dev/VolGroup00/LogVol00       6.7G       1.9G       4.5G 32.6%
RHEL47FV32:/dev/hda1                     98.7M      13.9M      79.7M 19.2%
RHEL47PV32:/dev/VolGroup00/LogVol00       6.7G       1.8G       4.6G 31.3%
RHEL47PV32:/dev/hda1                     98.7M       8.6M      85.0M 13.9%

It’s a bit noisy giving a warning about opening /dev/kqemu for each guest. I’ve grepped out the warnings above. The “Domain-0 seems to have no disk devices” warning is bug 538041.

Leave a comment

Filed under Uncategorized

Browsing guests using FUSE

What’s interesting about this screenshot is that I’m browsing into a guest filesystem using the GNOME file browser (nautilus), with the guest mounted using FUSE and libguestfs. You can visit directories, open files, and drag files out of the guest (but not drag them into the guest yet — we haven’t enabled file writes at the moment).

(This is /var/log from a Fedora 12 Alpha guest, displayed in a Fedora 11 host. Notice the “Free Space: 5.7 GB” label which accurately shows the amount of free space in the guest filesystem).

Update

FUSE support via the guestmount command is in libguestfs 1.0.77.

1 Comment

Filed under Uncategorized

Graphical “virt-df”


(Click to enlarge if WordPress.com squashes the image)

What I learned yesterday:

  • Programming Gtk apps is horrible. Gtk leaves far too much fiddly detail up to the programmer, and doesn’t have sensible defaults.
  • Programming Gtk apps with OCaml has an impedance mismatch (Gtk is imperative and highly mutable, OCaml is functional and mostly immutable). Luckily this isn’t Haskell, so we can drop into mutable code when we need to.
  • Programming with threads and locks is time-consuming for even the simplest things.
  • On the other hand, it is possible to write a decent multi-threaded app which accesses libvirt and libguestfs and leaves the user interface responsive. Here is a proof by existence of that.

The code will appear in ocaml/examples in the libguestfs source repo shortly. The code is available here.

Exercises for the programmer:

  • Display the mount point of each partition.
  • Double click on partition to display the tree of files inside.

8 Comments

Filed under Uncategorized