Tag Archives: virt-rescue

Tip: Get Ctrl-C working in virt-rescue

We need to fix virt-rescue so that ^C works. It needs some Perl hacking though.

In the meantime, Ray Strode just pointed out that this works:

><rescue> setsid bash < /dev/ttyS0 

Leave a comment

Filed under Uncategorized

Freezing filesystems

Ric Wheeler and Christoph Hellwig were quick to point out I was wrong about something: Linux now has a standard API for freezing or “quiescing” filesystems.

Quiescing a filesystem lets you take a consistent snapshot or backup at the block device level. If your server uses SAN storage, then probably your SAN lets you take snapshots of the SCSI LUNs at any time. But if you try doing this while the server is under load you’ll (at best) get a “crash consistent” snapshot, where the journal has to be replayed when the copy of the filesystem is mounted, and at worst you’ll get data corruption, particularly with ext3 defaults.

Quiescing tells the filesystem to make things consistent at the disk / block device level. The journal won’t need to be replayed, and the superblock is marked as if you’d unmounted the device. A snapshot taken at this stage will be consistent, at least at the filesystem level (applications don’t know what is happening, so you could still see things like half-written transactions in databases).

The downside to quiescing a filesystem is that it generally causes writes to be blocked, eventually bringing the whole system to a grinding halt. SAN snapshots can be done very quickly though, so the time between a “freeze” and “thaw” operation is usually brief.

Very recent versions of util-linux-ng have an fsfreeze command that lets you freeze or thaw filesystems at the command line. Use with care!

Freezing filesystems also has an application for virtual machines. Our new guest agent will support freezing filesystems so that you can coordinate a consistent backup or snapshot from outside the guest.

If you have Rawhide and the most recent virt-rescue you can play with freezing filesystems without breaking anything:

$ rm -f test.img
$ truncate -s 1G test.img
$ virt-rescue test.img
><rescue> mkfs.ext4 /dev/vda
><rescue> mount /dev/vda /sysroot

From another window you can see that the image is not consistent. If you were to snapshot the image now the filesystem would at least require journal recovery when mounted:

$ file test.img
test.img: Linux rev 1.0 ext4 filesystem data (needs journal recovery) (extents) (large files) (huge files)

But by issuing fsfreeze in the guest we can make it consistent:

><rescue> fsfreeze -f /sysroot
$ file test.img
test.img: Linux rev 1.0 ext4 filesystem data (extents) (large files) (huge files)

.. allowing us to take a snapshot or copy of the block device (test.img) in a consistent state.

Leave a comment

Filed under Uncategorized

virt-rescue doc refresh

virt-rescue is like a Rescue CD, but for virtual machines, and without the need for a CD. I refreshed the virt-rescue documentation yesterday to make it clearer and provide more examples.

I also added the -m option, --append option and --selinux option. The first allows you to provide more memory to the virtual appliance. The second lets you specify extra Linux boot arguments. The third enables SELinux support.

Leave a comment

Filed under Uncategorized

guestfish kicks ass .. maybe

Glauber “glommer” Costa says guestfish is “the most kicking ass software ever”. Cheers Glauber. I wrote virt-rescue for glommer because he wanted to “fsck” interactively to fix his broken virtual machine. Libguestfs wasn’t designed for making ad-hoc interactive changes, but glommer gave me the idea that we could reuse the libguestfs appliance quite easily to make an interactive rescue shell for virtual machines, analogous to the rescue CDs that are common for physical machines. And so virt-rescue came about.

Another couple of articles have appeared about libguestfs recently. This article [pdf] is the better one written by Red Hat’s Sadique Puthen for Linux For You magazine.

The not so good one appeared in SearchEnterpriseLinux.com and is little more than blogspam, and I speak as someone who used to write similar trash (it’s very attractive to Google, and before Red Hat I used to work in SEO). I don’t think the author actually tried guestfish or libguestfs, it seems he just copied stuff out from our manpages. I doubt SearchEnterpriseLinux care since it is my opinion that they exist to make widely syndicated spam articles just to attract page views by misleading users of search engines.

11 Comments

Filed under Uncategorized

Tip: virt-rescue, not just for virtual machines!

Virt-rescue isn’t just for virtual machines. You can run it on any raw file or disk image.

We are trying to make virt-df use the same calculations and therefore display the same output as df. So today I wanted to find out what the real “df” command would show on some test filesystems I had prepared. Virt-rescue “to the rescue”:

$ virt-rescue tools/test.img
[...]
Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> mkdir /sysroot/lv1
><rescue> mkdir /sysroot/lv2
><rescue> mkdir /sysroot/lv3
><rescue> mount /dev/VG/LV1 /sysroot/lv1
><rescue> mount /dev/VG/LV2 /sysroot/lv2
><rescue> mount /dev/VG/LV3 /sysroot/lv3
><rescue> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/dm-1              31M   28K   30M   1% /sysroot/lv1
/dev/dm-2              31M  395K   29M   2% /sysroot/lv2
/dev/dm-3              62M   36K   59M   1% /sysroot/lv3
><rescue> df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/dm-1               8192      11    8181    1% /sysroot/lv1
/dev/dm-2               8192      11    8181    1% /sysroot/lv2
/dev/dm-3              16384      11   16373    1% /sysroot/lv3
><rescue> df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/dm-1                31728        28     30064   1% /sysroot/lv1
/dev/dm-2                31729       395     29696   2% /sysroot/lv2
/dev/dm-3                63472        36     60160   1% /sysroot/lv3

Leave a comment

Filed under Uncategorized

Tip: VM won’t boot, troubleshoot with guestfish

Unbootable virtual machine? Here are three useful guestfish commands to help. (You can also consider using virt-rescue).

1. Edit /boot/grub/grub.conf

$ guestfish -i Rawhide

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ls /boot/
System.map-2.6.32.1-9.fc13.x86_64
System.map-2.6.32.3-21.fc13.x86_64
System.map-2.6.33-0.40.rc7.git0.fc13.x86_64
config-2.6.32.1-9.fc13.x86_64
config-2.6.32.3-21.fc13.x86_64
config-2.6.33-0.40.rc7.git0.fc13.x86_64
[...]

Use the “edit”, “emacs” or “vi” commands to edit grub.conf:

><fs> vi /boot/grub/grub.conf

From here you can change the boot kernel, change it to boot in single user mode, enable the grub menu, remove the “rhgb quiet” option so you can see boot messages, and much more.

2. Look at the /init script

When the kernel panics because it cannot mount root, it’s often because the initrd or initramfs is broken in some way. Two commands help here:

><fs> initrd-list /boot/initramfs-2.6.33-0.40.rc7.git0.fc13.x86_64.img | less
><fs> initrd-cat /boot/initramfs-2.6.33-0.40.rc7.git0.fc13.x86_64.img init | less

The first command lists all the files in the initrd, which lets you see if the right drivers got included for the (virtual) hardware. The second command lists out the init script — which is the shell script that runs first before the OS proper starts to boot.

3 Comments

Filed under Uncategorized

virt-rescue

Virt-rescue is a new tool proposed for inclusion in libguestfs. It lets you get a rescue shell on your virtual machine, good for making quick, interactive, unstructured fixes:

$ virt-rescue F11x64

Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you will be able to examine them.

><rescue> /sbin/e2fsck /dev/vg_f11x64/lv_root
[...]
><rescue> mount /dev/vg_f11x64/lv_root /sysroot
><rescue> ls /sysroot/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var
><rescue> sync
><rescue> umount /sysroot
><rescue> exit

Of course we encourage you to continue using libguestfs and guestfish for making properly structured changes through a stable, programmable API!

Leave a comment

Filed under Uncategorized