In the meantime, Ray Strode just pointed out that this works:
><rescue> setsid bash < /dev/ttyS0
Quiescing a filesystem lets you take a consistent snapshot or backup at the block device level. If your server uses SAN storage, then probably your SAN lets you take snapshots of the SCSI LUNs at any time. But if you try doing this while the server is under load you’ll (at best) get a “crash consistent” snapshot, where the journal has to be replayed when the copy of the filesystem is mounted, and at worst you’ll get data corruption, particularly with ext3 defaults.
Quiescing tells the filesystem to make things consistent at the disk / block device level. The journal won’t need to be replayed, and the superblock is marked as if you’d unmounted the device. A snapshot taken at this stage will be consistent, at least at the filesystem level (applications don’t know what is happening, so you could still see things like half-written transactions in databases).
The downside to quiescing a filesystem is that it generally causes writes to be blocked, eventually bringing the whole system to a grinding halt. SAN snapshots can be done very quickly though, so the time between a “freeze” and “thaw” operation is usually brief.
Very recent versions of util-linux-ng have an fsfreeze command that lets you freeze or thaw filesystems at the command line. Use with care!
Freezing filesystems also has an application for virtual machines. Our new guest agent will support freezing filesystems so that you can coordinate a consistent backup or snapshot from outside the guest.
If you have Rawhide and the most recent virt-rescue you can play with freezing filesystems without breaking anything:
$ rm -f test.img $ truncate -s 1G test.img $ virt-rescue test.img ><rescue> mkfs.ext4 /dev/vda ><rescue> mount /dev/vda /sysroot
From another window you can see that the image is not consistent. If you were to snapshot the image now the filesystem would at least require journal recovery when mounted:
$ file test.img test.img: Linux rev 1.0 ext4 filesystem data (needs journal recovery) (extents) (large files) (huge files)
But by issuing fsfreeze in the guest we can make it consistent:
><rescue> fsfreeze -f /sysroot
$ file test.img test.img: Linux rev 1.0 ext4 filesystem data (extents) (large files) (huge files)
.. allowing us to take a snapshot or copy of the block device (test.img) in a consistent state.
I also added the
--append option and
--selinux option. The first allows you to provide more memory to the virtual appliance. The second lets you specify extra Linux boot arguments. The third enables SELinux support.
Glauber “glommer” Costa says guestfish is “the most kicking ass software ever”. Cheers Glauber. I wrote virt-rescue for glommer because he wanted to “fsck” interactively to fix his broken virtual machine. Libguestfs wasn’t designed for making ad-hoc interactive changes, but glommer gave me the idea that we could reuse the libguestfs appliance quite easily to make an interactive rescue shell for virtual machines, analogous to the rescue CDs that are common for physical machines. And so virt-rescue came about.
The not so good one appeared in SearchEnterpriseLinux.com and is little more than blogspam, and I speak as someone who used to write similar trash (it’s very attractive to Google, and before Red Hat I used to work in SEO). I don’t think the author actually tried guestfish or libguestfs, it seems he just copied stuff out from our manpages. I doubt SearchEnterpriseLinux care since it is my opinion that they exist to make widely syndicated spam articles just to attract page views by misleading users of search engines.
Virt-rescue isn’t just for virtual machines. You can run it on any raw file or disk image.
We are trying to make virt-df use the same calculations and therefore display the same output as df. So today I wanted to find out what the real “df” command would show on some test filesystems I had prepared. Virt-rescue “to the rescue”:
$ virt-rescue tools/test.img [...] Welcome to virt-rescue, the libguestfs rescue shell. Note: The contents of / are the rescue appliance. You have to mount the guest's partitions under /sysroot before you can examine them. ><rescue> mkdir /sysroot/lv1 ><rescue> mkdir /sysroot/lv2 ><rescue> mkdir /sysroot/lv3 ><rescue> mount /dev/VG/LV1 /sysroot/lv1 ><rescue> mount /dev/VG/LV2 /sysroot/lv2 ><rescue> mount /dev/VG/LV3 /sysroot/lv3 ><rescue> df -h Filesystem Size Used Avail Use% Mounted on /dev/dm-1 31M 28K 30M 1% /sysroot/lv1 /dev/dm-2 31M 395K 29M 2% /sysroot/lv2 /dev/dm-3 62M 36K 59M 1% /sysroot/lv3 ><rescue> df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/dm-1 8192 11 8181 1% /sysroot/lv1 /dev/dm-2 8192 11 8181 1% /sysroot/lv2 /dev/dm-3 16384 11 16373 1% /sysroot/lv3 ><rescue> df Filesystem 1K-blocks Used Available Use% Mounted on /dev/dm-1 31728 28 30064 1% /sysroot/lv1 /dev/dm-2 31729 395 29696 2% /sysroot/lv2 /dev/dm-3 63472 36 60160 1% /sysroot/lv3
Unbootable virtual machine? Here are three useful guestfish commands to help. (You can also consider using virt-rescue).
$ guestfish -i Rawhide Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems. Type: 'help' for help with commands 'quit' to quit the shell ><fs> ls /boot/ System.map-184.108.40.206-9.fc13.x86_64 System.map-220.127.116.11-21.fc13.x86_64 System.map-2.6.33-0.40.rc7.git0.fc13.x86_64 config-18.104.22.168-9.fc13.x86_64 config-22.214.171.124-21.fc13.x86_64 config-2.6.33-0.40.rc7.git0.fc13.x86_64 [...]
Use the “edit”, “emacs” or “vi” commands to edit grub.conf:
><fs> vi /boot/grub/grub.conf
From here you can change the boot kernel, change it to boot in single user mode, enable the grub menu, remove the “rhgb quiet” option so you can see boot messages, and much more.
When the kernel panics because it cannot mount root, it’s often because the initrd or initramfs is broken in some way. Two commands help here:
><fs> initrd-list /boot/initramfs-2.6.33-0.40.rc7.git0.fc13.x86_64.img | less ><fs> initrd-cat /boot/initramfs-2.6.33-0.40.rc7.git0.fc13.x86_64.img init | less
The first command lists all the files in the initrd, which lets you see if the right drivers got included for the (virtual) hardware. The second command lists out the init script — which is the shell script that runs first before the OS proper starts to boot.
$ virt-rescue F11x64 Welcome to virt-rescue, the libguestfs rescue shell. Note: The contents of / are the rescue appliance. You have to mount the guest's partitions under /sysroot before you will be able to examine them. ><rescue> /sbin/e2fsck /dev/vg_f11x64/lv_root [...] ><rescue> mount /dev/vg_f11x64/lv_root /sysroot ><rescue> ls /sysroot/ bin dev home lib64 media opt root selinux sys usr boot etc lib lost+found mnt proc sbin srv tmp var ><rescue> sync ><rescue> umount /sysroot ><rescue> exit