Tag Archives: rhel

Odd/scary RHEL 5 bug

Yesterday my colleague gave me a RHEL 5 VM disk image which failed to boot after converting it using the latest virt-v2v.  Because it booted before conversion but not afterwards, the fingers naturally pointed at something that we were doing during the conversion process. Which is not unusual as v2v conversion is highly complex.

Screenshot_xen-pv-rhel5.8-x86_64
The “GRUB _” prompt after conversion

The thing is that we don’t reinstall grub during conversion, but we do edit a few grub configuration files. Could editing grub configuration cause this error?

I wanted to understand what the grub-legacy “GRUB _” prompt means. There are lots and lots and lots of people reporting this bug (eg), but as is often the case I could find no coherent explanation anywhere of what grub-legacy means when it gets into this state. Lots of the blind leading the blind, and random suggestions about how people had rescued such machines (probably coincidentally), but no hard data anywhere. So I had to go back to first principles and debug qemu to find out what’s happening just before the message is printed.

Tip: To breakpoint qemu when the Master Boot Record (first sector) is loaded, do:

target remote tcp::1234
set architecture i8086
b *0x7c00
cont

After an evening of debugging, I found that it’s the first sector (known in grub-legacy as “stage 1”) which prints the GRUB<space> message. (The same happens to be true of grub2). The stage 1 boot sector has, written into it at a fixed offset, the location of the /boot/grub/stage2 file, ie. the literal disk start sector and length of this file. It sends BIOS int $0x13 commands to load those sectors into memory at address 0x8000, and jumps there to start the stage 2 of grub. The boot sector is 512 bytes, so there’s no luxury to do anything except print 5 characters. It’s after the stage2 file has been loaded when all the nice graphical stuff happens.

Unfortunately in the image after conversion, the stage2 data loaded into memory was all zeroes, and that’s why the boot fails and you see GRUB<space><cursor> and then the VM crashes.

The mystery was how conversion could be changing the location of the /boot/grub/stage2 file so that it could no longer be loaded at the fixed offset encoded in the boot sector.

This morning it dawned on me what was really happening …

The new virt-v2v tries very hard to avoid copying any unused data from the guest, just to save time. No point wasting time copying deleted files and empty space. This makes virt-v2v very fast, but it has an unusual side-effect: If a file is deleted on the source, the contents of the file are not copied over to the target, and turn into zeroes.

It turns out if you take the source disk image and simply zero all of the empty space in /boot, then the source doesn’t boot either, even though virt-v2v is not involved. Yikes … this could be a bug in RHEL 5. Grub is generating a bootloader that references a deleted file.

This is where we are right now with this bug. It appears that a valid sequence of steps can make a RHEL 5 bootloader that references a deleted file, but still works as long as you never overwrite the sectors used by that file.

I have written a simple test script that you can download to find out if your RHEL ≤ 6 virtual machines could be affected by this problem. I’m interested if anyone else sees this. I ran the test over a selection of RHEL 3 – 5 guests, and could not find any which had the problem, but my collection is not very extensive, and there are likely to be common modes in how they were created.

The next steps will likely be to test a lot more RHEL 5 installs to see if this bug is really common or a strange one-off. I will also probably add a workaround to virt-v2v so it doesn’t trim the boot partition — the reason is that we cannot go back and fix old RHEL 5 installs, we have to work with them if they are broken. If it turns out to be a real bug in RHEL 5 then we will need to issue a fix for that.

3 Comments

Filed under Uncategorized

libguestfs RHEL 7.1 preview packages (yes, really)

RHEL 7 isn’t out yet, but if you’re using the the RHEL 7 RC, you’re on one of our beta programs, or you can wait for RHEL or CentOS 7.0 to be released, then you can upgrade libguestfs with these RHEL 7.1 libguestfs preview packages.

Amongst the new features:

Leave a comment

Filed under Uncategorized

virt-builder RHEL 7 release candidate

You can now install RHEL 7 release candidate (very unofficially) through virt-builder on Fedora 20).

Just do:

$ virt-builder rhel-7rc
[   0.0] Downloading: ***
[   1.0] Planning how to build this image
[   1.0] Uncompressing
[   6.0] Opening the new disk
[  53.0] Setting a random seed
[  53.0] Setting passwords
Setting random password of root to ***
[  53.0] Finishing off
Output: rhel-7rc.img
Output size: 6.0G
Output format: raw
Total usable space: 4.8G
Free space: 4.0G (82%)

To be honest with you I couldn’t get networking to work, so if it works at all for you then let us know how. The network worked once I supplied the right qemu options.

Leave a comment

Filed under Uncategorized

RHEL 7 beta is out

In case you didn’t see the announcement, RHEL 7 beta is available. I’ve been running RHEL 7 in several virtual machines for a while, and it works great for me.

The DVD image is ftp.redhat.com/pub/redhat/rhel/beta/7/x86_64/iso/rhel-everything-7.0-beta-1-x86_64-dvd.iso but it’s probably better to use the following virt-install command:

virt-install -n rhel7betax64 -r 4096 \
  --cpu=host --vcpus=4 \
  --os-type=linux --os-variant=fedora19 \
  --disk path=/dev/vg_data/rhel7betax64 \
  -l ftp://ftp.redhat.com/pub/redhat/rhel/beta/7/x86_64/os/

Leave a comment

Filed under Uncategorized

CentOS Dojo and Barbecue (UK)

It looks like I might be doing a short talk at the CentOS Dojo and Barbecue at Aldershot, UK, Friday 12th July 2013.

It’ll probably be about scripting/programming libvirt and the virt tools, but mainly it’ll be a chance for Q&A about any virtualization topic in RHEL / CentOS.

Also they have a BBQ — with beer! Sadly since I’m driving there I won’t be able to drink any of the beer.

(Thanks Karanbir Singh, Justin Clift)

Leave a comment

Filed under Uncategorized

libguestfs on RHEL 5

I’ve spent the past couple of weeks [sic] getting a modern libguestfs into RHEL 5. RHEL 5 was first released in March 2007 and is thus over 5 years old.

It’s hard work.

Firstly RHEL 5 lacks virtio-serial, which is the mechanism used by the library to talk to the appliance. We used to use a TCP connection, but we gave that up since it’s slow, awkward and possibly insecure. To get RHEL 5 going again, I had to forward port the old TCP code into a modern branch of libguestfs.

Secondly RHEL 5 lacks a lot of features that libguestfs presents through the API. I got most of them working by carefully fixing and rewriting code, but some of them are impossible to do (eg. btrfs support).

Altogether I wrote about 30 patches, with about half going upstream, and the other half sitting in an “old Linux”-only side branch.

The release notes are attached below, and the packages should be available in EPEL 5 in a few days.


libguestfs for EPEL 5
by Richard W.M. Jones

This package contains modern libguestfs for EPEL 5. Although libguestfs is supplied in RHEL 5, the version supplied is very old and is only intended to be used with virt-v2v from RHEL 5. It is only available in the RHEL-5-V2V channel.

Therefore there exists a need for a modern libguestfs for RHEL 5 users. This package supplies this through the community-supported EPEL repository.

(1) Installation

After enabling the EPEL repository, install libguestfs in the normal way using yum. No post-installation configuration should be needed.

You must have a non-Xen kernel installed on the host. Note that the non-Xen kernel just needs to be installed, it does *not* need to be running (you can run libguestfs under Xen if you want). The non-Xen kernel is used to boot the libguestfs appliance.

You must use this package together with the version of qemu-kvm from RHEL 5 (usually called kvm-83-NNN.el5). Alternate versions of qemu, KVM, using $LIBGUESTFS_QEMU or QEMU wrappers, will probably not work.

Hardware virtualization will greatly improve the performance of libguestfs, but it is not required. libguestfs cannot use hardware virtualization when run in a Xen domain (not even dom0). For more performance tips, read guestfs-performance(1).

(2) Getting help

This package is supported voluntarily by the EPEL community and by the upstream libguestfs maintainers.

It is not supported by Red Hat.

It cannot be used with the virt-v2v package from RHEL 5. If you need to use virt-v2v on RHEL 5, use the libguestfs package from the same RHN channel. However we would advise you to use RHEL 6 or later because that version of virt-v2v is substantially more powerful.

To get help with this package, contact the libguestfs maintainers. See http://libguestfs.org for links to the mailing list, IRC, and how to file bugs.

(3) Features not available

The following features are not compatible with RHEL 5 and have been removed completely:

  • PHP bindings
  • GObject, GObject introspection, anything that uses GObject Introspection such as the Javascript bindings
  • Erlang bindings
  • The btrfs filesystem.
  • guestfs_fstrim
  • guestfs_wipefs
  • Various APIs that take UUIDs (eg. guestfs_mkswap_U)
  • virtio-scsi
  • Internationalization of any non-C programs.

(4) Features that may not work

The following features may not work in this version of libguestfs:

  • LUKS (encrypted filesystems within guests)
  • MD (RAID devices within guests)
  • FUSE, guestmount, the mount-local API. These appear to have some problems because of the older version of FUSE in RHEL 5.
  • virt-sysprep --script option (because it requires FUSE).
  • The ntfsclone_* APIs only partially work. In particular, the metadataonly flag is not working, and maybe other parts of this API won’t work.
  • guestfs_vfs_label cannot read labels from NTFS filesystems.
  • guestfs_blkid may return fewer fields than expected on a more modern Linux system.
  • guestfs_txz_in and guestfs_txz_out don’t work.
  • guestfs_utimens doesn’t let you set the time on a symbolic link (this is a limitation of the RHEL 5 kernel).
  • guestfs_mkswap_label followed by guestfs_swapon_label does not work. The reasons are not well understood.
  • Adding drives using non-virtio-blk interfaces.

(5) Other notes

An artificial pause of 1 second has been added after unmounting any filesystem.

*Without* the 1 second pause, the following test case will fail sometimes:

  guestfish <<EOF
  sparse test1.img 1G
  run
  part-disk /dev/sda mbr
  mkfs msdos /dev/sda1
  mount /dev/sda1 /
  touch /foo
  umount /
  mkfs ntfs /dev/sda1
  # Next command would fail:
  mount /dev/sda1 /
  vfs-type /dev/sda1 -x
  EOF

With the 1 second pause, this case passes. The reason is not understood.

Leave a comment

Filed under Uncategorized

libguestfs 1.16.15 preview packages for RHEL 6.2

I’ve updated the libguestfs preview packages for RHEL 6.2/6.3. These contain many new features and bug fixes compared to RHEL 6.2.

An easy way to use these packages is to run this command, which creates a yum repository pointing to the preview packages:

# cat <<EOF > /etc/yum.repos.d/libguestfs-RHEL-6.3-preview.repo
[libguestfs-RHEL-6.3-preview]
name=libguestfs RHEL 6.3 preview - x86_64
baseurl=http://people.redhat.com/~rjones/libguestfs-RHEL-6.3-preview/
enabled=1
gpgcheck=0
EOF

Leave a comment

Filed under Uncategorized