Tip: Updating RHEL 7.1 cloud images using virt-customize and subscription-manager

Red Hat provide RHEL KVM guest and cloud images. At time of writing, the last one was built in Feb 2015, and so undoubtedly contains packages which are out of date or insecure.

You can use virt-customize to update the packages in the cloud image. This requires the libguestfs subscription-manager feature which will only be available in RHEL 7.3, but see here for RHEL 7.3 preview packages. Alternatively you can use Fedora ≥ 22.

$ virt-customize \
  -a rhel-guest-image-7.1-20150224.0.x86_64.qcow2 \
  --sm-credentials 'USERNAME:password:PASSWORD' \
  --sm-register --sm-attach auto \
  --update
[   0.0] Examining the guest ...
[  17.2] Setting a random seed
[  17.2] Registering with subscription-manager
[  28.8] Attaching to compatible subscriptions
[  61.3] Updating core packages
[ 976.8] Finishing off

1. You should probably use --sm-credentials USERNAME:file:FILENAME to specify your password using a file, rather than having it exposed on the command line.
2. The command above will leave the image template registered to RHN. To unregister it, add --sm-unregister at the end.

libguestfs, virt tools, virt-v2v, virt-p2v Summit video

The video is up here. It’s only available to Red Hat subscribers. You’ll need an RHN account of some sort for access.

The handouts which go with the talk are here.

Thanks to Sean Huck who did a good job editing the video into shape.

Half-baked ideas: verify the integrity of VMs

For more half-baked ideas, see my ideas tag.

This LWN article on OSSEC reminded me of an idea I had. We need a verify tool that can verify your VMs are not corrupted and don’t contain a rootkit. You can currently run a simple rpm -V command or one of the tools listed in that LWN article, but the problem is you have to run those commands inside the VM, thus relying on the VM itself not to have been corrupted. (You can also reboot the VM into a known-good state, eg. from a rescue ISO, but then you get downtime).

Obviously the answer is to examine the VM from the host, using libguestfs to grab the checksums directly from the filesystem.

You can get the checksums easily this way, but what do you compare them against and how do you know the checksums are good?

You would need to ask the distribution for a list of known-good checksums for the packages they publish. In fact you can do this reasonably easily. That information is available in the raw RPMs, or Red Hat’s RHN, and I’m quite sure you can get it from the Debian repos too. Windows? I don’t know specifically, but I guess either Microsoft publish this or you could derive it in some way.

So now if you are presented with some file from the VM and its checksum, like:

e2ed3c7d6d429716173fbd2d831d6e2855f1d20209da1238f75d1892a3074af5 /sbin/ldconfig

in theory we can verify this file was distributed in the signed package glibc-2.11.1-4.x86_64.rpm built on Thu 18 Mar 2010 04:51:51 PM GMT by a Fedora builder. (Using virt-inspector you can work out that the VM is a Fedora instance).

There’s still a subtle problem with this which I can’t work out. What happens if the attacker doesn’t directly install a rootkit, but instead replaces a binary with a version which has a known vulnerability. Say, a known root exploit in a package which the distro had previously shipped and later found to be vulnerable? This would allow the attacker to revisit the machine and acquire root, and run any software they want entirely in memory (libguestfs only sees the disk). For that you’d need not just a big database of all the files that distributions have shipped, but a list of files that have been obsoleted for security reasons.

There’s also a second problem: Although we can enumerate the goodness (all files are files that the distribution has shipped), we don’t know what to do with all the other files. Like user files, configuration files, /tmp, or files that just shouldn’t be there. To detect rootkits amongst those files, it’s starting to look like we’d have to enumerate badness and we don’t want to go there.