Clean up your spec files!

Modern RPMs don’t need any of the following. You can just delete them:

• %clean
• %defattr
• rm -rf $RPM_BUILD_ROOT or rm -rf %{buildroot}
• BuildRoot
• Group (thanks bochecha)

Nice RPM / git patch management trick

As far as I know, this trick was invented by Peter Jones. Edit: Or it could be ajax?

Parted in Fedora uses a clever method to manage patches with git and “git am”.

%prep
%setup -q
# Create a git repo within the expanded tarball.
git init
git config user.email "..."
git config user.name "..."
git add .
git commit -a -q -m "%{version} baseline."
# Apply all the patches on top.
git am %{patches}

The background is that there is a git repo somewhere else which stores the unpacked baseline parted tarball, plus patches (stored as commits) on top.

I assume that Peter exports the commits using git format-patch. At build time these are applied on top of the tarball using git am.

There are two clear advantages:

• No need to have lots of duplicate %patch lines in the spec file.
• git-am restores permissions and empty files properly, which regular patch does not do.

With libguestfs in RHEL 6 we have roughly 80 patches, so managing these patches is very tedious, and this will greatly simplify things.

Half-baked idea: “Try this patch” tool for RPMs

For more half-baked ideas, see my ideas tag.

RPM has some nice features for easily rebuilding packages. You can, for example, easily structure a source tarball so that an end user can build RPMs from it in a single step, and you can also easily rebuild an RPM from a source RPM. (See my recent notes on how to do all that here).

However for a lot of end users even these simple commands are too complex. And applying a patch to an RPM is beyond even that stage.

Here’s the idea: it’s a “try this patch” graphical tool. It takes a patch from a pastebin or email, and tries to apply it to an installed package. It downloads the source, attempts to apply the patch, rebuilds a new binary RPM, and installs it. (Of course it may not be possible to apply the patch, in which case it should either give the user a very simple message about what went wrong, or help more advanced users to manually fix rejects).

With this tool I could in confidence ask a user: “try this patch and tell me if it works”.

All the user has to do is to drag the patch file into the “try this patch” tool, and it will do the rest. If the patch doesn’t fix the problem, the tool lets the user “yum downgrade” to the previous version.

See also: A “view source” button for Fedora

Tip: Install RPMs in a guest

This script lets you install RPMs in a Fedora or RHEL guest (Update: offline guest in case that is not clear). It works by installing a “firstboot”-type script that actually does the install (avoiding various pitfalls of installing RPMs directly from libguestfs).

You use it like this:

# ./install-rpms.sh F14x64 xbill-2.1-2.fc11.x86_64.rpm
Uploading /etc/init.d/installrpms (firstboot script) ...
Uploading xbill-2.1-2.fc11.x86_64.rpm to /var/lib/installrpms/xbill-2.1-2.fc11.x86_64.rpm ...

Multiple RPMs can be given on the command line, and it can be used incrementally.

This is not quite a complete usable solution yet. What is really needed is a way to determine the dependencies between RPMs, and also determine what needs to be updated in a guest. These are jobs that can probably be done through the yum API.

#!/bin/bash -
#
# Install RPMs at next boot on a RHEL or Fedora VM.  This uploads the
# RPMs to the VM and creates a 'firstboot' script to install them when
# the VM boots next time.  You can use this script incrementally.
# Each time it runs, it adds further RPMs.  The RPM list is cleared
# when the VM boots.
#
# For more information, see
# https://rwmj.wordpress.com/   http://virt-tools.org/
#
# Usage:
#   install-rpms.sh GuestName *.rpm

# Parse command line.
if [ $# -lt 2 ]; then
    echo "install-rpms.sh GuestName *.rpm"
    exit 1
fi

set -e
guest="$1"; shift

# Create a temporary working directory.
tmpdir=$(mktemp -d)
trap "rm -rf '$tmpdir'" EXIT INT QUIT TERM

# Start up guestfish in remote control mode.
unset GUESTFISH_PID
eval `guestfish --listen -d "$guest" -i`
if [ -z "$GUESTFISH_PID" ]; then exit 1; fi
trap "guestfish --remote exit" EXIT INT QUIT TERM


# Check guest uses RPM for package management.
root=`guestfish --remote -- inspect-get-roots`
pkgfmt=`guestfish --remote -- inspect-get-package-format "$root"`
if [ "$pkgfmt" != "rpm" ]; then
    echo "$0: $guest: guest does not use RPM (package format = $pkgfmt)"
    exit 1
fi

# Upload/overwrite firstboot RPM installer script.
# This should run early (before network starts).
cat > $tmpdir/install.rc <<'EOF'
#!/bin/sh
#
# chkconfig: 345 15 85
# description: Install RPMs at next boot
#
### BEGIN INIT INFO
# Short-Description: Install RPMs at next boot
# Description: Install RPMs at next boot
### END INIT INFO

. /etc/rc.d/init.d/functions

[ -d /var/lib/installrpms ] || exit 0

start ()
{
    if [ `ls -1 /var/lib/installrpms/*.rpm 2>/dev/null | wc -l` -gt 0 ]; then
        echo -n $"Installing new packages: "
        yum -y install /var/lib/installrpms/*.rpm >> /var/log/installrpms.log
        if [ $? -eq 0 ]; then
            success;
            rm /var/lib/installrpms/*.rpm
        else
            failure
        fi
    fi
}

case "$1" in
  start)
        start
        ;;
  stop)
        # nothing
        ;;
  *)
        echo "Usage: $0 {start|stop}"
        exit 1
        ;;
esac
EOF
echo "Uploading /etc/init.d/installrpms (firstboot script) ..."
guestfish --remote -- upload $tmpdir/install.rc /etc/init.d/installrpms

# Set the script to run at boot.
guestfish --remote -- chmod 0755 /etc/init.d/installrpms
guestfish --remote -- ln-sf /etc/init.d/installrpms /etc/rc2.d/S15installrpms
guestfish --remote -- ln-sf /etc/init.d/installrpms /etc/rc3.d/S15installrpms
guestfish --remote -- ln-sf /etc/init.d/installrpms /etc/rc5.d/S15installrpms

# Make the RPMs directory.
guestfish --remote -- mkdir-p /var/lib/installrpms
guestfish --remote -- chmod 0755 /var/lib/installrpms

# Upload the RPMs.
for f in "$@"; do
    b="$(basename $f)"
    echo "Uploading $f to /var/lib/installrpms/$b ..."
    guestfish --remote -- upload "$f" /var/lib/installrpms/"$b"
done

Don’t forget your Epochs

I had a puzzler today. The RPM spec file contained:

BuildRequires: qemu-kvm < 0.12.2.0

The version of qemu-kvm was 0.12.1.2-… and you would think that 0.12.1.2 < 0.12.2.0. It should build, right? But rpmbuild consistently refused to find that qemu-kvm package.

My first thought was that RPM somehow needs to have a ≥ relation in order to find a package at all. I tried:

BuildRequires: qemu-kvm >= 0.12.1.0
BuildRequires: qemu-kvm < 0.12.2.0

but this failed in an even stranger way. It finds the right package, but then rejects it:

DEBUG util.py:256:  2:qemu-kvm-0.12.1.2-2.91.el6.x86_64
DEBUG util.py:256:  No Package Found for qemu-kvm < 0.12.2.0

But the clue to the answer is right there. qemu-kvm has an Epoch of 2 (hence the package name given is 2:qemu-kvm-0.12.1.2-…).

The fix is to write:

BuildRequires: qemu-kvm >= 2:0.12.1.0
BuildRequires: qemu-kvm < 2:0.12.2.0

But note a huge, hidden gotcha in RPMs with Epoch. You write:

BuildRequires: qemu-kvm >= 0.12

and it’s effectively meaningless. Any qemu-kvm version (with epoch 2) will match this, even version 0.11. In fact I could have written:

BuildRequires: qemu-kvm >= 0.95

and that would still have pulled in 2:qemu-kvm-0.12.1.2.

Half-baked ideas: verify the integrity of VMs

For more half-baked ideas, see my ideas tag.

This LWN article on OSSEC reminded me of an idea I had. We need a verify tool that can verify your VMs are not corrupted and don’t contain a rootkit. You can currently run a simple rpm -V command or one of the tools listed in that LWN article, but the problem is you have to run those commands inside the VM, thus relying on the VM itself not to have been corrupted. (You can also reboot the VM into a known-good state, eg. from a rescue ISO, but then you get downtime).

Obviously the answer is to examine the VM from the host, using libguestfs to grab the checksums directly from the filesystem.

You can get the checksums easily this way, but what do you compare them against and how do you know the checksums are good?

You would need to ask the distribution for a list of known-good checksums for the packages they publish. In fact you can do this reasonably easily. That information is available in the raw RPMs, or Red Hat’s RHN, and I’m quite sure you can get it from the Debian repos too. Windows? I don’t know specifically, but I guess either Microsoft publish this or you could derive it in some way.

So now if you are presented with some file from the VM and its checksum, like:

e2ed3c7d6d429716173fbd2d831d6e2855f1d20209da1238f75d1892a3074af5 /sbin/ldconfig

in theory we can verify this file was distributed in the signed package glibc-2.11.1-4.x86_64.rpm built on Thu 18 Mar 2010 04:51:51 PM GMT by a Fedora builder. (Using virt-inspector you can work out that the VM is a Fedora instance).

There’s still a subtle problem with this which I can’t work out. What happens if the attacker doesn’t directly install a rootkit, but instead replaces a binary with a version which has a known vulnerability. Say, a known root exploit in a package which the distro had previously shipped and later found to be vulnerable? This would allow the attacker to revisit the machine and acquire root, and run any software they want entirely in memory (libguestfs only sees the disk). For that you’d need not just a big database of all the files that distributions have shipped, but a list of files that have been obsoleted for security reasons.

There’s also a second problem: Although we can enumerate the goodness (all files are files that the distribution has shipped), we don’t know what to do with all the other files. Like user files, configuration files, /tmp, or files that just shouldn’t be there. To detect rootkits amongst those files, it’s starting to look like we’d have to enumerate badness and we don’t want to go there.

Tip: mock-build Rawhide packages on RHEL 5

The Fedora build system Koji runs on RHEL 5 Xen and builds everything on top of that using mock. This can lead to some rather difficult to debug problems where your package builds and tests OK for you on your local Rawhide machine, but fails in Koji. The reason it can fail in Koji is because it is running on the RHEL 5 Linux kernel (2.6.18). Your program, or any program you depend on during the build, might make assumptions about system calls that work for a Rawhide kernel, but fail for a RHEL 5 kernel.

Reproducing these bugs is difficult. Hopefully this posting should be a good start.

Koji is doing roughly the equivalent of this command (on a RHEL 5 host):

mock -r fedora-rawhide-x86_64 --rebuild your.src.rpm

That command doesn’t work straightaway. There are some things you have to install and upgrade first before that works:

1. Install RHEL 5 (or use CentOS or another no-cost alternative).
2. Install EPEL.
3. Install or update yum, python-hashlib, mock.
4. Use /usr/sbin/vigr to add yourself to the “mock” group.
5. The version of RPM from RHEL 5 is too old to understand the new xz-based compression format used by Rawhide RPMs. You have to build the Fedora 12 RPM (NB: Fedora 13 RPM definitely doesn’t work because it requires Python 2.6). The Fedora 12 specfile is a starting point, but it won’t work directly. There are some small changes you have to make, and a single patch to the source code, but hopefully those will be obvious. Update: Here for a short time is a scratch build of the Fedora 12 RPM made to work on RHEL 5.4. Once you’ve built the new rpm RPM (!), install it.

At this point you can use the mock command above to test-build SRPMs using the unusual RHEL 5 kernel / Rawhide userspace combination.

RPM dependency size viewer now available

The interactive RPM dependency size viewer (discussed earlier here and here) has now reached a stable 1.0 version. You can grab source and x86-64 RPMs from my website.

The diagram below shows an example, ocaml-camlp4-devel, a rather large OCaml package which I’m partly responsible for.

If you were to take a fresh Fedora 10 machine and yum install ocaml-camlp4-devel, you’d need a minimum of 404 MB (I mean, including things like glibc and the filesystem and everything else that camlp4 ultimately depends on). Where does the space go?

Click to enlarge

If you look at the top of the enlarged diagram, you can see that ocaml-camlp4-devel is not a tiny package, consuming 33 MB just for itself. The packages below it, ocaml, ocaml-camlp4, ocaml-runtime and glibc are ordered by the greatest to smallest total size (size inc. all dependencies) left to right. But the width shown is the incremental size, roughly speaking meaning if we got rid of that package, how much would we save.

We can see the ocaml-runtime pulls in util-linux-ng which has a huge chain of requirements (including python, interestingly). So one area to look at would be whether that dependency can be removed or narrowed down.

If you mouse-over a particular package, the colours change to show “parent packages” (above us, so sky blue), and dependent packages (below us, so grassy green):

Click to enlarge

There are still many ways to improve this viewer. I’m now out of time on this little project, but there is a git repository (kindly supplied by Red Hat / Jim Meyering) so if you want to hack away and supply patches, please send them. If you analyzed your own RPMs, let us know in the comments.

Size of RPM dependencies

Continuing the theme of minimal Fedora installs, I had a go at visualizing RPM dependencies and the size of those dependencies.

This problem is harder than I thought. The pretty filelight diagrams in my last post are possible because filesystems are simple trees (if you discount hard links).

However package dependencies are directed graphs, often containing loops and diamonds. If pam pulls in cracklib-dicts, that’s not good because cracklib-dicts is big. But if it was pulled in by another package already, then pam got it “for free” so do we need to worry? You can massively change your views on whether a particular package is excessively large just by changing the way you divide up these shared dependencies between packages.

What I’m working on is a more interactive tool that will let you explore these possibilities — so you will be able to “remove” a dependency and see how that changes the situation. Also you should be able to explore different ways of dividing up shared deps.

That part is not written yet, but I do have a visualisation of the dependencies of some packages already. Please keep in mind that the width of each bar is the incremental cost of the dependency (in terms of all extra data that it pulls in) and it does not mean that a particular package is bloated or excessive.

The rpmdepsize program is an ad-hoc mixture of python and OCaml. Caveat emptor.

auto-buildrequires

I just pushed a new release of auto-buildrequires, a super little tool for finding the BuildRequires that your package needs automatically. Just replace rpmbuild with auto-br-rpmbuild:

$ auto-br-rpmbuild -ta auto-buildrequires-0.9.tar.gz
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.wphzM5
[ lots of stuff here ... ]
+ exit 0
BuildRequires: bash = 3.2.30.fc10.x86_64
BuildRequires: binutils = 2.18.50.0.9.8.fc10.x86_64
BuildRequires: coreutils = 6.12.19.fc10.x86_64
BuildRequires: cpio = 2.9.90.2.fc10.x86_64
BuildRequires: diffutils = 2.8.1.21.fc9.x86_64
BuildRequires: elfutils = 0.140.1.fc10.x86_64
BuildRequires: file = 4.26.4.fc10.x86_64
BuildRequires: filesystem = 2.4.19.1.fc10.x86_64
BuildRequires: findutils = 1:4.4.0.1.fc10.x86_64
BuildRequires: gawk = 3.1.6.2.fc10.x86_64
BuildRequires: gcc = 4.3.2.7.x86_64
BuildRequires: glibc-devel = 2.9.3.x86_64
BuildRequires: glibc-headers = 2.9.3.x86_64
BuildRequires: grep = 2.5.1a.61.fc10.x86_64
BuildRequires: gzip = 1.3.12.7.fc10.x86_64
BuildRequires: kernel-headers = 2.6.27.19.170.2.35.fc10.x86_64
BuildRequires: make = 1:3.81.14.fc10.x86_64
BuildRequires: net-tools = 1.60.91.fc10.x86_64
BuildRequires: sed = 4.1.5.11.fc10.x86_64
BuildRequires: tar = 2:1.20.5.fc10.x86_64

It currently lists everything that your package needs. It doesn’t know which dependencies are in the core mock / Koji system, because these don’t seem to be listed anywhere as far as I can tell.

The program works using an LD_PRELOAD hack that intercepts any system calls that try to read or write files, and at the end a small Perl script works out which installed packages each file belongs to.

There are currently some problems. For example, autoconf tends to check for a C++ and FORTRAN compiler if one is installed, even if it is never requested or used, so you’ll get these as false requirements. Nevertheless it’s a great tool that I have been using all the time as a replacement for plain rpmbuild and as a final check that I’ve got the BuildRequires approximately right.

Developers, please see the git repository.