Tag Archives: fedora

Half-baked ideas: Demand-revealing referenda applied to Fedora features

For more half-baked ideas, see the ideas tag

Recently Lennart proposed dropping — and then unilaterally dropped a few days later — support for TCP wrappers in systemd. I haven’t used TCP wrappers for a long time, but there are some who do, and for those people dropping features like this provokes strong feelings. How should we conduct a survey or vote to decide what features to add or drop in software projects?

One way would be for all users [however that is defined] to have a vote. The problem with that is that a feature which few people use, but which really matters for those that use it would probably get dropped by a simple majority vote.

A better idea would be to use an economic system called a revealed preference. The idea is by asking people to risk their own money on the outcome of a vote, you hope to get a truer picture of their feelings. This technique also excludes moaners with lots of time on their hands to argue on mailing lists.

Chris Dillow (who incidentally writes a really great blog) has a worked example of a demand-revealing referendum which you should go and read.

Let’s try this with the systemd / TCP wrappers example. I’m going to have six voters. Four are mostly apathetic about the feature. But two of them use it, and one of those is going to have to change his whole infrastructure around if TCP wrappers goes away.

But first I have to assign a cost to this feature1. Unlike Chris’s Trident example, removing TCP wrappers from systemd is cheap. But it’s not completely free, assuming that Lennart is going to have to write some code, communicate the change, update documentation and so on. I’ll say it costs £12, which is £2/voter.

Let’s see how our six people might vote:

        Cost  Benefit
Alice   £ 2   £    5
Bob     £ 2   £   10
Charlie £ 2   £    2
Diane   £ 2   £    2
Eleanor £ 2   £  -50
Fred    £ 2   £-1000
TOTAL   £12   £-1031

Alice and Bob perceive some small benefit to the change because they think it’ll make systemd cleaner. Fred is the one who is going to have to make significant changes to his company network, and he’s not happy. Charlie and Diane are completely neutral.

The net benefits are calculated by subtracting the benefit from the cost:

        Cost  Benefit  Net benefit
                     (Benefit - Cost)
Alice   £ 2   £    5   £    3
Bob     £ 2   £   10   £    8
Charlie £ 2   £    2   £    0
Diane   £ 2   £    2   £    0
Eleanor £ 2   £  -50   £  -52
Fred    £ 2   £-1000   £-1002
TOTAL   £12   £-1031   £-1043

One thing you should notice from the “TOTAL” row is that there is no (expressed) net benefit to the change. Fred’s large negative vote has soured the whole thing. It sounds unfair that Fred is able to block this, but read on …

All we’ve done so far is asked people to guess numbers. To make it a revealed preference, we have to get people to pay real money. In this case, we’re going to ask some people to pay what is called a Clarke tax.

The tax is paid only by those who “win” (or get their way). Eleanor and Fred in this example get their way and we keep TCP wrappers in systemd. They pay the social cost of their winning that is incurred by the rest of the voters. To calculate the tax you have to remove Eleanor and Fred from the table to find the net benefit without them:

        Net benefit
Alice   £    3
Bob     £    8
Charlie £    0
Diane   £    0
TAX     £   11

Eleanor and Fred have to pay £11 in tax. (I’m unclear if this is split equally or pro-rata according to their vote). They pay this real money to Alice and Bob. Even after paying the tax, Eleanor and Fred and still better off (according to their claim). Alice and Bob have been compensated for their lost benefit.

The bids in the auction are sealed — ie. people shouldn’t be able to collude. Let’s imagine however that Alice estimated Fred’s £1000 cost and tried to neutralize it by claiming a £2000 benefit to the change. Alice would win (total net benefit becomes positive £952), but she and the other winners would have to pay a tax of £1046. This is a costly victory, but the money goes some way to compensating Fred for the changes he has to make to his company network.

1One issue with this is the estimate of the cost of the feature. I’m sure systemd developers will claim that although dropping TCP wrappers costs a bit of money in the short term, it pays dividends in the long term because of reduced code maintenance and bug reports. In other words that the cost is negative. You have to be able to provide credible costs for this to work, but I think you can express that by having the feature developers join in the voting process, in other words, revealing their true preferences as well.

Further reading

Leave a comment

Filed under Uncategorized

Analysis of the size of libguestfs dependencies

In libguestfs ≥ 1.26 we are going to start splitting the package up into smaller dependencies. Since the full libguestfs package has lots of dependencies because it has to be able to process lots of obscure filesystems, the question is how best to split up the dependencies? We could split off, say, XFS support into a subpackage, but how do we know if that will save any space?

Given the set of dependencies, we want to know the incremental cost of adding another dependency.

We can get an exact measure of this by using supermin to build a chroot containing the set of dependencies, and a second chroot containing the set of dependencies + the additional package. Then we simply compare the sizes of the two chroots. The advantage of using supermin is that the exact same script [see end of posting] will work for Fedora and Debian/Ubuntu since supermin hides the complexity of dealing with the different package managers through its package manager abstraction.

The results of this, using the libguestfs appliance dependencies, on Fedora 20, sorted by dependency size, with my comments added:

  1. gdisk adds 25420 KB

    This is a surprising result in first place, since gdisk is a fairly small, unassuming C++ program (only ~11KLoC). My initial thought was it must be something to do with being written in C++, but I tested that and it’s not true. The real problem is that gdisk depends on libicu (a Unicode library) which adds 24.6 MB to the appliance. [Note: this issue has been fixed in Rawhide.]

  2. lvm2 adds 19432 KB

    The default disk layout of many Linux distros uses LVM so this and similar dependencies have to stay in base libguestfs.

  3. binutils adds 16604 KB

    This is a sorry tale. The one file we use from binutils is /usr/bin/strings (33KB). Unfortunately this single binary pulls in a huge dependency (even worse, it’s a development package, and this causes problems on production systems). I don’t really understand why strings is included in binutils.

  4. gfs2-utils adds 9648 KB
  5. zfs-fuse adds 5208 KB

    Split off in the proposed reorganization.

  6. ntfsprogs adds 4572 KB
  7. e2fsprogs adds 4312 KB

    Most Linux distros use ext4, and we want to support Windows out of the box, so these are included in base libguestfs.

  8. xfsprogs adds 3532 KB

    Split off in the proposed reorganization.

  9. iproute adds 3180 KB

    We use /sbin/ip to set up the network card inside the appliance. It’s a shame this “better” replacement for ifconfig is so large.

  10. tar adds 2896 KB
  11. btrfs-progs adds 2800 KB
  12. openssh-clients adds 2428 KB
  13. parted adds 2420 KB
  14. jfsutils adds 1668 KB
  15. genisoimage adds 1644 KB
  16. syslinux-extlinux adds 1420 KB
  17. augeas-libs adds 1404 KB
  18. iputils adds 1128 KB
  19. reiserfs-utils adds 1076 KB
  20. mdadm adds 1032 KB
  21. strace adds 976 KB
  22. lsof adds 972 KB
  23. vim-minimal adds 912 KB
  24. rsync adds 812 KB
  25. libldm adds 616 KB
  26. psmisc adds 592 KB
  27. nilfs-utils adds 520 KB
  28. hfsplus-tools adds 480 KB

The test script used to produce these results:

#!/bin/bash -

# NB: For this program to work, you must have the following
# packages (or as many as possible) installed locally.
pkgs='acl attr augeas-libs bash binutils bsdmainutils btrfs-progs
bzip2 coreutils cpio cryptsetup cryptsetup-luks diffutils dosfstools
e2fsprogs extlinux file findutils gawk gdisk genisoimage gfs2-utils
grep grub grub-pc gzip hfsplus hfsplus-tools hivex iproute iputils
jfsutils kernel kmod less libaugeas0 libcap libcap2 libhivex0 libldm
libpcre3 libselinux libsystemd-id128-0 libsystemd-journal0 libxml2
libyajl2 linux-image lsof lsscsi lvm2 lzop mdadm module-init-tools
mtools nilfs-utils ntfs-3g ntfsprogs openssh-clients parted pcre
procps procps-ng psmisc reiserfs-utils reiserfsprogs rsync scrub sed
strace syslinux syslinux-extlinux systemd sysvinit tar udev ufsutils
util-linux util-linux-ng vim-minimal vim-tiny xfsprogs xz xz-utils
yajl zerofree zfs-fuse'

# These are the packages (from the above list) that we want to test.
testpkgs="$pkgs"

# Helper function to construct an appliance and see how big it is.
function appliance_size
{
    set -e
    supermin --prepare -o /tmp/supermin.d "$@" >&/dev/null
    supermin --build -f chroot -o /tmp/appliance.d \
      /tmp/supermin.d >&/dev/null
    du -s /tmp/appliance.d | awk '{print $1}'
}

# Construct entire appliance to see how big that would be.
totalsize=`appliance_size $pkgs`

# Remove each package from the list in turn, and find out
# how much extra that package contributes.
for p in $testpkgs; do
    opkgs=
    for o in $pkgs; do
        if [ $o != $p ]; then opkgs="$opkgs $o"; fi
    done
    size=`appliance_size $opkgs`
    extra=$(($totalsize - $size))

    echo $p adds $extra KB
done

1 Comment

Filed under Uncategorized

Red Hat speakers at FOSDEM 2014

If you’re planning to be at FOSDEM in Belgium this weekend, then do come and see some of the Red Hat folk who are giving talks. The full list is here or use this handy printed guide [PDF].

(Thanks Garrett)

Leave a comment

Filed under Uncategorized

fedpkg recipes

At some point I intend to implement a view source tool, but in the mean time here are some useful recipes.

fedpkg is a great little command line tool for checking out, building and modifying Fedora packages. On this page I’m collecting a few simple recipes, useful for all Fedora users. You will need to yum install fedpkg but apart from that no other preparation is required. All the recipes use binutils as an example, but you can use any package name.

Do you have a recipe using fedpkg? Post it in the comments.

Upgrade to the Rawhide version of a Fedora package

fedpkg clone -a -B binutils
cd binutils/master

master corresponds to Rawhide. There are also directories for each branch, eg. f20 for Fedora 20. At this point you may need to install build dependencies:

sudo yum-builddep binutils.spec

Build the package on your local machine. This builds it against whatever version of Fedora is installed on your machine:

fedpkg local

To install the package:

sudo yum install */*.rpm

Apply a source patch to a Fedora package

Note: One aim of the proposed view-source tool will be to make this almost completely automated.

fedpkg clone -a -B binutils
cd binutils/fXX

(Choose your current Fedora version instead of fXX, or use master for Rawhide)

Apply the patch:

cp /tmp/test.patch .
rpmdev-bumpspec -r binutils.spec
vi binutils.spec

Near the top of the spec file, add:

Patch9999: test.patch

In the %prep section, add:

%patch9999 -p1

Now build and install the patched package as usual:

sudo yum-builddep binutils.spec
fedpkg local
sudo yum localupdate */*.rpm

Examine the source of a package

Sure, you can look at the upstream package sources, but often you want to look at the older / patched source as it applies to your version of Fedora, since that might be different …

fedpkg clone -a -B binutils
cd binutils/fXX
fedpkg prep

Change fXX to your version of Fedora. The prep subcommand will unpack the tarball and apply all the patches.

2 Comments

Filed under Uncategorized

Alignment errors on Fedora ARM

Here’s what you should do if you get a compile error like this on Fedora ARM:

error: cast increases required alignment of target type [-Werror=cast-align]
  1. If it’s an easy fix with little chance of breaking things, then fix it.
  2. If it’s on a performance critical path, especially if it causes a measurable slow down, then fix it.
  3. Otherwise, disable the warning. One way is to add:
    #pragma GCC diagnostic ignored "-Wcast-align"
    

BTW I will delete any comment on this post unless you show you have read the next part:

The background is that on certain architectures (ARM and MIPS are the main ones) the processor cannot load or store values from memory which is not aligned. Say you had a protocol which sent a 1 byte length field followed by a 4 byte data field:

+-------+-------+-------+-------+-------+- - - -
| len   | <---------- data -----------> |
+-------+-------+-------+-------+-------+- - - -

If you loaded this into malloc’d memory and used a C struct like this to access it:

struct {
  uint8_t len;
  int32_t data;
} __attribute__((packed));

then accesses to p->data might trap. The malloc’d memory, hence the struct, is aligned, and so the data field is not aligned.

What happens in Fedora is the kernel gets called. It sees a FAULT_CODE_ALIGNMENT fault, looks at the failed load instruction, emulates it, then returns back to your code. This fixup step is slow.

However it’s not a bug. Fedora only runs on recent ARM chips that can now do some unaligned accesses in hardware and always generate traps for ones they can’t handle.

Your code will still run fine assuming you don’t change the /proc/cpu/alignment setting. But if these fixups are frequent it is plausible that they could cause a performance problem.

Fixing these can be hard, and is more likely to create bugs (as we found out when trying to fix alignment bugs in hivex).

Since the most popular development architecture (x86) has always been able to handle unaligned accesses in hardware, developers are going to keep adding unaligned accesses to their programs. In the Intel Haswell chips, there isn’t even a performance penalty (in fact, it’s likely to be faster to squeeze your structs as much as possible, even if it makes them unaligned). ARM has gradually been adding the ability to handle unaligned access too, so eventually one hopes the problem will go away for developers.

5 Comments

Filed under Uncategorized

Fedora 21 has a working OCaml ARM64

Update: Thanks to Peter Robinson, there is now a build of OCaml for aarch64 in the Fedora repository.

I have backported the upstream ARM64 support into Fedora 21’s OCaml, so you can now use it to generate native ARM64/AArch64 binaries. If you don’t have hardware, use qemu to emulate it instead.

6 Comments

Filed under Uncategorized

How to run aarch64 binaries on an x86-64 host using qemu userspace emulation

Note: This is the Fedora version of this Debian document so full credit must go to Debian and SuSE for assembling the bits.

Qemu (not quite upstream) now has ARM 64 bit emulation. It only does the userspace emulation at the moment, although system emulation is being worked on. This is much faster than the ARM “Foundation Model” (basically ARM’s proprietary emulator).

You will need to clone SuSE’s ARM64 qemu git repository and compile it statically linked (configure it with --target-list=arm64-linux-user --static). Static glib2 for Fedora is here.

You will also need to download this Fedora 19 arm64 disk image and convert it into a root filesystem (unfortunately this process requires, just temporarily, 14 GB of free disk space — should have used virt-sparsify!!):

mkdir /tmp/arm64
cd /tmp/arm64
tar zxf F19-aarch64-efi.tar.xz
virt-tar-out -a ./F19-aarch64-efi/aarch64-efi.img / - |
    sudo tar xf -
rm -rf F19-aarch64-efi

So now what you should have is a statically linked qemu-arm64 binary (that’s the userspace arm64 emulator), and a root filesystem containing lots of arm64 binaries. To run them we’ll need to set up a systemd binfmt handler, which will notice when we’re about to try to run an arm64 binary and inject our emulator so it’s possible to run it on our host (which is obviously not arm64 since that hardware is unobtainium for most mortals).

Copy the statically linked qemu-arm64 binary into the chroot, ie:

cp ./arm64-linux-user/qemu-arm64 /tmp/arm64/

Create a binfmt file /etc/binfmt.d/qemu-arm64.conf with the content below. Note that it refers to the absolute path /qemu-arm64 which (will be) correct once we chroot into the arm64 filesystem.

# AArch64 binaries.
:qemu-arm64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/qemu-arm64:OC
$ sudo service systemd-binfmt restart

Check that /proc/sys/fs/binfmt_misc/qemu-arm64 has been created with the expected contents.

Now you’re ready to go!

$ cd /tmp/arm64
$ sudo mount -o bind /dev ./dev
$ sudo mount -o bind /dev/pts ./dev/pts
$ sudo mount -o bind /proc ./proc
$ sudo mount -o bind /sys ./sys
$ sudo chroot .
# file bin/ls
bin/ls: ELF 64-bit LSB executable, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux
3.7.0,
BuildID[sha1]=0x31bcaf7de1bb3c6a4400983ac31f0dff1bbfc394, stripped
# /bin/ls
bin   dev  home  lib64	     media  opt   qemu-arm64  run   srv  tmp  var
boot  etc  lib	 lost+found  mnt    proc  root	      sbin  sys  usr
# uname -a
Linux choo.home.annexia.org 3.8.0 #1 SMP Thu Oct 10
14:11:18 UTC 2013 aarch64 aarch64 aarch64 GNU/Linux

And somehow convince yourself you really are running arm64 binaries!

One problem I had is the supplied chroot is quite minimal and I’m not exactly sure how to install more. But it’s fun to play and a lot faster than the Foundation Model.

Update #1

# arch
aarch64

Update #2

The /etc/yum.repos.d/stage4.repo file in the disk image above is broken. Peter Robinson notes that you can use http://arm.koji.fedoraproject.org/aarch64/stage4/ as a yum repo for installing more packages.

Update #3

Outside the chroot — when you’re running commands in the chroot — you’ll see qemu interpreting the arm64 binaries in ps output:

$ ps ax | grep qemu
26051 pts/14   S      0:03 /qemu-arm64 /bin/bash -i
28170 pts/14   S      0:00 /qemu-arm64 /bin/su rjones
28171 pts/14   S      0:01 /qemu-arm64 /bin/bash
28226 pts/14   Sl+    0:03 /qemu-arm64 /bin/git clone git://libvirt.org/libvirt.git
28228 pts/14   S+     0:12 /qemu-arm64 /usr/libexec/git-core/git index-pack --stdin -v --fix-thin --keep=fetch-pack 28226 on choo.home.annexia.org

7 Comments

Filed under Uncategorized