How to run aarch64 binaries on an x86-64 host using qemu userspace emulation

Note: This is the Fedora version of this Debian document so full credit must go to Debian and SuSE for assembling the bits.

Qemu (not quite upstream) now has ARM 64 bit emulation. It only does the userspace emulation at the moment, although system emulation is being worked on. This is much faster than the ARM “Foundation Model” (basically ARM’s proprietary emulator).

You will need to clone SuSE’s ARM64 qemu git repository and compile it statically linked (configure it with --target-list=arm64-linux-user --static). Static glib2 for Fedora is here.

You will also need to download this Fedora 19 arm64 disk image and convert it into a root filesystem (unfortunately this process requires, just temporarily, 14 GB of free disk space — should have used virt-sparsify!!):

mkdir /tmp/arm64
cd /tmp/arm64
tar zxf F19-aarch64-efi.tar.xz
virt-tar-out -a ./F19-aarch64-efi/aarch64-efi.img / - |
    sudo tar xf -
rm -rf F19-aarch64-efi

So now what you should have is a statically linked qemu-arm64 binary (that’s the userspace arm64 emulator), and a root filesystem containing lots of arm64 binaries. To run them we’ll need to set up a systemd binfmt handler, which will notice when we’re about to try to run an arm64 binary and inject our emulator so it’s possible to run it on our host (which is obviously not arm64 since that hardware is unobtainium for most mortals).

Copy the statically linked qemu-arm64 binary into the chroot, ie:

cp ./arm64-linux-user/qemu-arm64 /tmp/arm64/

Create a binfmt file /etc/binfmt.d/qemu-arm64.conf with the content below. Note that it refers to the absolute path /qemu-arm64 which (will be) correct once we chroot into the arm64 filesystem.

# AArch64 binaries.
:qemu-arm64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/qemu-arm64:OC

$ sudo service systemd-binfmt restart

Check that /proc/sys/fs/binfmt_misc/qemu-arm64 has been created with the expected contents.

Now you’re ready to go!

$ cd /tmp/arm64
$ sudo mount -o bind /dev ./dev
$ sudo mount -o bind /dev/pts ./dev/pts
$ sudo mount -o bind /proc ./proc
$ sudo mount -o bind /sys ./sys
$ sudo chroot .
# file bin/ls
bin/ls: ELF 64-bit LSB executable, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux
3.7.0,
BuildID[sha1]=0x31bcaf7de1bb3c6a4400983ac31f0dff1bbfc394, stripped
# /bin/ls
bin   dev  home  lib64	     media  opt   qemu-arm64  run   srv  tmp  var
boot  etc  lib	 lost+found  mnt    proc  root	      sbin  sys  usr
# uname -a
Linux choo.home.annexia.org 3.8.0 #1 SMP Thu Oct 10
14:11:18 UTC 2013 aarch64 aarch64 aarch64 GNU/Linux

And somehow convince yourself you really are running arm64 binaries!

One problem I had is the supplied chroot is quite minimal and I’m not exactly sure how to install more. But it’s fun to play and a lot faster than the Foundation Model.

Update #1

# arch
aarch64

Update #2

The /etc/yum.repos.d/stage4.repo file in the disk image above is broken. Peter Robinson notes that you can use http://arm.koji.fedoraproject.org/aarch64/stage4/ as a yum repo for installing more packages.

Update #3

Outside the chroot — when you’re running commands in the chroot — you’ll see qemu interpreting the arm64 binaries in ps output:

$ ps ax | grep qemu
26051 pts/14   S      0:03 /qemu-arm64 /bin/bash -i
28170 pts/14   S      0:00 /qemu-arm64 /bin/su rjones
28171 pts/14   S      0:01 /qemu-arm64 /bin/bash
28226 pts/14   Sl+    0:03 /qemu-arm64 /bin/git clone git://libvirt.org/libvirt.git
28228 pts/14   S+     0:12 /qemu-arm64 /usr/libexec/git-core/git index-pack --stdin -v --fix-thin --keep=fetch-pack 28226 on choo.home.annexia.org