How to run aarch64 binaries on an x86-64 host using qemu userspace emulation

Note: This is the Fedora version of this Debian document so full credit must go to Debian and SuSE for assembling the bits.

Qemu (not quite upstream) now has ARM 64 bit emulation. It only does the userspace emulation at the moment, although system emulation is being worked on. This is much faster than the ARM “Foundation Model” (basically ARM’s proprietary emulator).

You will need to clone SuSE’s ARM64 qemu git repository and compile it statically linked (configure it with --target-list=arm64-linux-user --static). Static glib2 for Fedora is here.

You will also need to download this Fedora 19 arm64 disk image and convert it into a root filesystem (unfortunately this process requires, just temporarily, 14 GB of free disk space — should have used virt-sparsify!!):

mkdir /tmp/arm64
cd /tmp/arm64
tar zxf F19-aarch64-efi.tar.xz
virt-tar-out -a ./F19-aarch64-efi/aarch64-efi.img / - |
    sudo tar xf -
rm -rf F19-aarch64-efi

So now what you should have is a statically linked qemu-arm64 binary (that’s the userspace arm64 emulator), and a root filesystem containing lots of arm64 binaries. To run them we’ll need to set up a systemd binfmt handler, which will notice when we’re about to try to run an arm64 binary and inject our emulator so it’s possible to run it on our host (which is obviously not arm64 since that hardware is unobtainium for most mortals).

Copy the statically linked qemu-arm64 binary into the chroot, ie:

cp ./arm64-linux-user/qemu-arm64 /tmp/arm64/

Create a binfmt file /etc/binfmt.d/qemu-arm64.conf with the content below. Note that it refers to the absolute path /qemu-arm64 which (will be) correct once we chroot into the arm64 filesystem.

# AArch64 binaries.
:qemu-arm64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/qemu-arm64:OC
$ sudo service systemd-binfmt restart

Check that /proc/sys/fs/binfmt_misc/qemu-arm64 has been created with the expected contents.

Now you’re ready to go!

$ cd /tmp/arm64
$ sudo mount -o bind /dev ./dev
$ sudo mount -o bind /dev/pts ./dev/pts
$ sudo mount -o bind /proc ./proc
$ sudo mount -o bind /sys ./sys
$ sudo chroot .
# file bin/ls
bin/ls: ELF 64-bit LSB executable, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux
3.7.0,
BuildID[sha1]=0x31bcaf7de1bb3c6a4400983ac31f0dff1bbfc394, stripped
# /bin/ls
bin   dev  home  lib64	     media  opt   qemu-arm64  run   srv  tmp  var
boot  etc  lib	 lost+found  mnt    proc  root	      sbin  sys  usr
# uname -a
Linux choo.home.annexia.org 3.8.0 #1 SMP Thu Oct 10
14:11:18 UTC 2013 aarch64 aarch64 aarch64 GNU/Linux

And somehow convince yourself you really are running arm64 binaries!

One problem I had is the supplied chroot is quite minimal and I’m not exactly sure how to install more. But it’s fun to play and a lot faster than the Foundation Model.

Update #1

# arch
aarch64

Update #2

The /etc/yum.repos.d/stage4.repo file in the disk image above is broken. Peter Robinson notes that you can use http://arm.koji.fedoraproject.org/aarch64/stage4/ as a yum repo for installing more packages.

Update #3

Outside the chroot — when you’re running commands in the chroot — you’ll see qemu interpreting the arm64 binaries in ps output:

$ ps ax | grep qemu
26051 pts/14   S      0:03 /qemu-arm64 /bin/bash -i
28170 pts/14   S      0:00 /qemu-arm64 /bin/su rjones
28171 pts/14   S      0:01 /qemu-arm64 /bin/bash
28226 pts/14   Sl+    0:03 /qemu-arm64 /bin/git clone git://libvirt.org/libvirt.git
28228 pts/14   S+     0:12 /qemu-arm64 /usr/libexec/git-core/git index-pack --stdin -v --fix-thin --keep=fetch-pack 28226 on choo.home.annexia.org
About these ads

6 Comments

Filed under Uncategorized

6 responses to “How to run aarch64 binaries on an x86-64 host using qemu userspace emulation

  1. You can skip all the mounting silliness by:
    systemd-nspawn -D /tmp/arm64

    • rich

      I agree that way would be better. However some commands fail inside the container:

      # su - rjones
      su: avc.c:74: avc_context_to_sid_raw: Assertion `avc_running' failed.
      Aborted (core dumped)
      

      I’ll see if I can track this one down …

  2. Pingback: Fedora 21 has a working OCaml ARM64 | Richard WM Jones

  3. I’m using this on an F19 host and can’t see the network from within the chroot, which prevents package installation from the repos. Attempting to install packages from outside the chroot with ‘yum –installroot=…’ fails because of db version mismatches (I suspect but haven’t checked that the stage4 ‘rpm’ is pre-F19 or uses a different db version than the F19 x86_64 rpm). Is there a way to gain network access from inside the chroot?

    • rich

      Network works fine for me on F20. Note that qemu emulates the network, I think by thunking system calls and using SLIRP, so various things (like ping) definitely won’t work. Try testing “ssh”, “telnet” etc. For example from the chroot:

      # ping 8.8.8.8
      ping: cap_get_proc: Function not implemented
      

      but

      # telnet onuma 22
      Trying [IPv6 addr] ...
      Connected to onuma.
      Escape character is '^]'.
      SSH-2.0-OpenSSH_5.3
      
      • You’re right … the inability to do basic network things like ‘ip addr’ was leading me in the wrong direction — the real problem was the resolver configuration. Happily yum installing now!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s