Raspberry Pi 4 running Fedora 32

I got Fedora 32 installed on an RPi 4 8GB, booting off USB, with UEFI and ACPI. I followed Robert Grimm’s instructions here, and had an additional set of complications summarised here. There’s not much to say except that it was fiendishly complicated. But it works beautifully now, and is reasonably quick too especially when you consider how little it cost.

So let’s talk about costs (all include tax and delivery):

Raspberry Pi 4 8GB	£77.33
Case	£10.99
SanDisk 500GB SSD x 2	£149.98
small SD card needed for booting	£free

Only one of the SSDs is actually used, but if you follow Robert’s instructions you will need two. I didn’t have any external USB SSDs that were both USB 3 and not spinning hard disks, so I had to buy these, but I’ll be able to reuse one in a future project. The SD card is required to work around a bug in the UEFI firmware, but I happened to have one lying around.

AMD Seattle LeMaker Cello

I was a bit optimistic when I said:

the LeMaker Cello is available for preorder with delivery next month.

back in March 2016 (sic).

But hey, better late than never.

AMD seem to have decided to give up on ARM, making this board now only a historical curiosity. And look at that heatsink! I suspect these early chips have cooling problems.

A couple of new ARM server hardware announcements

Firstly Jon Masters demonstrates running RHEL 7.2 with a (very very nearly) unmodified upstream kernel on SBSA/SBBR standards supporting hardware:

Secondly the LeMaker Cello is available for preorder from http://www.lenovator.com/product/103.html with delivery next month.

AMD Huskyboard (video)

This board has the AMD “Seattle” A1100 64 bit, 8 core ARM.

Not very serious quest to run Linux on every architecture

Just looking at the Debian Ports page, how many of these architectures do I own?

Arch		Notes
Alpha	✗	Good, working Alpha hardware is getting hard to obtain these days.
amd64 (x86-64)	✓	Obviously …
ARM (32 bit)	✓	I don’t really care about anything less than ARM version 7 these days, although I do have a Sheevaplug which I think is v5 or v6. My “go-to” 32 bit ARM board is the Cubietruck.
ARM (64 bit)	✓	I don’t own, but have under my desk at work, the Applied Micro X-gene (picture), and a lovely piece of kit it is too.
HP-PA	✗	I would love to get my hands on some HP-PA RISC hardware, but it’s even harder to get hold of than Alpha. Edit: If you really want HP PA then the HP C8000 workstation is available on eBay for $300 or so.
i386	✓	I own a Lenovo X60s laptop, which is one of the few 32 bit Intel processors with (very slow) hardware virtualization
ia64	✓	HP Itanium RX2620
M68K	✗	As I started out my professional career writing 68k assembler, I’d love to buy an MVME crate, but incredibly they go for upwards of $1000 on ebay, even 20 years after they stopped being made.
MIPS	✓/✗	I just bought the MIPS Creator CI20. I don’t own any Big Endian MIPS hardware, nor any 64 bit MIPS hardware.
OpenRISC	✓	I have this burned into an FPGA, but in storage.
PPC (32 bit)	✗	I recently sent a couple of old 32 bit Mac G4s to recycling.
PPC/POWER (64 bit)	✓/✗	I own a Mac G5 running Linux (picture). However I don’t own, and as far as I know cannot get my hands on any of the more interesting POWER7 or POWER8 hardware, although I have remote access to it through Red Hat.
s390/s390x	✗	Similar to ppc64, no possible way to own it, but I am able to remote access it through Red Hat.
SPARC (32 bit)	✓	Currently sitting in storage, but it’s there and it used to run Linux, albeit with only 128 MB of RAM
SPARC (64 bit)	✗	Is it possible to buy 64 bit SPARC hardware for a reasonable price?

Looks like I’m about half way there!

libguestfs now works on 64 bit ARM

Pictured above is my 64 bit ARM server. It’s under NDA so I cannot tell you who supplied it or even show you a proper photo.

However it runs Fedora 21 & Rawhide:

Linux arm64.home.annexia.org 3.16.0-0.rc6.git1.1.efirtcfix1.fc22.aarch64 #1 SMP Wed Jul 23 12:15:58 BST 2014 aarch64 aarch64 aarch64 GNU/Linux

libvirt and libguestfs run fine, with full KVM acceleration, although right now you have to use qemu from git as the Rawhide version of qemu is not new enough.

Also OCaml 4.02.0 beta works (after we found and fixed a few bugs in the arm64 native code generator last week).

Fedora 21 has a working OCaml ARM64

Update: Thanks to Peter Robinson, there is now a build of OCaml for aarch64 in the Fedora repository.

I have backported the upstream ARM64 support into Fedora 21’s OCaml, so you can now use it to generate native ARM64/AArch64 binaries. If you don’t have hardware, use qemu to emulate it instead.

How to run aarch64 binaries on an x86-64 host using qemu userspace emulation

Note: This is the Fedora version of this Debian document so full credit must go to Debian and SuSE for assembling the bits.

Qemu (not quite upstream) now has ARM 64 bit emulation. It only does the userspace emulation at the moment, although system emulation is being worked on. This is much faster than the ARM “Foundation Model” (basically ARM’s proprietary emulator).

You will need to clone SuSE’s ARM64 qemu git repository and compile it statically linked (configure it with --target-list=arm64-linux-user --static). Static glib2 for Fedora is here.

You will also need to download this Fedora 19 arm64 disk image and convert it into a root filesystem (unfortunately this process requires, just temporarily, 14 GB of free disk space — should have used virt-sparsify!!):

mkdir /tmp/arm64
cd /tmp/arm64
tar zxf F19-aarch64-efi.tar.xz
virt-tar-out -a ./F19-aarch64-efi/aarch64-efi.img / - |
    sudo tar xf -
rm -rf F19-aarch64-efi

So now what you should have is a statically linked qemu-arm64 binary (that’s the userspace arm64 emulator), and a root filesystem containing lots of arm64 binaries. To run them we’ll need to set up a systemd binfmt handler, which will notice when we’re about to try to run an arm64 binary and inject our emulator so it’s possible to run it on our host (which is obviously not arm64 since that hardware is unobtainium for most mortals).

Copy the statically linked qemu-arm64 binary into the chroot, ie:

cp ./arm64-linux-user/qemu-arm64 /tmp/arm64/

Create a binfmt file /etc/binfmt.d/qemu-arm64.conf with the content below. Note that it refers to the absolute path /qemu-arm64 which (will be) correct once we chroot into the arm64 filesystem.

# AArch64 binaries.
:qemu-arm64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/qemu-arm64:OC

$ sudo service systemd-binfmt restart

Check that /proc/sys/fs/binfmt_misc/qemu-arm64 has been created with the expected contents.

Now you’re ready to go!

$ cd /tmp/arm64
$ sudo mount -o bind /dev ./dev
$ sudo mount -o bind /dev/pts ./dev/pts
$ sudo mount -o bind /proc ./proc
$ sudo mount -o bind /sys ./sys
$ sudo chroot .
# file bin/ls
bin/ls: ELF 64-bit LSB executable, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux
3.7.0,
BuildID[sha1]=0x31bcaf7de1bb3c6a4400983ac31f0dff1bbfc394, stripped
# /bin/ls
bin   dev  home  lib64	     media  opt   qemu-arm64  run   srv  tmp  var
boot  etc  lib	 lost+found  mnt    proc  root	      sbin  sys  usr
# uname -a
Linux choo.home.annexia.org 3.8.0 #1 SMP Thu Oct 10
14:11:18 UTC 2013 aarch64 aarch64 aarch64 GNU/Linux

And somehow convince yourself you really are running arm64 binaries!

One problem I had is the supplied chroot is quite minimal and I’m not exactly sure how to install more. But it’s fun to play and a lot faster than the Foundation Model.

Update #1

# arch
aarch64

Update #2

The /etc/yum.repos.d/stage4.repo file in the disk image above is broken. Peter Robinson notes that you can use http://arm.koji.fedoraproject.org/aarch64/stage4/ as a yum repo for installing more packages.

Update #3

Outside the chroot — when you’re running commands in the chroot — you’ll see qemu interpreting the arm64 binaries in ps output:

$ ps ax | grep qemu
26051 pts/14   S      0:03 /qemu-arm64 /bin/bash -i
28170 pts/14   S      0:00 /qemu-arm64 /bin/su rjones
28171 pts/14   S      0:01 /qemu-arm64 /bin/bash
28226 pts/14   Sl+    0:03 /qemu-arm64 /bin/git clone git://libvirt.org/libvirt.git
28228 pts/14   S+     0:12 /qemu-arm64 /usr/libexec/git-core/git index-pack --stdin -v --fix-thin --keep=fetch-pack 28226 on choo.home.annexia.org