Tag Archives: virt-tools

Fedora 21 chrooted on an aarch64 Nexus 9


A while back I bought a Nexus 9, mainly because it has a weird processor that emulates a 64 bit ARM (aarch64). Google seem to have abandoned this platform entirely, just 6 months after I got it, so fuck you too Google. Anyway here’s how I installed a Fedora 21 aarch64 chroot on the device, using virt-builder and virt-tar-out and a bunch of unnecessary hassle.

First I ran virt-builder, which takes under a minute to produce a Fedora 21 aarch64 disk image. I then used virt-tar-out to convert all the files in that disk image into a tar file:

$ virt-builder --arch aarch64 fedora-21
$ virt-tar-out -a fedora-21.img / chroot.tar

Copy this file over to the N9, and unpack it. I have rooted my N9, so I can do this as root to preserve all the permissions etc:

# mkdir root
# cd root
# tar -xf /sdcard/Download/chroot.tar
# cd ..

And how can there not be a tar utility in Android?? I had to build a static ‘tar’ for aarch64 using my existing aarch64 server, to run the above command. And and and how can there be no chroot utility either!? I ended up compiling that myself too yada yada.

After all that you can do:

# mount -o bind /dev root/dev
# mount -o bind /proc root/proc
# mount -o bind /sys root/sys
# PATH=/usr/bin:/bin LD_PRELOAD= chroot root /bin/bash

which gives me at least a Fedora 21 shell on Android.

Edit: A few further notes:

  1. When setting up a non-root user account inside the chroot, give it the same UID, GID and groups as the ordinary non-privileged Android user account. In particular it must be in the inet group, else network access is blocked.
  2. You may need to set up /etc/resolv.conf by hand in the chroot.

1 Comment

Filed under Uncategorized

libguestfs works on MIPS Creator (mipsel)

[Previous post about the MIPS Creator CI20]

Slowly, of course.

I had to compile supermin & qemu from upstream and download (but not install) a qemu-compatible Debian kernel. Then setting the following environment variables allows make quickcheck to pass:

$ cat localenv
export SUPERMIN=/home/rjones/d/supermin-mipsel/src/supermin
export LIBGUESTFS_HV=/home/rjones/d/qemu-mipsel/mipsel-softmmu/qemu-system-mipsel
export SUPERMIN_KERNEL=/home/rjones/d/libguestfs-mipsel/kernel/boot/vmlinux-3.16.0-0.bpo.4-4kc-malta
export SUPERMIN_KERNEL_VERSION=3.16.0-0.bpo.4-4kc-malta
export SUPERMIN_MODULES=/home/rjones/d/libguestfs-mipsel/kernel/lib/modules/3.16.0-0.bpo.4-4kc-malta/

Leave a comment

Filed under Uncategorized

virt-builder Debian 8 (Jessie) image

Debian 8 was released a couple of days ago, and you can now install it through virt-builder.

Use --notes to read the release notes:

$ virt-builder debian-8 --notes

To build an image:

$ virt-builder debian-8 \
    --firstboot-command "dpkg-reconfigure openssh-server"

To boot it under libvirt:

$ virt-install --import \
  --name debian-8 --ram 2048 \
  --disk path=debian-8.img,format=raw --os-variant=debianwheezy

(At some point --os-variant=debianjessie will work, but virt-install doesn’t support it yet)

Update: This is how I ended up running Debian 8:

$ virt-builder debian-8 \
    --size=30G \
    --root-password PASSWORD \
    --edit '/etc/apt/sources.list: s/wheezy/jessie/g' \
    --run-command '
      apt-get -y install debian-keyring debian-archive-keyring
      apt-key update
    ' \
    --install emacs,nfs-common,sudo \
    --edit '/etc/ssh/sshd_config:
              s/^#PermitEmptyPasswords no/PermitEmptyPasswords yes/' \
    --firstboot FIRSTBOOT.sh
    --run-command 'update-rc.d virt-sysprep-firstboot defaults' \
    --run-command 'killall dbus-daemon cgmanager ||:'

Leave a comment

Filed under Uncategorized

New in virt-v2v

Leave a comment

Filed under Uncategorized

virt-builder: Fedora 21 ppc64 and ppc64le images

virt-builder now has Fedora 21 ppc64 and ppc64le images available, and you can run these under emulation on an x86-64 host. Here’s how to do it:

$ virt-builder --arch ppc64 fedora-21 \
    -o fedora-21-ppc64.img


$ virt-builder --arch ppc64le fedora-21 \
    -o fedora-21-ppc64le.img

To boot them:

$ qemu-system-ppc64 -M pseries -cpu POWER8 -m 4096 \
    -drive file=fedora-21-ppc64[le].img \
    -serial stdio

Oddly the boot messages will appear on the GUI, but the login prompt will only appear on the serial console. (Fixed)

Libvirt also has support, so with a sufficiently new version of the toolchain you can also use:

$ virt-install --import --name=guestname \
    --ram=4096 --vcpus=1 \
    --os-type=linux --os-variant=fedora21 \
    --arch=ppc64[le] --machine pseries \
$ virsh start guestname

It’s quite fun to play with Big Iron, even in an emulator that runs at about 1/1000th the speed of the real thing. I know a lot about this, because we have POWER8 machines at Red Hat, and they really are the fastest computers alive, by a significant multiple. Of course, they also cost a fortune and use huge amounts of power.

Some random observations:

  1. The virt-builder --size parameter cannot resize the ppc64 guest filesystem correctly, because Anaconda uses an extended partition. Workaround is to either add a second disk or to create another extended partition in the extra space. (Fixed)
  2. The disks are ibmvscsi model (not virtio or ide). This is the default, but something to think about if you edit or create the libvirt XML manually.
  3. Somehow the same CPU/machine model works for both Big Endian and Little Endian guests. It must somehow auto-detect the guest type, but I couldn’t work out how that works. Anyway, it just works by magic. it’s done by the kernel
  4. libguestfs inspection is broken for ppc64le
  5. Because TCG (qemu software emulation) is single threaded, only use a single vCPU. If you use more, it’ll actually slow the thing down.

Thanks: Maros Zatko for working out the virt-install command line and implementing the virt-builder script to build the images.

Leave a comment

Filed under Uncategorized

Tip: Wake up a guest from screen blank

A few years ago Dan Berrange added a way to send fake keyboard events to libvirt guests. You can use this to inject just a press on the Left Shift key to wake up a guest from screen blank. Very useful if you need to take a screenshot!

$ virsh send-key guest KEY_LEFTSHIFT
$ sleep 1
$ virsh screenshot guest /tmp/screenshot.ppm

Update: A word of warning though. If you try this for Windows guests you’ll hit this message:


The solution is to hit other keys randomly. Grrr.

Leave a comment

Filed under Uncategorized

Edit UEFI varstores

See end of post for an important update

UEFI firmware has a concept of persistent variables. They are used to control the boot order amongst other things. They are stored in non-volatile RAM on the system board, or for virtual machines in a host file.

When a UEFI machine is running you can edit these variables using various tools, such as Peter Jones’s efivar library, or the efibootmgr program.

These programs don’t actually edit the varstore directly. They access the kernel /sys/firmware/efi interface, but even the kernel doesn’t edit the varstore. It just redirects to the UEFI runtime “Variable Services”, so what is really running is UEFI code (possibly proprietary, but more usually from the open source TianoCore project).

So how can you edit varstores offline? The NVRAM file format is peculiar to say the least, and the only real specification is the code that writes it from Tianocore. So somehow you must reuse that code. To make it more complicated, the varstore NVRAM format is tied to the specific firmware that uses it, so varstores used on aarch64 aren’t compatible with those on x86-64, nor are SecureBoot varstores compatible with normal ones.

virt-efivars is an attempt to do that. It’s rather “meta”. You write a small editor program (an example is included), and virt-efivars compiles it into a tiny appliance. You then boot the appliance using qemu + UEFI firmware + varstore combination, the editor program runs and edits the varstore, using the UEFI code.

It works .. at least on aarch64 which is the only convenient machine I have that has virtualized UEFI.

Git repo: http://git.annexia.org/?p=virt-efivars.git;a=summary


After studying this problem some more, Laszlo Ersek came up with a different and better plan:

  1. Boot qemu with only the OVMF code & varstore attached. No OS or appliance.
  2. This should drop you into a UEFI shell which is accessible over qemu’s serial port.
  3. Send appropriate setvar commands to update the variables. Using expect this should be automatable.

Leave a comment

Filed under Uncategorized