SSH from RHEL 9 to RHEL 5 or RHEL 6

RHEL 9 no longer lets you ssh to RHEL ≤ 6 hosts out of the box. You can weaken security of the whole system but there’s no easy way to set security policy per remote host. Here’s how to set up ssh so it works for a RHEL 5 or RHEL 6 host:

First edit your .ssh/config file, adding an entry for the host:

Host rhel5or6-host
KexAlgorithms +diffie-hellman-group14-sha1
MACs +hmac-sha1
HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

(The lines except the first “Host” line should be indented. WordPress screws up the formatting …)

That’s not enough on its own, because RHEL 9 also maims the openssl library by disabling SHA1 support by default. To fix that, create /var/tmp/openssl.cnf with:

.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes

Now you can ssh to RHEL 5 or RHEL 6 hosts like this:

OPENSSL_CONF=/var/tmp/openssl.cnf ssh rhel5or6-host

Thanks Laszlo Ersek for working out most of this. Related bugs:

2064740 – RFE: Make it easier to configure LEGACY policy per service or per host

2062360 – RFE: Virt-v2v should replace hairy “enable LEGACY crypto” advice which a more targeted mechanism

Mini Cloud/Cluster v2.0

Last year I wrote and rewrote a little command line tool for managing my virtualization cluster.

Of course I could use OpenStack RDO but OpenStack is a vast box of somewhat working bits and pieces. I think for a small cluster like mine you can get the essential functionality of OpenStack a lot more simply — in 1300 lines of code as it turns out.

The first thing that small cluster management software doesn’t need is any permanent daemon running on the nodes. The reason is that we already have sshd (for secure management access) and libvirtd (to manage the guests) out of the box. That’s quite sufficient to manage all the state we care about. My Mini Cloud/Cluster software just goes out and queries each node for that information whenever it needs it (in parallel of course). Nodes that are switched off are handled by ignoring them.

The second thing is that for a small cloud we can toss features that aren’t needed at all: multi-user/multi-tenant, failover, VLANs, a nice GUI.

The old mclu (Mini Cluster) v1.0 was written in Python and used Ansible to query nodes. If you’re not familiar with Ansible, it’s basically parallel ssh on steroids. This was convenient to get the implementation working, but I ended up rewriting this essential feature of Ansible in ~ 60 lines of code.

The huge down-side of Python is that even such a small program has loads of hidden bugs, because there’s no safety at all. The rewrite (in OCaml) is 1,300 lines of code, so a fraction larger, but I have a far higher confidence that it is mostly bug free.

I also changed around the way the software works to make it more “cloud like” (and hence the name change from “Mini Cluster” to “Mini Cloud”). Guests are now created from templates using virt-builder, and are stateless “cattle” (although you can mix in “pets” and mclu will manage those perfectly well because all it’s doing is remote libvirt-over-ssh commands).

$ mclu status
ham0                     on
                           total: 8pcpus 15.2G
                            used: 8vcpus 8.0G by 2 guest(s)
                            free: 6.2G
ham1                     on
                           total: 8pcpus 15.2G
                            free: 14.2G
ham2                     on
                           total: 8pcpus 30.9G
                            free: 29.9G
ham3                     off

You can grab mclu v2.0 from the git repository.

Nexus 9 ssh on the go

The Nexus 9 is an odd, compromised tablet, and way too expensive, but combined with the folio keyboard & pocketwifi it makes a nice ssh terminal for use on the road.

Various ssh apps like ConnectBot have terrible external keyboard support. So I compiled a static dropbear binary and static busybox, and I’m using those with Android Terminal Emulator.

The tablet has a 64 bit ARM processor (actually it’s way stranger than that – it uses a proprietary VLIW core with Transmeta-style code morphing in software). I used my AArch64 Fedora machine to compile the static binaries which I copied across.

I changed the default shell to busybox ash and added a bunch of start-up scripts to make Android more bearable.
It all works except nsswitch (user & DNS resolution) because of glibc static brokenness.

nbdkit now supports cURL — HTTP, FTP, and SSH connections

nbdkit is a liberally licensed NBD (Network Block Device) server designed to let you connect all sorts of crazy disk images sources (like Amazon, Glance, VMware VDDK) to the universal network protocol for sharing disk images: NBD.

New in nbdkit 1.1.8: cURL support. This lets you turn any HTTP, FTP, TFTP or SSH server that hosts a disk image into an NBD server.

For example:

$ nbdkit -r curl url=http://onuma/scratch/boot.iso

and then you can read the disk image using guestfish, qemu or any other nbd client:

$ guestfish --ro -a nbd://localhost -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

/dev/sda mounted on /

><fs> _

If you are using a normal SSH server like OpenSSH which supports the SSH File Transfer Protocol (aka SFTP), then you can use SFTP to access images:

$ nbdkit -r curl url=sftp://rjones@localhost/~/fedora-20.img

I’m hoping to enable write support in a future version.

It doesn’t work at the moment because I haven’t worked out how to switch between read (GET) and write (POST) requests in a single cURL handle. Perhaps I need to use two handles? The documentation is confusing.

qemu 1.5.0 released, with ssh block device support

qemu 1.5.0 has been released, featuring ssh support so you can access remote disks over ssh, including from libguestfs.

Here’s how to use this from guestfish:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -a ssh://onuma/mnt/scratch/winxp.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 May 20 19:47 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
-rwxrwxrwx  1 root root      47564 Apr 14  2008 NTDETECT.COM
drwxrwxrwx  1 root root       4096 Oct 11  2011 Program Files
drwxrwxrwx  1 root root       4096 Oct 11  2011 System Volume Information
drwxrwxrwx  1 root root      28672 Oct 11  2011 WINDOWS
-rwxrwxrwx  1 root root        211 Oct 11  2011 boot.ini
-rwxrwxrwx  1 root root     250048 Apr 14  2008 ntldr
-rwxrwxrwx  1 root root 1610612736 Oct 11  2011 pagefile.sys

Use guestfish, virt tools with remote disks

New in libguestfs ≥ 1.21.30 is the ability to use guestfish and some of the virt tools with remote disks.

Currently you can use remote disks over NBD, GlusterFS, Ceph, Sheepdog and (recently upstream) SSH.

For this example I’ll use SSH because it needs no setup, although this requires absolutely the latest qemu and libguestfs (both from git).

Since we don’t have libvirt support for ssh yet, so this only works with the direct backend:

$ export LIBGUESTFS_BACKEND=direct

I can use a ssh:// URI to add disks with guestfish, guestmount and most of the virt tools. For example:

$ virt-rescue -a ssh://localhost/tmp/f17x64.img
[... lots of boot messages ...]
Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> mount /dev/vg_f17x64/lv_root /sysroot
><rescue> cat /sysroot/etc/redhat-release
Fedora release 17 (Beefy Miracle)

Apart from being a tiny bit slower, it just works as if the disk was local:

$ virt-df -a ssh://localhost/tmp/f17x64.img
Filesystem                           1K-blocks       Used  Available  Use%
f17x64.img:/dev/sda1                    487652      63738     398314   14%
f17x64.img:/dev/vg_f17x64/lv_root     28316680    4285576   22586036   16%

$ guestmount -a ssh://localhost/tmp/f17x64.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ cat /tmp/mnt/etc/redhat-release
Fedora release 17 (Beefy Miracle)
$ guestunmount /tmp/mnt

libguestfs and qemu ssh driver

The qemu ssh block device is now up to version 7 … although sadly not upstream yet.

Nevertheless by applying this patch to libguestfs you can use libguestfs to access remote disks over ssh:

$ export LIBGUESTFS_QEMU=~/d/qemu/qemu.wrapper
$ export LIBGUESTFS_BACKEND=direct
$ ./run ./fish/guestfish

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> add /tmp/f17x64.img readonly:true format:raw \
        protocol:ssh server:onuma
><fs> run
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs> inspect-os
/dev/vg_f17x64/lv_root
><fs> inspect-get-product-name /dev/vg_f17x64/lv_root
Fedora release 17 (Beefy Miracle)
><fs> list-filesystems
/dev/sda1: ext4
/dev/vg_f17x64/lv_root: ext4
/dev/vg_f17x64/lv_swap: swap
><fs> mount /dev/vg_f17x64/lv_root /
><fs> cat /etc/redhat-release
Fedora release 17 (Beefy Miracle)

Everything just works as if this were a local disk.

There are a couple of minor caveats (the major caveat being none of this is upstream): Firstly you have to have ssh-agent set up. Secondly the remote host must be in your known_hosts file (if not, do ssh remotehost first to add it).

QEMU ssh block device

I wrote a small patch (intro, patch) which adds a Secure Shell (ssh) block device to qemu. With this patch you could access a remote disk image or device by doing:

qemu -drive file=ssh://host/path/to/file,if=virtio,cache=none

QEMU ssh’es into “host” and opens /path/to/file. For the initial version of this patch you will need to set up ssh-agent access to the remote server.

The motivation behind this patch is to allow libguestfs to access remote disks using ssh the same way we already do with NBD. Secure Shell is ubiquitous, so for the majority of users libguestfs-over-qemu/ssh would let them use disks remotely with zero configuration.

Tip: Use libguestfs on VMware ESX guests

You can use libguestfs, guestfish and the virt tools on VMware ESX guests quite easily. However it’s not obvious how to do it, so this post explains that.

You will need:

• libguestfs tools installed on a Linux machine
• sshfs installed on the same Linux machine
• ssh access to the VMware ESX storage (find the root password from the administrator)
• the name of the guest and the name of the storage volume that the guest is stored on

The guest must be shut down (more on this later).

First of all, make sure you are able to ssh as root to the VMware ESX storage. It will look something like this:

$ ssh root@vmware
root@vmware's password: ****
Last login: Wed May  4 20:47:50 2011 from [...]
[root@vmware ~]# ls -l /vmfs/
total 1
drwxr-xr-x 1 root root 512 May 10 09:22 devices
drwxr-xr-x 1 root root 512 May 10 09:22 volumes

Now you should create a temporary mount point, and mount /vmfs from the VMware ESX storage server using sshfs. The command is quite simple and you don’t need to be root on the Linux side:

$ mkdir /tmp/vmfs
$ sshfs root@vmware:/vmfs /tmp/vmfs
root@vmware's password: ****
$

In another window you can navigate to the guest. For example if the guest was called “test” and it lived on volume “Storage1” then:

$ cd /tmp/vmfs/volumes/Storage1/test
$ ls -l
total 1718720
-rw------- 1 root root 8589934592 May 10 09:48 test-flat.vmdk
-rw------- 1 root root       8684 May 10 09:37 test.nvram
-rw------- 1 root root        469 Apr  4 08:16 test.vmdk
-rw------- 1 root root          0 May 11  2010 test.vmsd
-rwxr-xr-x 1 root root       2666 May 10 09:37 test.vmx
-rw------- 1 root root        259 May 11  2010 test.vmxf
-rw-r--r-- 1 root root      53966 May 11  2010 vmware-1.log
-rw-r--r-- 1 root root      78771 May 11  2010 vmware-2.log
-rw-r--r-- 1 root root      56483 Apr  4 08:15 vmware-3.log
-rw-r--r-- 1 root root      56305 May 10 09:37 vmware.log

The critical file is guestname-flat.vmdk which is the flat disk image. You can just open this for read or write using guestfish, virt-df, virt-filesystems or other libguestfs tools or programs.

For example:

$ guestfish --rw -i -a test-flat.vmdk

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
/dev/VolGroup00/LogVol00 mounted on /
/dev/vda1 mounted on /boot

><fs> touch /tmp/hello
><fs> ll /tmp
total 20
drwxrwxrwt.  3 root root 4096 May 10 14:48 .
drwxr-xr-x. 24 root root 4096 May 10 14:36 ..
drwxrwxrwt   2 root root 4096 Apr  4 13:16 .ICE-unix
-rw-r--r--   1 root root    0 May 10 14:48 hello

Notice that guestfish determined the guest operating system and lets you edit the disk.

$ virt-filesystems -a test-flat.vmdk --all --long -h
Name                     Type       VFS  Label Size Parent
/dev/sda1                filesystem ext3 /boot 102M -
/dev/VolGroup00/LogVol00 filesystem ext3 -     7.1G -
/dev/VolGroup00/LogVol01 filesystem swap -     768M -
/dev/VolGroup00/LogVol00 lv         -    -     7.1G /dev/VolGroup00
/dev/VolGroup00/LogVol01 lv         -    -     768M /dev/VolGroup00
/dev/VolGroup00          vg         -    -     7.9G -
/dev/sda2                pv         -    -     7.9G -
/dev/sda1                partition  -    -     102M /dev/sda
/dev/sda2                partition  -    -     7.9G /dev/sda
/dev/sda                 device     -    -     8.0G -

$ virt-df -a test-flat.vmdk -h
Filesystem                                Size       Used  Available  Use%
test-flat.vmdk:/dev/sda1                   99M        12M        81M   13%
test-flat.vmdk:/dev/VolGroup00/LogVol00
                                          6.9G       1.1G       5.5G   16%

With libguestfs we usually allow you to read guests which are running. The results might be inconsistent at times, but it generally works. However VMware itself doesn’t allow running guests to be read. If the guest is running you can see that VMware prevents access:

# file test-flat.vmdk
test-flat.vmdk: writable, regular file, no read permission

Whereas when the same guest is shut down, reads (and writes) are allowed:

# file test-flat.vmdk
test-flat.vmdk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, starthead 0, startsector 208845, 16563015 sectors, code offset 0x48

This is a limitation of VMware and nothing to do with libguestfs.

A note on performance: I run this from my home to a VMware server which is a third of the way around the planet over plain 2Mbps ADSL. It’s noticeably slower than accessing local disk images, but still very usable. sshfs appears to be very efficiently implemented. It is far faster and more convenient than copying the whole disk image around.