Caseless virtualization cluster, part 5

My caseless virtualization cluster is now complete. 32 cores (arguably), 64 GB of RAM, for about £1300:

20140428_123312

The power supplies cause a real wiring nightmare! It would be great to have a better solution to delivering power:

20140428_123326

20140428_123333

It runs almost silently.

I plan to encase the whole thing in a metal case, firstly to make it more portable, and secondly to reduce the amount of RF given off. You would probably not be able to legally run this in a commercial environment because of EMC regulations. You definitely would not be allowed to sell it.

The next problem is management software. While it’s certainly possible to log in to each of the four individual hosts and run virsh commands, that’s going to get tedious rather quickly.

The problem is that all “solutions” to this are rather heavyweight. I could manage the hosts using Puppet and install OpenStack, but it would probably take longer to set that up than the time saved. There’s a lot of cloud software out there, but not much that nicely manages 4 hosts without requiring huge dependencies. What I really want is a small command line tool that uses libvirt remotely so I don’t have to install anything on the hosts.

23 Comments

Filed under Uncategorized

23 responses to “Caseless virtualization cluster, part 5

  1. I started in on a web interface using Libvirt, Python and Cherrypy, but am not far enough along to publish just yet. Kind of lightweight so far, but still not minimal.

  2. Chuck

    There are a few ways to handle the manage multiple hosts issue. DSH (http://www.unixlore.net/articles/running-remote-commands-multiple-servers-ssh-dsh.html) or a similar program is one way to do it if you’re issueing the same command across all the hosts. If not, perhaps configuring a gnuscreen or tmux session that, when launched, ssh’s to each host in a different window. I do this for a few systems that I manage generally at the same time and it works well.

  3. I wish Penguin had truely OpenSourced BProc, as it gives a great abstraction for a shared nothing cluster. However, I went through “shared nothing diskless booting” http://adam.younglogic.com/2012/03/shared-nothing-diskless-boot/ a while back. Which sounds like what you want.

  4. Stephen Smoogen

    rich. I would recommend ansible. It is light weight(ish), is python and works over ssh.

  5. Rich, FWIW, I have the same need as you (managing multiple virt hosts with a light weight CLI) and I find ‘virsh’ quite convenient in combination with libvirt URI aliases — http://libvirt.org/uri.html#URI_config, you can replace commands like:

    $ virsh -c qemu+ssh://host1/system list --all
    $ virsh -c qemu+ssh://host2/system list --all
    

    with:

     $ virsh -c host1 list --all
  6. Pingback: Caseless virtualization cluster: power usage | Richard WM Jones

  7. http://www.theforeman.org – we do support libvirt, provisioning, puppet is optional and we do have cli tool.

    • I mean we can do bare metal fairly well.

    • rich

      Now I’m actually going to read the Foreman quickstart page.

      • lzap

        Ping me if you have questions, since Foreman is good enough to install & manage Red Hat OpenStack Distribution, it can provide reliable solution to your custom libvirt-based cluster as well. By that I mean you can either install hypervisors on bare metal AND you can also install & manage VMs on them if you want.

        Btw this script will give you foreman stable/rc/nightly using your virt-builder very quickly🙂 Today I’d recommend to spawn rc (1.5 RC2), we are very close to the 1.5 release, lots of new features.

        https://github.com/lzap/bin-public/blob/master/fvb

  8. adamwill

    “The power supplies cause a real wiring nightmare!”

    Did you use modular power supplies? That’d help at least a bit, and the Corsair ones aren’t much more expensive than the non-modular versions. Basically they put sockets on the PSU and make the cables detachable, and you only have to connect the ones you actually need (probably only ATX and one SATA per system, in your case – you could leave out quite a few).

  9. Pingback: Setting up virtlockd on NFS | Richard WM Jones

  10. Pingback: Mini-cluster (mclu) command line tool | Richard WM Jones

  11. Pingback: Using LVM’s new cache feature | Richard WM Jones

  12. Pingback: Cluster performance: baseline testing | Richard WM Jones

  13. Regarding looking for a lightweight solution to manage your system; have you looked at Ganeti? It’s not quite perfect but it should be simple to build up a stock image for your cluster and deploy it to hardware. It basically does what you said and uses lib off to manage a group of hosts.

  14. Pingback: Mini Cloud/Cluster v2.0 | Richard WM Jones

  15. Pingback: Gigabyte MP30-AR0 | Richard WM Jones

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s