My caseless virtualization cluster is now complete. 32 cores (arguably), 64 GB of RAM, for about £1300:
The power supplies cause a real wiring nightmare! It would be great to have a better solution to delivering power:
It runs almost silently.
I plan to encase the whole thing in a metal case, firstly to make it more portable, and secondly to reduce the amount of RF given off. You would probably not be able to legally run this in a commercial environment because of EMC regulations. You definitely would not be allowed to sell it.
The next problem is management software. While it’s certainly possible to log in to each of the four individual hosts and run virsh commands, that’s going to get tedious rather quickly.
The problem is that all “solutions” to this are rather heavyweight. I could manage the hosts using Puppet and install OpenStack, but it would probably take longer to set that up than the time saved. There’s a lot of cloud software out there, but not much that nicely manages 4 hosts without requiring huge dependencies. What I really want is a small command line tool that uses libvirt remotely so I don’t have to install anything on the hosts.
Richard WM Jones 
I started in on a web interface using Libvirt, Python and Cherrypy, but am not far enough along to publish just yet. Kind of lightweight so far, but still not minimal.
There are a few ways to handle the manage multiple hosts issue. DSH (http://www.unixlore.net/articles/running-remote-commands-multiple-servers-ssh-dsh.html) or a similar program is one way to do it if you’re issueing the same command across all the hosts. If not, perhaps configuring a gnuscreen or tmux session that, when launched, ssh’s to each host in a different window. I do this for a few systems that I manage generally at the same time and it works well.
I wish Penguin had truely OpenSourced BProc, as it gives a great abstraction for a shared nothing cluster. However, I went through “shared nothing diskless booting” http://adam.younglogic.com/2012/03/shared-nothing-diskless-boot/ a while back. Which sounds like what you want.
Thanks. By the way, dnsmasq is way easier to configure for both dhcp and tftp.
rich. I would recommend ansible. It is light weight(ish), is python and works over ssh.
Rich, FWIW, I have the same need as you (managing multiple virt hosts with a light weight CLI) and I find ‘virsh’ quite convenient in combination with libvirt URI aliases — http://libvirt.org/uri.html#URI_config, you can replace commands like:
with:
That’s a great tip. I had no idea that these URI aliases existed, thanks.
Pingback: Caseless virtualization cluster: power usage | Richard WM Jones
http://www.theforeman.org – we do support libvirt, provisioning, puppet is optional and we do have cli tool.
I mean we can do bare metal fairly well.
Now I’m actually going to read the Foreman quickstart page.
Ping me if you have questions, since Foreman is good enough to install & manage Red Hat OpenStack Distribution, it can provide reliable solution to your custom libvirt-based cluster as well. By that I mean you can either install hypervisors on bare metal AND you can also install & manage VMs on them if you want.
Btw this script will give you foreman stable/rc/nightly using your virt-builder very quickly 🙂 Today I’d recommend to spawn rc (1.5 RC2), we are very close to the 1.5 release, lots of new features.
https://github.com/lzap/bin-public/blob/master/fvb
“The power supplies cause a real wiring nightmare!”
Did you use modular power supplies? That’d help at least a bit, and the Corsair ones aren’t much more expensive than the non-modular versions. Basically they put sockets on the PSU and make the cables detachable, and you only have to connect the ones you actually need (probably only ATX and one SATA per system, in your case – you could leave out quite a few).
That would have been a good idea. I wasn’t aware that modular PSUs existed.
https://en.wikipedia.org/wiki/Power_supply_unit_%28computer%29#Modular_power_supplies
Pingback: Setting up virtlockd on NFS | Richard WM Jones
Pingback: Mini-cluster (mclu) command line tool | Richard WM Jones
Pingback: Using LVM’s new cache feature | Richard WM Jones
Pingback: Cluster performance: baseline testing | Richard WM Jones
Regarding looking for a lightweight solution to manage your system; have you looked at Ganeti? It’s not quite perfect but it should be simple to build up a stock image for your cluster and deploy it to hardware. It basically does what you said and uses lib off to manage a group of hosts.
FYI I’m using mclu to manage the cluster. It is very simple (under 1000 LoC) and has worked well for a few weeks.
Ikea encasement http://helmer.sfe.se/
Pingback: Mini Cloud/Cluster v2.0 | Richard WM Jones
Pingback: Gigabyte MP30-AR0 | Richard WM Jones
Pingback: Intel NUC | Richard WM Jones