Caseless virtualization cluster, part 1

This is my slightly mad plan to build a 32 core, 64 GB virtualization cluster for as little money as possible.

I bought the first “layer” of this infinitely expandable cluster design to check that all the parts work together (in fact they don’t — see below).

20140413_154143

In the box: Gigabyte 970A-DS3P AMD 970 motherboard, AMD FX 8320 8 core processor, Crucial Ballistix BLS2C8G3D169DS3CEU 16GB RAM, Corsair Builder Series CX 430W PSU.

The total cost was £304 (includes sales tax and delivery). The cost per core + 2 GB RAM is a very reasonable £38.

I’m planning to run the cluster caseless (or at least, I’m first going to examine the heat and EM-radiation by running this first layer caseless to see if it is feasible). And diskless, using PXE or a cheap USB key to boot, with the OS and guests located on a fast NFS server.

To stack up caseless motherboards in the final cluster, I’m using these aluminium stand-offs. Each stand-off is 1″ high. [Edit: See comments for a cheaper alternative]

20140413_154643

20140413_154902

Unfortunately even with 3 inches of stand-offs, the clearance over the processor fan wouldn’t be very much. If I go for 4 spacers (4″) then the total height of the final four board cluster would be more than a foot!

20140413_155728

The second problem is that I’d forgotten that being AMD there is no integrated graphics [not quite true, see comments]. These boards appear not to boot without a graphics card. The card will be completely useless in normal operation, just taking up space and power and adding to the price per core.

Another issue is whether I should just purchase one PSU per motherboard, or invest in Y-splitters such as this one. It’s not clear to me that a Y-splitter can power the CPU.

Thanks to Karanbir Singh for suggesting these processors. They are very cheap per core.

14 Comments

Filed under Uncategorized

14 responses to “Caseless virtualization cluster, part 1

  1. Nice set up, but…
    problem with lack of integrated graphic, isn’t AMD’s problem, but your choice problem. AMD has great series of APU with great Radeon Chips integrated in FM2 and FM2+ socket processors. http://www.engadget.com/2014/01/17/amd-kaveri-review-round-up/

    • rich

      Yes, correct. For the second layer I have chosen this motherboard: GIGABYTE GA-78LMT-USB3 which has on-board graphics. Unfortunately I believe it won’t be able to drive the CPU at full power, although it does seem to be compatible.

      Edit: Kaveri isn’t useful to me because it only has 4 cores.

  2. problemchild68

    Richard, not sure where exactly the board i stopping in the BIOS.
    It may be an idea to boot the system with a video card go into the BIOS turn off stop on error or anything that looks like stop on no video then do a headless install of RHEL then test that you don’t actually need the card … you may need to put a USB to serial on the outside though ??!

    Also use 3mm threaded rod rather than stand-offs as it’s more flexible and cheaper!!

    Good luck

    • rich

      Yeah .. unfortunately it turns out I don’t own a PCI-Express video card! I threw one away from an old computer about 2 years ago since I thought I’d never need it. I had to order a very cheap one (£18).

      Thanks for the tip about threaded rod. I’ve got about 100 of these aluminium stand-offs now (bought in bulk they were quite cheap), so I’ll use those, but hopefully this information will be useful for others.

      • problemchild68

        Just checking there and I see that the CPU is AM3+ pin out so many of those boards should be able to drive it and obviously most have integrated graphics. What was the Motherboard choice based on compatability,performance features etc?
        Also I think if performance is a issue that you may be as wise to abandon the very low profile ( hard to get with that stonking fan any way) and add a couple of extra Gig-E ports maybe as a dual(this may be full height) from EBAY. I use them and it makes a poverty-Cluster(TM) absolutely thunder along 🙂
        More Power less ££££…..Again good luck

      • rich

        The goal is most cores, RAM and performance for least cost.

  3. lzap

    Nice. Why NFS over iSCSI or something else? Are these development purposes, or any other reasons?

    • rich

      Well it’s a good question. I’d prefer to use NFS (v4) because it’s a lot easier to manage than iSCSI. Also it is used by RHEV-M in some configurations to store VM images so (presumably) performance is good for this application.

      However I will need to test it once I’ve actually got the hardware working.

      The machines are for virtualization development and testing, particularly for testing virt-v2v which requires me to have a lot of different hypervisors and guests installed for continuous integration.

      (Edit: Also because it’s cool to have a 32 core cluster, right?)

      • problemchild68

        I only have 20 cores over a few i7s and AMDs ….now feeling terribly inadequate 🙂
        Your CPU looks quite healthy against the competition especially for the money
        http://www.cpubenchmark.net/high_end_cpus.html

        what Hypervisors are you testing BTW?

      • rich

        virt-v2v supports two flavors of Xen, and VMware ESX. It is my hope that because this uses AMD cores, nested virtualization will work well, and I will use KVM as the L0 hypervisor allowing me to run the other hypervisors in guests. However I won’t really know if this works / performs well until I test it. It may be that I have to run some of the test hypervisors on baremetal on some of the motherboards.

  4. Pingback: Caseless virtualization cluster, part 2 | Richard WM Jones

  5. Jorge Luis Andrade Escobar

    Hi Rich!

    Did you use GIGABYTE GA-78LMT-USB3, with the same CPU and Memory ?

    Is it compatible with RHEL6, have you tested it ?

    Thank You

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s