Caseless virtualization cluster, part 2

The second layer goes on:


I’ve changed motherboards. As outlined in part 1 I bought a motherboard without onboard graphics, which means I’m waiting for a cheap PCI-Express graphics card so I can turn on the first layer. This time I switched to the cheaper, more compact GIGABYTE GA-78LMT-USB3, which comes with on-board graphics. Thanks again Karanbir for this excellent suggestion.

The cost of the second layer is £289.96 (includes VAT and delivery).

I’m going to use the same motherboards etc for the third and fourth layers, so if you were building this cluster the total cost would be:

Part Qty. Cost
Crucial Ballistix BLS2C8G3D169DS3CEU 16 GB, Corsair CX 430
4 £1,159.84
Power strip 1 £10 (est.)
8 port gigabit ethernet switch, cables 1 £40 (est.)
Stand-offs or equivalent 1 £50 (est.)
USB keys for booting

4 £40 (est.)


  1. All prices include tax and delivery.
  2. The system is diskless so this does not include a fileserver that you will need to provide.
  3. The cost per core is around £38.


Filed under Uncategorized

12 responses to “Caseless virtualization cluster, part 2

  1. problemchild68

    I see the new board is MUCH smaller and a bit cheaper.
    some of those 24port Gig-E switches are like £70+vat and may make a better buy especially if you are doing performance tests and would like to have more NICs per board.

    Can you tell me more about the idea of nesting hypervisors I was under the impression that this worked on both INTEL and AMD equally is there bias for AMD here for performance or is the reason to find this out ?

    • rich

      New board: Yes smaller and cheaper. However there is a question as to whether it can deliver enough power to drive all 8 cores at full speed. TBH I don’t know the answer, it’s one of the things I have to measure.

      Re nesting: As you may or may not know, Intel and AMD implement virtualization completely differently. It turns out that AMD’s implementation made it far simpler to add nested virt. kvm_amd nested virt has just worked since ~2008. kvm_intel nested is a work in progress, which has had a multitude of bugs. Whether it works at all will depend on the particular Intel processor you have. So for people who seriously need nested virtualization, I would currently recommend AMD.

  2. Pingback: Caseless virtualization cluster, part 3 | Richard WM Jones

  3. problemchild68

    I head the same then I heard that the Intel one now offered better results.
    Fundamentally you are never sure until you have done it yourself (SHOW ME THE MONEY!!).
    I’ve actually followed you on this vein as I wanted to experiment with KVM and VMware so this whole thing fits my remit great. I actually sprung up to
    £350 per node as I had a SSD into the price and a better motherboard to support Sata III (not an issue for you !!). Additionally I’m using the threaded rod I mentioned and probably 2 extra GIG-E ports so more bells and whistles.

    Also your NFS v4 rates are very good limited by the single NIC on the Reads I think ?

  4. Pingback: Caseless virtualization cluster, part 4 | Richard WM Jones

  5. Why the USB keys? If you’re using a file server anyway couldn’t you just PXE boot from it?

    • rich

      Yes … but USB keys do give you somewhere to store the OS which isn’t RAM and isn’t some sort of stateless NFS set up.

  6. Zack B

    Out of curiosity (and apologies if this has been mentioned already), do you know if these motherboards and processors in your current configuration support IOMMU aka AMD-vi? From what I’ve been reading, it looks like they probably do? Thanks for the great write up!

    • rich

      Neither “AMD-vi” nor “IOMMU” appear (case insensitively) in dmesg. So I would say that either the processors do not support it, or it may be disabled in the BIOS. The processors are AMD FX8320 in case you want to look them up on AMD’s site.

  7. adriyas

    So nice to see this. I know this post is a year older. I end up using VMWare Workstation on my workstation linux machine to achieve the nested virtualization. Though I tried nested on KVM under CentOS6 last year but unsuccessfull (even activating other CPU features), probably because I use intel proc (I guess). Would like to try now using AMD.
    May I know the status if you’re able to use the nested virt using your specs.

    Thanks and more power.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.