Re: [Hampshire] Virtualization Project advice

Imran Chaudhry wrote:
> Hey all, I'm embarking on a project involving virtualization and
> thought I'd consult the list in a wisdom-of-crowds fashion :-)

Hi Imran,

> At my workplace we use virtualization to support test and development
> of our products. That is, we have a team of about 8 staff creating VMs
> of a custom Linux distribution and sometimes many such VMs connected
> in virtual networks. There is also a requirement to add VMs of 4 or 5
> other staff (each of those with up to 200Gb of VMs) to this workload.

Yes, I'm vaguely familiar with it ;)

> Basically, I'm wondering what folks use to provide a reliable,
> fast and highly-available virtualization infrastructure for
> internal use only to serve the above usage scenario. In my research,
> it seems the big virtualization vendors are geared to folks
> using constantly running VMs that are serving websites, databases etc.

At the University of Southampton we currently use Microsoft Virtual
Server for this purpose but we're in the process of migrating to VMWare
ESX ("VMWare Infrastructure") 3 on a number of servers for both
"production" virtual machines and development/testing virtual machine
services.

> Our existing infrastructure is creaking a bit - solely for VMWare we
> have a single Dell
> 1750 1U with Dual Xeon 2.4Ghz (with hyper-threading stuff it acts like
> 4 cores), 4 GBs of RAM (10Gb of swap) and SCSI discs in hardware
> RAID5 using the Dell hardware raid gubbins. We have problems with slow
> response, weird ARP issues and suchlike. I should add that it has a
> Gigabit ethernet nic but our network is not all gigabit yet but thats
> coming soon. This is with VMWare Server 1.0.4 (Free edition).

Although old a PE1750 is still a very good server and that specification
isn't "low-end". I'd recommend on keeping it to supplement what you want
to buy in future. It lacks on-CPU virtualisation extensions but it will
still do a good job.

Are you still using the Debian Xen server for "internal" VMs?

> We look to spend time to save money so the free versions of software
> from the "big 2 or 3" look very attractive. I have been looking at
> VMWare (Free or ESXi), Citrix Xen (XenServer 5 Express) and
> VirtualIron. My users need some kind of GUI client or agent to perform
> management and admin of their own VMs/networks (and ideally kept
> separate from everyone else). This means a Linux GUI client (for
> Ubuntu/Fedora would be lovely) is a MUST. I take it as read that each
> vendor has a Windows client. We're using the free VMWare Workstation
> at the moment which has a Linux client. I'm finding it hard to find
> out who exactly has a Linux client and their websites are often
> confusing if one is new to virtualization. (sometimes Wikipedia can
> help cut through the marketing puff in these kinds of cases :-)

You have a number of options open to you. I'm not going to mention
paravirtualisation options or "operating system level" options because
they won't work for your custom linux setup (unless you heavily modify
the kernel which I doubt you'll want to do...). These options are all
free to use:

- VMWare ESXi

This option is easy to install, easy to use and is totally free. This is
probably the best performing solution as well. However it lacks a Linux
GUI client, is limited when it comes to permissions (in that is has
virtually none) and has local usernames and passwords (unless you hack
it!).

- VMWare Server

Will be about 20% on average slower than the ESXi solution. Easy to
install however, easy to use and again is totally free. It has a Linux
client (and a Windows client) and permissions are slightly better
(either "Private" or "Shared" virtual machines). The new version, 2,
also has a web interface. It uses whatever authentication system the
host operating system uses.

- VirtualBox 2.0

VirtualBox will perform about the same as VMWare Server, perhaps a
little slower. It is easy to setup however, easy to use but networking
can be a little tricky on GNU/Linux. I'm not sure how multi-user works
with VirtualBox.

If you use the official version (not the Open Source version) you can
use an RDP client to connect to the console of all of the virtual
machines - very nice feature.

- Xen on GNU/Linux

This is much harder to setup and maintain (unless you use something like
CentOS 5x). Performance isn't bad but it isn't fantastic either. It
won't be as fast as ESXi. Managing virtual machines however and
accessing their consoles is tough - there are a number of third party
add ons but none of them are really up to scratch. If you use
CentOS/RHEL then you can use the Red Hat GUI tools.

I also advise not going near Xen - the open source version has never
really seen much love and attention. Most work goes into the commerial
XenServer.

- KVM on GNU/Linux

You could use KVM on Ubuntu 8.04. This has graphical tools and setup
isn't that difficult. Networking is a bit of a pain but it possible to
get working. Performance might be a little better than Xen but KVM isn't
really very mature yet. Support for KVM will be limited.

- XenServer

I can't really tell you much about this because I've never used it. I
guess you should just try an evaluation of it! Should be as fast as ESXi
if their claims are to be believed.


> Regarding hardware I was looking at going one of two ways -
>
> a) expensive - a Dell 1950 + DAS (direct attached storage) such as
> their MediaVault MD1000 + some fast SAS discs. This particular Dell
> has support for VMWare ESXi at the firmware level which effectively
> turns it into a "VMWare appliance" with the resulting VMs having a
> smaller footprint.
>
> b) cheaper - several cheaper Dells such as the PowerEdge T105 with say
> 8Gb RAM each and fast SATA drives acting as a cluster. That is, each
> node with their own VMs running but the whole cluster with one central
> management console. Both XenSource and VMWare have features to allow
> moving VMs between nodes in clusters without interruptions to the
> running VM.

I don't think a PE1950 is really suited to what you want to do. The
PE1950 is the "high-end" 1U Dell server and is designed for
mission-critical roles. Also ESXi in firmware will be no different to
just installing it - it's simply a gimmick to help Dell sell the server
to virtualisation enthusiasts.

I strongly recommend using an AMD Opteron for virtualisation because of
the much better virtualisation performance the vastly better inter-core
memory bandwidth. Eventually the Intel Core i7 will erode this advantage
but until then I'd buy Opteron (it'll be cheaper too!).

I also recommend buying tower servers since they're cheaper unless you
really do need a rack server. Based on my experience of where you work
I'd recommend a tower server. This is where Dell let you down though
since the only Opteron tower they sell is rather limited.

However Dell's only rack-mountable 1U Opteorn server is also rather
limited. If you have the money then a Dell PowerEdge 2970 might be a
good solution. It supports two AMD Opteron series processors and six SAS
disks - but it is rather expensive.

Keep all this in mind when considering a server. I'd also strongly
recommend looking at HP (or even IBM) servers before considering Dell's
range.

Finally, those live migrations only work if you pay for them. Xen and
VMWare only offer live migration on their pay-for products. They also
don't really do automatic fail over (unless you consider restarting
"fail over") making them less useful.

Cheers,

David