Free VMware hypervisors: Server vs. ESXi
Past year, I’ve been watching threads like this constantly popping up on virtualization forums here and there, with people asking for advices and wondering what is the real difference between VMware ESXi and VMware Server. I used to promise myself to share my own experiences sometimes – and I finally got time to do this.
It is no secret that many virtualization enthusiasts and small companies looking to try out virtualization usually do not have neither budget for paid hypervisor, nor fancy host hardware. So they need free hypervisor which is able to work on hardware they already have.
As for vendor choice, based on feedback I am getting, most people are still unaware about other free options like Citrix XENServer or Microsoft Hyper-V, and are only considering VMware offerings. While I do plan to extensively evaluate other hypervisors down the road, personally I have been using VMware all this time, and this article covers free VMware hypervisors only.
So, what is the best free VMware hypervisor for low-end servers? I’ve made my choice long ago, and vSphere release did not make me re-consider it: my favorite is still VMware Server 2.0. I’ve tried to use free ESXi too, but Server 2.0 has been consistently beating ESXi for my specific needs.
Here’s the summary table outlining how these two hypervisors compare with each other in my opinion, and for my specific needs which are: free hypervisor, small 5 hosts environment, cheap low end servers, local storage.
VMware ESXi 4.0 vs. VMware Server 2.0
Cost. Both hypervisors are free, but you have to keep in mind that depending on the underlying OS you choose for your VMware Server 2.0 host, you may have to pay for OS license. I plan to give some tips on best OS for VMware Server 2.0 in my following article, so stay tuned if you are interested!
Abstraction layer present. Most ESXi fans would state that ESXi is the only “bare metal” type hypervisor without another abstraction layer for the application to pass through to get to the hardware. While it cannot be argued that ESXi installs directly on bare metal, I consider the latter statement to be pure marketing. Just like other hypervisors and “full” ESX, ESXi does have the same abstraction layer in a form of underlying operating system – which is proprietary Linux-based OS packaged with hypervisor code. Note that Linux distributives used for service console applications are different: Red Hat for ESX, and BusyBox for ESXi.
Hardware support. While VMware did great job with adding more supported hardware with ESXi 4.0, it still does not support a lot of hardware (for instance, very commonly used ICH-8 chipset I happened to have on couple of my servers). On the other hand, Server 2.0 can be installed on pretty much any hardware (only limited by underlying operating system requirements).
Memory footprint. This one actually got me surprised. Based on all those “smallest foorpring” marketing claims, I expected ESXi 4.0 hypervisor to provide smaller resource usage footprint on the host than Server 2.0, but it appeared to be quite opposite. Check out this freshly booted hypervisor statistics:
Resource usage after reboot
As you can see, the difference for memory usage is pretty dramatic. While it does not worth paying attention in case of 64GB host, for low end hosts based on old hardware with 2-4GB of RAM it will definitely make difference. Anyway, if I got you interested in such a low memory footprint for VMware Server 2.0 host – check back next week. I plan to provide more details on my host setup, along with some recommendation for OS selection and configuration.
Manageability and control. Without a doubt, unrestricted access to underlying console gives me much more control over my host in case of Server 2.0. I can always kill or restart misbehaving process as needed with Server 2.0, and cannot do much if ESXi becomes unresponsive (but do a hard reset). I can completely integrate the host into my existing system (for instance, Active Directory).
API access. Free version of ESXi has all APIs restricted to read-only operation, so you cannot do much with them. On the other hand, VMware Server 2.0 has all APIs open for full access, and because it is the same VI API as regular ESX hosts are using, many great free tools work with Server 2.0 just fine. I plan to have another post talking about tools I am using to manage my Server 2.0 hosts soon, so be sure to check back in a couple of weeks if you are interested.
Backup performance. Server 2.0 wins hands down here, with backup and restore speed capping at storage and/or network speed. ESXi was always known for slow file management interfaces, which can hardly provide faster speed than 5-10MB/s for upload (restore), and about 20MB/s for download (backup). Moreover, the throttling seems to affect even “full” ESX version with vSphere release, as my recent testing showed.
VM performance. Obviously Server 2.0 is not as well optimized as ESXi 4.0, besides host OS may add noticeable toll depending on which one you are using. Common opinion is that ESXi provides about 10% better VM performance than VMware Server 2.0. I would say it is probably worst-case scenario, but still as you can see the difference is not incredible, moreover – it is hardly noticeable. And it does not really matter for most workloads too. I’ve seen small companies with whole production environments with DC, Exchange, SharePoint, SQL, DNS, RRAS and other services and applications running on a single VMware Server 2.0 host with 4GB RAM without an issue, and no complains on VM performance and responsiveness (except at reboot).
Stability. We all know that even best software has bugs. Among all hypervisors, ESXi is smallest, so logically it has less bugs and thus is more stable. Besides, as enterprise product, it definitely gets tons more testing. But, with good underlying OS, I am getting great stability with VMware Server 2.0 too. My hosts have been rock solid, and I cannot really be more happy with their long term stability.
Security. Without a doubt – out of box, and for inexperienced administrator, ESXi is a much better choice in terms of security. Reasons are obvious: the more code and services – the larger attack surface is. But again, it is all in your hands: select proper OS, perform minimal install (only necessary components and services), patch your box regularly – and this will give your Server 2.0 hosts very decent security.
So for my environment, VMware Server 2.0 wins 6:4, and will definitely remain my hypervisor of choice (at least until I finish evaluation of competitive free hypervisors). I really think that VMware has done totally awesome job with VMware Server 2.0, this is truly one of my favorite software products to date.
Thanks for reading, this web site was brought to you by VMware Server 2.0 
Does the VMware Server memory usage include memory consumed by host OS?
I assume so, because when using Windows 2003 as host OS, the host’s memory usage after reboot shows 803MB. I will compare different host OS in the next article.
I think you missed one small big thing: Balloon driver.
What you’ve exposed here is actually true. But, in terms of memory, You’ll have to add about 200MB for a Win2k3 server of memory to the 134MB that VMw Server 2.0 is showing, and almost the same if you want a Fedora host or something like it. It is still half of what you’ve on ESXi, true, Actually if you leave ESXi iddle for a few mins after reboot it narrows that down to 600MB (which IMO, still too high).
BUT, When you beging starting up VMs, you’ll find you can only fit your available RAM divided the ammount of RAM VM’s (you have 2GB host ram, ergo you’ll be able to host 4 512MB aprox, 3 and a little, actually)
. In ESXi, Balloon driver will fire up and start freeing up unused mem plus Page Shares. Using this technique, in my case, I’m overcommiting about 1GB of ram in a 2GB host withouth performance drop.
Besides that, you were OK at your thoughts
So let’s make a tie with 5:5 =D
Regards
Aldo Zampatti
Hi Aldo, thank you for your comment.
Actually, VMware Server also has the balloon driver. It is disabled by default, but you can enable if from web UI under host settings. I prefer to always enable it (in least aggressive mode). I agree that it does make difference for host capacity for low-memory hosts.
Concerning the memory usage – the host in question is Linux-based. I have a few Windows 2003 SP2 based hosts too, and as I’ve said in my previous comment, memory usage is 803MB after reboot there. I will post more data on how different host OS compare to each other later this week.
As for ESXi 4.0 memory usage, the metrics above are actually from well after reboot (I have series of screenshots after reboot and initially, memory usage is even higher than indicated above).
I prefer ESXi because it has:
- memory overcommit
- better network management (with VLANs, load balancing and fault tolerance)
- better monitoring and alarms
Also there are a lot of free Vitual Appliances that I can easily download and install on ESXi.
Hi Andrew, thank you for your comment. I totally agree that depending on specific needs and situation, ESXi might be a better choice.
You are right on networking, it is better in ESXi. However, please note that VMware Server 2.0 also has the memory overcommit feature. And there are free monitoring tools available, which provide much better monitoring and alarming than ESXi built-in.
The bottom 3 items in your initial comparison chart are sort of significant, aren’t they? I find the idea of building a virtual infrastructure on top of an OS to be completely counter-intuitive. All OS’s have issues, they need to be maintained in various ways, and that means your entire virtual infrastructure is hostage to that OS. The ESXi ‘OS’, by comparison, is super thin, will run on a USB key, and will run and run and run without any more updates than, say, your BIOS would need. Not to mention that the whole balloon thing sort of makes the differences in the way they use memory completely irrelevant. I also prefer the ESXi client, and would flat-out disagree with your assessment of the mangeability of ESXi. But it is definitely true that it was extremely easy to back up those 2.0 VM files. But I’ll take the performance, reliability, security, and interface of ESXi, and deal with running old-school network backups.
Steve, thank you for your comment.
I disagree with small disk footprint makes ESXi more secure, this is only true in theory and in ideal world. In real world, we all are still seeing awful security holes being continuously fixed (check out the patch notes for ESXi patches) – despite that small disk footprint.
I also disagree with “will run and run and run without any more updates than, say, your BIOS would need” – since its release, ESXi has had 13 patches, totaling over 2.7 GB. A little bit too much for a BIOS comparison.
I definitely agree with you that ESXi has some very good things, I wish I could use it, however backup performance does not allow using it in my lab. This really is one of the deciding factors for me. Sorry, but I am not building my infrastructure on something I am not able to back up properly.
Anton, thanks for typing this up. I’ve been meaning to do the same. I’m running both (server 2.0 and ESXi 4.0) across 15 servers and I’m seeing the same results. I’ve been trying to figure out the high idle mem usage on ESXi and it just seems to be part of the OS. I’ve done all the recommended tweaks and none seem to get it under 650mb with no VM’s running.
A few things to add to the above list: You get more local disk space with ESXi than a linux host(for obvious reasons, smaller footprint). SNMP on ESXi is subpar with no get requests. Also, raid control for most older raid controllers on ESXi is non-existent. From a real world remote data center setup, this is a deal killer(free hypervisor or not). If a disk drops you need to know.
Also, on Server 2.0 I noticed my servers started creeping up in cpu load after a few VM’s were added. The best tweak in the word was moving the .vmem files into /dev/shm. Dropped my server load by almost 75%. Well worth reading up on: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=844
In the end, ESXi is great for brand spanking new in house servers. But for legacy hardware, Server 2.0 seems to be the best fit so far. The key is sliming down the linux install to almost nothing.
Setup:
ESXi 4.0
Vmware server 2.1
Veeam FastSCP
Vmware Converter
Vsphere Client(works great on Vmware server 2.1 as well)
Thanks for great comment, I learned from it!
Very much appreciated.
In a previous comment you stated that Vmware Server 2.0 also has the memory overcommit feature. Could show were that was an added feature. And do you know if it is the same ratio as esxi?
Hi Davis, I have just noticed your comment. This setting is available in VMware Server web-UI. It has 3 modes determining how agressive memory should be overcommitted. Unfortunately, I do not have a server handy to tell you the exact location of this setting, but there are very few options in web UI, and it is hard to miss.
In the web-ui after logging in, over on the right choose edit host settings and under the heading Additional Memory you can change the memory overcommit. It’s referred to swapping memory in the web-ui but it’s the same thing