ESX4 U1: Hot Add backups for everyone!
I have received some truly great news from VMware about ESX 4 Update 1, which I am sure many Veeam Backup and Replication customers will appreciate.
Recently we had to contact VMware Storage Alliance team to share a growing number of complaints from our customers about vStorage API Hot Add processing mode not being available for vSphere license levels lower than Advanced.
Indeed, if you look at the vSphere licensing structure, Hot Add feature which this backup mode relies upon is not available for the lower versions:
When developing Veeam Backup and Replication 4.0, we found that on lower vSphere licenses, vStorage API fails to process VMs in the hot add mode with the corresponding (and very clear) licensing error, saying that your license level does not support devices hot add. By all means, this looked like intentional restriction, so we thought “oh well”, and simply documented this in the User Guide.
However, those of our customers who are 100% virtual, and those who simple wanted to take the chance of moving yet another physical server out of the server room, really wanted to use this mode. Other customers were hoping to use this backup mode as workaround to the famous ESX4 slow network backup issue (now addressed). In the end, hot add mode was the recommended workaround is VMware’s support KB article about ESX4 slow service console disk reads. So the interest to this processing mode have been huge on our Veeam Backup & Replication forums, with part of the reason being that Veeam is the first 3rd party vendor to implement vStorage Hot Add backups.
Thus, as more and more customers started to express their frustration due to this limitation, I have decided to share and discuss this issue with VMware people. It did not take long to hear back: I got a very prompt response confirming that VMware is well aware about this issue. This appeared to be unintentional licensing glitch with Hot Plug device support removed from lower versions vSphere.
And the good news is, this glitch is fixed with ESX 4.0 Update 1!
I have just got off my testing lab, and I can confirm that this is true. Veeam Backup & Replication 4.1 “Virtual Appliance” mode started to work fine on vSphere Essentials and vSphere Standard labs once I upgraded to ESX4 U1. So now everyone can enjoy all the benefits of the “Virtual Appliance” mode, no matter of VMware vSphere license level!
What’s so cool about this mode? Basically, it allows for direct storage access through ESX I/O storage stack (instead of network stack) by mounting backed up VM disks directly to the virtual Veeam Backup server. Couple that with completely re-written iSCSI initiator in ESX4 (my own testing showed about 15% performance increase comparing to ESX 3.5), and you will understand why some customers observe similar or even faster backup performance (on 4 vCPU Veeam Backup VM) than with older physical backup server hooked up into the SAN fabric via HBA for direct storage access.
But what makes this mode especially useful for some customers, is its performance with NFS storage. As you know, while NFS is in fact a “shared” storage, you cannot use direct-from-SAN processing mode with it, because the latter is only available for FC and iSCSI SAN storage. Before, customers with NFS storages had to struggle with absolutely abysmal speed of network backups, which were always much slower for NFS storage (even for ESX 3.5 hosts). And now, the same customers are reporting up to 3 times increase in full backup speed. Add to that most typical 10x speed increase from native changed block tracking support Veeam Backup 4.0 provides, and you get up to 30x faster backup speed with your good old NFS!
I’d say, pretty cool result for simply upgrading two products from v3 to v4: VMware ESX(i) and Veeam Backup and Replication. 
Major kudos to VMware for allowing this feature on all vSphere license levels, and making fast vSphere backups available for everyone. This great change definitely made VMware ESX(i) even better virtualization platform for many customers out there.