Under the guise of science (lots of graphs, configuration parameters and techno speak must mean they are impartial, right?) the “performance team” at VMware has published compelling performance data for ESX 3.5 versus XenServer 5.0 and native, for virtualized XenApp workloads.  

Congratulations are in order.  The VMware team has done a fabulous job of searching to find a single instance of a set of parameters for ESX & XenServer that, under a carefully crafted set of “simulated user behaviors” shows ESX outperforming XenServer for the XenApp workload. 

As a former academic, I’d give this mumbo jumbo an F grade.  Bad science, bad scientists, uneven playing field:

  • First, the VMware claims are not independently reproducible.  Like every claim on performance that VMware makes, only they can make it and nobody can refute it, because nobody else can publish results for comparative tests between VMware and any other product.Their EULA forbids it.  So, these results are true by definition, from your pals at the VMware ministry of truth.  By contrast, an open, independent set of tests run by Project Virtual Reality Check, a benchmarking project conducted jointly by two Citrix/VMware solution providers in the Netherlands finds results wholly at odds with VMware’s.  Our own performance tests have also been independently validated by the Tolly Group (to whom VMware also denied permission to publish comparative results against ESX).   Project VRC concludes that
    • XenServer supports between 118-128% more users per host than VMware ESX for XenApp VMs configured with 1 vCPU.  For example: Test 5 on XenServer and Test 8 on ESX (which were identical tests) tested 4 VMs with 1 vCPU and 4Gb of memory per VM, and shows that XenServer’s optimal user workload is 86.5 users whereas ESX is just 38 users
    • XenServer supports between 42-68% more users per host than VMware ESX for XenApp VMs configured with 2 vCPUs.  For example Test 8 on XenServer and Test 16 on ESX (which were identical tests) tested 4 VMs with 2 vCPUs and 4Gb of memory per VM, and shows that XenServer’s optimal user workload is 124.5 users whereas ESX is just 82.5 users.
  • Second, the VMware “study” is not a thorough exploration of a valid set of parameters for the Terminal Services / XenApp workload.  Instead, it is a narrow look at a particular set of configurations which are not reasonable in practice:
    • No test of 32 bit workloads - the primary candidates for server consolidation for this workload because a 32 bit OS exhausts its memory at 4 GB and a modern server can pack hundreds of GB and many cores.  Our work in this area has shown a compelling benefit to virtualizing TS/XenApp 32 bit workloads on XenServer, and an equally compelling set of reasons not to use ESX for this purpose.
    • Unrealistic configuration - The server used in the tests is certainly punchy - the machine had 64 GB RAM and 4 processors–each with 4 cores (16 total processor cores).  Anyone familiar with 64b TS/XenApp knows this machine could easily  support hundreds of XenApp sessions.  But the “scientists” at VMware don’t.  They instead chose to run exactly  one VM (with only 2 vCPU’s and using only 25% of the available memory) and XenApp at minimal levels of concurrency (i.e. 10-40 users).  No multi-VM scenarios, no tests at useful user-counts.  Based on their measurements they appear to gleefully extrapolate deeper into the realm of fiction to proudly pronounce their horse the winner. 
      **
      In our own work in this area, we found XenServer and other virtualization platforms to be roughly equal for this rather absurd set of parameters.  But for high user counts, the numbers are radicaly different.  We suitably anonymized the non-XenServer results, which are reproduced from the Tolly report.  Quiz of the day, which result do you think is ESX?:
  • Third, even VMware’s users and partners are challenging their “results”.  This study is so one sided that the majority of the blog followups on VMware’s site from its partners and customers point out how ridiculous they are.  Since the “performance team” may well redact them, here are a couple, saved for posterity:

VMware’s continued blunders in the performance arena are nothing short of embarrassing.    So I’ve decided to issue an open challenge to VMware CTO Stephen Herrod: Steve, it’s time to rein in the monkeys behind the keyboard, end VMware’s indefensible EULA restrictions and allow independent performance comparisons of your products with others, by third parties with a vested interest in accuracy and independence.  This sort of nonsense does nothing for VMware’s brand credibility, its customers, channel partners or competitors other than give us all a hearty laugh at your expense.