Subsequent to my recent posting of our performance benchmarking of several hypervisors for single-server density for hosted virtual desktops, which found that XenServer and Hyper-V handily lead the pack in VDI performance, there have been some great follow-up postings. One in particular, from VMware’s Eric Horschmann, deserves a proper response. He writes:

“The great thing about tranparent page sharing is that it has almost no impact on performance. In fact, for many workloads, TPS can INCREASE performance because the hypervisor will have less memory to manage. You can see results from TPS performance testing we published that show just how well it performs in the VMware paper here. The combination of TPS and VDI is extremely powerful in letting you run many more client VMs with no performance impact. It’s the closest thing to a “free lunch” in software you’re likely to find. I’m still waiting to hear directly from someone at Citrix on the settings they used in this testing.”

It’s time for VMware to “put up or shut up” on TPS, providing real data that validates its claims instead of promising benefits to customers that are not backed by empirical data from benchmarks by a disinterested third party.

Let’s be blunt: When VMware is involved there is no such thing as a “free lunch”. What matters more, perhaps, is how much you’re being charged for features that don’t deliver benefits.

  • Neither the paper to which Horschmann refers, nor any other results of which I am aware show any actual benefit of TPS for VDI. VMware markets TPS as a key differentiator, but has to date offered zero evidence in support of its benefits for VDI. Our results show that TPS offers no benefit for VDI, contrary to Horschmann’s claim. (one caveat: we assume that the Project VRC Benchmarks are representative of a VDI workload).
  • We are not the only ones sceptical of its benefits. Project VRC team in its VRC phase I benchmarks for virtualized XenApp / Terminal Services workloads recommended TPS be turned off. However, in their phase II update report they reversed their position saying: “After extensive testing of the vSphere update, tests have shown TPS does not influence performance (not even slightest). As a result, Project VRC will not recommend disabling TPS anymore. Although disabling TPS does not hurt performance, it is now proven that it also so does not improve performance: With both 4VM and 8VM tests TPS_ON and TPS_OFF score identical on vSphere.” Now, my friends at VMware may want to diss Project VRC, but if they do, they will need to take on analysts such as Simon Bramfitt who agrees with the VRC methodology and recommends its adoption as a standardized benchmark.
  • Neither Hyper-V R2, nor XenServer 5.5 supports TPS, so by Horschmann’s logic we ought to expect them to under-perform. Our tests included vSphere 4.0 and other hypervisors, and our ESX tests evaluated performance both with and without TPS.
  • To get the benefit of TPS, one needs to assume that it is possible to share code pages across multiple Windows VMs, thereby saving memory, but with Windows Server 2008 and later, areas of memory that could be attacked through buffer overruns use address space layout randomization (ASLR) to minimize the probability that an attacker can correctly guess the locations of specific segments of code. ASLR limits the benefits of TPS because it leads to a different memory layout for each VM on the same host, making it difficult or impossible to share code pages between them. I have seen VMware field reps recommend that customers turn off ASLR so that they can get savings using TPS, which would in turn void the protection that address space randomization can offer. TPS may be useful in some cases, but if your system is insecure as a result, is it really worthwhile?
  • VMware claims that TPS still performs well in the presence of ASLR, but there is no doubt that the “free lunch” that Horschmann claims is far from free:
    • TPS itself is computationally intensive. The hypervisor runs through all system memory, computes a hash on each page, identifies common pages across VMs, and then rearranges their page tables to indirect to a common set of pages. On a modern server we support hundreds of VMs, and within the recommended TPS scan window of 1 hour, we also expect many VMs to boot or shutdown, so the system will rarely be stable enough for TPS scans to complete. VMware acknowledges the high cost and enables users to modify TPS parameters, with recommendations that you tinker carefully so as not to destabilize the system. It’s a black art that probably helps in some well understood scenarios. For the VDI benchmarks either:
      • TPS doesn’t have a chance to help due to the CPU scheduling challenges VMware has recently acknowledged when Intel Hyperthreading is used (Read: their performance is dominated by CPU utilization) or,
      • TPS works better in theory than in practice or we would have seen tangible evidence by now of superior VM density with its use. We haven’t. In fact, the only thing we have consistently seen is superior density from XenServer and Hyper-V.
    • TPS costs real money: Though neither XenServer nor Hyper-V currently supports TPS, both are free, and they still offered significantly better desktop VM density than the nearest competitor. You can freely download XenServer or Hyper-V and save yourself about $3,000 per server.
  • Finally, Horschmann says that he is still waiting for us to publish the details of the configuration used in our tests. We have done that for XenServer and Hyper-V, but obviously cannot do so for the nearest competitor. Specifically, if it were the case that the competitor was a VMware product, the VMware EULA would prohibit publication without VMware’s approval. The focus of our study was XenServer and Hyper-V, and we determined that they lead in VDI density.

Bottom Line: Is VMware over-emphasizing TPS because it is the only feature that other vendors don’t have today? That would seem a risky strategy. If you’re a VMware customer, you should demand concrete evidence of the purported benefits of their stack, given that at least two free competitors – XenServer and Hyper-V, outperform VMware for VDI.