Comments on the “Application Virtualization 2008-2009 Performance Assessment” published by Devil Mountain Software

Remember the kid in grade school who would announce that his dad was a CIA agent or something like that in order to win friends and influence people… or at least 4th graders?  You know, the kind of scenario where you were filled with the sense of “does this kid even know how silly he looks”?  The problem was that he would always avoid discovery by saying “Yep, his job at the post office is just a cover”.  This last statement was used to inject just enough uncertainty to make the story linger as plausible, at least until the tale gets so absurd that no one can take it any longer, or the kid’s little sister shows up and blows the whole fantasy.  VMware’s latest string of sponsored “independent” analysis comparing their View product line with XenDesktop, and now a heavily biased assessment of Application Virtualization solutions, have left me with that same “does this kid know…” feeling.

The “uncertainty injector” for VMware’s tall tales remains their End-User license agreement that basically forbids any 3rd party from publishing comparative data without written permission from VMware.  This written permission usually comes at the cost of a watered down analysis that leaves the reader with little to no usable information and a bad impression of the independent analyst.  This latest publication from Devil Mountain Software is a perfect example.  Devil Mountain Software’s test utilities are used by many as an automated method of benchmarking Microsoft Office run-time performance on a given platform; Citrix has even used OfficeBench in the past for some in-house testing.  The problem I have with the report is not necessarily with the tools used, or even the data presented for the tests run, but rather the narrow point of view driving the tests and the broad and sometimes contradictory conclusions.

But first, how has the blog community responded to the report?

The primary public outlet, with the largest number of public comments for this report has been a February 5, 2009 blog post by Ruben Spruijtunder the BrianMadden.comcommunity site.

Head-to-head performance analysis of App-V, SVS, ThinApp and XenApp

Ruben is a well respected subject matter expert with regards to the Application Virtualization space.  I think he has served the community and any application virtualization prospects well with his latest “Application Virtualization Solutions Overview and Feature Comparison Matrix” comparisons.  He is a true customer champion, and as such ultimately qualified his posting of the Devil Mountain Software document as being in the spirit of providing an open forum where the community could have an opportunity to provide a “peer review” of the analysis.

The current conclusion of this blogosphere peer review would indicate that the analysis is based upon faulty methodology, blatantly influenced by VMware, who appears to have commissioned the test.

What is wrong with the report?

As I have already said, I won’t argue about the tools used or the validity of the data.  I do however have a few comments and questions about the methodology, presentation and the conclusions in light of the data.

First, some comments on how VMware ThinApp is incorrectly positioned in light of all other vendors in the report:

The installation comparisons indicate that the analyst either does not understand what this technology is used for, or DMS is intentionally misleading the reader with regards to feature parity across the solutions tested.

The analyst’s notes on the comparative complexity of installation would leave an uneducated reader with the sense that ThinApp as installed provides the same functionality as the other vendors tested, including Citrix XenApp.  This is far from true in that Citrix XenApp provides an integrated management system that delivers the benefits that enterprise application deployments require, such as;

o   Controlled access to and distribution of virtualized applications,

o   Virtualized application license monitoring and control,

o   Target platform control (necessary to provide a predictable and supportable user experience),

o   Local or remote execution fallback,

o   …and many other features not available in VMware ThinApp, which is basically a packaging utility in comparison to every other vendor referenced in the report.    I’ve skipped all of the marketing speak and “cool feature name” details of what is involved here in order to keep this post as concise as possible.  For a detailed description of XenApp capabilities please check out the XenApp product pageand the XenApp blogs.

So, in order for the comparison to reflect parity between all vendors, DMS should have either explained the necessity of installing additional 3rd party management infrastructure and tools that are required to bring ThinApp up to parity, or the test should have compared a different set of vendors that do not include enterprise class management.  Instead, DMS concludes that 3 out of 4 vendors tested include components that unnecessarily complicate the installation of the solutions.  This conclusion presents the reader with an inevitable bait and switch scenario since it completely disregards the integration effort required before ThinApp can be used as part of a secure and managed solution.

The Tables: Bar charts are good; numbers would be more useful…

First of all, I have to thank VMware and DMS for allowing Citrix XenApp to score as the second best solution, even though I can only guess what the conversation with VMware was like during the disclosure of the data.

But let’s move on to how the results are presented…

“Cached + Uncached / 2” is an irrelevant formula, but it is the foundation of all of the conclusions in the report.

The tables clearly compare the difference in results between “uncached” and “cached” execution for all 4 vendors.  In real world deployments of application virtualization it is a widely acknowledged best practice to pre-deploy the virtualized packages to the end-point to avoid longer than necessary initial execution times when possible.  In general the “streaming” component of all of these solutions is best understood as the primary mechanism for application self healing and real-time application updates and roll-back, not the initial delivery. 

For the sake of continuity, let’s agree with the report’s assumption that all applications use streaming as the initial deployment mechanism.   In this case, although the uncached performance data is valuable to understand, when building a formula to determine the average execution time “cached + uncached /2” would be incorrect, since the reality is not “/2” but rather “/n” where n represents the total times cached execution takes place after the initial, and singular, instance of an uncached launch. 

The real formula would need to be “uncached + (cached * n) /n”

In my personal experience with our Citrix in-house implementation the formula would be easily “uncached + (cached * 1000)/1000” making the difference in “raw script performance” between Citrix and VMware 4%, not 18% as stated in the report. This 4% delta will decrease to 2% over the next 6 months as I continue to use my streamed applications.  Ultimately the delta in performance will approach 0% depending on the lifecycle of a given application.

*Now let’s look at the tables:*

My quick take on the tables is this: There are several charts in the analysis that communicate the relative performance across all 4 solutions.  In most cases, the difference between vendors is significant enough for the reader to understand the relative value of each solution.  Unfortunately (or maybe fortunately?) from a Citrix XenApp perspective, the chart values between Citrix XenApp and VMware ThinApp (the concluded winner) are so close that publishing the actual numbers would have been the only way to clearly demonstrate the difference.  My take from the bar charts presented is that XenApp is within a 1-5% margin of ThinApp for every test.  Again, it would be really nice to have the actual numbers so that readers can determine, if over a 70 second period of time, a 3 second difference (or is it 2?, do I hear 4?) is enough of a performance “hit” to justify the whole-sale trade-off of any management whatsoever in order to make the ThinApp solution as DMS concludes “the logical choice”.

%Execution Overhead: (Figure 1.1, Page 3)

Citrix and VMware practically tie for cached execution, in spite of the fact that Citrix is handling a much higher load of communication and monitoring with regards to centralized management, which ThinApp does not provide – period.

 OfficeBench Completion Times (Sec): (Figure 3.1, Page 11)

4% difference between ThinApp and Citrix? 70 seconds vs. 67 seconds?  Streaming overhead is only realized on initial launch of non-cached applications.  As stated earlier, streaming is primarily an update/synchronization mechanism, not a “deployment mechanism”

OfficeBench CPU Consumption: (Figure 3.2, Page 11)

Again this is a statistical tie between the 2 vendors for a single application instance in Windows XP.  Because the ThinApp embedded agent must load with each application, this would probably demonstrate Citrix as the better performer as more applications are introduced to the test, even if we stick with Windows XP SP3 only and completely ignore the ramifications of the ThinApp “embedded agent” in a Terminal Services execution environment.  But neither of these latter scenarios was considered.

Combined Working Set – Office + Agent (Mbytes): (Figure 3.3, Page 12)

Office + Agent (Mbytes), nearly a tie between Citrix XenApp and VMware ThinApp as tested.  But because the ThinApp “embedded agent” loads for each application instead of only once as in the case of every other vendor tested including Citrix XenApp, this is where the ThinApp story falls apart quickly, most likely with the addition of as little as one more virtual application.  The implications of this unacknowledged “embedded agent” overhead could prove radically misleading in all but the simplest implementations, not to mention what the impact in a multi-user environment such as Terminal Services or even a VDI model would be.

Streaming-Related Network Overhead – Avg Mbps: (Figure 3.4, Page 12)

Citrix XenApp wins! But again it is a statistical tie with VMware ThinApp for a single application over a Gigabit connection.  The real world question left unanswered by this particular test is what the launch time numbers begin to look like for an update over a high latency WAN connection of 1.5 Mbps or even lower (which is the real world manifestation/concern for this scenario).

Comments on the methodology and testing parameters:

Regarding the methodology used for testing, there are several aspects that are not valid from an enterprise perspective:

  • There is no indication that Devil Mountain Software attempted to contact any other vendor in the report in an effort to ensure the validity of the tests. 
  • The test should have included benchmarks in 64-bit environments for those solutions that support 64-bit such as Citrix XenApp. 

This one in particular stands out in the report since, from a basic comparison perspective, it is obvious that the analyst is not aware of even the most basic facts regarding the capabilities of XenApp and did not even read any of our documentation, nor did they attempt to contact Citrix in an effort to clearly understand basic system requirements and platform support.   The report states:

“As of this writing, only ThinApp supports deployment onto 64-bit versions of Windows XP, Server and Vista…”  

This is simply not true. XenApp Application streaming supports 64-bit environments and has since its initial release.  Had the analysts bothered to read any of our documentation, even a cursory review of the README and Installation Guides would have enabled the analyst to avoid making such a blatant mistake, one that in the eyes of the community immediately invalidated the report.

  • What are the impacts in Terminal Services and VDI environments?  The tests should have included Microsoft Terminal Services since this has historically been the target platform for application virtualization solutions.      
  • Why only one instance of Office?  Isn’t the whole point of application virtualization the ability to run several simultaneous applications that may not normally work otherwise?
  • The tests should have included a more complex set of applications and application dependencies where inter-isolation communication mechanisms would impact the results.  All four solutions provide some form of inter-isolation communication although the performance characteristics across the group vary significantly.
  • Why is management of the applications disregarded?  DMS should have at least added a 3rd party management agent to the ThinApp build, but that probably would have changed their numbers for CPU and Memory overhead if measured properly. 
  • If the comparison of managed application virtualization solutions was not the focus of the report then other unmanaged “embedded agent” like solutions should have been the chosen vendors in order to provide a more direct comparison to the ThinApp offering.

One final word about the quality of the document itself…

In general these types of sponsored documents are inherently questionable, although the best ones do show a degree of contrition with regards to the positives of all of the solutions tested in an effort to establish credibility of the final conclusions.  This document lacks this basic tactic, and in the end that exaggerates the obvious and invalid bias toward the VMware solution.

I hate to beat a dead horse but basic proofreading of the doc may have also helped establish some credibility.  In a couple of introductory paragraphs (page 3 and page 5) DMS states that their goal was to analyze the 5 leading application virtualization solutions, but the published results only include 4 solutions.  This begs the question “Was it 4, or 5, vendors?  If it was 5, what happened to the data for the 5th vendor and who are they?  If it was 4 but the document reviewers were not concerned enough about the numbers to fix the typo, then how can we trust the rest of the numbers in the document?”   My last nit in this regards is “who owns the copyright?”  I assume that it is Devil Mountain Software, Inc. since the document is posted to their website and the front page copyright states “2008 – 2009 Devil Mountain Software, Inc.”  I am only confused because the copyright on the last page is “2004 Competitive Systems Analysis, Inc.”.  Okay, this may be a stupid nit, but again, it is indicative of the overall attention to detail regarding this report.

And finally, I called Devil Mountain Software to clarify some of these questions and to find out if they would be willing to run another set of tests across all of these vendors, which would include more meaningful parameters as well as input from all of the vendors willing to participate.  Honestly, my whole goal here is to rise above putting out “bad marketing collateral” disguised as scientific analysis, in order for all of us participating in the evolution of application virtualization to benefit, most of all the IT buyers who have to make the tough decision on which new technologies to embrace in an economy with no room for error.

Hopefully someone at Devil Mountain Software will call me back, and hopefully the caller ID will not display “So and so’s dad at the CIA”.

I look forward to all of your comments, let’s get this right!