A 3D benchmarking blog that doesn’t even look at the frame rate….

Over the last few weeks I’ve been blogging about tools and factors to consider when benchmarking and evaluating GPU sharing using the new vGPU technologies on Citrix XenServer and XenDesktop. I’ve been seeing a lot of users looking at frame rate (fps) metrics to compare technologies. This blog aims to step back and consider what the appropriate benchmarks are and how supportable various technology models are particularly with respect to their support ongoing for OpenGL and DirectX APIs.

A few months ago Gartner analyst and End User Computing Enthusiast Gunnar Berger put out a comparison of GRID ™ vGPU versus VMware’s vSGA technology using the Redway3d Turbine benchmark , which is particularly CAD focused. In contrast, gaming benchmarks such as Unigine Tropic are actually much lighter on GPU usage and quite heavy on CPU so you can hide a lot of limitations and differences between technologies. I’ve been looking recently at more CAD oriented benchmarks to complement Unigine whilst evaluating the differentiators in vGPU from other technologies, with a particular view to CAD related uses and 3D model visualization.

As I expected, GRID vGPU did indeed shine with the CAD oriented Redway3d benchmark. What was surprising was how poorly vSGA performed, the video is here. At times Gunnar had to resort to speeding up the video of the vSGA tests 10x times to keep up with vGPU. Something went very badly wrong in that vSGA benchmark. This was a test running on one of Dell’s top GPU servers, the R720, with plenty of RAM and CPU and with an NVIDIA K2 graphics card. How could such high quality hardware frequently used to support hundreds of VDI users, result in a test where a fairly average, standard CAD benchmark failed to support even a single user?

To be fair to VMware do make it clear that vSGA isn’t suitable for most serious CAD applications, and suitable for “review and lightweight use cases”  only and also clearly document these applications are not GPU vendor certified for use with vSGA. Nevertheless I’d have expected the theoretical behavior vSGA’s synthetic driver model for GPU sharing to be better.

Gunnar made an interesting observation in his video;  he had been unable to get vSGA working with Redway3d Turbine under XenDesktop. This was something we wanted to check out. When we looked into it we couldn’t find a problem in the XenDesktop stack but Redway3d seemed unable to use the vSGA VMware synthetic drivers. Given how badly vSGA appeared to behave with such a high specification of test rig, it has made me wonder if what was being compared was even vSGA. Given such poor performance I wondered if vSGA somehow falls back to some sort of software rasterisation.

The questions this test raises are concerning to the end CAD user and have made me consider the very architecture of GPU sharing technologies. Tim Mackey wrote a blog explain the different technologies and driver models a few weeks ago, which I believe is essential reading for anyone evaluating buying these technologies.

Citrix GPU technologies leverage the GPU vendors’ own drivers, and NVIDIA’s entire organization is devoted to integrating the latest cutting edge technologies, OpenGL APIs etc. As such the application support is provided by NVIDIA without any synthetic virtualization drivers.

Anyone working in CAD knows the nightmare of cross-organisational issues, the sheer overhead of software certification and paperwork associated with data part export compliance to allow an issue to be looked at by another organization (and the woe when one team is in the US and the other isn’t!). The cost and delay of such an issue plus the vast resources consumed regression testing and re-integrating a component fix in these highly regulated industries is daunting at the best of times. The simplicity of the vGPU architectural model removing a certification layer seems so elegant in this context.

Why does Redway3d matter?

Redway3d is probably a less familiar name to most end users than applications such as Dassault SolidWorks and Siemens PLM NX. But Redway3d are a significant component supplier in CAD/CAE/AEC. Most CAD products use the same components for large parts of the infrastructure of the product and add value in the GUI or technologies such as simulation solvers.

Redway3d is a rendering and visualization component embedded in a large number of CAD applications and platforms. Many benchmarking programs are actually quite old and some are not very realistic for CAD users. Redway3d utilizes the CPU and GPU in a way that many CAD users. Redway’s SDK is a key component in a number of top CAD programs including Missler’s Topsolid and Airbus. They integrate against leading geometric kernels such as Siemens PLM’s Parasolid (which underlies NX, Solidworks, Ansys Workbench, SolidEdge) and have strong links to Dassault (CATIA) and their customer base.

Redway’s Turbine Demo has been around for several years and is in wide use, proven to work on physical systems. In fact it was used a few weeks ago by Dane Young to demonstrate XenDesktop 7.1 on HP Moonshot architecture. It is actively regression tested at NVIDIA and one of the applications they frequently issue performance enhancements for. Which does together with the other evidence suggest the problem is in the virtualization stack.

As such, a problem with Redway3d integration within a virtualization stack is very concerning as it could mean end users are restricted in their choice of software and also face potential uncertainty, delays and costs resolving issues.

If I were looking to buy a solution I’d be seriously quizzing the sales teams involved and be including this benchmark in my evaluation, not to compare the frame rate but to understand the ongoing driver support liabilities.

So what is going wrong in Gunnar’s trial of vSGA?

I actually contacted Gunnar and asked whether my theory of some sort of fallback to software rasterisation was plausible and he thought not. Apparently he had monitored the GPU usage to eliminate that possibility himself and the GPU was at times used, pretty much, at maximal capacity. I’m still also puzzled as to why the vSGA drivers failed under XenDesktop.

I did look into the demo’s OpenGL support, and was glad to see it covered key CAD functionality such as Hidden Line that the gaming benchmarks and alike tend to overlook, however I failed to find sufficient details to grasp if the vSGA test performance would affect other OpenGL based applications.

One of the reasons for writing this blog is that I’m hoping someone may have a better knowledge of VMWare’s stack and its interaction with Redway3D’s applications or Redway’s OpenGL usage and enlighten me, so please do comment as I’m still bamboozled!