Today Citrix announced that we have partnered with McAfee to advance the state of the art in secure virtual infrastructure. In this post I aim to provide more detail on the partnership announcement and the technologies involved, and to place them within the broader context of security for virtualization, with reference to Hoff’s superb discussion of the key issues facing security in a virtualized environment in his “Four Horsemen Of the Virtualization Security Apocalypse”. Our goal is to deliver technologies that make concrete progress toward the taming of the Four Horsemen.

Our partnership will deliver innovation in virtualization and security by delivering endpoint security technology that:

  • is of immediate practical value – security specifically optimized for virtualization – yet hypervisor independent
  • offers the industry powerful new tools for securing virtual environments – hypervisor native detection dramatically shifts the power of detection in favor of the virtual infrastructure
The problem

Today, securing guests in virtualized environments doesn’t scale well, and it’s fragile. It relies on the use of traditional endpoint security (EPS) solutions (AV, firewall etc) within each guest, and/or inserting virtual-appliance (VA) based network security functions in the data path to/from the network or storage. This is particularly acute in VDI, where large numbers of guests can be hosted per server.

  1. The former results in hundreds of copies of the EPS agents, signature files etc per server, wasting CPU, memory and storage. Synchronized AV signature file updates and hard disk scans hammers VM performance and overloads storage and the network. It’s bad, but until today was the state of the art.
  2. The latter requires the hypervisor to provide a privileged API to security VAs so they can inspect traffic. It suffers from three major drawbacks:
    1. Software based processing of each packet/block is slow, and any stateful filtering is vulnerable to the dynamic relocation of a VM – ie leaving the state behind as the VM moves to a new server
    2. The trend in desktop virtualization is to dynamically compose a VM from golden images of the OS & apps, and a virtualized user profile, to reduce storage cost, and improve performance and manageability. Traditional EPS relies on being installed in the OS, where it can continually check storage etc as the desktop evolves through its life. But now we might build a virtual desktop for a user session and then discard it on log-off, but still want to protect it fully.
    3. The rapid drive towards hardware-virtualized I/O that directly plugs into the guests, bypassing the hypervisor and therefore eliminating any opportunity to inspect traffic in software.
  3. The drive toward hardware I/O virtualization is motivated by the increasing density of guests/server – XenDesktop supports up to 16 VMs/server on modern Intel Xeon 5500 series or later servers, so there is a really urgent need to address this issue.
What have we done?

We have entered into a strategic partnership with McAfee to develop an endpoint security solution for XenDesktop that addresses security at two crucial layers of the software stack. Any solution that fails to address both layers fails to address the key concerns in virtualized security:

  1. An optimized virtual infrastructure security service that is hypervisor independent
  2. A hypervisor-native detection service that enables a quantum leap forward in secure virtualization, expressed via an open API to third party detection and remediation tools such as McAfee’s.

In the first category, the solution

  • Enables complete integration of endpoint security into the lifecycle of virtual desktops
  • We are developing an open, standards-based, hypervisor-agnostic set of virtual desktop security APIs to allow all vendors of virtualization-ready EPS to integrate their solutions with XenDesktop, and Citrix will offer a new category of Citrix Ready certification for EPS vendors.
  • We address key needs for scalability, without compromising on the need for per-guest protection, thereby making desktop virtualization robust, scalable and enterprise-ready for the most demanding organizations.
    The partnership will give customers the most comprehensive protection for application and desktop delivery of any solution in the industry. Crucially, this allows us jointly to meet customer needs to
  • Simplify the management and efficiency of EPS for XenDesktop’s by using dynamic composition of the user’s virtual desktop from centrally managed golden images combined with user-specific applications and profile. This reduces the number of managed images, simplifies patch management and allows IT to ensure users always use latest secure OS and apps.
    • Integration with McAfee MOVE – virtualization-optimized EPS makes the execution of EPS security tasks (detection, remediation) a property of the virtualization platform as opposed to within each virtual desktop
    • By tightly coupling EPS management with virtual desktop lifecycle management we eliminate potential for error, mis-configuration and poor performance that results from deploying EPS individually to each virtual desktop
    • Leveraging the existing management tools for EPS, and not tools that are specific to XenDesktop, simplifies management of EPS for all desktops, native and virtual. McAfee ePolicy Orchestrator is customers’ preferred way to manage security for their McAfee secured desktops today.
  • The solution will allow virtual desktop security to scale to tens or hundreds of thousands of desktops
    • In particular, for VDI desktops, the partnership will allow XenDesktop on the customer’s preferred hypervisor (XenServer, Hyper-V or ESX) to play a key role in threat detection and reduce the need for complex detection and remediation per virtual desktop; instead it offers a significantly faster, lightweight capability per guest, with core detection and prevention capabilities executed in a per-host service VM. Other benefits include reduction in storage, reduced network and storage traffic associated with AV scans, and centralization of the EPS threat library update.

In the second category, namely hypervisor-native detection, Citrix and McAfee are integrating a novel form of in-hypervisor detection into XenServer and XenClient to offer dramatically superior security to virtualized workloads than is possible with the state of the art today. The in-hypervior detection capability will also be contributed to Xen.org, together with a new open (non-GPL, open source) API that will allow the security ecosystem broadly to exploit and add value to all Xen based clouds and clients. McAfee will offer products that exploit the new API to offer its customers a leapfrog capability for protecting virtualized infrastructure.

The technology involves the addition to Xen of an ability to ensure that the state of a guest VM is as intended, as opposed to trying to find attackers. For example, when a new app starts in a VM, new pages are added to the guest’s page tables. By defining triggers associated with key low-level primitives within the hypervisor, and enabling it to then perform simple checks (such as computing a hash on the code pages of the app) we can facilitate granular, dynamic run-time checks that allow IT administrators to be certain that their VMs have not been compromised. The technique used is called white-listing, and relies on the hypervisor’s ability to use dynamically gathered data to check that the system is in the right state, rather than trying to find bad guys by detecting changes in guest behavior or state. This, combined with an ability to exploit Trusted Platform Modules, and Intel TXT / AMD SVM features to build a platform-based root of trust, will reassert Xen’s leadership in the domain of secure virtual infrastructure. It is of immense importance in the context of XenClient, where our partnership with Intel will enable corporations to ensure that enterprise desktops can be securely delivered to devices outside the traditional security perimeter of the enterprise, where they can be co-located on the same device as an untrusted user personal VM.

The technology we are pursuing is entirely different from, and vastly superior to, the memory inspection APIs that VMSafe offers today. Poking about in memory in the hope of finding an attacker is a bit like looking for the proverbial needle in a haystack, but with the additional complication that the needle can be split into many parts, and can disguise itself as a piece of straw. Our technology offers a much richer interface and a positive attestation as to the state of the guest. This critically important at a time when rapidly self-modifying attacks are making the job of attack detection increasingly difficult.

The Context

Some time ago I was lucky enough to survive a virtual debate with the fearsome Chris Hoff about the role of hypervisors in securing virtualized enterprise infrastructure.

Hoff’s dissection in “Four Horsemen of the Virtualization Security Apocalypse” of the security issues facing virtualized infrastructure identifies several categories of security management that virtualization turns on its head. Here’s a more complete discussion (for the record, he missed one Horseman):

  • Securing applications that run on virtualized infrastructure
  • Securing the virtual infrastructure itself
  • Virtualizing the security functions in the infrastructure

The key disagreement was the role of the hypervisor vendors in delivering technology to secure virtualized workloads. My position then, as now, was that virtual infrastructure vendors are not smart enough about security to do a good job of delivering “built in” capabilities to secure VMs, and that what was required was an open set of APIs that would allow security ISVs to add security juice that could take advantage of the hypervisor’s privileged position to deliver enhanced security. I pointed out that VMware’s position was muddying the waters – that through its acquisitions of Blue Lane and Determina it was trying to be both a virtual infrastructure and a security vendor, and by virtue of its competitive stance, would be unlikely to offer sufficient richness or business opportunity to its security ISV competitors.

VMware’s VMSafe 1.0 initiative, announced to great fanfare in Cannes, in February 2008, has yet to deliver any substantially new solutions in the area of security, and indeed fails to address any of the issues raised by Hoff or the challenges that result from the trend toward increasing hardware support of virtualization. VMSafe 1.0 offers APIs that allow “helper” virtual appliances to gain access to

  • Network packets traversing the virtual switch
  • Block I/O traffic to/from storage
  • Guest VM Memory

Neither VMSafe or other initiatives to date address key challenges in security:

  • With SR-IOV devices now shipping from Intel (as demo-ed using [XenServer:http://www.xenserver5.com] at Synergy), direct hardware support for I/O virtualization leaves the hypervisor in no position to inspect network and even block traffic into/out of a VM.
  • No major security vendors have offered solutions that take advantage of the VMSafe cability that allows an external agent to peek into guest memory. Why? It’s incredibly difficult to detect attacks at this layer, and nearly every in-OS endpoint security solution requires a much more semantically rich interface to understand what the guest is doing, than simply having an ability to poke about in guest memory. Moreover, the black-hat community has pointed out the obvious risks of any API that exposes guest memory to another VM, particularly given VMware’s spotty security record.
    Neil MacDonald of Gartner accurately summarizes that while VMSafe is a good first start, it is no panacea, and Hoff agrees that solutions offered to date for VMSafe 1.0 are not really moving the world forward:

There is lots of effort going on to try to force-fit entire existing markets of solutions in order to squeeze a little more life out of investments made in products, but expect some serious pain in the short term because you’re going to be dealing with all of this for the next couple of years for sure.

Now, to put the security/protection debate into stark context from a vendor perspective (ie: as a vendor, you can’t hide, so you might as well declare everything and get on with it), Justin Foster of Developing Security highlights the key challenge:

I challenge VMware, Citrix or Microsoft to introduce a common technique for running security agents outside of the virtual machine where the policy and even the security software could be carried with the VM (in a OVF or similar). Extend this support to a wide range of physical hardware and delivery models (including IaaS) and we could have VMs of any nature (desktop or server) runnable in a wide variety of environments with closely coupled, but isolated security in tow.

I believe the Citrix and McAfee partnership substantially changes the landscape, and look forward to an ongoing debate as we deliver the technologies to market. What we are working on with McAfee will

  1. Address the urgent need to optimize per-endpoint security, independent of the hypervisor
  2. Enable customers, clouds and community partners to use the Xen hypervisor as a new vantage point in the battle for protection of enterprise workloads, by exploiting opportunities in the execution stack to implement positive attestation as to the posture of the workload. We are actively developing a powerful new open source virtual switch, maintained at openvswitch.org that will allow security virtual appliances to gain access to network traffic, and the virtual switch to control (through deep packet inspaction and programmable forwarding rules) per-flow based access to the virtual infrastructure. Though our virtual switch (which will also work for Linux/KVM) offers a more powerful set of open interfaces than VMware’s vDS, we fully recognize that we are still as an industry in the early stages of having a comprehensive set of security solutions. But we’re making excellent progress.