I am writing this on a Windows 7 desktop running as a VM on a XenServer pool running a pre-release version of XenServer. In XenServer Engineering this system is known as XenIA. It is an example of a so-called “dog-fooding” system, whereby members of the engineering teams who create a product also use that product in their daily work.
Citrix has a long history of using its own products, usually in one of two ways: as a corporate IT led deployment of shipping versions of Citrix products such as XenApp, NetScaler, CloudBridge or CloudPlatform ; or as more local deployments within engineering teams, often of pre-release versions. XenIA is an example of the latter. It is run by the XenServer QA team for use by members of the XenServer engineering group.
The primary aim of XenIA is to provide XenServer staff with an insight into our customers’ experience of XenServer. In other words, to gain a measure of empathy with our customers. To do this we need first-hand experience of deploying and administering a production XenServer pool running live workloads that matter to people. We also want to understand the impacts of our product quality, performance, stability etc on the end-user experience.
We run two XenServer pools, one containing the last GA version of XenServer kept up-to-date with all hotfixes, and one with the current pre-release version of XenServer. Various workloads that are critical to the XenServer product group are run on these pools, including virtual desktops. Although XenIA does not use a full XenDesktop deployment (RDS or VDI) , it does re-use the virtual desktop “brokering” functionality of XenDesktop to provide easy access to persistent VMs. Members of the XenServer product group are encouraged to use these VMs as their primary desktop for day-to-day work, whether in the office or out.
As mentioned, a key goal of XenIA is to provide the XenServer product group with an insight into what it takes to administer XenServer production pools. Our QA team own XenIA and as such gain experience of installing, upgrading, hotfixing and generally maintaining a production system and providing a service based on it. Their findings are injected into XenServer roadmapping decisions, bug trackers and product enhancement proposals.When a new release of XenServer is imminent, we upgrade one of the pools to use the new version, thereby validating the upgrade path from the previous XenServer version, and exposing the newly-installed candidate builds to live usage and hence, in essence, a real-time soak test. When a new Hotfix for the last GA version of XenServer is imminent, the final validation step is to apply it to the appropriate pool to ensure it runs successfully there first.
Crucially, although the system is run by XenServer QA, members of our development and product management teams often participate in administration tasks such as upgrades. We want as many members of the product group as possible to gain a better degree of empathy with our customers, not just as end-users but as system admins as well.
Of course, an additional benefit of XenIA is that by its very nature it is a live system that has to run 24×7. This gives us a degree of soak testing that complements our existing but more static automated soak tests. XenIA has various monitoring and checking capabilities built into it including memory leak detection, with more planned (including use of 3rd party tools integrated to XenServer). This makes XenIA a powerful part of our Continuous Integration (CI) suite. Examples of defects found by XenIA, before release, are a nasty upgrade bug between XenServer 6.1 and 6.2 caused by the addition of a new piece of VM metadata, various permissions defects and usability issues.
XenIA is run by and for the XenServer product group and focuses on XenServer largely in isolation from other Citrix products.
One of the inspirations for XenIA is another internal “dog-fooding “ system known as RTST (“Real Time Stress Test”). This has been run by our colleagues in the XenDesktop product group for a number of years, providing XenDesktop and XenApp virtual desktops running on a variety of different hypervisors.
The XenServer engineering team also runs a Citrix CloudPlatform deployment on XenServer.
XenServer staff gain empathy with our customers’ experience of administering and using XenServer through our use of an internal deployment of XenServer with live and business-critical workloads.
Findings are fed back as defects and product improvement requests, or more generally as context for roadmapping decisions.