Engineering teams at Citrix are constantly innovating. We wanted to share an insider look into some of the more impactful innovations that are internal to Citrix development teams.

These innovations do not offer anything to end users directly. They improve developer productivity and our internal testing process. This allows engineer to release higher quality products more frequently, which indirectly benefits end users.

End-to-end automated testing leads to higher quality code! This means more stable and feature-rich products for customers, as well as happier and more efficient engineers. The XenApp and XenDesktop team utilizes many forms of automation at various levels of development. We recently started taking existing system tests and begun providing them “as-a-service” internally, so that Citrix engineers can run them on demand. This service transformation is worth sharing, so others can implement it and customers can see how we drive product quality and offer reliable cloud solutions.

The first transition to the on-demand service model was the XenDesktop Cloud BVT test which needed to scale to meet demand while reducing its management overhead. The second transition, spurred by the success of the BVT, was a scalability test. The team desired quicker turnaround time between requesting the test and getting the results. Both transitions have increased efficiency and improved product quality by making the automation more easily accessible as a service.

BVT-as-a-Service:

DISCLAIMER: This is an internal, engineering only service targeted towards Citrix Development teams. This is not an external service offering.

The XenDesktop Cloud BVT is an automated build verification test (BVT) service created to allow developers to quickly test any new code changes as part of new feature development. The test is especially critical as a passing report is a gating requirement for all submissions to the mainline code branch. Therefore, the test is required to be stable and reasonably efficient or the team’s productivity is significantly impacted. Prior to the “as-a-Service” changes, each engineering team had their own hardware dedicated to running these types of automation. Each team generally underutilized their hardware and during a closedown were under resourced. This was not the best use of infrastructure.

Before

Under the new approach, rather than each development team triggering the BVT, the test automation is triggered automatically for each new build on each stream once the build is ready. The test can also be triggered on demand from the Continuous Integration (CI) system’s webpage where additional customizations can be specified (such as enabling beta features). In both cases, summarized results that contain links to detailed reports, product tracing and methods to get additional help are received via email and are also available on the CI webpage.After

By integrating the automation framework and CI system (we actually install a CI agent on each automation controller!) we gain several benefits for the service.

  1. Priority Queuing – The co-located CI agents are placed into a worker pool in the CI system. This allows jobs to be queued and run by the first available automation controller making efficient use of resources.
  1. Build Chains – By creating a new build job that is executed at the end of each product layout we can automatically trigger any required automated tests without waiting for an engineer to manually take action.
  1. Code Sync – When a build chain triggers an automated test, the CI agent syncs the associated stream to the automation controller. Because we store automation code in the main product code branch engineers can make BVT breaking changes by simply updating the associated tests in the isolation of their own stream without assistance from the automation team.

At this point, the BVT-as-a-Service is executing ~40-110 runs a day which adds up to 1000’s of BVTs per month providing fully automated coverage for over a hundred code streams while keeping the main code branch stable!

Scalability and Performance Testing -as-a-Service:

DISCLAIMER: This is an internal, engineering only service targeted towards Citrix Development teams. This is not an external service offering.

We had such great success with the BVT-as-a-Service we decided to expand the service model to include  scalability and performance testing! We started with a registration scale test followed by an end user session launch test either of which could take up to a week to get results on. Using the same mechanisms as described above with the BVT, we were able to offer this test as an on demand service with results in just a few hours.

One notable difference between this test and the BVT is that its environment is dynamically created fresh each run. This consumes a bit more time to deploy but is a tradeoff that allows for much more flexible control of the test environment. For example, engineers can specify the scale of the test desired or even the software versions used throughout the test environment.

By offering these tests as a service, engineers get fully automated results allowing them to quickly and confidently make changes to areas of the product that can affect scalability and performance. Detailed reports (examples below) allow engineers to measure the product performance from build-to-build on any stream against known baselines to ensure we are always improving and never regressing in terms of scalability and performance.

STAT

Summary:

Based on our experiences using this approach we have plans to improve it going forward including an automated BVT result analysis tool to aid in triaging failures and tracking metrics. As well as converting additional scalability tests to the service model.

Conducting complex automated system tests as-a-Service requires many moving parts and is by no means fool proof. However, by doing so we enable our team to efficiently test code in a self-service model that best fits our agile development practices. These are a few of our as-a-Service success stories, let’s hear yours!

Summit banner