As a Technical Relationship Manager, I spend a lot of time researching the more complex questions asked by customers that do not fit into the traditional Break-Fix reactive support model. One that I was asked recently sparked a good bit of thought, research and a debate within the TRM Team at Citrix and I wanted to share my findings.
The customer question was: What is Citrix’s recommendation for the maximum number of servers, users etc for a XenApp 6.5 Farm?
There are some published documents that discuss XenApp 6.5 Farm scalability which you can read here:
Planning Server Functions: http://support.citrix.com/proddocs/topic/XenApp 6.565-planning/ps-planning-infrastructure-servers-v2.html
XenApp 6.5Enterprise Scalable XenApp 6.5 Deployments: http://support.citrix.com/article/CTX131102
Speeding up Farm Deployments with XenApp 6.5– Part 3: /blogs/2011/09/23/speeding-up-Farm-deployments-with-XenApp 6.5-6-5-part-3/
Pedal to the Metal: Bare Metal Scaling of XenApp 6.5 Hosted Shared Desktops: /blogs/2013/03/28/pedal-to-the-metal-bare-metal-scaling-of-XenApp 6.5-6-5-hosted-shared-desktops/
After reading these documents I drew a these conclusions:
- The largest XenApp 6.5 deployment we have tested internally was 1000 servers in a single Farm.
- Memory consumption of the IMA Service on the Zone Data Collector is going to be a critical bottleneck as Farm size increases.
- It is the fine detail of how the Farm has been deployed and how it is used that will determine overall scalability.
GENERAL FARM LIMITS
We recommend no more than 5 zones per Farm due to the mesh network nature of Zone Data Collector communications – adding zones exponentially increases the network chatter and bandwidth required to maintain inter-zone communications. I wrote about this a couple of years ago here: /blogs/2012/11/26/some-XenApp 6.5-6-5-Zone Data Collector-replication-calculations/.
If you need more than 5 zones you probably need to look at your original design and consider whether to implement multiple XenApp 6.5 Farms. I recommend reading more about this here: /blogs/2013/05/01/xenapp-farm-and-zone-design-v2013/.
There are no hard limits for the number of servers in a zone, although you may hit a CPU or memory limit on the Zone Data Collector for the zone (discussed below), note this comment from eDocs (http://support.citrix.com/proddocs/topic/XenApp 6.565-planning/ps-planning-data-collectors-planning-v2.html): Likewise, CPU usage is not significant. A data collector hosted on a dual-processor server can support over 1000 servers in its zone. In general, CPU usage increases as the number of servers in a zone increases, the number of zones increases, and the number of users launching applications increases.
There are no limits for the number of worker groups in a Farm.
There are no limits for the number of servers in a worker group.
Total number of servers
We have tested XenApp 6.5 to ensure scalability and performance with up to 1,000 XenApp 6.5 servers in a single Farm. Some of the key metrics found during testing:
- Application publishing to worker groups and load balancing policies had no measurable impact on application enumeration or load balancing times.
- The number of worker groups had minimal impact on discovery times for the management console. Adding 200 worker groups increased discovery time by 2.5 seconds, while 500 worker groups increased the time by 4.2 seconds.
- Worker groups and their memberships are cached in memory in every IMA service for performance. This results in an increase in memory consumption of 8 KB for every worker group in the Farm.
From this information you can see that there no known hard architectural limits. I accept that because we have not tested beyond 1000 servers, some customers may consider this a “limit” because that is what we have tested and validated. It is of course unlikely that there will be many customers that need a Farm of this size; a quick poll of my fellow TRMs suggested there might only be a handful of XenApp 6.5 Farms of this size globally.
However, if we look in more detail at a XenApp 6.5 Farm, there are a couple of variables that need to be considered that could limit maximum Farm size:
- XML broker capacity
- IMA memory consumption on the Zone Data Collector
- CPU utilisation on the Zone Data Collector
XML BROKER CAPACITY
The XML broker is responsible for communications between StoreFront (or Web Interface) and the Zone Data Collector – authenticating users and returning the list of available resources to StoreFront. For each action undertaken by the XML broker a thread is started. The XML broker has a hard limit of 16 concurrent worker threads and therefore 16 concurrent XML operations. In a very busy Farm it would be possible for the XML broker to reach this limit which would then deny users access to resources for short periods of time.
Are you monitoring XML broker load in your production XenApp 6.5 Farms? If not, read the excellent XenApp 6.5 Operations Guide here: http://support.citrix.com/article/CTX133540 (page 8 describes XenApp-specific metrics to monitor).
The performance counter you are looking for is Number of busy XML threads and our Operations Guide recommends a Yellow warning the number of threads is 10 or more, and a Red warning if the number of threads reaches 16.
You can increase your XML thread capacity by load balancing multiple XML brokers, for example by using NetScaler and the XenApp 6.5 configuration wizard built into NetScaler. This is discussed here: http://support.citrix.com/proddocs/topic/netscaler-traffic-management-10-5-map/ns-lb-XenApp 6.5-wizard-tsk.html
IMA MEMORY CONSUMPTION
IMA memory consumption on a Zone Data Collector grows in a predictable way as objects are added to the Farm. Objects are:
- published resources
- worker groups
Using some numbers published by Citrix (here: http://support.citrix.com/article/CTX131102) we can predict the memory consumption of a Zone Data Collector in a Farm, and also then predict how memory consumption will increase as the Farm grows. Here are the memory footprints per Farm object:
A published application with 32-bit icon consumes ~65kB of memory
A session consumes 1.6KB
A server consumes 210KB
A worker group consumes 8KB
Using numbers from an anonymous customer deployment:
|Memory consumption per object||Number of objects||Memory per object|
|PUBLISHED APPS (65 KB)||400||400×65=26,000|
|SESSION (1.6 KB)||30000||30000×1.6=48000|
|SERVER (210 KB)||1300||1300×210=273000|
|WORKER GROUP (8 KB)||16||8×16=128|
|TOTAL MEMORY CONSUMPTION||347128 KB|
So total memory would be a minimum of 347128 KB or 338MB of memory. The IMA service is a 32-bit process so can address a maximum of 2GB of memory. However, before you assume this scenario allowed for about 1.6GB of headroom, remember that 338MB number assumes a steady state; i.e. no changes to the Farm, no new sessions, no enumeration through StoreFront, and no admin consoles open etc. Each of these actions will add (temporary) load to the Zone Data Collector.
I know from recent experience that AppCenter sessions add to this baseline memory consumption, and searches within AppCenter (for example, a Helpdesk Analyst searching for a user session) can temporarily balloon the memory consumption of the IMA server on the Zone Data Collector. For this reason I recommend that you monitor the memory consumption of the IMA service to establish a baseline value for your environment.
You may have read all of this wondering when I was going to mention the Farm Datastore. In the first two drafts of this post I did not have a section on the Datastore and it was only when working on the third draft that I noticed this omission. I do have some justification for this: I have been working within Citrix’s Services organisation for 5 years, and I have not encountered a problem that I could trace to Datastore performance. I have worked on issues caused by Datastore unavailability, but never Datastore performance which was caused by Farm size. And remember that with the optimisations made to the IMA service in XenApp 6.5, the load on the Datastore is much reduced when compared to earlier versions of XenApp 6.5.
- Citrix have only tested XenApp 6.5 up to 1000 servers – therefore for some customers this will the only acceptable farm size limit
- XenApp 6.5 will scale beyond this number but your experience will depend upon how many objects exist in the Farm
- How you operate XenApp 6.5 will have a large bearing on scalability – consider what other tools and processes are making calls to the IMA service on the ZONE DATA COLLECTOR – each will increase the CPU or memory requirement for this process
- You should be monitoring the number of busy XML threads on your XML brokers (and other metrics)
- Keep the number of zones at 5 or less
- You should be monitoring the memory consumption of the IMA Service on your Zone Data Collectors.
- You should be monitoring the CPU utilisation of the IMA Service on your Zone Data Collectors.
Related Code Changes
Are you operating XenApp Farms of this size? I would like to make you aware of two recently-released fixes for XenApp 6.5 that will improve the performance of the IMA service in these very large environments:
LC1741 – High CPU Utilization on the IMA Process
LC1203 – High Memory Utilization on the IMA Process
These two fixes were released together in a limited release hotfix XA650R05W2K8R2X64016, available here: http://support.citrix.com/article/CTX142348. These fixes are also included in Hotfix Rollup Pack 6 which is currently available as a BETA from here: http://support.citrix.com/article/CTX142585