The “layers of cake” presents great opportunities for Citrix Delivery Center storage optimization.  Rather than store separate images for each machine, you can have ONE image for maintaining the operating system, ONE location which holds the application images, and ONE location for the User Profile.  This is valuable for XenApp, but it is especially important in XenDesktop where the number of machine images multiplies to be a really big number.

This post describes the “problems” of getting applications into this layered world and discusses multiple solutions; including several that are not ideal.  It concludes with short term and long term recommendations.

Here’s a slide that I presented at the Citrix Synergy conference in Las Vegas last week.

Even has candles!  Provisioning Services provides the bottom layer.  Application Streaming provides the middle layer and Citrix Profile Manager provides the per-user settings as the top layer of this multi-layer cake.  Getting this to move from “vision” to “reality” though involves system design and piecing this into your larger data center configuration is a complex operation.  The benefit though is increased flexibility and improved management of your systems as you are managing only images and assigning these as workloads to machines.

Why does Application Streaming provide the middle layer

In this discussion, the “Streaming” aspects of Application STREAMING are really overrated.  The important fact is that the base operating system image focuses on being the operating system and the application layer focuses on providing applications.  Application Streaming is well suited to this need primarily because of the just in time application delivery – delivery based on which USERS will actually visit the machine.  To implement the separation in the layers of cake, the lowest layer cannot have knowledge of the application set and this means that the applications have to be RUNTIME provided to the execution machine.  The Citrix XenApp publishing does exactly this, both for Stream to Server (XenApp) and for Stream to Desktop (XenDesktop).  In both cases, the machine is “vanila”.  A user logs in, application icons are placed onto the machine via the Plugin for Hosted Apps (aka PNAgent) and eventually the user clicks on something to cause the application to run.  With “streamed” applications, the applications are presented to the user “just in time” for their use.

Where it gets complicated

WOW, this is awesome.  I can have my cake and eat it too!  The details though are subject suitable for significant debate of best solution.  In particular the delivery of applications to the middle (application) layer.

Provisioning Sevices will provide the base operating system image to the bottom layer and this layer will be absent most applications.  Sure, some foundation applications that are used by all users may be in this layer, but the advantage in the layers of cake is separating the applications from the operating system and this motivates us to want the apps delivered via Application Streaming regardless of benefits of isolation.

Provisioning Services implements ONE to MANY by having a single disk image that it distributes to MANY “machines”.  These machines may be real machines, or they may be virtual machines.  Either way, they get a MAC address and this defines them as a “machine”.  Since this is a disk image and used by many, PVS must implement “write-back” cache so that machines using the shared image can update that image at runtime.  Important changes to the single image will be performed in “single” mode of PVS, but runtime changes to the disk must be maintained so that PVS can runtime support the temporary changes that the host operating system and applications will make the the “disk”.   For each machine that PVS is running, it maintains a write-back cache.

The PVS write-back cache is “temporary”.  When the machine shuts off, it is discarded.  This is what we “want”, but it also “must be” because PVS is a block based implementation of a disk and it really has no idea what is stored in what block on the disk.  It doesn’t even know what the “files” are.  With numerous machines “writing” conflicting changes to the disk at the same time, the only way out is to write cache these and then discard the cache when the machine shuts off, or in the XenDesktop case, when the user logs off.

Why is this a problem

If the Application Streaming application content (RadeCache) is runtime populated, then users will experience a first time application launch penalty each time the machine is booted.  In the XenDesktop case, this will be experienced on each logon.  The RadeCache must be populated and while it holds only a portion of what the application “installed”, it is still enough data that the population of this space is a sore-point for first time use of applications.  In server case, or stream to a physical desktop, the first time launch is a one time cost.  The second time launch is much more important.  BUT, in the layers of cake the first time launch penalties become a every logon penalty.  The first experience on each logon is that the user behavio will be “slow” and then this “slow” performance will be repeated each time the user logs off and logs back in.  NOT GOOD.

It gets worse.  The PVS system write-back cache is about to get hit with LOTS of stuff.   As an example, consider MS Office 2007 on the notebook from whcih I’m writing this post.  The populated stuff to support MS office is about 400MB of disk space.  This will populate into the PVS write back cache and that’s alot of disk space if you multiply it by 1000 machines.  More to consider, the Application streaming System does a network read to get the data from the Application Hub, it then writes it to the “local” disk, which in this case, isn’t local.  The PVS write-back system then has to send that data across the network, back to the PVS server, who has to maintain the data on a per-machine basis; and potentially later, send that data BACK across the network to the execution machine to support execution.  I’m not sure of the numbers, but if you load it heavily enough, this will start to be a burden on almost any network.

How to solve

First a bit of background on the streaming system storage of application content.  The disk storage is the key here.  Everything that represents the application gets added to the streaming execution cache, which defaults to disk location, \program files\citrix\RadeCache.  There is a GUID beneath here to hold the execution content for each “profile” or execution “target” which represents the installed application that the streaming system is keeping track of.  Technically, a versioned GUID, GUID_version where version is used to allow “hot” update of applications.

You COULD populate the RadeCache into your base PVS disk, but this rather defeats the separation between Operating System and Application images.  This solution is “out”.

The better solution says that the RadeCache should not be stored on the same disk volume that is the bottom layer.  Instead, the RadeCache should be moved to another disk volume and this disk volume should be common and shared across all the using systems (XenApp servers or XenDesktop clients).  Now, the maintenance of the operating system image can be ONE; and done via Provisioning Services and the maintenance of the applications can be ONE; and done via the Application Streaming profiler + the shared RadeCache execution space.

Getting more complicated

Much of what follows are multiple approaches to solving this problem.  The “correct” answer will depend on your network configuration and the tools that you have available to assist with representing a shared network resource as a local system disk volume.


The easy solution.  In XenApp for example, even if using PVS as boot source, you likely still have a physical disk volume on the execution machine.  If yes, configure that space as write-back cache for PVS and the complications are improved.  The extra network hops will go away.   I’m not positive, but it is likely that this write back cache still resets on each boot, so there will still be first-time launch penalties.  With that, we should instead focus on machines that have no physical disk as the solution here will work for both configurations.

We need a central source that can be mounted into all execution machines and treated as streaming runtime execution storage.  We want the “RadeCache” to be “fully populated” so that the streaming system NEVER needs to do a runtime cache fill.  In this way, the “streaming” aspects of Application Streaming are handed off to other technologies that will make the RadeCache appear as present even when they are remoted.  A side benefit of this is that App Streaming file based cache fills then get replaced with a page based system.

Application Hub

In a way, we already have a central space that holds the application content, it is the Application Hub.   Generally, all the profiles are stored beneath some “top” location and each profile contains the execution content to support that application on what ever the execution system may be.  One complication is that the “execution stuff” is stored inside a .CAB file.  While this is handy for compression and handy for portability to offline execution, it is not handy for access at runtime, pretending that the cab contents are actually present on the referencing machine.  It WOULD SURE BE HELPFUL if the CAB file were replaced with a subdirectory that has the same name, but is an uncabbed representation of the execution content.   In this way, the streaming system could redirect the local machine RadeCache\GUID_v to the Application Hub, profilename\GUID_v directory.  Do this and the problem is SOLVED.  Only problem, it doesn’t work.

Redirecting the execution content to network source

The “LINKD” command in Vista command prompts provides the foundation.    You need to use the /D option to create a directory junction.  Cool part here is that you can redirect the directory space to another volume and that volume CAN BE ON A NETWORK!  My experiments say that this works on Windows Vista, but that it does not work on Windows XP.  Given most XenDesktop configurations are Windows XP, this is a complication.  Note that the foundation reparse points that make this magic possible having been present in NTFS since Windows 2000.  The ability to convince the OS that this is present appears to only work on Vista and above.  I wrote some programs to exercise this back in the early Tarpon days; none bore fruit, but then again, Windows 2000 was a required operating system at the time, so more options may be available.

But wait, I can mount a volume into an empty directory on Windows XP.  Yes, yes, you can, but you can only mount the ROOT of that volume.  On Vista, you can mount subdirectory to subdirectory -> Much more useful!

The foundation commands are: As an admin:

CD \Program files\Citrix\RadeCache

mkdir \\apphub\share\profiles\profilename\GUID_v  (and then uncab the cab contents into that dir).

MKLINK /D GUID_v \\apphub\share\profiles\profilename\GUID_v

DIR GUID_v  ( Local directory, then you see everything on the network server )

In theory, DONE!  The streaming system will redirect stuff into the RadeCache, the I/O manager and NTFS will convert those into reparse points where they will ultimatley go back down the stack, this time as a network file system operation and all will be happy.

But, runtime, the Application Streaming system gets unhappy.  I have tested this only with the non-released streaming client that I’m using on all my machines, including the machine I’m using to write this post.  There is some chance that this will actually work on the currently released stuff.  The stuff “you don’t have yet” does registry MOUNTS rather than loading of registry information.  This is far more efficient at runtime, but it is also dependent on the operating system supporting the mount and when asked to mount a registry hive from a non-local disk location, the system responds, very simply, “NO”.  DUDE!  Nothing is local!  You think C: is better than that redirected space, trust me, it’s all bad!  Please push on and give it a shot!  Nope.  Doesn’t happen.

Using a second disk volume

Provisioning Server focuses on providing the operating system image; read this as “one volume” per machine.  What if you had a second “local” disk volume?  Yes, that would work!  You could tell the Application Streaming system to move the RadeCache to the second disk volume and all would be happy!  The streaming system already comes with a utility to move the cache, it is ClientCache.exe and is included and installed with the streaming client.  Here’s a link that describes how to use it.  So far so good.  The streaming system though will insist that this “new” location be on a “local” disk.

The second disk volume would contain only a fully populated view of a conceptual RadeCache of a machine that has ALL applications fully streamed.  Requires the admin to create this second disk image and requires that this image be maintained when applications are updated, but all of these things are containable and might even make sense as we all get used to maintaining the layers of cake.

How do you MOUNT a second disk volume?

Nicely, this disk volume does NOT have to exist at machine boot.  It only needs to exist BEFORE the user has a chance to launch any “streamed” applications.  If we assume that they do that in their startup folder, then this means that we need the second disk in place as a function of logon.  It could be in a system logon script.

The mounted disk volume must be LOCAL.  The streaming system insists on it.  It isn’t that the streaming system is against storing execution content on remote systems, it just insists on it – which is interestingly very similar to the registry mount rocks I threw earlier in this post.

The streaming system file system filter driver observes all file system volume mounts.  If it’s “local” and if its “NTFS”, it gets involved in the I/O stack and can use this space as a place to store the execution content.

The App Streaming system spends alot of time lying to applications about what is present.  Our trick is to LIE to the LYING system to fool it into believing a disk volume is “local” when everything is really remote.

How to lie?

Some customers have elegant disk systems that allow them to mount extra “drives”.   If so, then the ideal such system would allow mount of space into N machines where there is ONE backing store.  In utopia, this space is read/write.  Only the Streaming Caching service will write to it and any writes done for any user on any given machine would then benefit all users on ALL machines.  ELEGANT!

I belive such systems exist, but I have not yet seen it successfully implemented.  One complication is that the streaming system insists that the disk volume be formatted NTFS.  If the file sytem isn’t NTFS, we get out of the way.  In this case, we don’t want to get out of the way and given numerous machines accessing a single shared source, the odds of this having NTFS as the file system are … remote.  It will though have most everything we need.  If you have one of these, shoot me a line and we’ll work with you to get the streaming filter willing to put the streaming cache into that space.


XenServer, VMWare, Hyper-V

The machine virtualization technologies also provide tools to mount disk volumes.  If we can mount a single disk volume, local disk, into multiple machines, ideal.  If this supports WRITES, better.  At logoff, would need to merge this common RadeCache back to a master store to become the base image for additional users.  All of this would work – but it is all very involved.

Ideal world

In an easier world, the Application Streaming system itself would permit redirecting the RadeCache execution space to a central network location, on the Application Hub.  That does not exist.  The fact that I’m talking about it though should tell you that I want to solve this problem.  I’ve been avoiding talking about it because I am not yet satisfied with the answer.  This post though attempts to discuss your NEAR term solutions.

What should I do?

Near term, populate the RadeCache into the base PVS image.  I know this isn’t ideal, but it is a workable solution and solves the performance aspects.  Longer term, investigate mounting .ISO and VHD and look into abilities to leverage storage arrays to provide “local” disks that are really remote.   Longer term, understand that this is a common need and solving it, is broadly important.

Joe Nord

Product Architect- Application Streaming and User Profile Manager

Citrix System, Fort Lauderdale, FL