Windows Side by Side is a complicated beast and a source of much confusion in Windows system administration.  Today’s post provides a foundation in WinSxS and goes into considerable detail about how this works with Citrix App Streaming.


DLL Hell is the root of all evil and the Microsoft C runtime is the root of DLL Hell.  In case anyone is counting, that’s “2” statements describing the underworld and I’m just getting started!  For a detailed review of DLL Hell, please read the earlier post and then head right back.

Okay – Microsoft compiler group implements lots of versions of the VC5 and VC6  runtimes, all of which have the same filename.  While these SHOULD be stored with individual application programs, this was not standard procedure in that day, a time of really scarce RAM.  Standard procedure calls for applications to all use the same single DLL runtime and to INSTALL and REPLACE *THE* C runtime with a newer and better version as a function of their own installation.

Unfortunately, this upgrade could and did break existing applications, which were dependent on the behavior of the earlier version of that version of the runtime.  Perhaps dependent on the broken behavior of the earlier runtime, but still dependent.  Install an application, get a fixed runtime, break an unrelated application that has been working for years.  DLL Hell was born.


I really feel for the Windows kernel team.  I imagine that it went something like this.  The compiler team invented this bastion of DLL Hell, then the core team got asked to “fix it”.  My reading of other posts on the Internet says that WinSxS first appeared on the scene with Windows Me.  I really liked that version of Win 9x.   Important point is that this attempt to fix DLL Hell has been around for a long time and has been present in pretty much all versions of Windows NT from Windows 2000 and above.

More recently, I understand that Microsoft are starting to give up on WinSxS and move back to straight DLL Loads.  IMHO, an excellent idea.

What does it do

Consider that TWO applications both install MSVCRTD.DLL as the C compiler runtime for Visual C 6.  In THEORY, each application installer inspects the file version data of THE runtime as it is found in the \Windows\System32 directory.  If they are installing a “newer” one, they replace the existing MSVCRTD.DLL with the newer one.  Recall that this often craters the first application.

“newer” is defined based on the file version data of the runtime DLL.  Here’s an example:

>filever msvcrtd.dll
msvcrtd.dll: Microsoft (R) C Runtime Library, version 6.00.8168.0

Important part is the “8168”.  In concept, this is the eight thousand one hundred and sixty eighth build of the version 6.00 C runtime.  In reality, they likely started counting at 8000, but it doesn’t matter.  The key item is that the number only gets “bigger”.  Bigger number equals “newer” version of the runtime.

With WinSxS, when installers “replace” the runtime, instead of replacing, what happens is that a subdirectory is created for the specific version of the runtime that the installer “installs”.  Now, both “old” and “new” versions of the single runtime file MSVCRTD.dll exist and with a little help from the loader, can be separately loaded.

At application execution, when the application requests THE runtime be loaded, WinSxS makes it possible to tell the system to load a SPECIFIC version of the version 6 runtime.

BTW – The problems of DLL Hell are broader than just the VC runtime.

How do you tell the system that you need this help?

Manifest files

Manifest files are files on disk, just simple TEXT files which happen to have XML data inside and exist on disk with .manifest extension.  By convention an executable program named foo.exe can have an accompanying foo.exe.manifest file which gives the loader instructions on which DLLs to load for this exe.   Side note is that manifest files can do much more.

Cool part of .manifest files being text file additions is that this could be added to existing programs to make them work in a conflicting world, all without application update.

Manifests can also be EMBEDDED into the EXE itself, as a resource – like an icon.

Modern versions of Visual Studio make this automatic – the produced EXE includes manifest data embedded in the EXE to tell the loader, I need *THIS* version of the runtime and don’t give me any impersonators!  With RAM plentiful in modern worlds, this statement of specific versions of C runtime has become standard procedure and modern versions of Visual Studio include this data as standard procedure in the production of applications.  I’ll add that this is often included even where no assistance is really needed.

Watching it work

Put your Windows Loader hat on.  You load a .EXE into memory and are getting ready to create a thread to run it.  When do, you observe manifest data.   Because you are the one IMPLEMENTING WinSxS, it is your job to go find the right version of the runtime (any DLL really) and bring that into memory to become part of the application.

You’re doing this as THE loader and when you get done finding and loading all the DLLs, you finally unleash the application to run state.

This all works swell.  Until you add Application Virtualization.

Application Virtualization and WinSxS

In Application Streaming, the executable is launched by … the application virtualization system, link.  In the case of Citrix App Streaming, the streaming client eventually makes a call to CreateProcess, with the start it “suspended” flag specified.   Here, the application is brought completely into memory, but it doesn’t start yet.  The start is delayed until the application virtualization system has a chance to install “hooks” to mask the applications vision to the machine and establish the isolation layers of glass.  Once these are in place, the application state is changed to runnable and … it’s off to the races at near native execution speeds.

If you’re the Windows loader, the application that we have asked you to start is something like

C:\Program files\Citrix\RadeCache\GUID_V\Device\C\Program Files\Bar\Foo.exe.

Notice that this is the really long path; the system loader isn’t confused by isolation and there isn’t any isolation yet.  Only after the executable is “started”, do the hooks get in place to mask everyone’s vision.  When the application asks, “where did I come from”, the isolation system converts the really long name to the short version and the application continues happy.  If the application runtime loads a DLL or accesses a file from the virtual space, the isolation system converts that to the long path and sends it on it’s way.

Observe that without isolation in place, all the DLL fixups needed to bring the EXE into memory have to be solvable, and solvable without a “cache fill”.   To do this, the isolation system inspects the PE Header in the .EXE and forces items into the execution cache that will soon be needed by the Windows loader when the system  is soon asked to execute the application.  This is … standard procedure and is done for every application launch under Application Streaming.

The WinSxS activity though is a trouble spot

Recent customer inquiry says that with cache full, application runs.  With cache empty, application refuses to run.  What happened?  I have to infer a bit because I have not torn the application apart, but what I expect happened is WinSxS manifest data.  Customer was able to tear this apart to identify ONE single DLL that was the culprit.  In filemon traces, see a reference to the DLL with “file not found” as the file open RC, then the application launch unravels.

Update: Hypothesis later confirmed.

At runtime, the application .exe.manifest, or resource embedded equivalent told the WinSxS loader to bring in a specific version of the DLL, from a place outside of pre-populated files.  Since this DLL was part of the isolated application and since the cache was empty, the DLL was not found and the loader had no choice but to give up.

Since the loader is not isolated, the loaders attempt to load the WinSxS dependent DLL failed – and it flunked application load.  Solution, place single DLL into RadeCache via pre-launch script and application runs, and runs reliably.

Why is this not more prevalent?

Because the primary place for WinSxS is solving the MS C invented problem of DLL Hell.  Here, Application Streaming “handles it” and supports the WinSxS dependent application execution by ensuring all the SxS dependent DLLs are in place before the application is run.

We do this by installing, locally installing, the common runtime DLLs as a function of the streaming client installation.  In a way, this is cheating, but the isolation system itself needs the DLLs anyway, so it works out to be no spilled milk because we have to install the runtimes anyway.  Also, installing the runtime is often a “nothing” because other components have already installed it, but we do have to go through the motions.

BTW – It is for this reason that App Streaming installs from a .EXE rather than from a .MSI.  We do install from a MSI, but first, the EXE installer places the Visual Studio 2005 and 2008 runtimes onto the machine where they will be seen by the streaming system itself, and the applications that we run.  This makes issues of WinSxS largely go away.

It gets easier

For a time, we had to include more than one version of the 2005 runtime.  Fortunately, Microsoft are now promoting the idea that having multiple versions of the same C runtime installed on the same system is a bad idea.  In this, they are saying that WinSxS is a bad idea – and – I agree.  Much better to fix it by building your own version of the C runtime if you need to, but if you can run with THE runtime use the most current.  With decades done now, the runtime versioning problems that were the root of DLL Hell are largely ironed out.

The official story I get is that if a C runtime has a security problem, and if that security problem is fixed, then eliminating WinSxS really helps move applications up to a more secure version of the compiler runtime.  Does the make the system more secure?  That’s debatable because you don’t really know if the specific application was calling the “fixed” API, but you can argue the opposite pretty easily.  Moving to a more secure runtime SHOULD make things more secure and pretty likely won’t make them worse.  A “cost” of doing this is a risk that DLL Hell will resurrect.

In today’s “modern time”, the horrible history of the VC5 and VC6 runtimes are largely behind us.  The tradeoff says that the runtime is pretty stable and now we have problems of the solution (WinSxS) causing headaches where left alone, none would exist.

WinSxS is changing so that it will ignore the application requests for a specific C runtime and instead load a more current version that is compatible with that old version.  Winner.  Things evolve.

I extend compliments to the present Microsoft teams that inherited this complicated puzzle and have done a pretty admirable job in retiring the old problems without inventing too many new ones.  All while arguably making the system more secure by getting rid of old compiler runtimes which have some known issues.

I also expect to receive notes to the effect of “close, but it really works this way”.   I look forward to reading…

Joe Nord