Introduction

Over the last year, a number of people have asked me for ideas regarding how to create a pool of 2 (or so) XenServers without having to spend thousands of pounds on shared storage.

The simplistic answer is that you don’t need any shared storage in order to create a XenServer Pool, but without it you have export and re-import VMs if you want to “load balance” your XenServers. Of course, the export/import approach is slow, and this is usually the problem that people are trying to solve.

The profile of user that I am thinking about in this document may have access to a team XenServer pool somewhere (with properly funded shared storage), but they primarily use a XenServer under their desk. They now want to put another XenServer under their desk and share out the VMs.

Example scenario

Consider 2 test engineers who each have a Dell Precision 380 with 4GB memory. Although these machines are getting a bit old, they are capable of running XenServer, and can each run <4 VMs.

A new project comes along which requires greater use of Windows 2008 R2, and Windows 7, but there is only budget to buy 1 Dell T3400 with 8Gb of memory and faster processors. One tester is all smiles, and the other one ends up with 2 Dell 380s and wants to pool them so that he/she can (manually) load balance the VMs.

Bearing in mind that in 6 months time there may be budget to upgrade the second tester, it would be useful if there were some pre-defined shared strorage solutions that might be adopted.

At present I suspect that many people assume that there is not a solution and either a) waste time inventing a “new” storage solution, b) waste time exporting and importing VMs, or c) waste resources by asymmetrically loading their XenServers.

In order to flush out any diverse ideas on how to create shared storage, I recently distributed a survey asking people to share their experiences and to let me know what shared storage solutions they are using. This document collates the feedback received.

Possible solutions

The “correct way”

Use an Equalogic, NetApp or similar appliance.

In my view this is an “enterprise class” solution is outside the scope of this document.

However, you should use something like this if a) you plan to run a production work load, b) any disk, data or VM loss would be expensive, c) you plan to do any performance measurement.

Advantages

-         Optimum performance with resilience to disk failure.

-         Supported by XenSource, and by hardware manufacturer.

-         This is usually the best solution in the long run.

Disadvantage

-         Initial cost.

OpenFiler “loopback” VM

The idea here is that you install a pre-built OpenFiler appliance VM on some local/un-shared storage on one of your XenServers. As well as a system disk, the OpenFiler VM has an large additional disk that will be used to provide the shared storage. The OpenFiler appliance then shares its large disk using NFS, and the XenServer uses this share to create an NFS based Storage Repository that can be shared.

This approach is fully described in “XenServer_Demo_and_Evaluation_Setup_Guide_v5.5.1” (CTX118735).

Advantages

-         Other than disk space this is almost a no-cost solution.

-         It is well documented in CTX118735.

-         XenSource make extensive use of OpenFiler servers when testing XenServer.

Disadvantages –

-         This solution is probably the least resilient to failures.

-         Not fast.

-         XenSource do not believe that OpenFiler would handle any significant stress .

-         The OpenFiler VM must be running before any other VMs can start, and in some cases it may be necessary to “repair” the shared SR before other VMs can start.

OpenFiler on a re-purposed PC

This is very similar to the OpenFiler “Loopback” VM (above), but in this case OpenFiler is installed on a physical device such as an old PC or server.

Download Openfiler from http://www.openfiler.com/ and install it on an old PC or server. OpenFiler can then be used to host an NFS server that is suitable use with XenServer.

 The installation process is described at http://www.openfiler.com/learn/how-to/graphical-installation, is generally quite straightforward, and should not scare those who fear LINUX.

Configuration of OpenFiler for use with XenServers is probably best described in “XenServer_Demo_and_Evaluation_Setup_Guide_v5.5.1”  (CTX118735).

The cost of SATA disks often makes it worth installing a SATA disk controller card if new or additional disk space is required.

System 3 are currently testing a Dell 380 to host 300Gb of shared storage for a pool of 2 Dell 380 XenServers. No performance data has been taken but the solution has been seen to work.

Advantages

-         It’s easy to set up.

-         A repurposed server, or the inclusion of a RAID card could offer better speed and resilience when compared with an OpenFiler “loopback” VM.

-         XenSource make extensive use of OpenFiler servers for testing XenServer.

-         Less demanding on older hardware

Disadvantages

-         XenSource do not believe that OpenFiler would handle any significant stress.

-         Old hardware may not be reliable or supportable.

FreeNAS

In many ways FreeNAS is similar to OpenFiler in that it is a cut down version of UNIX that has been configured for maximum NAS (Network Attached Storage) performance, and minimum exposure to UNIX.

One respondent who has tried FreeNAS said “… its setup can be completed in 10-15 minutes …. OS footprint is just 64MB, and boot to OS is just a less than 20sec process. The maintenance is almost zero.

Advantages

-         It’s easy to set up.

-         Less demanding on older hardware.

Disadvantages

-         Old hardware may not be reliable or supportable.

LINUX  on a re-purposed PC

Instead of installing OpenFiler on a re-purposed PC, most people who responded to my request for information used Linux to serve NFS; Ubuntu was the most popular distribution.

Installing Ubuntu is described https://help.ubuntu.com/9.04/installation-guide/i386/index.htmland https://help.ubuntu.com/community/Installation

Installation of NFS on Ubuntu is described here https://help.ubuntu.com/community/SettingUpNFSHowTo#Minimalistic%20NFS%20Set%20Up

The cost of SATA disks often makes it worth installing a SATA disk controller card if new or additional disk space is required.

Linux distributions that have been seen to work.

-         Ubuntu with an EXT4 filesystem, serving NFS.

-         Susie with a Razor filesystem, serving NFS.

-         Fedora with an XFS filesystem, serving NFS.

-         In System 3 we use an old Fedora 6 NFS server to provide shared storage to some of our XenServers, and iSCSI for VMware Virtual Infrastructure.

Advantages

-         A repurposed server, or the inclusion of a RAID card could offer better speed and resilience when compared with an OpenFiler “loopback” VM.

-         Of those who responded to my request for information, this was the most popular solution.

-         Depending on your hardware, this maybe the fastest solution for providing shared storage for small XenServer Pools. One respondent has done some performance comparisons using an HP Prolient DL380 (2GB mem) with an HP Smart Array P600 SAS RAID Controller.

-         Ubuntu with an EXT4 filesystem – 225MBps.

-         Susie with a Razor filesystem – Very good.

-         Fedora with an XFS filesystem – Good.

-         Free BSD with a UFS filesystem – Average.

Disadvantages

-         Old hardware may not be reliable or supportable.

-         Some people aren’t comfortable with  Linux

NFS service on a Windows 2008 R2 server.

Windows 2008 R2 now includes an NFS server. It should therefore be possible to host an NFS service on an existing Windows 2008 R2 Server.

In addition to the full graphical version of Windows 2008 R2 Server, there is also the lighter weight Core product which may offer increased performance for a given hardware platform.

NFS services have to be installed separately and the following documents describe how to install and configure NFS on Windows 2008 R2.

 http://technet.microsoft.com/en-us/library/cc753302%28WS.10%29.aspx

and on Windows 2008 R2 core

http://blogs.technet.com/jorke/archive/2008/09/10/configuring-nfs-on-windows-server-2008-core.aspx

Advantages

-         Some people feel more comfortable with a Windows solution

-         You may not need any additional hardware as you should be able to add NFS services to an existing Windows 2008 R2 server.

-         Windows 2008 R2 Server may also be used to create shared storage for Hyper-V clusters.

Disadvantages

-         If using old hardware, it may not be reliable or supportable.

StarWind Free iSCSI Target on a Windows.

Another solution that uses a Windows PC or Server is StarWind iSCSI Target. The free version, which supports up to 2 TB, can be downloaded from http://www.starwindsoftware.com/free

“StarWind Free software is provided only for personal use, demoing, training, or test and development and may not be used for commercial or production purposes”.

Advantages

-         Some people feel more comfortable with a Windows solution

-         You may not need any additional hardware as you should be able to add StarWind to an existing Windows PC or Server.

-         StarWind may also be used to create shared storage for Hyper-V clusters.

-         The inclusion of an iSCSI based Storage Repository allows you to enable XenServer’s High Availability option. While you are unlikely to need this in a small XenServer pool, it may be useful if you need to include HA in your testing.

Disadvantages

-         If using old hardware, it may not be reliable or supportable.

-         Some people feel more comfortable with a non-Windows solution

Windows Storage Server 2008

http://www.microsoft.com/windowsserver2008/en/us/wss08.aspx

One respondent has just started using Windows Storage Server 2008 on a Dell T3500.

The primary purpose of this is to provide clustered storage for a Hyper-V / SCVMM development environment, but it may also be an option for XenServer.

Advantages

-         Some people feel more comfortable with a Windows solution

-         Developers and Testers may have a suitable MSDN licenses that would allow them use this solution in a development or test environment.

-         Windows Storage Server may also be used to create shared storage for Hyper-V clusters.

-         The inclusion of an iSCSI based Storage Repository allows you to enable XenServer’s High Availability option. While you are unlikely to need this in a small XenServer pool, it may be useful if you need to include HA in your testing.

Disadvantages

-         Some people feel more comfortable with a non-Windows solution

-         I have no feedback on performance to date. 

Buffalo TeraStation appliance

One respondent has tried a  Buffalo Terastation but did not continue as write performance was too weak.

Other Considerations / Recommendations

-         Make sure that the network between the XenServers and the storage server is as fast as possible. Ideally they should be connected to the same Gigabit switch.

-         Consider putting the storage traffic on a separate “private” network, if you have sufficient NICs in your servers.

-         In Chalfont System 3 we have a Dell PowerEdge 1600 SC server running Fedora 6. It would have been prohibitively expensive to upgrade it with 1.5 TB of SCSI disks so we left the SCSI boot disk in place and added 4 x 500Gb SATA disks attached to a new “Adaptec SERIAL ATA II RAID 2820SA ROHS with battery backup”. We chose this SATA card because it offered RAID, and because it supported “Write Back”. My understanding was that this would be faster than “Write Through”, although this indicates that this is debatable.

This storage server currently only supports 2 x VMware ESX hosts using iSCSI, and it is fast enough. We have not done any recent performance measurements on this system.

-         It would be nice to be able to make objective recommendations about which of the above solutions was “best”, but with the available information, I can’t. 

What I can say is that of the people who responded to my survey:-

o   Most were using Ubuntu on a re-purposed PC.

o   OpenFiler is probably easier to set up if you are a UNIX rooky.

o   There are Windows solutions, but at least one respondent rejected these because it was felt that they would not be as fast as an Ubuntu solution.

Hopefully someone will have measured this and will be willing to share their results.

Acknowledgements

With thanks to all those who responded to my initial survey.

Thanks also to those who may provide feedback or additional content.