In the upcoming introduction to XenServer networking document Citrix is about to release, we talk about the sometimes overlooked topic of cabling XenServer.

It’s easy to forget that XenServer isn’t just a software application: it is also virtual switch. When you add a XenServer host to your environment, you are actually adding a Layer 2 network switch, albeit a virtual one, to your network infrastructure.

As a result, making sure you physically integrate it correctly is critical. If not, your hosts may not be able to send or receive traffic!

A key point to note:

XenServer cannot detect if you make any errors while setting up the physical network.

Consequently, if a XenServer host expects to, for example, be able to contact a specific gateway using a certain NIC, XenServer cannot indicate the cabling is incorrect. If you receive errors, they might not indicate network configuration as the cause.

What does “correct” physical integration mean?
This means configuring the switches XenServer connects to according to Citrix guidelines in CTX123158 -- Considerations for XenServer Switch Ports and cabling XenServer correctly.

What’s the big deal about cabling?
If you are installing XenServer on previously used hosts, ones that are already integrated into your network infrastructure, you need to look at their cabling. If you want to pool those hosts and you have multiple physical networks or VLANs, you can’t just  leave the hosts hooked up to their existing switches – especially if their NICs plug into different networks.

For example, you want to virtualize a physical Microsoft Exchange server, and, through a happy coincidence, its existing hardware meets XenServer’s hardware requirements. You decide to reuse the box you are currently hosting Exchange on and virtualize Exchange on it.

Also, happily, you have other servers you want to virtualize have the same hardware, so you decide to reuse their hardware and pool these servers.

Well, even though the Exchange server box is connected to the physical networks it requires, you cannot just continue with your deployment without reviewing that box’s physical networking configuration.
Why not? Because as soon as you pool these hosts, the Exchange server will take on the network configuration of the pool master, regardless of what physical networks it is connected to.

How should I do it then?
A subtle, yet important, point about connecting a host’s NICs to switches is that all networking configurations and physical connections must match the pool master. Like many things in XenServer, consistency is the guiding principle. When networking, all hardware and settings must match across hosts in a pool.

Not only should all NICs on each host in the pool have the same NICs and networking settings, but when you pool hosts, the corresponding NICs on each host must connect to the same physical network. That is, NIC 1 on host 3 must connect to a switch with the same physical connectivity as NIC 1 on the pool master.

For example, look at this crazy “spaghetti monster” diagram:

In this diagram, each NIC on the hosts is connected to a different physical network, as represented by the switches. However, NIC1 on Host 2 connects to the same physical network as NIC 1 on the pool master.

There are two reasons physical networking in a pool must match:
1. XenServer automatically synchronizes the member servers (slaves) in the pool so they use the same network settings as the pool master.  (So, even if you intend for the members to use different physical connections, XenServer won’t let you.)

2. Without matching network settings, live migration becomes a nightmare. Think about how awful it would be if you had to calculate and manually configure a VM’s networking settings every time there was an HA event or WLB optimized your servers.

It is also important to note that if you change the physical configuration of anything to do with networking (e.g., NICs, cabling, switches, etc), you need to a) be very cautious as you do so and b) make sure you make the change on all hosts in pool. I’ll describe this more in my next post.