Cloud computing and virtual computing have been good at breaking down the traditional silos in IT shops.  But when it comes to cloud networking, a new kind of sub-optimal division of labor has been becoming prevalent.  This is the divide between networking in the hypervisor, and networking in the physical boxes.  Between virtual switches in hosts, and physical ones in the network.  Between provider networks and (multi-)tenant networks.

Because of this divide, the virtual and the physical weren’t talking.  The virtual switches treating the physical network more or less as “dumb connectivity”, and the physical switches mostly helpless in their attempts to sniff-n-snoop proprietary encapsulated packets.

There are only two problems with this setup.  It doesn’t perform and it doesn’t scale.

This is where the new VXLAN proposal comes in.  You can read the proposal here.


It almost doesn’t matter what scheme we used.  The key is that several industry leaders from the physical world, including Citrix, and several industry leaders from the virtual world, including Citrix, are coming together in support behind this.  The physical and virtual networks are going to talk, ladies and gentlemen.  And good things will happen.

Reading this proposal you’d identify it as a simple MAC in UDP or L2-in-L4 scheme.  And you’d be right.  But there are several implications of VXLAN.


Yes this an example of an overlay network.  Overlay networks have an “outer” provider network, and an “inner” tenant network overlaid on it.

The choice of the inner network is readily made – it is L2 – offering the Ethernet service model to the tenant.  It is the choice of the outer network that is crucial.  Pick L2 for the outer, and you’d be insisting on the entire physical provider network to present itself as a giant, flat L2 domain.  How does it scale?  Very very carefully.  Read: not impossible, but few have.  You quickly head down the path of “out with those spanning trees, in with, umm let me see…”.

Pick L3 for the outer network, and we’d be doing better.  Now the provider has to figure how to scale a large L3 network.  A far better understood problem with multiple solutions.  But there is even more goodness when the outer network carries L4 attributes.  The extra header fields provide a degree of freedom to the control plane that is useful in many scenarios.  Better balancing of aggregated links for example.

There is more to scaling networks than having a 24-bit segment identifier though.  It is the control planes and protocols at both the virtual plane, as well as the physical plane that will need to engage to optimize a VXLAN environment.  Turning tenant broadcast into provider multicasts, for example, with 16M segments in mind, requires a pretty robust IGMP and supporting stack in the outer network.  To take advantage of the all that extra flexibility in the L4 wrapper requires intelligence in the virtual switches and their controller.

Mixed Virtual-Physical Tenants

More often than not, I find cloud consumers aka tenants, are able to take quite a bit of their infrastructure into the virtual world, but there are some items that remain in the physical world; a specific firewall or load-balancer, for example.

By standardizing the VXLAN frame format, not only does it allow physical “connectivity” providers to participate in these networks, but physical instances of network services can now also join virtual tenant topologies.  One could expect a NetScaler SDX or NetScaler MPX physical device to participate in one or more virtual tenant networks using VXLAN as the means of sharing tenant context.

Bridging Across Data Centers

A standard format helps not only within the data center, but in cloud bridging scenarios.  NetScaler Cloud Bridge connects tenant networks in multiple different data centers, at layer 2.  With VXLAN, and a cloud bridge that also understands the VXLAN frame format, it can now bridge VXLAN segments and non-VXLAN networks.  The difference between intra-DC and inter-DC bridging being in the control plane and the optimization across a WAN link.

In summary, the standardization of VXLAN at the intersection of the physical edge and the virtual edge of the network is going to make more open, more flexible, and a lot more scalable cloud networking a reality.

Its all good.