Ain’t nothing like playing the Lego game of Application Networking architecture — anonymous application networking architect
While L3-networking could be seen as optimising connectivity between IP addresses, application networking focuses on building a network that creates the best application performance. One important building block in a global scenario is to provide multiple points of presence for network services and then to apply a service discovery scheme that enables users to use the “best” one. This post tries to give an overview of techniques applicable in this space.
Most enterprise customers run a corporate WAN—often on a global scale—by connecting their business locations’ LANs to their two or often more, data centers. For the purpose of this post, let me call this “Intranet.” In this network, routing is controlled by the customer typically by using some sort of dynamic routing protocol.
Assuming that certain network services are provided in different locations, a desirable system property is, very often, to use the application PoP (point of presence) that is closest to the actual network location of the client system. This is to minimize delay and to avoid traffic zig-zag between locations (e.g., a customer might have a documentation database that is replicated to each geo and wants Asian-connected employees to use the Asian version of this service rather than the North American one or a customer has two data centers co-located in two office campuses in two cities of the same country).
The picture below is a visual representation of this scenario. Assume that local gives better performance; your system should map the user that is connected to DC1 with the VIP in DC1.
The typical semi-automatic, consulting answer (after “it depends”) is to use GSLB for this. This post is about looking at and discussing alternative options.
Option A: User-manual Selection
In this option, service locality is achieved by presenting the user all VIPs (maybe in a nice web page providing some visual guidance such as a globe) and have him select the one she believes to be appropriate.
Option B: Static Pointers
In a corporate setting, managed computers typically “know” where they are which can be exploited to statically direct users to services local to the end point. For example, an Active Directory Group Policy could be used to use a local URL or a local DNS could resolve a global URL to a local VIP while DHCP could make sure a local DNS is used.
Option C: Client Software Dynamics
In this modern day and age, even web applications can do almost anything on the client. Moreover, many endpoints actually do know their geographic location. So, with this option you could e.g. compute the geographic distance to your PoPs, measure ping RTT or download a file from each to determine bandwidth. Obviously, you need a certain control of the app about it. And, although I did not PoC this yet, I am pretty sure you can do some simple yet effective magic with NetScaler’s responder actions to incorporate LB parameters in this game. The upside is that you can basically do whatever you want and you are “aware of the PoP or failover state you are in.”
Option D: Global Site Load-Balancing Static Proximity / GSLB Content-Switching
In this variant, a static rule set or proximity data base is used to map the client IP address to a PoP. In its pure — static proximity — form, the client IP is mapped to a location id first and then the location id is mapped to a PoP. Also, GSLB Content-Switching can be used, utilizing (boolean) policy expressions to map client IPs to GSLB vservers.
But wait, the client IP? Not really. With GSLB the client IP address is the ADNS-facing IP address of the DNS resolver chain. In other words, specifically in an Internet scenario, this could be an IP address that is very distinct from the real client who could be behind a chain of forward proxies and another chain of DNS forwarders. Recent DNS extensions can help with this by adding the original client IP subnet into the DNS request but the general observation here that — while absolutely applicable to solve the general problem, it requires a lot of coordination between proxies, DNS and network layout. Also, the GSLB configuration logic is (partially) mirroring the network layout and — in that sense — static redundancy, i.e., if the network changes and the config is not adapted, the mapping results become sub-optimal.
The same is true if — for some reason — the DNS structure changes.
Option E: Global Site Load-Balancing Dynamic Proximity
GSLB – in general—uses dynamic DNS records to direct traffic destined for a shared FQDN to multiple IPs based on VIP and network state and policy. It can be used to support a multiple of use cases such as—most prominently—disaster recovery. Basically, it works by NetScaler either intercepting DNS requests or by serving as an authoritative name server.
One possible way of mapping a request for a name that is under GSLB’s control to a VIP is by trying to stay close to the DNS resolver. This can be done by all NetScalers pinging a certain responder and then sending the VIP that has the lowest RTT between the last hop of the client’s DNS resolver chain and the NetScaler serving as an ADNS.
While – in theory — brilliant and quite easy to configure, this setup needs a significant delay differences to work reliably and is thus more a really global scenario, i.e., it is unlikely to get good results if you have two PoPs in darkfiber distance. In other words, it is better suited for a London/HongKong/Chicago type scenario. In addition it has the same issue that it measures distance to the DNS resolver which might be somewhere else entirely.
Option F: Static Anycast
Impressed if you made it this far. Actually, this option is the inspiration for this blog post. Since my university days, I have always been fascinated with Anycast and I have always been somewhat surprised it is so rarely applied in Enterprise networking. In my career, I have only see it done once and – in this case – for DNS. A couple of weeks back, I stumbled over a blog post from LinkedIn engineering about how anycast even works in the Internet and even for TCP, i.e., HTTP based services and not only for DNS.
First, what is Anycast? Anycast is simply the offering of the same IP address in multiple network locations which results in the Internet Routing table holding multiple routes to this IP. If the routing metrics — or network distance — to these multiple instances is different, the better one is always taken. If the hop metric is the same, routers do flow-based ECMP (equal cost multi-path). Traditionally, this concept is often believed to be impure because the router “thinks” it talks to the same end point over different paths. Most famous example is Google’s 220.127.116.11 and 18.104.22.168 Anycast DNS IP.
In Intranet networking (as opposed to Internet networking) there is a high level of control over the routing dynamics and routing table sizes. Also, I would guess routes are typically a lot more stable, hop distances smaller and more distinct in case of anycast. Many scenarios are really quite simple — not much policy, just shortest path. In these places, the local DC typically means you are connected on the same core and thus just one (L3) hop away from the local instance and then 2 or 3 hops away from the other instances.
Admittedly, it might not be a good fit for every network topology but the properties of Anycast to solve this problem are definitely intriguing. First, the configuration is really simple. All you need is a way to distribute routes which you very probably have. Then you have to see that the anycast IP instances ideally show diverse metrics (if they don’t, you might have a semantic issue with what you want “the most local” to actually mean or worse – with multiple equal cost routes, you might get into an ECMP scenario which might very well break your application.).
Option G: Anycast with Route Health Injection or Router Route Monitoring
One of the bigger advantages of a load balancer is that it “knows” if a service is up or not. This service (or VIP) state can be time-series’ed or upstreamed by SNMP traps. In case of a VIP being unavailable, it can be upstreamed by taking down the port (TIMEOUT or RST) or even turning off ARP. In this scenario, this would have the downside that different AnyCast instance could not back each other up and clients associated (by proximity) with a service instance would loose service instead of using an instance further away. To alleviate this, to solutions popped into my mind.
1. Use NetScaler dynamic routing with Route Health Injection (RHI) where the NetScaler stops announcing the Anycast VIP to upstream routing when the VIP is going out of service. By means of dynamic routing, the upstream routing agent then removes the route and traffic gets redistributed to the remaining instances.
2. Some customers networking department do not like non-routers to inject routes. In that case, customers tell me that modern routers can have static routers and their own monitoring to check availability which would give them the same functionality with a slightly different angle on how to operate it.
An interesting side effect of this could be to connect the NS to different routers and use this as a routing-spaced multi-chassis LAG. (Not saying I recommend this in general but it is an intriguing Gedankenspiel.)
A note about persistence
Persistence can be tricky to achieve. In general, in scenarios such as this, VIP failure should be a rather catastrophic event because the single service PoP is already designed on a resilience level as if it would be the only instance.
Setting aside persistence issues caused by services going down and coming up again (which can be helped by “disable when down”), the different options here have different persistence properties. Option A is typically made to be PoP persistent till the client has roamed and picks up a new IP address (with a different URL, different Cookies, etc.). For option B, persistence is similarly easy to achieve because the PoP is “wired into the endpoint.” In option C, all depends on the client software that can be explicitly aware of PoPs and handle failover, flapping and roaming gracefully. In option D, persistence can be controlled by GSLB logic plus “disabled when down.”
Due to the dynamic nature of option E, persistence can fail in certain cases if the client software would keep resolving the name. Truth be told, this is typically not the case in practice, which makes persistency not much of an issue for GSLB. What makes options F and G special is that in the worst-case scenario, with either a failing piece of the service or a failing piece of the network, the services might flap because the end point does not “know” to which PoP it actually talks to. Largely simplified, if route convergence stability is an issue in your network, Anycast is probably not for you or you might only want to use it for service discovery. Also, if you look at the picture at the top of this post, equal cost scenarios can completely screw with persistence and have to be avoided, specifically when the local VIP is down and the remote VIPs are equally far away.
The options presented here all solve the service proximity problem and have different properties in how they achieve customer success criteria. In all this application networking Lego dreaming, I am a big believer in simplicity, so you should not let yourself get carried away in the flow of technical awe but keep asking yourself that you might not need Active/Active multi-PoP VIPs at all and — for disaster recovery — choose simple static reconfiguration thereby immediately freeing yourself of all the challenges presented here and throwing the ball in the AD’s team court. For two-site, dark-fiber DCs I like Anycast as a nice additional layer of redundancy.
If you can justify the effort however, it is certainly a lot of fun to plan and implement. If you are interested in laying out your corporate application delivery templates or enrich it with client proximity, Citrix Consulting Services — which I coincidentally work for — is very happy to help you in this quest. We create the solution with minimum complexity for you. While doing so, we follow a clear and structured approach to unearth your requirements and we are brave enough to make clear recommendations to maximize your benefit from our experience.