What happens when a Kubernetes cluster fails or becomes unreachable? The apps in that cluster would fail, too, resulting in an interruption of services offered to the clients.

To ensure resiliency, savvy operators deploy many copies of the same app across multiple Kubernetes clusters in multiple data centers and clouds. With this kind of multi-cluster strategy, you can:

  • Roll out new versions of the app in a canary fashion, to select customers
  • Improve user experience by routing user traffic to the closest copy of the app with the least amount of latency (also known as proximity routing)
  • Upgrade a Kubernetes cluster without fear of affecting the availability of the apps running inside the cluster
  • Divert some traffic to a different cluster when a cluster is at capacity
  • Deploy apps in an active/passive configuration when they can’t be deployed in active/active fashion

To achieve these kinds of deployment, the front-end proxies should be configured with relevant configuration to divert traffic for applications running in each of the Kubernetes clusters. This is a tedious job and prone to error. In this blog post, we’ll look at a solution to mitigate these pain points and ease the configuration of proxies as per the required deployment strategy.

Why Citrix ADC?

Citrix ADC has a powerful traffic management solution (called GSLB) for load balancing an app that is deployed across multiple sites. GSLB balances the load across the sites by directing client requests to the closest or best performing site or to surviving sites if there is an outage.

When the app is deployed in multiple Kubernetes clusters, each cluster can be a site GSLB can load balance. GSLB provides sophisticated algorithms to distribute the load across different Kubernetes clusters for use cases such as canary deployments, disaster recovery, and high availability.

Citrix’s multi-cluster load balancing solution provides a Kubernetes-native interface (using kubectl) that operators can use to manage traffic to apps deployed across multiple clusters. Custom resource definitions (CRDs) are used to configure GSLB on the Citrix ADCs. The Citrix multi-cluster controller enables the Citrix ADC GSLB to make load balancing decisions on any service exposed outside the cluster, including Ingress, type LoadBalancer and NodePort.

This solution works when services inside the cluster want to send traffic to other services. The GSLB decision will send the traffic to the best cluster based on load and availability.

Figure 1: Flow diagram of Citrix GSLB enabled multi-cluster K8s solution

The diagram above shows:

  1. Application providers will apply yaml for ingress or service in each of the Kubernetes cluster (Cluster 1 and Cluster 2). Service cold is shown in Figure 1.
  2. The admin will apply the same global traffic policy (cold1) and appropriate global service entry (cold.default.east.cluster1 and cold.default.west.cluster2) yamls in each of the clusters.
  3. The GSLB controller running in each of the cluster will listen for these CRDs and push config to Citrix ADC GSLB Device D1. D1 will sync the config to D2.
  4. The client request for a particular application will be directed to D1 or D2.
  5. D1 will take the load balancing decision and direct the client to one of the clusters.

Citrix ADC supports failover (maintaining a backup in case the primary fails), canary (distribution of traffic in a controlled manner to a newly brought-up cluster or application version), and local-first (preferring services local to the caller in latency critical use-cases) deployment modes. The distribution methods available are round robin, staticproximity, and RTT. Ingress, load balancer kind of services, or any K8s object used to route traffic to a cluster would be monitored for its health with regular health checks.

While there are other ways to address the need to maintain multiple clusters such as federation, external DNS, and more, Federation doesn’t enable distribution of traffic across clusters. The external dns solution has the limitation of updating the dns record for a domain with a single cluster ingress endpoint, even when ingress is deployed across multiple Kubernetes clusters.

A Sample Deployment

Let’s say we have an application deployed across two clusters — the east and west clusters. Our app is exposed using a host0based ingress object — app1.example.com. The app primarily resides in the east cluster; but when the east cluster fails, the traffic has to be served by the west cluster.

The Citrix multi-cluster controller is deployed in each cluster. This controller listens for two CRDs: Global Traffic Policy and Global Service Entry. The same Global Traffic Policy is applied across both clusters, while Global Service Entry is cluster-specific. The multicluster controller configures the GSLB devices based on these policies.

The Global Traffic Policy will have the deployment configuration details. For example:

  • The Ingress host entry for the application (i.e. app1.example.com)
  • Which clusters will serve as the primary endpoint and which clusters will have the backup role
  • Monitor to probe the health of the ingress
Figure 2: GTP yaml definition applied on each kubernetes cluster

As per the yaml definition in Figure 2, ingress ingress1 in the default namespace applied on Kubernetes cluster ‘cluster1’ in the east region will be serving as the primary. And ingress ingress2 in the default namespace applied on Kubernetes cluster ‘cluster2’ in the west region will be the backup.

Global Service Entry will have the following information regarding the ingress endpoint:

  • Front-end ingress IP
  • Front-end ingress port
Figure 3: GSE yaml definition applied on Kubernetes cluster cluster1 in east region

Figure 3 is the global service entry applied on cluster cluster1 in the east region for ingress ingress1 in the default namespace; ipv4address and monitorPort are the front-end ingress ip and port in cluster1, respectively, in the east region.

Figure 4: GSE yaml definition applied on Kubernetes cluster cluster2 in west region

Figure 4 shows the global service entry applied on cluster cluster2 in the west region for ingress ingress2 in the default namespace; ipv4address and monitorPort are the front-end ingress ip and port in cluster2, respectively, in the west region.

The Citrix multi-cluster controller in each cluster will listen for these two CRDs and configure the master Citrix GSLB ADC device accordingly. The Citrix GSLB device will have the endpoint information of all the ingresses deployed across clusters. Other Citrix GSLB ADC devices will be synchronized with the master using the GSLB AutoSync feature.

For example, with the above yamls applied, during normal operation, clients will resolve app1.example.com using the GSLB device as the Authoritative DNS server to the east cluster (10.102.217.10). If the east cluster fails, then clients will receive 10.102.151.10 in response to the DNS query.

Conclusion

You can easily achieve load balancing of traffic for applications deployed across multiple Kubernetes clusters by adopting the proposed Citrix GSLB-enabled multi-cluster K8s solution. This solution will enable the customers to deploy their application in different cloud providers and direct the traffic to any cloud provider as per their requirement. All the pain points in configuring the proxies are abstracted away with the help of the two CRDs introduced.

For more information about this solution, check out https://github.com/citrix/citrix-k8s-ingress-controller/blob/master/docs/multicluster/multi-cluster.md.