In a microservices architectural framework, a monolithic application is divided into multiple, loosely coupled, highly maintainable services that you can deploy independently. The main advantage of microservices architecture? Continuous development and faster deployment because the codebase and scope are smaller.
But this poses a challenge for app developers. Along with development, testing must be quicker. This sometimes leads to bugs that aren’t discovered in development, testing, or staging environments.
Canary deployment enables users to deploy the latest version of an app into in the production environment by sending a small fraction of traffic to this updated version. A rollout mechanism is usually triggered if this traffic exhibits undesirable behavior. It’s possible, even in canary deployment, that a small subset of users might be affected. No matter how much we try to test/simulate traffic beforehand, your production environment will have some users who provide inputs that lead to an undesirable behavior.
One way to reduce this risk before end users experience any impact is traffic mirroring. Also known as shadowing, traffic mirroring is a deployment strategy in which the same live traffic destined to the production server is cloned and forwarded to the latest version of the service.
The response from the latest version is analyzed for errors but is not sent back to the end user, so any issues observed in processing mirror traffic won’t have an impact on the production environment. Because production traffic isn’t affected, the mirrored traffic can be analyzed for performance, errors, and exceptions.
Istio refers to traffic mirroring as fire and forget. You can achieve HTTP traffic mirroring in Istio with Citrix ADC deployed as an ingress gateway, as well as a sidecar; citrix-istio-adaptor v1.2.0-beta provides support for HTTP traffic mirroring.
In this post, I’ll provide an overview of traffic mirroring when Citrix ADC is deployed as a sidecar to the application pod.
Configuring Traffic Mirroring
Get started by automatically injecting Citrix ADC CPX as a sidecar to applications. Label the namespace that you’ll use for application deployment with cpx-injection=enabled.
kubectl label namespace <namespace> cpx-injection=enabled
In this blog, namespace mirror will be used.
$ kubectl label namespace mirror cpx-injection=enabled
For this post, I’ve used httpbin as a sample application. This httpbin application is deployed with two versions. First, whole traffic is routed only to the initial version (v1) of the httpbin service. Then a rule will be applied to mirror traffic to the latest version (v2) as shown here:
In this example, I’m deploying v1 and v2 together. However, in real life, v1 will always be up and running, and v2 will be deployed later. Deploy version v1 and v2 of “httpbin” application using this in the “mirror” namespace. Please note, a destination rule is created to specify subsets of httpbin. Because CPX uses 80 port for internal use, httpbin app is being exposed via 9080. Learn more here.
CPX will be injected automatically as a sidecar along with citrix-istio-adaptor in the application pod
kubectl get pods -n mirror
NAME READY STATUS RESTARTS AGE
httpbin-v1-6c56545d98-pj642 3/3 Running 0 1m
httpbin-v2-7d676rtd78-sa781 3/3 Running 0 1mYou can send requests to httpbin from outside the cluster (N-S) or within the cluster (E-W). In this example, I’m sending it from another microservice, sleep:
$ export SLEEP_POD=$(kubectl get pod -l app=sleep -n mirror -o jsonpath={.items..metadata.name
$ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -n mirror -o jsonpath={.items..metadata.name})
$ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -n mirror -o jsonpath={.items..metadata.name})$ kubectl exec -it $SLEEP_POD -c sleep -n mirror -- sh -c 'curl http://httpbin:8000/headers' | python -m json.tool
{
"headers": {
"Accept": "*/*",
"Host": "httpbin:8000",
"User-Agent": "curl/7.35.0"
}
}We can verify that traffic is hitting to v1 not v2 from log entries in the v1 and v2 pods.
Logs from V1_POD:
$ kubectl logs -f $V1_POD -c httpbin -n mirror
192.0.0.1 - - [23/Mar/2020:12:53:20 +0000] "GET /headers HTTP/1.1" 200 106 "-" "curl/7.35.0"Logs from v2_POD:
$ kubectl logs -f $V2_POD -n mirror -c httpbin
<none>Mirror Traffic to the Newer Version
Apply the below Route Rule, which has same VirtualService CRD name used earlier, to mirror traffic to httpbin:v2.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin
http:
- route:
- destination:
host: httpbin
subset: v1
weight: 100
mirror:
host: httpbin
subset: v2
mirror_percent: 100Apply the new route-rule using:
$ kubectl apply -f mirror-vs.yaml -n mirror
When the traffic is mirrored, requests are sent to the v2 service, with their host or authority headers appended with keyword -shadow. Responses to these mirrored requests are discarded. Currently citirx-istio-adaptor does not support the mirror_percent field, which is used to mirror only a certain percentage of traffic.
Send traffic to httpbin from another application.
$ kubectl exec -it $SLEEP_POD -c sleep -n mirror -- sh -c 'curl http://httpbin:8000/headers' | python -m json.tool
We can see access logs for both v1 and v2.
$ kubectl logs -f $V1_POD -c httpbin -n mirror
192.0.0.1 - - [23/Mar/2020:12:53:20 +0000] "GET /headers HTTP/1.1" 200 106 "-" "curl/7.35.0"
192.0.0.1 - - [23/Mar/2020:12:57:15 +0000] "GET /headers HTTP/1.1" 200 106 "-" "curl/7.35.0"$kubectl logs -f $V2_POD -c httpbin -n mirror
192.0.0.1 - - [23/Mar/2020:12:57:15 +0000] "GET /headers HTTP/1.1" 200 113 "-" "curl/7.35.0"To confirm that v2 is getting mirrored traffic, exec to “cpx-proxy” of the V2_POD and take tcpdump. You’ll see that the request header’s host field is appended with -shadow.
$ kubectl exec -it $V2_POD -n mirror -c cpx-proxy bash
# tcpdump -i any port 9080 -A
E....J@...]...M...M.*.#x........P...H...GET /headers HTTP/1.1
User-Agent: curl/7.35.0
Host: httpbin-shadow:8000
Accept: */*
In this example, all the traffic destined to version v1 of the service is getting cloned to version v2 in fire and forget mode. Even though this is a simple example, you can see how traffic mirroring can support risk-free deployment and testing of the latest version of a microservice in production, with live traffic. You can use traffic mirroring in conjunction with a canary deployment to effectively roll out the latest versions of applications knowing users will experience no negative impact. And you can phase out the older version once the latest version is completely stable. Learn more about traffic mirroring.
