Application health is critical because downtime can lead to lost revenue and poor customer experiences. Without the appropriate app health checks and effective app monitoring in place, an ops team will likely face outages that leave apps inaccessible.

It’s important that your apps are resilient and that you can proactively monitor and test apps and their dependencies end to end so you can provide more robust software delivery to your end users. With several hundred (and growing) Citrix ADC counters available through the Citrix Observability Exporter, we’ve made app monitoring easier.

In this blog post, I’ll look at how to integrate Citrix ADC with the Citrix Observability Exporter and how to use Prometheus and Grafana to collect and display key app metrics. Together, the app information and the Citrix Observability Exporter data can help admins determine the overall health and serviceability of an application. I’ll also share recommended approaches for troubleshooting this logging infrastructure.

Architecture

You can see various modes of deploying Citrix Observability Exporter in the following diagram:

Citrix Observability Exporter is available in two forms:

• Bundled in Citrix Application Delivery Management (ADM) service for advanced analytics. This mode supports ADM analytics and Citrix Analytics service-related deployments.
• As a docker image, which can be deployed on a standalone Linux server or a Kubernetes cluster, a container-orchestration system.

Currently, Citrix ADC sends to the Citrix Observability Exporter application transaction (TX) information through a proprietary protocol called Logstream and telemetry data (time-series TSD) through the Avro/Json data format.

These data are then processed and, based on the configuration, are exported to various endpoints. In the example below, I’ll show you how Prometheus pulls data from Citrix Observability Exporter which can be visualized using Grafana.

Citrix ADC Health Monitoring Deployment with Prometheus, Grafana

Citrix ADC exports your application metrics to Citrix Observability Exporter, which listens for REST connections. Citrix Observability Exporter has a push-gateway server, which listens on the default port 5563 to serve Prometheus pull requests. It exports Citrix ADC metrics (time-series data) to Prometheus’s pull request. The duration depends on the scrape interval configured on the Prometheus server.

Configuration for Health Monitoring

Most of the solutions for monitoring available are generally very complex to setup and maintain. Here you can set up health monitoring of your applications deployed with Citrix ADC in five easy steps. For this example, I’m assuming that you’ve completed the application and its load balancing or content switching configurations on the Citrix ADC and are looking to extend it for observability.

Step 1: Set up Citrix Observability Exporter

Import the Citrix Observability Exporter as a standalone docker or on your Kubernetes environment from GitHub.

For your Kubernetes deployment, if you have your Prometheus server deployed in the same Kube cluster, you can configure annotations to enable Prometheus to automatically add Citrix Observability Exporter as a scrape target. Add the following lines in your Citrix Observability Exporter yaml file:

      annotations:         
        prometheus.io/scrape: "true"         
        prometheus.io/port: "5563"

A sample yaml file, which can be used to deploy Citrix Observability Exporter with Prometheus annotations, is available at https://github.com/citrix/citrix-observability-exporter/blob/master/deployment/coe-prometheus.yaml.

Step 2: Configure Citrix ADC to export metrics

Add the Citrix Observability Exporter IP/FQDN address as an HTTP service and bind it to the default “ns_analytics_time_series_profile” analytics profile; enable the Metrics export; and set the output mode to Prometheus.

enable ns feature AppFlow
enable ns mode ULFD
add server Citrix_Observability_Exporter_instance 192.168.1.102
add service Citrix_Observability_Exporter_metric_collector_svc_192.168.1.102 COE_instance HTTP 5563
set analytics profile ns_analytics_time_series_profile -collector COE_metric_collector_svc_192.168.1.102 -Metrics ENABLED -OutputMode Prometheus

Please note, to use a server URL (for example, http://COE.cluster.local:5563/), use “add server Citrix_Observability_Exporter_instance COE.cluster.local“; COE.cluster.local should resolve to the Citrix Observability Exporter IP address.

Step 3: Configure Prometheus to pull telemetry data

Running Prometheus on Docker is as simple as docker run -p 9090:9090 prom/prometheus. This starts Prometheus with a sample configuration and exposes it on port 9090.

To manually add the Citrix Observability Exporter as scrape target, edit the prometheus.yml to add the Citrix Observability Exporter IP/FQDN address and port 5563 as the scrape target.

scrape_configs:
  - job_name: COE
    static_configs:
      - targets: ['192.168.1.102:5563']

Step 4: Configure Grafana

You’ll need to add Prometheus as a data source on Grafana. Open the Grafana admin page and follow the steps given in the How to add Prometheus as a data source guide.

Step 5: Create Grafana Visualization

You can create a new Grafana dashboard and select the key metrics and the visualization-type appropriate for the data. In the following steps, we can see an example of adding ADC CPU metric to a new Grafana panel:

1. Specify the Panel Title “ADC CPU use”
2. In the Query tab, for query A, specify the metric “cpu_use”
3. Select the Visualization type on the settings tab

The following dashboard view shows Citrix ADC application health metrics.

Here we can visualize ADC health status like the CPU, memory, network data usage, and network issues like dropped packets. You can set alerts for application (LB/CS vserver) health and backend service status. You can use the traffic rate (HTTP Req/Sec) served and number of active client connections to make scale decisions.

You can use pre-built dashboard for Grafana available at https://github.com/citrix/citrix-observability-exporter/tree/master/dashboards.

Custom Dashboard with Node Exporter for Host Metrics

I have designed a custom dashboard that helps monitoring your ADC application health, Citrix Observability Exporter stats, host machine health, and endpoint consumption stats. You can download the Json file here. (The dashboard is developed on Grafana v7.0.3.)

Please note, you must complete the node exporter setup on the host machine to get host metrics using this dashboard. For more information to configure node exporter, go to https://devconnected.com/complete-node-exporter-mastery-with-prometheus/

Key Metrics

Citrix ADC Metrics

The following list shows some key Citrix ADC metrics that can be monitored. You can view the complete list of counters in the Prometheus/Grafana UI.

Troubleshooting Techniques

In this section I’ll cover troubleshooting techniques for Citrix ADC’s interoperability with the Citrix Observability Exporter.

To check if ADC is sending metric data logs to Citrix Observability Exporter, check the metrics data export logs on Citrix ADC for Prometheus Endpoint:

shell
/var/nslog/metrics_prom.log

If Citrix Observability Exporter fails to log data, you can contact Citrix Support with following files:

/cores/ (Here you will find core dump files, if any.)

/var/ulflog/ (Here you will find data logs and counter details.)

/var/log (Here you will find console logs, process logs, and more.)

Conclusion

With the information I’ve covered in this blog post, you can define a set of health rules, each with unique conditions, that map to your main performance indicators to effectively measure the health status of apps. Citrix Observability Exporter, integrated with Prometheus and Grafana, can help you to proactively monitor the performance of your application and provide alerts around performance before an actual outage happens.

Learn more about the Citrix Observability Exporter.