Thanks to everyone who attended the webinar “In-depth Troubleshooting on NetScaler using Command Line Tools” on 3/27. We had a full attendance and as a result there were a lot of questions posed and not enough time to cover them all. This blog will try to answer as many of those questions as possible.
Before we get to the Q&A let’s get a few links listed.
- Trial version of NetScaler VPX
- Slides from the session
- Link to the recording
- I was asked a couple of times what CLI tool I was using, it’s Cygwin and I highly recommend it for Windows users.
- For anyone looking for a trial version of NetScaler please click here.
- SSH private key creation and usage with NetScaler http://support.citrix.com/article/CTX109009 and configure the NetScaler to use them http://support.citrix.com/article/CTX109011
- NetScaler 10 command reference http://support.citrix.com/article/CTX132384
Also, all the options for nsconmsg are visible when using the -h switch.
The Q&A is a varied lot with some great questions and not all command line specific! We don’t mind at the NetScaler Central, happy to address as many queries as possible. I did take the liberty of paraphrasing some of the questions for readability.
There were some very specific questions in regard to NetScaler Gateway and Web Interface or StoreFront. Troubleshooting this can be difficult and the approach changes with each different problem. For latency and disconnect issues packet tracing at problem clients and NetScaler likely needed, STA issues = STA logging and comparison with NetScaler trace. I would look up specific errors on http://support.citrix.com and/or engage the Support team.
Education and shameless plugs
In terms of Education, please check here as Citrix Education provide multiple NetScaler courses. In regard to a specific question, CNS-301 covers Application Firewall. Also worth checking out Citrix Synergy in LA this May ’14. I personally will be working with my colleague Lucas Araujo on NetScaler Security sessions at both the TechEdge event on 5/5 and delivering some instructor-led and self-paced learning labs at the event from 5/6-8.
Questions and Answers
Q: How you can use a show lb server or service that has whitespace in the name?
> show lb vserver "LB RGB"
Q: the ns.log file wraps every hour, is there an easy way of reviewing all at the same time?
A: This depends on the file extension. If the ns.logs are gzipped, then go for zcat:
# zcat ns.log.*.gz
If the files are not gzipped, just ns.log.x then cat will do fine:
# cat ns.log.*
Q: Problems syncing a HA-Pair?
A: Check events in nsconmsg for failovers or nodes going down
# nsconmsg –K newnslog –d event | egrep –I 'node|init|failover|ha' # nsconmsg –K newnslog –d stats –g ha_ | more
Also check http://support.citrix.com/article/CTX121734
Q: Can you reset the HTTP hit counter on NetScaler 9.2 for a single VIP? Or do you need to restart the appliance?
A: You can do this on NetScaler 10.1:
> stat lb vserver [<name>] [-clearstats ( basic | full )]
Q: Why is there no nslookup utility?
A: Please use the dig utility in the shell # instead
Q: What commands are usable on VPX?
A: All commands apart from hardware specific ones are usable on VPX
Q: How do we delete old dumps (/var/core)
A: Drop to the BSD shell
# cd /var/core # ls # rm /var/core/1/*
Core dumps are stored in numbered directories so you many have to repeat this step
Q: Is the leading hyphen required for some of the options? I noticed I can run without (i.e., show interface summary)
A: That varies by command
Q: Could you pls show us how to unzip the old nslog file and read them?
# cd /var/log # zcat ns.log.gz # tar zxvf ns.log.gz
Q: Can I do at least basic health check of the NetScaler from Director?
A: No, integration with Director is for the HDX Insight part of the NetScaler Insight product.
Q: Is Citrix working to develop or release PowerShell commandlets for NetScaler?
A: You can leverage the NetScaler NITRO API to use PowerShell. Please see this informative blog post /blogs/2011/12/07/nitro-unleash-the-power-using-powershell/
Q: Maybe mention the ns.log shows CLI commands when the GUI is used. Useful to see what the CLI command was, and auditing.
A: Great suggestion. When issuing commands via the UI, you can monitor from the shell to see these live:
# tail -f /var/log/ns.log
Q: What is the difference between ns,conf and rc.conf?
A: ns.conf stores the NSCLI configuration commands and is read in on bootup with each command issued in turn sequentially. The rc.conf stores BSD commands and issues them on bootup also, a good example for rc.conf or rc.netscaler file is to store NetScaler Gateway portal page customizations.
Q: Is there a way or command to show latency of the connection?
A: Best way to track latency is to take a packet trace in nstrace format. This will give you the view of the NetScaler between client and backend. Using Wireshark you should be able to get a better view if the delays are client, NetScaler or server.
Q: Is there a NetScaler emulator available?
A: No, I’m afraid not. Please download a trial version of NetScaler VPX
Q: What is difference between NS.LOG and NEWNSLOG?
A: ns.log is a standard flat file storing events that occurred on the NetScaler. The file can be read using standard UNIX tools like grep, less, more, cat etc. Newnslog is a binary file and needs a special binary command nsconmsg to interrogate it. Logs and counters are written to newnslogs every 7 seconds.
Q: please share the URL to create private key and logon using a shell script
A: Links for SSH keys at the top of this page. The shell script is simple:
#! /bin/sh ssh firstname.lastname@example.org
Make sure the file has execute rights:
$ chmod +x file.sh
Q: I am having problem with number of connection – VPX 3000, with 3 varnish servers in URL hash LB, it’s capped at around 1600 concurrent connections, where to look?
A: 1600 concurrent connections should be possible, check newnslog for NIC related errors
# nsconmsg -K newnslog -d statswt0 | grep nic_err
Check for errors related to RL (Rate Limiting) in case you are hitting the 3000mb limit
Q: Can you touch on how to get newnslog off of the NetScaler and into a centralized logging setup like Splunk?
A: Splunk has a NetScaler app already http://apps.splunk.com/app/370/
Q: Is there a quick command line to back up the NetScaler so it can possibly be restored later?
> create system backup
Q: Where is the aaad.log file again?
# cat /tmp/aaad.debug
Q: Is there a way to digitally sign an archived log to prevent tampering?
A: Great idea but not that I’m aware of.
Q: Do we need a specific version to run SSH with a command remotely (as in the tips showed)? I tried on 120 and that didn’t work.
A: Not that I’m aware of. I tested on build 124.13 and it worked fine.
Q: How to access the BSD shell?
A: BSD is accessed by typing # and hitting enter once in the NSCLI >
Q: Question: When upgrading from 9.3.x to 10.1 different CTXs states you can just upgrade the whole way, other states you should upgrade to 10.0.x first and then to 10.1, what would be the correct way of upgrading? Will do it from CLI of course.
A: Tricky one to answer. The strict answer is yes as there’s no barrier to jumping from 9.3 directly to the latest 10.1 release. However some authors are more paranoid than others and prefer a “staged” upgrade approach.
Q: If you are running VPX’es on the SDX platform what would the stat ssl command show?
A: It will output as normal dependent on how SSL cores are dedicated to that VPX instance.
Q: Can you pass traditional tcpdump flags to the nstcpdump.sh command?
A: Mostly, this is covered extensively in the slides that were shared.
Q: When you say NetScaler kernel and BSD kernel, does that mean that the NetScaler does not run in BSD userspace? Is it similar to how you can enable Linux ABI in kernel on FreeBSD?
A: The NetScaler packet engine (NSPPE) runs in BSD userspace.
Q: and scripts to push load balance install??
Q: I’m a scripting type guy…. so are there any scripts which and be adjusted (shell scripts) which can be used for NetScaler?
A: If you are interested in scripting on the NetScaler there is always BASH and Perl installed on the appliance. If you want to interface from outside, then we have extensive APIs you can hook in to with the NITRO API http://support.citrix.com/proddocs/topic/netscaler-main-api-10-1-map/ns-nitro-wrapper-con.html and SDK https://www.citrix.com/downloads/netscaler-adc/sdks/netscaler-sdk-release-101.html
Q: can you more elaborate on vserver?
Q: Turning on Authentication enhanced feedback –> will that create noticeable performance overhead on NetScaler?
A: Enhanced feedback should not affect the performance of your NetScaler.
Q: what is packet engine?
A: A packet engine is the process running on a CPU core of the NetScaler with one purpose and one purpose only: to process network traffic!
Q: Is there a way to change the interface that the HA Heartbeat uses?
A: Yes, in the demo I showed changing the interface to turn on or off the HAmonitor option which listens for HA heartbeats on that interface:
> set interface 1/2 –hamonitor OFF
Q: Is it best practice to have the default route table as the same gateway as the SNIP or MIP?
A: Not totally sure I follow this one. The NetScaler will have one default route in it’s routing table and then will use the MIP or SNIP for a subnet as the IP it will use to reach a node in that particular subnet.
Q: Which log file will contains the failed command from ns.conf from boot time?
A: Try the ns.log
Q: Where does the persistence.log file reside?
A: That was an example file name, you can use anything you wish here. In the demo that will be written to the working directory at the command prompt. For me it was /home/andrews in Cygwin.
Q: I think I have a VPX with the same gateway IP I use for my VIP 🙁
A: Slide 64 of the presentation has details on checking the ns.log for IP conflicts.
Q: What is the minimum recommended NIC for VPX?
A: Difficult to answer, any 1Gb NIC should be fine. It’s really NIC compatibility with your hardware and hypervisor whether it’s XenServer, VMware or Hyper-V.
Q: can we use PuTTY?
A: Yes of course, any SSH tool will work fine to connect to the NetScaler.
Q: From the perspective of management is there any difference between VPX and MPX (besides SSL cards)?
A: In terms of management no difference at all!
Q: Does VPX also works on Hyper-V 2012?
A: Yes NetScaler VPX 10.1 runs on Hyper-V 2012. R2 support is on the roadmap.
Q: Is there a command to monitor the results of rewrites? Not just the hits, but the before and after of a rewrite action?
A: A packet trace is probably best here to see the result of the rewrite action live. You could also use the Rewrite action evaluator in the GUI.
Q: will there be an email address so we can suggest future webinars around NetScaler?
A: Sure, post them in the comments.
Q: The sysctl command says not found on the VPX. Is that only for the MPX?
> shell # sysctl -a netscaler
Q: The “show run” command output, could it be used to config a clean NetScaler instance i.e. to replace a virtual with a physical?
A: Yes, you can move the /nsconfig/ns.conf file from a physical to a virtual appliance and vice versa. I would recommend being aware of hardware differences between interfaces, VLANs, routes, IP ranges etc. Any commands you do not want to run can be hashed out with # character at the start of the line.
Q: Where do I find the domain join troubleshooting tools NetScaler 10.1 build 124? In build 119 there was a directory /opt/likewise/bin, which doesn’t exist in build 124 anymore. KB article: http://support.citrix.com/article/CTX133789
A: There’s a new tool to replace likewise utils called nskrb. See this article for details on how to use this new tool http://support.citrix.com/servlet/KbServlet/download/35879-102-706395/CTX139133_25th_September_2013.pdf
Q: When trying to login to web interface through NetScaler why are we getting “State Error”? Is there any command available to see that?
A: The NetScaler is performing ICA Proxy so will be proxying this information from Web Interface.
Q: Is there any command to view the active ICA sessions?
> show aaa session
Q: Is there a command that will list rewrite or responder policies with 0 hits? IE only list policies with 0 hits.
A: Funnily enough there are switches for nsconmsg to filter out 0 hits but none to focus on that I’m afraid. You could try:
# nsconmsg -K newnslog.ppe.0 -d stats -g pol_hits
The use another grep (-g) to filter some more.
Q: In the meanwhile are the NetScaler VPX licenses interchangeable with MPX licenses or do both environments have still their own licensing model?
A: Licensing is different between hardware and virtual appliances.
Q: How about improving performance for slow WAN connections (TCP windows size or other tuning tips)?
A: TCP Profiles will help here /blogs/2012/03/22/tune-netscaler-tcp-stack-to-suit-your-needs/
Q: How about throughput for each user session? How do you look for that, short of using insight?
A: Honestly, use Insight if you can. Makes your life much easier for this kind of thing.
Q: Does the command “”show run”” do a comprehensive configuration export?
A: Show run outputs the running config in memory. To commit the running config you must issue a save config command. You can also export /nsconfig/ns.conf
Q: How can we use the SNIP from the command line to test the opened firewall port?
A: The easiest way to do this is to setup a service for the particular port on that subnet and the NetScaler will try to use the SNIP to reach it. If you don’t get a reply then likely firewall port not open.
Q: When you have SSL all the way through to backend services does the NetScaler remove anything say proprietary client certificates before passing SSL back to backend services?
A: If you are running an SSL_Bridge Virtual Server then there will be no interference with the client certificate in the packet
Q: How can we will be able to check http servers client-server connection table using CLI?
> show persistentSessions
Pipe this through grep and filter for port 80 or the LB Virtual server name.
Q: Is there a specific log file for Load Balancing?
A: There’s no specific log file for load balancing, counters for LB will be written to the newnslog.
Q: How can I verify SSLv2 is enabled or disabled?
A: At the virtual server level run the show command and SSLv2 is listed in the output
> sh ssl vserver 1) Vserver Name: LB_VS_2 DH: DISABLED Ephemeral RSA: ENABLED Refresh Count: 0 Session Reuse: ENABLED Timeout: 120 seconds Cipher Redirect: ENABLED SSLv2 Redirect: ENABLED ClearText Port: 0 Client Auth: ENABLED Client Cert Required: Optional SSL Redirect: DISABLED Non FIPS Ciphers: DISABLED SNI: DISABLED SSLv2: DISABLED SSLv3: ENABLED TLSv1.0: ENABLED TLSv1.1: DISABLED TLSv1.2: DISABLED Push Encryption Trigger: Always Send Close-Notify: YES
Q: Does NetScaler Insight Center require the NetScaler it monitors to be hardware only? Or can it monitor a VPX?
A: Insight will work with physical or virtual NetScaler appliances.
Q: Does the aaad.debug have a logfile?
A: You can pipe the aaad.debug to a log using the pipe command or with tee to send to console and log at the same time:
# cat /tmp/aaad.debug | tee aaad.log
Q: Can NetScaler SDX be virtualized for a lab?
A: No, SDX cannot be virtualized as it’s a XenServer based platform itself.
Q: How to determine how much memory is allocated to this virtual instance?
# sysctl -a netscaler
Look for netscaler.hw_physmem_mb: 3786 and round up to the nearest Gigabyte
Q: How to determine using CLI how long the system is up?
Q: Unlimited ica sessions isnt true if you are utilizing insight correct?
A: I did a three-part blog series on Insight /blogs/2013/10/01/nstipster-series-netscaler-insight-part1/ Limits on Insight are based on the licenses on the NetScaler http://support.citrix.com/proddocs/topic/ni-10-1-map/ni-licensing-details-con.html
Q: Why should fail safe mode be on?
A: Fail-safe mode will elect a primary in a scenario where both NetScaler in a high-availability pair fail the health check http://support.citrix.com/proddocs/topic/ns-system-10-map/ns-nw-ha-cnfgrng-fail-safe-tsk.html