Blocking Robots



The Citrix NetScaler can be placed in front of a webserver farm that is running Apache. The same re-write rules that run on Apache, can be implemented on the Citrix NetScaler.

You can block a really annoying robot from retrieving pages of a specific webarea. This way you can ease up the traffic at some directories.

Example : This could be done by using a rule set which forbids the URLs of the web area /~quux/foo/arc/. This could also be accomplished by matching the User-Agent HTTP header information. In this example, the ip address to be blocked is 123.45.67.8 & 123.45.67.9.

Apache rewrite:

RewriteCond %{HTTP_USER_AGENT}   ^NameOfBadRobot.*
RewriteCond %{REMOTE_ADDR}       ^123\.45\.67\.[8-9]$
RewriteRule ^/~quux/foo/arc/.+   -   [F]



AppExpert rewrite:

add responder action act1 respondwith '<span class="code-quote">"HTTP/1.1 403 Forbidden\r\n\r\n"</span>'
add responder policy pol1 'HTTP.REQ.HEADER(<span class="code-quote">"User_Agent"</span>).STARTSWITH(<span class="code-quote">"NameOfBadRobot"</span>)&amp;&amp;CLIENT.IP.SRC.EQ(123.45.67.8)&amp;&amp;CLIENT.IP.SRC.EQ(123.45.67.9) &amp;&amp; HTTP.REQ.URL.STARTSWITH(<span class="code-quote">"/~quux/foo/arc"</span>)' act1
bind responder global pol1 100



Tap into the power of AppExpert!