Nginx had the ngx_http_limit_req_module which can be used to limit request processing rate, but most people seem to only used its basic feature: limiting request rate by the remote address, like this:

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
    ...
    server {
        ...
        location /search/ {
            limit_req zone=one burst=5;
        }

This is the example configuration taken from Nginx’s official document, the limit_req_zone directive takes variable $binary_remote_addr as a key to limit incoming request. The key fills a zone named one, which is defined by the zone parameter, it could use up to 10m memory. And the rate parameter says that the maximum request rate of each $binary_remote_addr is 1 per second. In the search location block, we can use limit_req directive to refer the one zone, with the bursts not exceeding 5 requests.

Everything seems great, we have configured Nginx against rogue bots/spiders, right?

No! That configuration won’t work in real life, it should never used in your production environments! Take these following circumstances as examples:

• When users access your website behind a NAT, they share the same public IP, thus Nginx will use only one $binary_remote_addr to do limit request. Hundreds of users in total could only be able access your website 1 time per second!
• A botnet is used to crawl your website, it use different IP addresses each time. Again, in this situation, limiting by $binary_remote_addr is totally useless.

So what configuration should we use then? We need to use a different variable as the key, or even multiple variables combined together (since version 1.7.6, limit_req_zone‘s key can take multiple variables). Instead of remote address, it is better to use request HTTP headers to distinguish users apart, like User-Agent, Referer, Cookie, etc. These headers are easy to access in Nginx, they are exposed as built in variables, like $http_user_agent, $http_referer, $cookie_name, etc.

For example, this is a better way to define a zone:

http {
    limit_req_zone $binary_remote_addr$http_user_agent zone=two:10m rate=90r/m;
}

It combines $binary_remote_addr and $http_user_agent together, so different user agent behind NATed network can be distinguished. But it is still not perfect, multiple users could use a same browser, same version, thus they send same User-Agent header! Another problem is that the length of $http_user_agent variable is not fixed (unlike $binary_remote_addr), a long header could use a lot of memory of the zone, may exceeds it.

To solve the first problem, we can use more variables there, cookies would be great, since different user sends their unique cookies, like $cookie_userid, but this still leaves us the second problem. The answer it to use the hashes of variables instead.

Thers is a third-party module called set-misc-nginx-module, we can use it to generate hashes from variables. If you are using Openresty, this moule is already included. So the configuration is like this:

http {
    ...
    limit_req_zone $binary_remote_addr$cookie_hash$ua_hash zone=three:10m rate=90r/m;
    ...
    server {
        ...
        set_md5 $cookie_hash $cookie_userid;
        set_md5 $ua_hash $http_user_agent;
        ...
    }
}

It is OK we used $cookie_hash and $ua_hash in the http block before they are defined in server block. This configuration is great now.

Let’s continue with the distributed botnet problem now, we need to take $binary_remote_addr out of the key, and since those bots usually don’t send Referer header (else you can found what unique about it by yourself), we can take advantage of it. This configuration should take care of it:

http {
    ...
    limit_req_zone $cookie_hash$referer_hash$ua_hash zone=three:10m rate=90r/m;
    ...
    server {
        ...
        set_md5 $cookie_hash $cookie_userid;
        set_md5 $referer_hash $http_referer;
        set_md5 $ua_hash $http_user_agent;
        ...
    }
}

前段时间配置 Nginx 时候碰到了一个问题：Nginx 模块内部暴露出来的资源无法访问。具体来说就是比如某个模块导出了一个资源（比如 /channels-stats）提供统计信息,如下：

location /channels-stats {
# activate channels statistics mode for this location
push_stream_channels_statistics;

# query string based channel id
push_stream_channels_path $arg_id;
}

而在配置文件中还有一下配置，如果访问的文件不存在，则重定向到使用 index.php 去进行访问

if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php/?s=$1 last;
break;
}

这个时候，/channels-stats 是无法被访问到的，原因应该是 /channels-stats 这样的有模块导出的资源并不属于 $request_filename 这个变量的范畴。导致实际访问的其实是 /index.php/?s=channels-stats 。

这个时候也不能用 $request_uri, $uri 等变量去替换 $request_filename 的位置，因为 if 能测试的就只是文件是否存在而已。

想了想，发现这段时间都忘了 Nginx 的 try_files 指令了，以及 if is evil 这条 Nginx 金句了，用下面这条可以替代上面的，并且可以保证  /channels-stats 仍然可以访问

try_files $uri $uri/ /index.php?s=$request_uri;

这里讲了不该用 if 的：
http://wiki.nginx.org/Pitfalls#Check_IF_File_Exists
http://wiki.nginx.org/IfIsEvil
更新：这里的问题其实是把”if (!-e $request_filename)”测试放到 location / 下面就很好解决了的，不知道当时为什么脑子短路了，没有发现。。。