El Cuchitril

Well not long ago DH contacted me with this dreaded message:

Hello,

I’m writing you about your domain:

foo.bar

Specifically the file:

xmlrpc.php

This file is used for modifying your wordpress install from 3rd party programs, like mobile site designers, desktop client programs, and pretty much anything besides the wordpress control panel under wp-admin.  Bots and hackers have been increasingly attacking these files listed above to try to either brute force into the wordpress installs, or cause other downtime and server issues.  Our security team is working on blocks and protections to prevent these, but in the meantime this is causing server issues and downtime for your site, so our oncall admin has had to disable the file by changing permissions to 200 so it isn’t accessible.  If you don’t use external programs to modify this blog, you likely won’t even notice any chance in behavior and it can remain.  If you do make use of this file, when you next need to use it it can be re-enabled by chmod. If you have further questions please let us know.

This was on august 3… I suppose we where part of a severe DDOS that were targeting third party sites… BTW this was fixed with mod_rewrite hack on .htaccess [1].

Two weeks ago we got report for another site I run, this time on a bigger VPS, that we where participating on an ongoing DDOS [2] that we also caught as our machine resources were drawn terribly.

Yesterday we got visits again, of course we were aware of the problem and originally chmoded 000 the file, dropped via iptables 3 subnets that where hammering it and all was good, but as you can read on several places disabling this file breaks several functions of WordPress [3].

So as we use nginx as front end, begin to investigate if like on apache or lighty could exist something like mod_evasive and indeed exists [2] it’s called  mod_http_limit_req_module so after reading the documentation, I searched for examples and find a very good one that inspired this setup [4] so we ended with the next changes on nginx.conf:

# Prevent DDOS
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/m;

So we are limiting request to 1 request for minute, but as you could imagine we don’t want this for all the site, so next on the site conf we added

location = /xmlrpc.php
{
limit_req zone=one burst=5;
}

So we had 1 connection for minute on this file and a burst of five, but as this guys where hammering at more than one connection per second, well I suspect going to get a lot of 503 messages, I only expect that normal sites and ourselves not need this file more than one time for minute… on the bright side CPU, IO, and other resources are normal now, BTW some better advice if you don’t manage a server but had only access to .htaccess can be found here [1]

Update: 26/10/2014 We ended having to tweak the limiting as WordPress itself uses to connect to the file so we ended limiting ourselves the fix was to change on the first one to limit to  six request per second instead of minute:

# Prevent DDOS
limit_req_zone $binary_remote_addr zone=one:10m rate=6r/s;

And on site config to change for:

location = /xmlrpc.php
{
limit_req zone=one burst=7 nodelay;
}

As the docs says if you want no delay you had to add and also thinking that most modern browsers make up to 16 simultaneous connections [6] we upped the number but we don’t expect to have 16 request per second to this file.

[1]https://wordpress.org/support/topic/resolving-xmlrpcphp-ddos-attack-with-htaccess-redirect
[2] http://blog.sucuri.net/2014/03/more-than-162000-wordpress-sites-used-for-distributed-denial-of-service-attack.html
[3] https://wordpress.org/support/topic/xmlrpcphp-attack-on-wordpress-38
[4] http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
[5] https://cowthink.org/flood-dos-protection-with-limit-req-in-nginx/
[6] http://kb.mozillazine.org/Network.http.max-connections-per-server