stop_sign_in_roadThe fail2ban suite is a very useful (if somewhat overcomplicated) tool in the battle against brute force login attempts.  Basically, fail2ban watches your log files, then executes specific commands (such as adding a firewall block for the offender’s IP address) when login failures cross a certain threshold, then reverses that block after a specified back-off period.  One of the servers I administer needs to run SSH on port 22, and fail2ban helped bring brute force attempts from over 70,000 per day (!!!) to around 20-30.  A dramatic decrease of 99.96% – not too shabby.

After this initial success, I expanded the use of fail2ban to other services, including POP3/IMAP, SMTP, WordPress (using a modified version of the “simple-login-log” plugin to log via syslog), and webmail.  Overall, this has been a huge help in minimizing the use of scarce human clock hours to review nuisance log entries.  If you run any authenticated service exposed to the Internet, I highly recommend deploying fail2ban or something similar as a component of your preventive/defensive measures.

Where this process broke down for me, however, was in addressing repeat offenders.  After the firewall block was removed for a given offender’s IP, it was not long before some of the IPs started brute forcing login attempts and were blocked again.  For quite a while, I was content to note the repeat offenders, add them to a permanent firewall block, and press on.  More recently, this chore became frequent enough that it made sense to build that functionality into the fail2ban configuration itself, removing the human (and my clock hours) from the loop.

I quickly found a helpful post from Lukas Camenzind that outlined the basics of what I wanted to accomplish.  His solution, however, implemented a “permanent block first” method, rather than a “repeat offender” one.  After a short bit of testing, I deployed the following solution, which fully addressed my requirements.  Of course, your experience may vary depending on configuration, but this should be enough to get a similar solution deployed in your environment.

First, create the /etc/fail2ban/action.d/iptables-repeater.conf file:

# Fail2ban configuration file
#
# Author: Phil Hagen <phil@identityvector.com>
#

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-REPEAT-<name>
              iptables -A fail2ban-REPEAT-<name> -j RETURN
              iptables -I INPUT -j fail2ban-REPEAT-<name>
              # set up from the static file
              cat /etc/fail2ban/ip.blocklist.<name> |grep -v ^\s*#|awk '{print $1}' | while read IP; do iptables -I fail2ban-REPEAT-<name> 1 -s $IP -j DROP; done

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D INPUT -j fail2ban-REPEAT-<name>
             iptables -F fail2ban-REPEAT-<name>
             iptables -X fail2ban-REPEAT-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L INPUT | grep -q fail2ban-REPEAT-<name>

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    <ip>  IP address
#          <failures>  number of failures
#          <time>

Note that the configuration above will block ALL traffic from offending IPs.

Next, add the appropriate stanza(s) to the /etc/fail2ban/jail.conf file:

[ssh-repeater]
enabled  = true
filter   = sshd
action   = iptables-repeater[name=ssh]
           sendmail-whois[name=SSH-repeater, dest=root, sender=root]
logpath  = /var/log/secure
maxretry = 21
findtime = 31536000
bantime  = 31536000

[wordpress-auth-repeater]
enabled  = true
port     = http,https
filter   = wordpress-auth
action   = iptables-repeater[name=wordpress]
            sendmail-whois[name=wordpress-repeater, dest=root, sender=root]
logpath  = /var/log/secure
maxretry = 35
findtime = 31536000
bantime  = 31536000

In the first jail, I’m using the existing “sshd” log watch filter, but waiting for 21 failed login attempts within a year. (This is three consecutive “basic” blocks, which trigger after seven failed attempts, per the defaults distributed with fail2ban.)  Of course, update this example to suit your distribution’s log location and your preferred service and other parameters.  The second jail uses a custom “wordpress-auth” filter and is slightly less aggressive – allowing 35 failed logins per year before invoking the ban-hammer. Duplicate the jail stanzas and adjust as needed for your systems.

After editing this file, just restart the fail2ban service and watch the contents of the /etc/fail2ban/ip.blocklist.ssh and /etc/fail2ban/ip.blocklist.wordpress files grow.

One debugging note – I found troubleshooting fail2ban quite difficult.  After a while, I found that stopping the service, then running “fail2ban-client -vvv -x start” would give me the verbosity needed to find syntax errors in the configuration files.

UPDATE (April 26, 2014): Per a great recommendation from Jan (below), added a grep to de-duplicate IPs on reboot/service restart.

UPDATE (April 28, 2013): After some great discussion below, I’ve updated the content and linked files per the following:

  1. Reverted to blocking all traffic after a block, rather than just the service the brute forcer attacked.
  2. The configuration now uses one ip.blocklist file and one iptables chain per jail, rather than sharing.  This prevents some duplicated rules and provides a cleaner segregation for types of blocks.
  3. Renamed jail to iptables-repeater.

Photo from Flickr user auve.

Tagged with →