Permanently Ban Repeat Offenders With fail2ban (UPDATED)

stop_sign_in_roadUPDATE: If you’re reading this, you may want to take a look at the “recidive” filter, which watches the fail2ban log itself and adds a more hardcore block on repeat offenders.  It’s not the exact same solution, but may meet your needs much more easily than the setup described here.  There are notes in the comments about this filter.  I’ll leave this post intact anyway, but maybe this will save you some time.  I will not be able to reply to support requests for this functionality, as I no longer use it myself.  Further comments are disabled on this post.

The fail2ban suite is a very useful (if somewhat overcomplicated) tool in the battle against brute force login attempts.  Basically, fail2ban watches your log files, then executes specific commands (such as adding a firewall block for the offender’s IP address) when login failures cross a certain threshold, then reverses that block after a specified back-off period.  One of the servers I administer needs to run SSH on port 22, and fail2ban helped bring brute force attempts from over 70,000 per day (!!!) to around 20-30.  A dramatic decrease of 99.96% – not too shabby.

After this initial success, I expanded the use of fail2ban to other services, including POP3/IMAP, SMTP, WordPress (using a modified version of the “simple-login-log” plugin to log via syslog), and webmail.  Overall, this has been a huge help in minimizing the use of scarce human clock hours to review nuisance log entries.  If you run any authenticated service exposed to the Internet, I highly recommend deploying fail2ban or something similar as a component of your preventive/defensive measures.

Where this process broke down for me, however, was in addressing repeat offenders.  After the firewall block was removed for a given offender’s IP, it was not long before some of the IPs started brute forcing login attempts and were blocked again.  For quite a while, I was content to note the repeat offenders, add them to a permanent firewall block, and press on.  More recently, this chore became frequent enough that it made sense to build that functionality into the fail2ban configuration itself, removing the human (and my clock hours) from the loop.

I quickly found a helpful post from Lukas Camenzind that outlined the basics of what I wanted to accomplish.  His solution, however, implemented a “permanent block first” method, rather than a “repeat offender” one.  After a short bit of testing, I deployed the following solution, which fully addressed my requirements.  Of course, your experience may vary depending on configuration, but this should be enough to get a similar solution deployed in your environment.

First, create the /etc/fail2ban/action.d/iptables-repeater.conf file:

# Fail2ban configuration file
#
# Author: Phil Hagen <phil@identityvector.com>
#

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-REPEAT-<name>
              iptables -A fail2ban-REPEAT-<name> -j RETURN
              iptables -I INPUT -j fail2ban-REPEAT-<name>
              # set up from the static file
              cat /etc/fail2ban/ip.blocklist.<name> |grep -v ^\s*#|awk '{print $1}' | while read IP; do iptables -I fail2ban-REPEAT-<name> 1 -s $IP -j DROP; done

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D INPUT -j fail2ban-REPEAT-<name>
             iptables -F fail2ban-REPEAT-<name>
             iptables -X fail2ban-REPEAT-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L INPUT | grep -q fail2ban-REPEAT-<name>

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    <ip>  IP address
#          <failures>  number of failures
#          <time>

Note that the configuration above will block ALL traffic from offending IPs.

Next, add the appropriate stanza(s) to the /etc/fail2ban/jail.conf file:

[ssh-repeater]
enabled  = true
filter   = sshd
action   = iptables-repeater[name=ssh]
           sendmail-whois[name=SSH-repeater, dest=root, sender=root]
logpath  = /var/log/secure
maxretry = 21
findtime = 31536000
bantime  = 31536000

[wordpress-auth-repeater]
enabled  = true
port     = http,https
filter   = wordpress-auth
action   = iptables-repeater[name=wordpress]
            sendmail-whois[name=wordpress-repeater, dest=root, sender=root]
logpath  = /var/log/secure
maxretry = 35
findtime = 31536000
bantime  = 31536000

In the first jail, I’m using the existing “sshd” log watch filter, but waiting for 21 failed login attempts within a year. (This is three consecutive “basic” blocks, which trigger after seven failed attempts, per the defaults distributed with fail2ban.)  Of course, update this example to suit your distribution’s log location and your preferred service and other parameters.  The second jail uses a custom “wordpress-auth” filter and is slightly less aggressive – allowing 35 failed logins per year before invoking the ban-hammer. Duplicate the jail stanzas and adjust as needed for your systems.

After editing this file, just restart the fail2ban service and watch the contents of the /etc/fail2ban/ip.blocklist.ssh and /etc/fail2ban/ip.blocklist.wordpress files grow.

One debugging note – I found troubleshooting fail2ban quite difficult.  After a while, I found that stopping the service, then running “fail2ban-client -vvv -x start” would give me the verbosity needed to find syntax errors in the configuration files.

UPDATE (April 26, 2014): Per a great recommendation from Jan (below), added a grep to de-duplicate IPs on reboot/service restart.

UPDATE (April 28, 2013): After some great discussion below, I’ve updated the content and linked files per the following:

  1. Reverted to blocking all traffic after a block, rather than just the service the brute forcer attacked.
  2. The configuration now uses one ip.blocklist file and one iptables chain per jail, rather than sharing.  This prevents some duplicated rules and provides a cleaner segregation for types of blocks.
  3. Renamed jail to iptables-repeater.

Photo from Flickr user auve.

176 comments

  1. Pingback: Surviving an attack from the Chinese – Tasos Sangiotis

  2. This is a great tool ..I have a question tho…even tho the system creates the blocked wordpress file, it does not create the ip.blocklist.*. Why might this be? I have a ton of addresses nightly trying to use ‘known hacks’. I was hoping that this would reduce/eliminate that..Any idea??

  3. had to change logpath = /var/log/secure to: logpath = %(sshd_log)s

    paths-common.conf sets sshd_log= %(syslog_authpriv)s

    ( paths-debian.conf sets syslog_authpriv = /var/log/auth.log )

  4. Hi,
    Great Post, working very effectively for me.
    Although I just noticed a few ip’s not banned due to this error, running on a VPS with Centos 6.7.
    2016-04-08 18:47:37,599 fail2ban.actions [2001]: ERROR Failed to execute ban jail ‘ssh-repeater’ action ‘iptables-repeater’ info ‘CallingMap({‘ipjailmatches’: <function at 0x13b8500>, ‘matches’:

  6. Thank you for this solution, I was looking for some step like a limit of attempts a hour, limit per day and limit per week. This is a little bit elaborate but it looks the only solution, anyway if I understand correctly fail2ban reads only the last file of the log, if I have a log rotate daily this solution how does it works? To make it work I’ll have to keep a single file with a year full of logs which it’s almost impossible with a normal volume of an average server
    Am I missing something?

    1. You may want to check the recidive filter instead – I think it would be better suited to your requirements.
      To your specific question, yes, fail2ban will only ready the log files that are available at startup. However, the original intent behind this repeat-offender methodology was to store the permanently blocked IPs in a different file that is read upon startup, removing the reliance on “live” log files.
      I hope that helps!

  7. Great resource! Reading your initial story matches mine exactly, as I have been using fail2ban successfully for years but sometimes the repeat offenders get annoying and plentiful.

    So please forgive me if I missed the answers to these two questions somewhere in the comments or whatnot but:

    1) As written, do these rules persist across reboots? Firewalls based on iptables are my kryptonite. 🙂

    2) If I do testing, or if a friend happens to set their password incorrectly and hits my site a few dozen times, and they get permablocked, what’s the best way to unblock them? Also, what’s the best way to make sure they don’t get re-blocked for past failures? I obviously want to block them (i.e. their IP address) for -future- failures, just in case they do it again, or their IP changes.

    Thanks again!

    1. Glad you found it helpful!

      Quick replies to your great questions:
      1) Yes, as written, these will persist across reboot, since the “actionstart” section reads from the /etc/fail2ban/ip.blocklist file upon startup. Note, however, that if you use the “recidive” method, the blocks will not persist. Not better or worse, just different.
      2) Removing permablocks is done manually, by removing the IP address from the /etc/fail2ban/ip.blocklist file, as well as removing the rule in iptables directly or via a command such as “fail2ban-client set <%JAILNAME%%gt; unbanip <%IP_ADDRESS%>”.

      1. Awesome! Removing an IP from a file and running that last command will work perfectly, thank you.

        As far as the “recidive” method…looks like I haven’t even moved the .rpmnew file in, so [recidive] isn’t even in my jail.conf file yet. Should be good at this point.

        As an aside, good design. I liked how it went back and started perma-blocking past offenders after a restart. Some of these brute-force attacks are nasty. Not so much worried about SSH (I don’t allow passwords, very restrictive), but Dovecot/IMAP.

      2. Also, as an aside, which plugin/resource are you using for your “Notify me of follow-up comments by email.” options on your blog? I’m running latest WordPress but haven’t seen that option. Figured it was a plugin or theme?

        1. No plugin, just the native functionality – not sure exactly which setting, unfortunately (this is not the admin account).

      3. I’ve tried several times to remove the perma-ban by doing the following:

        * Remove entry from /etc/fail2ban/ip.blocklist.dovecot
        * Remove entry from iptables with:
        — iptables -L -n
        — iptables -D fail2ban-REPEAT-dovecot 1
        Then running the list again to make sure it’s gone (it is)
        * Run the unbanip command:
        — fail2ban-client set dovecot unbanip 192.X.X.X
        — fail2ban-client set dovecot-repeater unbanip 192.X.X.X
        In each case, the IP address is returned.
        I then verify by running same commands again to get the error:
        “IP 192.X.X.X is not banned” in each case.
        * I even went so far as to remove all traces of 192.X,X,X from ALL log files in /var/adm – not a trace.

        But when I restart fail2ban, it perma-bans that IP address again. Am I missing something?

        1. Unsure based on the pasted info – is it possible that the IP is in more than one location? I’ve heard that some fail2ban implementations also re-read the entire log file upon startup, so that may be what’s happening. As long as you can ssh in before restarting the service (which should be a reasonably infrequent occurrence), you should be alright. (Or, perhaps more accurately, my use case has shown the above to be sufficient…)

          1. Looks like I’m running into a fail2ban bug. I’ve removed the IP address from everywhere, but it still perma-bans the IP. I think it’s not removing it from the sqlite database (I can see the IP address in there under the “bans”). So I’ll hit up the bug chat on that one. Works perfectly otherwise, thanks!

          2. aha!! OK, I’m not using any sqlite functions, so that’s likely the issue. Unfortunately, I’m not familiar with that functionality, though.

  9. Interesting read. I came up with a jury rigged system that is so far full proof. I posted it on my website ghost-house.co.uk. The outline is that it stores all banned IP’s in an SQL database and upon service/system restarts it automatically re-bans IP addresses. I’ve been testing it for a few weeks and so far blocked 550 IP’s from my little home server. On the home server I wrote extra scripts to block permanently or to un-ban after a set time with scripts that search and bans repeat offenders on a permanent basis.

    1. That looks quite robust – thanks for sharing! Could be leveraged by multiple servers pulling from same database as well – nice scalability!

      1. Thank You for the comment. Since posting, I have altered the system a little by adding a second table to my database called “compare”. When fail2ban does it blocking and adds the information to the database, it also adds the same reference to the second table. This means that after the first database unbans the IP address after 7 days, if the IP address then shows up again (referencing the compare database) it then has the option to permanently ban that address. Originally I was banning the IP for 180 days, but after just a couple of days I found that the database was getting pretty large and after 180 days it would probably contain thousands of entries which would slow down the search when adding/unbanning and anyone referencing the tables. So I scaled that back to just 7 days which so far has proved more successful with SQL retrieval. I wrote the front end to the database (http://wnsi.co.uk/banned.php) so that others could download the information via XML, JSON or good old CSV if they so wished 🙂 Its an ongoing little project with new ideas popping into my mind all the time lol.

        1. Me again (trouble). I’ve been playing with my little project some more and have integrated PHP graphs into the output to show different bits of information that is pretty useful. To date its showing that the biggest abuser is Vietnam which now allows me to have the option of blocking that specific country altogether (by never unlocking them).

          http://wnsi.co.uk/graph.php?type=current&theme=greentheme

          Its coming along. If people do get interested in this, I am willing to release my scripts that I have written for public use if needed. They are all written in PHP and although i try to keep my coding nice and tidy, it might mean a little deciphering though. But most of it is self explanatory.

          1. Nice graph!

            Personally I tend to use this shell script from http://www.cyberciti.biz/faq/block-entier-country-using-iptables/ for my country blocking. All I have to do is amend one line with the 2 character ISO code of the country/countries I wish to block and it rather nicely sets up the iptables entries for me.

            While integrated systems sound nice with this sort of stuff, splitting the country based blocking from the dynamic service based blocking works well for me.

    2. I initially tried with IP tables, but started to get into a headtizzle, so I moved over to using route to block IP’s, i found it to be a lot quicker and more efficient, i previously came from Windows systems so linux is still a learning curve for me, I’ve got most of it handled now, but still get messed up in the head with iptables lol. I have a fairly fast little home server but my internet is dead slow for upload, so i use the fastest possible ways of doing things where possible. The overall outcome though is pretty much the same.

      1. Hello Nicholas .
        Have you released the script yet? Could you please tell us about the process and a way we can contribute or follow the development ?

        1. Hi Mehmet, As we speak I am developing a management system. I have released a beta on my website http://wnsi.co.uk/bot2/index.php which allows a person to view stats, block IP’s and other stuff. Its still in beta and the back end for the script is still to be put together, but the basics of the scripts are already published on my website http://ghost-house.co.uk. Take a look and see what you think. If its useful I will zip up the source code and store it on my website for anyone to download and use. The new management system is basically a consolidation of all the scripts i have written since undertaking this little project, if all goes well, it should just be a case of dumping on a secure area of a web server (so the outside world doesn’t mess with stuff) and running it, plus an mysql server (the script sets up the database + tables) (hopefully) and if all goes well it should be a useful system. Please note though that I am a self taught web designer, so this will be my very first software release.

          1. Me again. I know i know i write too much lol. I’ll scale that down. I have now released my beta version of my fail2ban management system titled ‘botnet paradigm’ – has a catchy title. if you would like to test it out then it can be downloaded from http://ghost-house.co.uk I do hope that if you find it useful or at the very least intriguing that you’ll at the very least leave a comment of message me. Hope its of some use to you. Nicholas Weightman.

          2. well ,

            “Apologies the country Turkey has been deemed an origin country of spamming and hacking and therefore has been denied access to this site. I wish you all the best on your search if you are a genuine user”

            Ouch. Little too much ? 🙂 .
            Phil’s work has been proved useful, though I needed to use this on multiple servers sharing single database. Hopefully it will save me from those attackers.

  10. Hi Phil, please consider adding a note about the recidive filter since the comments are long. Many people may be happy just setting `enabled = true`.

    For others: By default it bans 5-times-in-a-day-offenders for a week (but does not persist across reboots).

  11. Pingback: Fail2ban + Tarpit – Antoine Aflalo

  12. Great tutorial. I’ve modified it a bit to add perma banned IPs to an ipset to prevent the iptables from getting large and hard to manage. Obviously, you’ll need ipset installed.

    I’m no expert on any of this, but this is working pretty good. I’ve modified the above as such:

    Action Start:

    actionstart = ipset –create fail2ban-REPEAT- iphash
    iptables -I INPUT -m set –match-set fail2ban-REPEAT- src -j DROP
    # set up from the static file
    cat /etc/fail2ban/ip.blocklist. |grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do ipset –add fail2ban-REPEAT- $IP; done

    Action Stop:

    actionstop = iptables -D INPUT -m set –match-set fail2ban-REPEAT- src -j
    ipset –flush fail2ban-REPEAT-
    ipset –destroy fail2ban-REPEAT-

    Action Ban:

    actionban = ipset –test fail2ban-REPEAT- || ipset –add fail2ban-REPEAT-
    # also put into the static file to re-populate after a restart
    ! grep -Fq /etc/fail2ban/ip.blocklist. && echo ” # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.

    Hope this helps someone else who was looking for this.

    2. Jamey,

      ipset is just what ive been looking for. Looks like you are missing DROP after -j in the actionstop line.

      My question for you is; I want to call this with multiple sections in jail.conf but that just causes ipset to overwrite its members and creates a separate “-m set –match-set fail2ban-REPEAT- src -j DROP” rule in the INPUT table.

      Do I have to create a separate repeater.conf action for each filter i want to use this for or is there a way to make each filter use the same ipset data without overwriting?

      1. I figured out what I needed to do. I had to modify your script a bit to get it to work correctly. For anyone interested in using ipset, here is what I had to do to get this working with multiple jails but only one data set of repeat offenders.

        First I change the action script for iptables-repeater.conf to use these;

        #Action Start
        actionstart = ipset -create fail2ban-REPEAT iphash
        iptables -I INPUT -m set –match-set fail2ban-REPEAT src -j DROP
        # set up from the static file
        cat /etc/fail2ban/ip.blocklist | grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do ipset -add fail2ban-REPEAT $IP; done

        #Action Stop
        actionstop = iptables -D INPUT -m set –match-set fail2ban-REPEAT src -j DROP
        ipset -flush fail2ban-REPEAT
        ipset -destroy fail2ban-REPEAT

        #Action Ban
        actionban = if [ipset -test fail2ban-REPEAT ] || [ipset -add fail2ban-REPEAT ]; then
        # also put into the static file to re-populate after a restart
        ! grep -Fq /etc/fail2ban/ip.blocklist && echo ” # fail2ban/$( date .+%%Y-%%m-%%d %%T. ): auto-add for repeat offender.” >> /etc/fail2ban/ip.blocklist
        fi

        Then I created an iptables-repeater2.conf and an iptables-repeater3.conf in /etc/fail2ban/action.d for use with two other jails in the jail.conf as the action line like
        action = iptables-repeater2

        but the iptables-repeater2.conf and iptables3.conf only have the #Action Ban section so it doesn’t overwrite the ipset data set members. Now i can use just one ipset data set for all perma-bans.

        I had to add some tags and a missing quote as well as make the OR operator statement in to an IF-THEN statement to get the ban actions working correctly.

        Thanks for putting me on the path to use ipset. I like it better than having this scrolling list of IP’s constantly being banned. Now I am going to consider using ipset to block foreign subnets from China and Russia to start.

        1. I tried your code, but characters got lost due to html parsing on this website.

          Any chance you could provide the code in another way? Pastebin, github?

  13. Is this still working with firewalld (CentOS 7)? Any alternatives that work with firewalld?

    1. I’m not sure if this exact implementation would work with firewalld or not, but the worse case would be replace the various iptables commands with their firewall-cmd commands.

      1. I can’t verify this right now. The FIrewallD documentation states:
        Pass a command through to the firewall. can be all iptables, ip6tables and ebtables command line arguments.
        This means we can simply make your script compatible be replacing all “iptables” commands in the action.d script with:
        firewall-cmd --direct --passthrough ipv4

        Right?

          1. Had a chance to give it a try. Let’s say that it does work, but it isn’t smooth. Offenders get banned and can’t connect anymore, on the other hand there seems to be some incompatibility with FirewallD:
            $ firewall-cmd --get-all-rules
            usage: see firewall-cmd man page
            Wrong usage of 'direct' options.

    2. I applied this on a ShellVM where I have CentOS 7 running with FirewallD and didn’t do any changes. It appears to work.

      1. Thanks for your confirmation! Do you receive the same errors when trying to read the firewalld configuration? My guess is that we need to adjust some of the commands to make it work. Currently afk, but will try next week.

  14. Hi Phil,

    thanks a lot for this article! I had a bash-cronjob previously, which crawled my fail2ban mails and counted the occourance of the same IPs in there, before this 😉 Much more elegant this way!

    Best regards,
    Marc

  15. Hey,
    thank you so much for posting this! I implented it in my server, which was bombarded with ssh login attempts, and it worked wonders. More than 100 banned IPs already. Now I can sleep with a little more peace, knowing that those persistent IPs get banned automatically 🙂
    Thank you!

    Cheers,

    JP

  16. Is there any fundementally healthy reason to block ALL traffic? Having a permaban scenario on ALL traffic world quickly rack up the IP’s it seems to me, and its usually botnets that do this sort of thing. It seems inevitable at some point somewhere someone (as in a legitimate user) is going to inherit a blacklisted IP. I’m not sure if that is a desireable thing.

    1. Good point, Kevin – there is no single answer to that, though… It’s really a matter of what kind of targeting list your system is on, and how intense/diverse the attacks against it are. You’re correct that the iptables rules can get pretty heavy, but personally, if those systems are still attacking, I prefer to block them entirely. No need to allow them any avenue to attack whatsoever.
      I do, however, go back every few months (really, just whenever I happen to think about it) and revisit the necessity of the rules. It’s pretty easy – just reset the packet counters on those rules and then remove blocks for whomever stays at zero for a few weeks.
      I haven’t had a situation where a legitimate user inherits the blacklisted IP, but you’re certainly correct that it’s possible.

      1. I have personally opted to only do any bans on Port 22 after three failed login attempts and I also have disabled root access in OpenSSH.
        This pretty much breaks the bute force hacking option with managable log-file pollution and no user intervention. (you can basically filter all login attempts that include root, wich is pretty much every login attempt ever 🙂 )

      2. Hi Phil,

        I am interested to know how you do this. I’m no iptables expert and sometimes wonder if the blocked IPs are still attempting to connect.

        The block files on the two systems I run this on contain a few hundred IP addresses. I’d like to reset every now and again, but only take out the “not trying anymore” addresses.

        1. I’m not sure there is an easy way to do that aside from manual review. You can determine if a rule is still being hit by examining the packet/byte counters at the beginning of each line from the command iptables -nvL, which is a good place to start.

  18. do i need to create the filter “wordpress-auth” ?

    [root@ ~]# service fail2ban restart
    Stopping fail2ban: [ OK ]
    Starting fail2ban: ERROR Found no accessible config files for ‘filter.d/wordpress-auth’ under /etc/fail2ban
    ERROR Unable to read the filter
    ERROR Errors in jail ‘wordpress-auth-repeater’. Skipping…
    [FAILED]

    1. That’s just an example, Noel – I created a custom WordPress filter, but the example configuration stanza is solely for illustrative purposes – you can use whatever existing filter(s) you like to create a persistent “repeater” jail.

      1. I have create a jail in jail.local:

        [nginx-badrequests-repeater]

        enabled = true
        port = http,https
        filter = nginx-badrequests
        action = iptables-repeater[name=BadrequestsRepeater]
        sendmail-whois[name=BadrequestsRepeater, dest=admin@domain.com, sender=root]
        logpath = /var/log/nginx/*access.log
        maxretry = 35
        findtime = 31536000
        bantime = 31536000

        But fail2ban don’t adds the same reference like ssh-repeater jail to the iptables:

        Chain fail2ban-REPEAT-sshRepeater (1 references)
        target prot opt source destination
        RETURN all — anywhere anywhere

  19. Hi Phil, great post, thanks!

    I would rather, however, to use /etc/hosts.deny instead of ip.blocklist.ssh file; keeping iptables clean of thousand blocked IPs. What do you think ?

    Regards

    1. That’s certainly a matter of administrator preference. The tcp_wrappers function does not extend to all network services, so it may or may not work for your particular daemon of interest. The Apache web server on RHEL/CentOS, for example does not use tcp_wrappers – so that method would be of significantly limited utility.

  20. I have a similar setup using the wordpress plugin. I found that by default if you trigger one ban in fail2ban, it won’t trigger any other bans. So, for example, if I trigger the wordpress ban I get blocked from http/https but will then NEVER get banned from SSH as long as the other ban is in place.

  21. Pingback: Answer: Permanent block of IP after n retries using fail2ban #fix #answer #development | SevenNet

  22. Hello,

    This never seemed to work before on 1.8.x, so I updated to 1.9.1 tonight and now get the following when (re)starting Fail2Ban:

    Stopping fail2ban: [ OK ]
    Starting fail2ban: ERROR NOK: (“Failed to initialize any backend for Jail ‘ssh-repeater'”,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    ERROR NOK: (‘ssh-repeater’,)
    (and this repeats about 20 more times)

    Any ideas why? When I set
    [ssh-repeater]
    enabled = false
    Then the error goes away and fail2ban starts just fine, so definitely something with the iptables-repeater.conf and the 0.9.x version

    1. Hi, TSH – I’m not sure what that error is, but my guess would be either a syntax error in the jail configuration or an underlying error with your fail2ban installation.

  23. Is this perhaps broken in fail2ban 0.9.1?

    For one thing the syntax you use for adding the date doesn’t work, at least on my box.

    My understanding is ‘%%’ is used for the actual PERCENT symbol, and a single ‘%’ is appropriate for the date.

    At least, that works in BASH scripts, I have searched the documentation for Fail2Ban, such as it is, and can’t seem to find where they use some sort of custom format for the $date function.

    At any rate, if as written, it doesn’t record the date and time as intended on my Fedora Linux system, and if I change it to the correct syntax the program says it’s an error.

    1. It’s possible that the syntax has changed with newer versions, @bill. This is confirmed to work on v0.8.14. The double percent signs are fail2ban’s syntax, which can be seen in numerous other action configuration files in /etc/fail2ban/action.d/. Just grep for “date” there, which should confirm if there is differing functionality in your version. I believe that fail2ban uses percent signs to signify a field, so the “%%” string would result in a single literal percent sign being passed to the date command.

  24. Pingback: Permanently Ban Repeat Offenders With fail2ban - MichaelOldroyd.co.uk

  25. Hey Phil,

    Just wanted to say thanks for this. I was struggling to fully understand exactly how fail2ban worked and was frustrated when trying to figure out how to ban repeat offenders. After reading your action and jail blocks I was able to finally figure out how it all tied together. After that I was able to modify your script slightly to get a jail set-up as I wanted it and it’s working perfectly. Plus it’s a great idea to store the banned IPs in a file so they can be readded after a flush.

    I’m using the scripts on a cluster of boxes and adding the blocked IPs from most of them back to the primary box every 15 minutes via cron then rsyncing the blocklist back to the others every hour so all the servers have the full list of blocked addresses. Most of the time the brute force attempts don’t move between boxes but every now and then there’s a clever so-and-so that’s got it running round robin between them all. This stops them dead in their tracks 🙂

    And the fact that you also included the WordPress jail here too is great, I have a different solution for blocking brute force attacks on my WP sites but I think using fail2ban directly is a better method so I’ll definitely be adapting that jail for my own needs too.

    1. Thanks for the feedback, Will. I’m glad the research was helpful for you! I’d be happy to hear how you modify the config to suit your needs, as well.

  26. it seem to be working in that it bans the offenders, however it doesn’t seem to be creating any ip.blocklist.* am I missing something? do I need to give some extra permissions to the folder /etc/fail2ban? at the moment I just have a default fail2ban install, I added this script and I added the definition part in jail.local for SSH-repeater only (I do not have wordpress on my server). I think if I can make this one work I should modify one for owncloud and web server admin interface.

    1. @gregor – It’s hard to say from your description, but it does sound like a permissions problem. You may also have selinux in play, but I’m afraid I can’t provide much detailed help.

      1. Was this resolved? I followed the guide but still have the same issue, no ip.blocklist.ssh is created.

  27. Hello Phil,
    Many thanks for your article.

    I am wondering if it possible to disable email notification in case of fail2ban restart?
    (for now when fail2ban restart we receive an email for every ip previously added in the ip.blocklist.xx.)

    It will be better if we receive an email notification only for new ip added. In my case for example I receive more than 200emails every time Fail2ban restart ^^.

      1. Hi Phil, thank you for your reply. could you tell me what you have changes exactly in the actionstart & actionstop, please?

        ps : it seems we can acesss the “older comments” pages anymore. I am always redirected to https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban. (a rule seems rewritte the url and strip the /comment-page-x/ part)

        ex : click “older comments” (https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/comment-page-2/#comments)

        will redirect to :
        https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/#comments

        Best regards,
        Xavier

        1. @xaa I just removed the “printf” lines from the “actionstart” and “actionstop” sections, which pipe to /usr/sbin/sendmail by default.
          Sorry about the comments – took a while to track that down, but seems it’s a conflict between the theme and WordPress’s comment pagination. I know it’s an ugly workaround, but I just removed pagination entirely. Thanks for the heads up!

  28. The ips aren’t getting removed from the flat file so they are effectively blocked forever or until they are removed manually from the flat file right? To automate something like:

    grep -vw IP ip.blocklist.TYPE > ip.blocklist.TYPE.tmp && mv ip.blocklist.TYPE.tmp ip.blocklist.TYPE

    and then maybe something to remove the ip from the live firewall

  29. i also discovered that when the fail2ban service is restarted since the blocklist ips are reloaded via iptables commands instead of fail2ban-client commands, as far as fail2ban is concerned they are not “banned”, so attempting to unban them via fail2ban-client commands fails. this also means that fail2ban can ban ips that are already blocked via previously loaded iptables commands.

    i’ve come up with the following changes to iptables-repeater.conf to implement a simple whitelist and to properly ban ips at fail2ban restart.

    “`
    — action.d/iptables-repeater.conf.orig 2014-10-09 09:36:30.000000000 -0400
    +++ action.d/iptables-repeater.conf 2014-10-09 15:10:47.773573056 -0400
    @@ -13,7 +13,7 @@
    iptables -A fail2ban-REPEAT- -j RETURN
    iptables -I INPUT -j fail2ban-REPEAT-
    # set up from the static file
    – cat /etc/fail2ban/ip.blocklist. |grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do iptables -I fail2ban-REPEAT- 1 -s $IP -j DROP; done
    + cat /etc/fail2ban/ip.blocklist. |grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do fail2ban-client set ssh-repeater banip $IP; done

    # Option: actionstop
    # Notes.: command executed once at the end of Fail2Ban
    @@ -37,7 +37,7 @@
    # unix timestamp of the ban time
    # Values: CMD
    #
    -actionban = iptables -I fail2ban-REPEAT- 1 -s -j DROP
    +actionban = ! grep -Fq /etc/fail2ban/ip.whitelist && iptables -I fail2ban-REPEAT- 1 -s -j DROP
    # also put into the static file to re-populate after a restart
    ! grep -Fq /etc/fail2ban/ip.blocklist. && echo ” # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.
    “`

  30. i’ve implemented your solution today on a centos 6.5 host and it’s working nicely. thanks for putting this together.

    i had written something similar myself in the past but it broke over time and i haven’t maintained it.

    i’ve also used the filter.d/recidive.conf in standard fail2ban installations that does some basic progressive banning of repeat offenders.

    it would be nice to see your scheme, maybe augmenting recidive, make it into the standard fail2ban releases.

    suggestions …

    – add a whitelist capability. as i learned during testing today, once my ip address ends up
    in the logs it doesn’t matter how many times i do ‘fail2ban-client set ssh-repeater unbanip’, the repeater jail is always going to ban it again until the findtime has expired or logs are cleared.

    – ban and whitelist based on ip ranges and change the simple greps in action.d/iptables-repeater.conf to something more sophisticated that can match against these ranges.

    if i get anywhere with adding these items myself i’ll feed it back to you.

    cheers!

    1. @tp – I’ve always used the built-in fail2ban whitelists, rather than re-inventing that functionality at the individual jail level.

  32. Thanks for the solution, it works good.

    I’m using it on a Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-64-generic x86_64) and one old Ubuntu 11.10 (GNU/Linux 3.0.0-12-server x86_64). I had to do some adjustments.

    1. date string
    If you copy iptables-repeater.conf from the link above there are not “‘” (ascii 39) surrounding the date string for date, it’s another character that is converted to “.” when I pasted it in unix.
    And I also had to change the date string from “date ‘+%%Y-%%m-%%d %%T'” to “date +’%%Y-%%m-%%d %%T'” to get it to work.

    2. duplicated repeater bans, and “returned 100” in log
    I got duplicated repeater bans after each restart of fail2ban so I added a prefix grep in the “actionban”, the first line for that rule:
    actionban = ! grep -Fq /etc/fail2ban/ip.blocklist. && iptables -I fail2ban-REPEAT- 1 -s -j DROP || /bin/true
    # also put into the static file to re-populate after a restart
    ! grep -Fq /etc/fail2ban/ip.blocklist. && echo ” # fail2ban/$( date +’%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist. || /bin/true
    I also added “|| /bin/true” to avoid error message in fail2ban.log that the action “returned 100” (I think that was only necessary on the older server, it was working but leaving the error messages).

    I tried to understand what Ian says in 1. but failed, can you try to explain more?

  33. Have you tried fail2ban’s recidive filter? How does that compare to your filter in stopping recidivism?

    1. Hi, Ben – I realize this comment was in the queue a while. I had not tried the recidive filter until you mentioned it, and wanted to give it a try before commenting. After a few weeks, I can say that I believe this is a better approach than what I’ve outlined here, and I’ve changed out my REPEATER-based blocks in favor of the built-in recidive filter.
      It still does not directly handle persistent blocks across reboots that I am aware, but it should be a more appropriate baseline on which to build that functionality.
      I won’t have a chance to properly test and validate that solution for quite some time, but will update this page when/if I am able.

      Thanks for the heads up!

  34. It appears that the repeat offender solution does not take the “ignoreip = ” into account. Anyone any comments or solutions on that?

    1. I have not observed this at all – an IP in the ignoreip list should never trip the block in the first place. I have tried and could not get this to happen.

  35. Do mind that you can get a 100 error in the fail2ban log if the name in the jail.local eg
    action = iptables-repeater[name=postfix] is to long. Don’t know the limit or details but my name was postfix-connection and that gets fail2ban-REPEAT prefixed so it was quite long.

    1. Good call, @AV – you’re correct. iptables has a 30-character limit on the chain name. You could trim the “-REPEAT” suffix if needed, though.

  36. Thanks for a great solution and post.

    It might be good to use jail.local as a file name, per the recommendation at the top of jail.conf

    1. That’s a great point about jail.local. I believe I originally started this idea before that recommendation was in the file (possibly before it existed?), but I should really port the configuration forward to the jail.local file.

  37. This was exactly what I searched for. I am using fail2ban under Debian 7 to block some annoying spammer that do only try out passwords via SMTP ports every 5 minutes and do not fall into my standard SASL fail2ban configuration.

    [sasl-repeater]
    enabled = true
    filter = sasl
    action = iptables-repeater[name=sasl]
    sendmail-whois[name=SASL-repeater, dest=root, sender=root]
    logpath = /var/log/mail.warn
    maxretry = 15
    findtime = 31536000
    bantime = 31536000

    Unfortunately I do not get the date notation working with this version of fail2ban and I added a whitelist because I had a false positive for myself once when changing my email passwords and I had to change them on all my devices after.

    My /etc/fail2ban/action.d/iptables-repeater.c now contains:

    actionban = ! grep -Fq /etc/fail2ban/ip.whitelist. && iptables -I fail2ban-REPEAT- 1 -s -j DROP
    # also put into the static file to re-populate after a restart
    ! grep -Fq /etc/fail2ban/ip.blocklist. && echo ” # fail2ban/$(date): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.

  38. 1. Long bans for repeat offenders

    What I do is set up a jail looking at /var/log/fail2ban.log itself. When the same IP address gets banned more than a couple of times by other filters, it triggers this jail which gives it a much longer ban.

    Because it would be unbanned when the most recent short ban is ended, it uses hosts.deny rather than iptables.

    It saves setting up more log files and the situation with restarting: the fail2ban.log will still have the necessary info to ban it for a long time.

    2. Protecting WordPress from bruteforcing logins

    Just set up a jail looking in the webserver access logs for anyone accessing wp-login.php say more than five times in a few minutes.

    There’s no need to install plugins creating / adding to other log files with this way.

    3. Protecting WordPress from spammers

    This is the one that needs a plugin 🙂 There’s one which writes to a log every time a comment is tagged as spam, either by Akismet (or other anti-spam plugin) or manually.

    Set up a jail watching it, and ban accordingly.

    1. Hi Ian – great info, thanks.
      There are some conditions in my environment that would preclude perfect parallel functionality (rotating the fail2ban log itself, for #1), but you’ve got a very elegant solution!
      I still contend the plugin for WordPress login bruting is more favorable, though, as I have anywhere between 60-100 http access log files (one per vhost), and I’d want to flag *only* on failed logins, not just page loads.

      Still, these are great customizations on the idea – thanks!

      -Phil

  39. Joel try this

    actionban = iptables -I fail2ban-REPEAT- 1 -s -j DROP
    if ! grep -Fq /etc/fail2ban/ip.blocklist.; then echo ” \t\t\t# fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.; fi

  40. I kept getting error 100 in my /var/log/fail2ban.log whenever it tried to add an IP, so my blocklist was never built. I had a heck of a time trying to figure it out even with fail2ban running as a user with the correct privileges, even root wouldnt work. Until I stumbled across this advisory http://lwn.net/Articles/440210/ about selinux and fail2ban – you apparently have to place your blocklist in /var/lib/fail2ban to able to write to it. .

    here’s my revised actionstart and actionban:

    actionstart = iptables -N fail2ban-REPEAT-
    iptables -A fail2ban-REPEAT- -j RETURN
    iptables -I INPUT -j fail2ban-REPEAT-
    # set up from the static file
    #prod
    cat /var/lib/fail2ban/ip.blocklist. |grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do iptables -I fail2ban-REPEAT- 1 -s $IP -j DROP; done

    actionban = ! grep -Fq /var/lib/fail2ban/ip.blocklist. && echo ‘ # fail2ban ‘”$(date)”‘ auto-add for repeat offender’ >> /var/lib/fail2ban/ip.blocklist.;iptables -I fail2ban-REPEAT- 1 -s -j DROP

  41. It needs the ip and name variables in the grep command too – grep-Fq ip /etc/fail2ban/ip.blocklist.name && … with the greater thans and less thans

  42. Thanks for the heads up, Justin! I’ve edited Jan’s comment to show the replacement strings.
    Jan – great improvement – I’ll try to integrate it above when I have a chance to test a few things.

  43. Excellent amendment Jan, but to get it to work properly on my Debian 7 box I had to amend it to add in the ip and name variables. I suspect this comment feature strips anything that looks like HTML. Phil may want to incorporate this amendment into his code, as it is an excellent improvement.

  44. Hi,
    Thank you for such a great solution. I restart my linux box one a week. When I do that I saw there are repeating ip’s populated in the blocklist file.

    So I have changed this in the action/iptables-repeater.conf file

    echo “<ip> # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.<name>

    with this

    ! grep -Fq <ip> /etc/fail2ban/ip.blocklist.<name> && echo “<ip> # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.<name>

  45. I gone through, followed the guide and half an hour later I have 30 IP’s in my blocklist.ssh and my /var/log/secure is finally calm and peaceful. This is a very good guide, easy to follow and understand. Thank you

  46. Hello All,

    Is there any chance I can make the action populate the iptables on an external firewall??

    Basically I have a UTM as firewall, then I have an asterisk server. How can I use the fail2ban on the asterisk to populate the iptables on the UTM?

    Regards,
    Roddy

    1. @Roddy: I think that would be a pretty elaborate setup, but you’d need to configure/permit non-interactive logins from the Asterisk server to the UTM firewall (e.g. ssh key-based authentication or similar). Then, instead of just running the relevant “iptables” command as I described, you’d run the command needed to run the command on the firewall.
      I’m not directly familiar with the UTM firewall, but in theory the above should work. Good luck!

      -Phil

  47. Oh what a relief. ipdeny seems to have gone bye bye with most of it’s country files empty so I started looking for a better mousetrap. And here it is.

    Rebuilt my personal mail/web/firewall machine last week and went live 48 hours ago. Implemented this 15 min ago and first run produced 8 hits of persistent ssh hack attempts.

    Thanks Phil.

  48. got it ! if you don’t have wordpress, then remove the second portion ! [wordpress-auth-repeater]

  49. Guys, I’m a complete Neub….
    I get errors: no such file wordpress-auth.conf and wordpress-auth.local
    how do i fix this ?

  50. Thanks! Very useful information and allready in use! 🙂 CHINANET is causing grayhairs with their bruteforce since 3 weeks now.

    1. That would definitely work, but IP geolocation is not accurate enough for such blanket blocks, in my opinion. Others may have operational needs to allow connections to certain countries where malicious activity happens to originate.
      Due to those shortcomings, I strongly recommend an observation-based active defense mechanism such as fail2ban – it’s far better to stop those that have conducted malicious activity than to make broad policy decisions that are based on flawed technical data (e.g. IP geolocation).

  53. If you want to use shorewall just replace the iptables command in the actionstart/actionban with “shorewall drop “.

    Phil for actionunban you can leave it blank rather than specifying a dummy executable. You do like your grep/awks :-). Alternative (assuming it doesn’t get filtered).

    while read IP ; do iptables -I fail2ban-REPEAT- 1 -s ${IP%%#*} -j DROP; done < /etc/fail2ban/ip.blocklist.

  54. Phil, This looks very promising and I’m eager to give your script a try. The current banaction in my jail.local is shorewall, which doesn’t seem to add chains to iptables. Will a mix of shorewall and iptables from your setup here jive well together?

    Thanks much.

    1. I have not used Shorewall myself, but it does use iptables under the hood. I would *assume* that as long as you start fail2ban after Shorewall, that the rules would insert in the appropriate locations. However, Shorewall might not be aware of those rules, so manual additions via the iptables command line might cause unpredictable behavior.
      However, it looks like there is a command line interface to Shorewall (http://www.shorewall.net/manpages/shorewall.html), so you may be able to modify the concept outlined here to suit your needs.
      Good luck – I’m interested to hear how you solve it!

  55. Thanks Phil 🙂
    I´ve used fail2ban sometime, but this permanent ban will be very usefull, there´s A LOT of chinese IP´s hammering.

    Looks like it does work now! Awesome 🙂

    /etc/fail2ban$ sudo cat ip.blocklist.ssh
    61.174.51.xx # fail2ban/2014-01-20 19:54:30: auto-add for repeat offender
    61.174.51.xx # fail2ban/2014-01-20 19:56:17: auto-add for repeat offender

  57. How this can be done with Ubuntu?
    There is no /var/log/secure

    But still, this is awesome!

    regards
    n00b 🙂

  58. I just followed your guide. Somehow fail2ban fails to restart after I’ve done the 2 modifications you suggested, however stopping and starting it works. Using Debian 7.1. Any idea?

  59. Phil,
    Thank you very much for this article. I had a pesky ip from China that was determined to continue trying to enter my ssh server. I followed this article in a debian wheezy install and it stopped them in their tracks.
    Thanks again.

  60. Fail2ban takes over your iptables completely. You lose all freedom while using it, which is why you should not be using it.

    First, you can’t explicitly ban an IP in fail2ban, it has to match a rule in a defined log.
    Second, if you block an IP manually in iptables, fail2ban will make sure to wipe that rule clean on the next restart, even if it’s saved in your sysconfig.

    So that means that it will secure against recurrent attacks but prevent you from having a secured system. Better use CSF to catch the problem as it happens instead of parsing log files.

    1. Sounds like your config needs a little more work. I have several very complex setups on multiple systems. I’ve not experienced by of the problems you describe.

  61. Phil,

    I must admit that there is a nuance to your solution that I had not considered and that is fail2ban “remembering” already-read log entries for as long as “findtime” (assuming it does so, and I would hope it does). So yes, as long as you don’t restart fail2ban, you would actually be operating on logs for longer than their rotation period.

    What’s funny is that I had already been thinking about a solution where fail2ban writes the actual log entries to a file (i.e. a journal) that cause temporary bans so that instead of just having a record of previous bans to look at, it had actual log entries. This actually seems like a feature that would do well to bake right into fail2ban.

  62. But the problem is that your “1 year” find time is being applied to /var/log/secure which is always just the “current” value of that file. If it’s on a 1 week (for example) rotation, you are only ever looking for repeaters in a 1 week window.

    FWIW, I’ve implemented the longer-window version of your concept here. It’s done by adding an entry to a log every time a temporary ban is applied and then looking for repeat offenders in that log maintained by fail2ban’s temporary bans.

    1. I see what you’re saying. This functionality all assumes fail2ban runs continuously for the duration of the find time. From my brief research, it looks like v0.8.6 added or improved functionality related to the re-banning of blocked IPs on restart – assuming the log data is available to support such a move.
      While a procedural shortcoming, I have found any IP that trips the perma-ban usually does so in rapid succession – certainly within the time frame that fail2ban continuously operates. Sorry that it seems I misunderstood your initial question – it’s just something that hasn’t been an issue on my systems so far. I like the elegance of your solution to use a temporary ban log file that you then examine for the permaban decision.

  63. Phil,

    I guess I’m missing where fail2ban is keeping it’s historical records. Both of the jail.conf entries for looking for repeat offenders within a year are using the operational log /var/log/secure.

    1. The solution above will track repeaters in the /etc/fail2ban/ip.blocklist.* files as they are identified. Then, when fail2ban restarts, it parses the contents of those files to populate iptables with previously-blocked IPs, while watching the designated log files for new offenders.

  64. Doesn’t this require that you keep a years worth of entries in /var/log/secure though? Clearly that doesn’t scale, especially for much busier log files like /var/log/messages.

    1. Actually, that’s the benefit of a solution like this, Brian. Fail2ban keeps the historical records in its own files, so you can clear logs on the operational host much more frequently.

    1. If I restart fail2ban, it rebans hosts which were already banned if a large enough findtime still qualifies the log entries which lead to that ban. Your action then writes another line for each such host even if it already was added to the file.

      1. Thanks for the great post, exactly what I was looking for.

        I can confirm the behavior of duplicate entries described by Jan.

        To overcome the problem, I adjusted the ‘actionban’ rule:

        actionban = iptables -C fail2ban-REPEAT- -s -j DROP && return 0
        ! grep -Fq /etc/fail2ban/ip.blocklist. && echo ” # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.
        iptables -I fail2ban-REPEAT- 1 -s -j DROP

        Besides checking whether a rule already exists (coming from the blocklist file), I switched the commands to have the result from iptables -I as exit status. Otherwise fail2ban adds ‘returned 100’ warnings the the IP is already present in the blocklist.

  67. Recommend using the recidive filter in the jail configuration to grab all repeat bans across all other jails.

    Also wishing people could contribute these kinds of things back to the fail2ban development effort rather than relying on users to search the entire internet for solutions.

  68. Nice one! But I observed that the action generates duplicates in the blocklist. The following grep checks for an existing entry prior to adding it:

    grep /etc/fail2ban/ip.blocklist. || echo ” # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.

  69. Pingback: How to install fail2ban on Ubuntu 12.04 for SSH and WordPress | Sweet Baby Talk Honey Boo Boo

  71. Thanks, Kevin – I’ll update the article body to reflect that now. (I see what you meant even though WP ate some of the text with format interpretation.)

  72. Also, you have an error in the following section:

    actionstop = iptables -D INPUT -j fail2ban-
    iptables -F fail2ban-
    iptables -X fail2ban-

    Should read:
    actionstop = iptables -D INPUT -j fail2ban-REPEAT-
    iptables -F fail2ban-REPEAT-
    iptables -X fail2ban-REPEAT-

  73. To avoid SELinux issues, changing the directory to one that fail2ban has default rights to write to, like /var/run/fail2ban/…, makes this process smoother.

    I ended up using /var/run/fail2ban/permblocks/

  74. It was selinux. Needed to grep fail2ban /var/log/audit/audit.log and pipe the output into audit2allow to create an selinux policy to allow the file write/append access.

    1. Excellent! I’m glad to hear it was a straightforward fix. The audit2allow tool has definitely taken a lot of frustration out of using selinux – hopefully I will have some time to focus on using it operationally soon.
      Thanks for following back up!

  75. @Jeff – selinux may be causing that error. I am not as versed in selinux as I’d like to be, but you may need to assign a role or context for the ip.blocklist.ssh file that allows the root user to write to it. Are you able to temporarily disable selinux with the “setenforce” command to test the block process?

  76. I’m also using Centos, but I get the following error when it tries to write to the file:
    echo “1.235.138.249 # fail2ban/$( date ‘+%Y-%m-%d %T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist.ssh returned 100

    Any ideas? Would selinux stop this from working?

  77. Thanks for the sharp eyes, Avi – I’ve corrected the formatting problems in the text.

    I am still working on the duplicate entry problem, but since this configuration is back to blocking all ports, I am pretty sure there shouldn’t be a way for that to happen. I’ll keep an eye on my logs to try and get a better handle on that.

  78. I’m subscribed. Good update. Here are a couple of comments.

    . looks like some conf to html error but the text looks like fail2ban-REPEAT- rather than fail2ban-REPEAT-. The download file is fine.

    Also, I did a quick glance through. I couldn’t figure out how you avoided duplicate iptables entries when reading ip.blocklist file and you have 2 repeater profiles.

  79. I’m not sure if you receive a notification when the post itself is updated, but I’ve made some updates to the post and linked files. Details are in the text – hope you find them helpful. Thanks for the discussion thread as well – it helped me to refine the approach and configuration.

  80. Well, I’m glad to hear that it’s working again. I’m not sure whether NTP plays into the process, but un-sync’ed time is often a problem in odd and unpredictable ways.

    I’ll send you an email momentarily. Just let me know what you’d like to edit and I’ll make the change.

  81. not sure. whatever it is, I’ve copied everything relevant above in my comment. After not getting it to work, I finally purged and restarted the entire thing and it seems everything is working again.

    One thought was it was because the time on my logs did not match up with my NTP date (syslogd wasn’t restarted after clock sync). I got everything synced, deleted the old logs with incorrect time and restarted fail2ban.

    Nothing else makes sense right now. Thanks

    ps. Phil. can you unicast me your private email. I have a request to edit my comment.

  82. Interesting. Just to clarify, the permaban filter+action pair was active during the entire period when those log entries occurred?

    I am not sure what might be causing both jails to stop blocking. You said that something had changed in your configs – what was the nature of those changes?

  83. @Phil. I ran the regex to show that the conf file was matching entries. I’ve also added the iptables-knownbad file that shows the action to blocks IPs and populate the ip.blocklist file. Finally, you can also see jail.local for both ssh and ssh-repeater. Despite several repeat failed ssh logins, you can see that neither ssh nor ssh-repeater was executed.fail2ban logs show nothing but status shows up as running.

  84. @Ben – are you testing with fail2ban-regex as well, or does this happen when you start the jail?

  85. @Avi: I believe that the fail2ban-regex utility only tests the regexes, and does not actually perform any blocks based on what it parses. Since no actions are taken, there would be no activity in the ip.blocklist file.

  87. I’ve got the same problem that AVI has. I get no errors. Over 200+ entries found. but they are not added to /etc/fail2ban/ip.blocklist. Actually, ip.blocklist isn’t even created. Nor are the found entries added to the iptables chain. The chain does exist in iptables.

    Thoughts?

  88. Guys. Need some help here. f2b is running but not blocking anything. Not sure what I changed that is causing this but I can’t spot it. Hoping you can pick up on what I did wrong.

    [ssh]
    enabled = true
    port = ssh
    filter = sshd
    logpath = /var/log/auth.log
    maxretry = 5
    findtime = 3600
    bantime = 3600

    [sshd-repeater]
    enabled = true
    filter = sshd
    action = iptables-knownbad[name=BAN, port=”all”]
    # sendmail-whois[name=ssh, dest=root, sender=fail2ban@mail.com]
    ignoreip = 127.0.0.1 127.0.0.0/8
    logpath = /var/log/auth.log
    maxretry = 25
    findtime = 604800

    www:/etc/fail2ban# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
    /usr/share/fail2ban/server/filter.py:442: DeprecationWarning: the md5 module is deprecated; use hashlib instead
    import md5

    Running tests
    =============

    Use regex file : /etc/fail2ban/filter.d/sshd.conf
    Use log file : /var/log/auth.log

    …. (a lot of stuff here)…

    58.17.89.122 (Fri Apr 26 05:27:46 2013)
    58.17.89.122 (Fri Apr 26 05:27:49 2013)
    58.17.89.122 (Fri Apr 26 05:27:52 2013)
    58.17.89.122 (Fri Apr 26 05:27:56 2013)
    58.17.89.122 (Fri Apr 26 05:27:59 2013)
    58.17.89.122 (Fri Apr 26 05:28:03 2013)
    58.17.89.122 (Fri Apr 26 05:28:06 2013)
    58.17.89.122 (Fri Apr 26 05:28:10 2013)
    58.17.89.122 (Fri Apr 26 05:28:14 2013)
    58.17.89.122 (Fri Apr 26 05:28:17 2013)
    58.17.89.122 (Fri Apr 26 05:28:20 2013)
    58.17.89.122 (Fri Apr 26 05:28:23 2013)
    58.17.89.122 (Fri Apr 26 05:28:27 2013)
    58.17.89.122 (Fri Apr 26 05:28:31 2013)
    58.17.89.122 (Fri Apr 26 05:28:34 2013)
    58.17.89.122 (Fri Apr 26 05:28:38 2013)
    58.17.89.122 (Fri Apr 26 05:28:41 2013)
    58.17.89.122 (Fri Apr 26 05:28:45 2013)
    58.17.89.122 (Fri Apr 26 05:28:48 2013)
    58.17.89.122 (Fri Apr 26 05:28:52 2013)
    58.17.89.122 (Fri Apr 26 05:28:55 2013)
    58.17.89.122 (Fri Apr 26 05:28:58 2013)
    58.17.89.122 (Fri Apr 26 05:29:02 2013)
    58.17.89.122 (Fri Apr 26 05:29:05 2013)
    58.17.89.122 (Fri Apr 26 05:29:09 2013)
    58.17.89.122 (Fri Apr 26 05:29:12 2013)
    58.17.89.122 (Fri Apr 26 05:29:16 2013)
    58.17.89.122 (Fri Apr 26 05:29:20 2013)
    222.42.146.13 (Fri Apr 26 07:54:19 2013)
    222.42.146.13 (Fri Apr 26 07:54:27 2013)
    222.42.146.13 (Fri Apr 26 07:54:31 2013)
    [6]
    [7]
    [8]
    [9]
    [10]

    Date template hits:
    37119 hit(s): MONTH Day Hour:Minute:Second
    0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
    0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
    0 hit(s): Year/Month/Day Hour:Minute:Second
    0 hit(s): Day/Month/Year Hour:Minute:Second
    0 hit(s): Day/Month/Year Hour:Minute:Second
    0 hit(s): Day/MONTH/Year:Hour:Minute:Second
    0 hit(s): Month/Day/Year:Hour:Minute:Second
    0 hit(s): Year-Month-Day Hour:Minute:Second
    0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
    0 hit(s): Day-Month-Year Hour:Minute:Second
    0 hit(s): TAI64N
    0 hit(s): Epoch
    0 hit(s): ISO 8601
    0 hit(s): Hour:Minute:Second
    0 hit(s):

    Success, the total number of match is 989

    However, look at the above section ‘Running tests’ which could contain important
    information.

    As you can see, there are 28 entries for 58.17.89.122 but it did not make the ip.blocklist file.

    www:/etc/fail2ban# cat action.d/iptables-knownbad.conf | grep -v ‘^#’ | egrep -v “^[[:space:]]*$”
    [Definition]
    actionstart = sleep $( perl -e ‘print rand(10);’ )
    iptables -N f2b-
    iptables -A f2b- -j RETURN
    iptables -I INPUT -j f2b-
    # set up from the static file
    cat /etc/fail2ban/ip.blocklist |grep -v ^\s*#|awk ‘{print $1}’ | while read IP; do iptables -I f2b- 1 -s $IP -j DROP; done
    actionstop = sleep $( perl -e ‘print rand(10);’ )
    iptables -D INPUT -j f2b-
    iptables -F f2b-
    iptables -X f2b-
    actioncheck = iptables -n -L INPUT | grep -q f2b-
    actionban = iptables -I f2b- 1 -s -j DROP
    # also put into the static file to re-populate after a restart
    echo ” # fail2ban/$( date ‘+%%Y-%%m-%%d %%T’ ): auto-add for repeat offender” >> /etc/fail2ban/ip.blocklist
    actionunban = /bin/true
    [Init]
    name = KNOWN_BAD

    www:/etc/fail2ban# iptables -L
    Chain INPUT (policy DROP)
    target prot opt source destination
    f2b-BAN all — anywhere anywhere
    f2b-RC tcp — anywhere anywhere multiport dports www,https,smtp,submission,pop3,pop3s,imap2,imaps,sieve
    f2b-SMTP tcp — anywhere anywhere multiport dports www,https,smtp,submission,pop3,pop3s,imap2,imaps,sieve
    f2b-ssh tcp — anywhere anywhere multiport dports ssh
    f2b-IMAP tcp — anywhere anywhere multiport dports www,https,smtp,submission,pop3,pop3s,imap2,imaps,sieve
    f2b-BAN all — anywhere anywhere
    ACCEPT all — anywhere anywhere
    REJECT all — anywhere loopback/8 reject-with icmp-port-unreachable
    ACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
    ACCEPT tcp — anywhere anywhere tcp dpt:www
    ACCEPT tcp — anywhere anywhere tcp dpt:https
    ACCEPT tcp — anywhere anywhere tcp dpt:smtp
    ACCEPT tcp — anywhere anywhere tcp dpt:submission
    ACCEPT tcp — anywhere anywhere tcp dpt:ssmtp
    ACCEPT tcp — anywhere anywhere tcp dpt:pop3s
    ACCEPT tcp — anywhere anywhere tcp dpt:sieve
    ACCEPT tcp — anywhere anywhere tcp dpt:imaps
    ACCEPT tcp — anywhere anywhere tcp dpt:ssh
    ACCEPT icmp — anywhere anywhere icmp echo-request
    LOG all — anywhere anywhere limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: ‘
    DROP all — anywhere anywhere

    Chain FORWARD (policy DROP)
    target prot opt source destination
    DROP all — anywhere anywhere

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    Chain f2b-BAN (2 references)
    target prot opt source destination
    DROP all — h31-3-245-178.host.redstation.co.uk anywhere
    DROP all — 226-121-162-69.reverse.lstn.net anywhere
    RETURN all — anywhere anywhere
    RETURN all — anywhere anywhere

    Chain f2b-IMAP (1 references)
    target prot opt source destination
    RETURN all — anywhere anywhere

    Chain f2b-RC (1 references)
    target prot opt source destination
    RETURN all — anywhere anywhere

    Chain f2b-SMTP (1 references)
    target prot opt source destination
    RETURN all — anywhere anywhere

    Chain f2b-ftpd (1 references)
    target prot opt source destination
    RETURN all — anywhere anywhere

    Chain f2b-ssh (1 references)
    target prot opt source destination
    RETURN all — anywhere anywhere

  89. First It needs to load fail2ban (fail2ban-client reload) and then load the others rules …
    It is tricky ..

  90. Fixed …

    Chain fail2ban-KNOWN_BAD (1 references)
    pkts bytes target prot opt in out source destination

    1104 115K RETURN all — * * 0.0.0.0/0 0.0.0.0/0

    1. Great! I’m glad to see it’s working. What was the problem you found or the fix that took care of it?

  91. It seems that would happen if there was no fail2ban-ssh chain on the INPUT table. This would be expected if the “actionstart” commands did not execute or complete successfully, as you’ve previously indicated.

  92. I changed something and I see this in the fail2ban log. Any ideas?

    2013-04-24 00:18:39,631 fail2ban.actions.action: ERROR iptables -D INPUT -p tcp –dport ssh -j fail2ban-ssh
    iptables -F fail2ban-ssh
    iptables -X fail2ban-ssh returned 100

  93. Hello,

    I have

    root@server:~# iptables -L -n -v

    ….

    Chain fail2ban-apache (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-apache-badbots (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-apache-nohome (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-apache-noscript (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-apache-overflows (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-ssh (1 references)
    pkts bytes target prot opt in out source destination

    Chain fail2ban-ssh-ddos (1 references)
    pkts bytes target prot opt in out source destination

    …..

    But I do not have Chain fail2ban ssh-repeater (?)

    root@server:~# fail2ban-client status
    Status
    |- Number of jail: 8
    `- Jail list: apache-noscript, apache-badbots,
    apache-nohome, ssh-ddos, apache-overflows, ssh-repeater, apache, ssh
    root@server:~#

    Best Regards,

  95. Avi – I thought about this and if you’re blocking offenders on all ports, there should theoretically never be a case where a blocked from ssh would be able to hit postfix. Since the IP is blocked, it shouldn’t have the opportunity to brute another service. Though I haven’t tested it myself, I can’t think of a way that would happen.

    Agree that blocking all ports seems best, but I have an idea that may help accomplish that more elegantly. Have to carve off the time to work on it, however. I will update this post if I get something working.

    Lin – I’m afraid I’m not sure what the problem may be. If grep’ing for “fail2ban” yields no results, that means none of your iptables-based jails have initialized. That seems like there is something going on with fail2ban itself.

  96. I do agree that you probably want to block all ports for repeat offenders even if they are only hitting sshd. I see Phil updated the iptables-knownbad file but I’ve keep it more like the original.

    However, If I repeat this for say postfix and use the same knowbad config file, I end up with duplicate blocks for the same IP. This is because we read the blocklist file for each one. I ended up creating another file ‘iptables-knowbad-alt’ which does not read the blocklist file a second time and polluting iptables with duplicate entries. However, I haven’t solved adding duplicate entries. So if someone tries to do this both and ssh and postfix, I end up with the same IP being added to the blocklist file.

    Adding the dport solves this problem since you only block specific ports. But I would like to block all ports for known offenders.

    Thanks

  98. root@server:~# iptables -L -n -v | grep ‘repeater’
    root@server:~#

    The chain does not appear. No errors are shown. (?)

  99. I have found that it may take several seconds for the chain to appear, due to concurrency problems with iptables commands across multiple jails. Not sure if that’s what you’re seeing, but it may be worth looking into.

  100. Hello,

    1. ssh-repeater is enabled

    root@server:~# fail2ban-client status
    Status
    |- Number of jail: 15
    `- Jail list: …., ssh-repeater, …..
    root@server:~#

    but I do not see any Chain fail2ban-ssh-repeater
    # iptables -L -n -v

    2. Newest modifications are

    while read IP; do iptables -I fail2ban- 1 -s $IP -j DROP; done
    changed
    while read IP; do iptables -I fail2ban- 1 -s $IP -p tcp –dport -j DROP; done

    actionban = iptables -I fail2ban- 1 -s -j DROP
    changed
    actionban = iptables -I fail2ban- 1 -s -p tcp –dport -j DROP

    action = iptables-knownbad
    changed
    action = iptables-knownbad[name=KNOWN_BAD, port=ssh]

    Nevertheless Most of ssh ip offenders are sending SPAM as well.
    DDOS and SLOW DDOS 80/443

    Best Regards,

  101. Originally, it blocked all ports, but I reduced that to just SSH this morning based on a comment above. You’d only need to alter the two iptables DROP commands in the iptables-knownbad.conf script to broaden that scope.

    As for blocking after failures on other services, I also updated the article to reflect that you’d need to duplicate the appropriate service stanza in the jail.conf file, modify the findtime value and using the iptables-knownbad action (with necessary parameters).

    As with any system-level customization, there is seldom a “one-size-fits-all” solution, so a some tweaking to fit your requirements would be needed.

  102. >How to use this for blocking other ports (http login failures etc)
    I think that it should be improved for blocking all ports to users misusing SSH

    Best Regards,

  103. Phil, thanks for this great article and timing. I implemented fail2ban and was missing this and ran into your article on a search. One thing I wanted to confirm. It seems like your action.d/knownbad file is generic for all protocols, even though your jail.local file seems to suggest ssh. It seems like your implementation here is to block all ports for users misusing SSH. I think its a fine implementation, but you should highlight this in your article. You should also suggest
    . Changing the log file depending on linux flavors (for example, I’m on debian and the file to check for ssh is /var/log/auth.log
    . How to use this for blocking other ports (http login failures etc)

    Can you also explain this statement you make
    “This is three consecutive “basic” blocks, which trigger after seven failed attempts”

    Looking through the code I couldn’t figure out where this happened. Can you elaborate on this please.

    Thanks much

    1. Hi, Avi – I’m glad this was helpful.

      Yes, you’re correct that while brute force attacks are detected from SSH logs, this will block all traffic from offending IPs. I’ve updated the text and files to perform a block on the SSH port only. Also good point on log location – this was adopted from a CentOS installation. I’ll add a note about adopting the jail.conf section to handle other authentication failures – I was actually about to do this for the SMTP server on one of my systems as well.

      In terms of the block behavior, the SSH defaults distributed with fail2ban are to block after seven failed attempts within five minutes. Therefore, by blocking after 21 failed attempts (maxretry = 21) within one year (findtime = 31536000 seconds), the third time an IP triggers a block, the ssh-repeater stanza kicks in and they’ll get the permaban.

      I hope that clarifies how it’s working a bit. I’ll update the text to reflect that a little more clearly.

  104. No errors and Now It is being tested
    I’ll post results …

    You should consider adding the file for download it

    Best Regards,

  105. Gah.. More WordPress formatting problems. I just did a vimdiff on this and my live version, and it should be all corrected now.

  106. Not working yet …

    root@server:~# fail2ban-client -vvv -x start
    ..
    ..
    ..
    DEBUG Reading /etc/fail2ban/action.d/iptables-knownbad
    DEBUG Reading files: [‘/etc/fail2ban/action.d/iptables-knownbad.conf’, ‘/etc/fa
    il2ban/action.d/iptables-knownbad.local’]
    ERROR Error in action definition iptables-knownbad
    DEBUG ‘%’ must be followed by ‘%’ or ‘(‘, found: ‘%T\’ ): auto-add for repeat o
    ffender” >> /etc/fail2ban/ip.blocklist’
    ERROR Errors in jail ‘ssh-repeater’. Skipping…
    root@server:~#

  107. Hello, Lin – great catch, and I’m sorry for the oversight. I corrected the file above, which should fix your problem. Again, my apologies.

  108. Hello,

    I implemented it but ….

    root@server:~# /etc/init.d/fail2ban restart
    Restarting authentication failure monitor: fail2ban failed!
    root@server:~#

    root@server:~# fail2ban-client -vvv -x start
    ..
    ..
    ..

    DEBUG Reading /etc/fail2ban/action.d/iptables-knownbad
    ERROR Error in action definition iptables-knownbad
    DEBUG File contains no section headers.
    file: /etc/fail2ban/action.d/iptables-knownbad.conf, line: 10
    “actionstart = sleep $( perl -e ‘print rand(10);’ )\n”
    ERROR Errors in jail ‘ssh-repeater’. Skipping…
    root@server:~#

  109. Your article got me well on the way to the solution, so thanks as well. Let me know what your results are – I’d be interested to hear them!

  110. Hi Phil. Thanks for the comment on my blog. Your solution sounds interesting, I will give it a shot on my systems 🙂 Cheers, Lukas

Comments are closed.