Fighting SMTP AUTH Brute Force Attacks with iptables string and recent

Brute Force Attacks on SMTP and SUBMISSION are on the rise. Here's a proof of concept that uses iptables Recent String module to counteract them. I tested it with Postfix version 2.11 on Ubuntu Trusty.


Be aware, that I did not tested this firewall rules on a real production server. Also be warned that using the iptables string module maybe a performance overkill, however it should be faster then using fail2ban for the job, but you may consider using both methods.

The iptables recent string module matches strings. Based upon these matches iptables may trigger actions, e.g. block the IP causing the match. My idea was to let the string filter look out for SMTP AUTH errors and then let it block the client.

To test this I simulated an SMTP AUTH session that failed:

# telnet 25
Connected to
Escape character is '^]'.
250-SIZE 10240000
250 DSN
334 VXNlcm5hbWU6
334 UGFzc3dvcmQ6
535 5.7.8 Error: authentication failed:

With the error code 535 5.7.8 Error: authentication failed: from that session I created the following iptables rules:

iptables -I OUTPUT -p tcp --sport 25 -m string --to 90 \
     --string "535 5.7.8 Error: authentication failed:" --algo bm -m recent \
     --rdest --name SMTP_AUTH_ERROR --set
iptables -I OUTPUT -m recent --name SMTP_AUTH_ERROR --rdest \
     --rcheck --seconds 180 --hitcount 3 -j DROP
iptables -I OUTPUT -p tcp --sport 25 -m string --to 90 \
     --string "535 5.7.8 Error: authentication failed:" --algo bm -j LOG \
     --log-level info --log-prefix "SMTP_AUTH_ERROR "

When the strings module catches an SMTP AUTH error, it logs the following line:

2014-03-25T14:38:11.238206+01:00 mail kernel: [2435436.682421] SMTP_AUTH_ERROR \
     IN= OUT=eth0 SRC= DST= LEN=109 TOS=0x00 PREC=0x00 TTL=64 \
     ID=16338 DF PROTO=TCP SPT=25 DPT=55028 WINDOW=227 RES=0x00 ACK PSH URGP=0

iptables also writes down the attackers IP address in the modules recent file:

# cat /proc/net/xt_recent/SMTP_AUTH_ERROR
src= ttl: 64 last_seen: 4945785611 oldest_pkt: 9
4945669673, 4945671995, 4945674395, 4945719287, 4945721259, \
4945723545, 4945780221, 4945783217, 4945785611

This might be called an adaptive firewall or with logging only and alarming/monitoring a "poor mans IDS".

Further Improvement

I didn't test TLS connections, because my LOG doesn't show any signs of SMTP AUTH Brute Force over TLS enctypted connections. That doesn't mean there aren't any. At the moment I don't even know if it's possible to filter any relevant strings during TLS sessions.

Also, iptables recent string might be good at fighting Brute Force Attacks on other mail protocols, such as POP3 or IMAP. As I am no firewall expert I leave this to my colleague Michael Schwartzkopff. Any optimized rules and comments are welcome!

Robert Schetterer, 27 Mar 2014