NOTE: openbl has been discontinued. So, I leave this article for the archive…
Since I am a very lazy person, I am quoting the introduction of https://openbl.org:
The OpenBL.org project (formerly known as the SSH blacklist) is about detecting, logging and reporting various types of internet abuse. Currently our hosts monitor ports 21 (FTP), 22 (SSH), 23 (TELNET), 25 (SMTP), 110 (POP3), 143 (IMAP), 587 (Submission), 993 (IMAPS) and 995 (POP3S) for bruteforce login attacks as well as scans on ports 80 (HTTP) and 443 (HTTPS) for vulnerable installations of phpMyAdmin and other web applications.
The traditional method to protect a server from SSH bruteforce attacks is fail2ban
; I hate fail2ban
:
- It is a very ugly (they call it legacy) Python script which runs as root; 24/7.
- It parses logfiles in order to derive and apply firewall rules.
- It eats ressources, as they were burgers.
- It is ugly to setup, due to its legacyness
- ...
Since I run my own mailserver (under FreeBSD of course), I am familiar with so called public blacklists. That is some list, were admins (= some script) report abusers automatically. These lists are updated frequently and can be used by poor guys like me, who want to benefit from the gathered data (= IP addresses of abusers). The question was, are there any of those blacklists especially for SSH brute-forcers?
There are. I have found two which are usable:
https://www.openbl.org/lists/base_all.txt
Update: This link is dead now!!- https://www.spamhaus.org/drop/edrop.txt
The task is, to add these IP addresses to a firewall block rule and update it, let's say, once a day. Since I failed in verifying ip addresses in POSIX shell, I wrote a little, sweet and tiny (= ugly) Python script, which downloads the ip list asynchronously (just because I can!). It concatenates the two lists together, verifies each ip address to be valid, and adds them to my ipfw
firewall rules.
IMPORTANT: It is very, very, very important to verify, that the downloaded strings are actual ip addresses. Imagine you download shellcode, e.g. ; rm -rf /*
, which is being executed on your server...
Python Script
Enough bullshitting now. Let's dive into the code... Here is the script, which is explained using the really awesome asciidoctor
features.
#!/usr/bin/env python3
import argparse
import asyncio
import logging
import shlex
from subprocess import run
from ipaddress import ip_address, ip_network
import aiohttp # <1>
DEFAULT_BLs = ( # <2>
"https://www.openbl.org/lists/base_all.txt",
"https://www.spamhaus.org/drop/edrop.txt",
)
FIREWALL = 'ipfw' # <3>
IPFW_TABLE = '5'
async def ips_fetch(session, url): # <4>
logging.info('Reading BL: {}'.format(url))
async with session.get(url) as r:
if r.status != 200:
return False
return await r.text()
async def bls_fetch(bls, session):
results = await asyncio.gather(
*[ips_fetch(session, bl) for bl in bls]
)
return results
def ip_verify(addr): # <5>
try:
return ip_address(addr)
except ValueError:
try:
return ip_network(addr)
except ValueError:
return False
def fw_ip_add(ip, args): # <6>
if args.f == 'ipfw':
run(['ipfw', '-qf', 'table', '5', 'add', shlex.quote(str(ip))])
elif args.f == 'dummy':
pass
else:
print('Firewall not supported')
exit(1)
def fw_prepare(args): # <7>
if args.f == 'ipfw':
run(['ipfw', '-qf', 'add', 'deny', 'ip', 'from', 'table(5)', 'to', 'me'])
run(['ipfw', '-qf', 'table', shlex.quote(str(args.ipfw_table)), 'flush'])
elif args.f == 'dummy':
pass
else:
print('Firewall not supported')
exit(1)
def logging_init(loglevel): # <8>
# From python docs. No magic stackoverflow involved. :)
# https://docs.python.org/3/howto/logging.html#logging-to-a-file
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
print('Invalid log level: "{}"'.format(loglevel))
exit(1)
logging.basicConfig(
format='%(asctime)s %(levelname)s: %(message)s',
level=numeric_level,
)
def parse_args(): # <9>
parser = argparse.ArgumentParser()
parser.add_argument(
'-d',
action='store_true',
help='Dry run, does not alter firewall rules'
)
parser.add_argument(
'-b',
metavar='URL',
nargs='+',
default=DEFAULT_BLs,
help='Specify blacklists to process'
)
parser.add_argument(
'-f',
metavar='FIREWALL',
default=FIREWALL,
help='Specify firewall backend [default: ipfw]'
)
parser.add_argument(
'--ipfw-table',
metavar='TABLE',
default=IPFW_TABLE,
help='Specify table for ipfw [default: 5]'
)
parser.add_argument(
'-l',
metavar='LEVEL',
type=str,
default='INFO',
help='CRITICAL, ERROR, WARNING, INFO [default], DEBUG'
)
return parser.parse_args()
def main():
args = parse_args()
logging_init(args.l)
if not args.d:
logging.debug('Preparing firewall tables')
fw_prepare(args)
# Async magic!
loop = asyncio.get_event_loop() # <10>
with aiohttp.ClientSession(loop=loop) as session:
ips = loop.run_until_complete(
bls_fetch(args.b, session)
)
loop.close()
ips = ('\n'.join(ips)).splitlines() # <11>
# Poor man's comment remover
ips = map(lambda ip: ip.split('#')[0].split(';')[0].strip(), ips)
ctr = 0 # <12>
for ip in ips: # <13>
ip_verified = ip_verify(ip)
if ip_verified is False:
logging.debug('Invalid ip address: "{}"'.format(ip))
logging.debug('Continuing with next ip')
continue
if not args.d:
logging.debug('Blocking "{}"'.format(ip_verified))
fw_ip_add(ip_verified, args)
ctr += 1
else:
logging.debug('Would block "{}"'.format(ip_verified))
logging.info('{} ips blocked'.format(ctr))
if __name__ == '__main__':
main()
Thanks to asciidoctor
and the jekyll-asciidoc
plugin, I am now able to comment on selected lines of code, which is awesome. Thanks to the creators of asciidoctor
, keep on the awesome work! EDIT: Migrated to markdown, works as well. :)
- Load the https://aiohttp.readthedocs.io[
aiohttp
library]. There are no technical reasons againstrequests
or the stdlib. I just wanted to try the new asynchronous capabilities of Python 3.5. One can replace it safely with the http library of choice; but the script has to be adapted then... - These are the blacklists which I use per default. Since I like Python programming a lot, I decided to make the blacklists configurable, and overwritable by the commandline options.
- My collegues still use
iptables
, so I designed the script to be extendible to other firewall backends. I useipfw
on FreeBSD, it should not be difficult to adapt the script to, e.g.iptables
. - To use the
async def
methods, a recent Python implementation has to be used, I use 3.5 currently. These two methods download the ip list from the configured blacklists. I have mostly copied the code from theaiohttp
documentation. In the end, thebls_fetch
method returns a string that contains the concatenated content of all configured blacklists. - That's the actual part why I have chosen to implement this script in Python. The gathered ip addresses (and address ranges) can simply be verified by instantiating an ipaddress object. When it fails, an exception is raised. I trust the Python stdlib to be correct.
- Just add the given ip address to the firewall table by calling the
ipfw
binary. Please note, that all given strings MUST be sanitized using theshlex
module when calling a binary!! - Do some preparations, add the lookup table, flush it, make coffe, ...
- Initialize logging framework. I want logging, when this crap is run by
cron
. - That's the argument parser, most variables can be overwritten on the commandline.
- That's needed because of
aiohttp
; it might be possible to replace it with a simple call torequests.get()
or something... - String post processing. The downloaded string has to be converted to the proper Python data types, and most importantly, comments should be removed in advance.
- A simple counter, to count how many ip addresses have been added to the block list.
- That's the main loop of my script. It iterates over the list of ip addresses, calls the verify methods and, if true, adds them to the firewall block table.
And finally add this script to cron, e.g. like this:
/etc/crontab
# ...
@daily root nice -n 5 /usr/local/bin/openbl.py
I have tested the script for two days now. It seems to work quite well; it indeed blocks potential abusers:
$ ipfw -cde list
....
01200 417 25127 deny ip from table(5) to me
....
So, I hope this was useful, and I hope my server is now more secure, unless someone hacks openbl.org and locks me out of my server... :D