Hey! It's been a while since there's been activity, huh? Well, worth first saying (even if almost 2 months late) Happy New Years to all my readers.

I recently became a founding member of a multinational group of FOSS Developers and Advocates called Neo-Sekai Club (more about that in a future post of mine!), and I'm in charge of many, many tasks, System Administration and Network Engineer being two of these tasks. In this blogpost, I go over real boots-on-the-ground experience of trying Fail2ban on production FreeBSD servers used by Neo-Sekai Club to see whether the thing everyone uses is really always the thing that works!

It goes without saying that the moment your server touches the internet, you're going to start having script kiddies and botnets hammering on your SSH, firing off port scans, and all sorts of other things. This totally sucks, and this is why we use firewalls. At Neo-Sekai Club, we deploy all of our servers using only BSDs. In this on-the-ground test, this is from real experience using a FreeBSD 14.3 server that we host our web services from.

We use Packet Filter (pf) as our firewall, which is a fantastically simple firewall ported over from OpenBSD and is just as much of a breeze to use over on FreeBSD. Packets get scrubbed in on our interface, logged out to the pflog, and if that's not a port we've whitelisted in the firewall, we block it. Pretty simple system to use! However, that's just not enough in a real-world scenario when it comes to SSH bruteforce attempts. If this was a Linux server, here's now where we're getting into using a system like Fail2ban. So… what exactly is Fail2ban?

Fail2ban is a Python-based security system that integrates into your firewall and provides an IP blacklist based on a number of failed SSH attempts within an alloted timeframe, and then issues bans on IPs who've failed SSH enough times for a period specified by the System Administrator. In the World of Linux, this is a very common anti-bruteforce system that System Administrators use to stop bruteforce attacks in their tracks, and it integrates quite nicely into Linux's firewall system iptables. Here's the thing however, we're not on Linux, and we don't get to use iptables on FreeBSD… so what's it like to use Fail2ban with pf on FreeBSD?

Firstly, Fail2ban uses a jail system, not the same as the jail system provided in FreeBSD, in which IPs can no longer access a service monitored by Fail2ban once their IP is banned. However, there's almost no good information out there to use Fail2ban with pf on FreeBSD! Sure, our SysAdmin Team found setup guides and we followed those to the absolute T, but boy was it a headache for us to do it… Okay, great, now we've got the Fail2ban up and running, added as a service to rc… And nobody's getting blocked. Why?

We weren't really able to conclusively figure out as to what was going wrong with Fail2ban, but our presumption had to do with pf somehow, or with our Fail2ban configuration. We'd found before in FreeBSD's forums¹ a link to using Fail2ban bans with pf on a fairly helpful FreeBSD-centric blog² and, huh… Fail2ban was trying to use iptables. What? That's so weird! We did almost everything identically to this blog's config too! We even overrid the Fail2ban config defaults to say "Hey, when you're banning an IP, PLEASE use pf to do this! Thanks!" But alas, IPs were not being banned. It worked when we sat there and added these IPs to the banlist ourselves using Fail2ban manually, but not automatically. Hm, odd.

This was a massive problem that we immediately had to take to the CISO. We have a bruteforce prevention system, and it isn't even working after we set it up correctly, this is not good! It was decided among the SysAdmins and Network Engineers that we were going to completely migrate over to blacklistd to properly enforce our IP bans, or that was the hope. In all honesty, we were afraid that this system wouldn't work either, or it'd be esoteric. This was our security we were gambling on here! But, did we really have a reason to be afraid?

So what even is blacklistd? Well, in versions of FreeBSD prior to 15, blacklistd is a similar service to Fail2ban but provided natively in FreeBSD to do the same task. While Fail2ban is in Python, blacklistd is in C, which has far superior performance with a much smaller resource footprint, something we absolutely loved! A lot of the code in blacklistd also comes from NetBSD's blocklistd, and blocklistd is now what the daemon is called in FreeBSD 15; and surprisingly this was a hell of a simple system for us to setup. Firstly, the config is dead simple in contrast to Fail2ban, we just added an anchor for it to our pf.conf and modified the blacklistd.conf to increase the bantime and change the maximum number of tries. The OS even provided a fantastic and sane example config that required very little work from us to tweak! After that, we enabled it in the init system, enabled it in our sshd_config, and restarted everything.

Immediately, blacklistd got to work. Specifically, unlike Fail2ban, IPs got banned. Shot down right then and there after meeting blacklistd's ban criteria! We started to see immediate results. I really do mean immediate. IPs were already starting to get shot down by blacklistd and added to the pf table. It was a serious boost to our peace of mind, and quite wild how little work it took to actually start blacklisting IPs trying to bruteforce SSH. Now of course, it goes without saying that blacklistd works at the socket level, it's a system that works between Layers 3 and 4 upon direct socket data whereas Fail2ban merely greps logfiles and acts upon events, meaning it's acting upon Layer 7 data to then make Layer 3 and 4 decisions.¹ There are cases where Fail2ban might be the choice and blacklistd isn't, such as services running in Jails that blacklistd then cannot monitor if you're attempting to run a singular firewall system. It does go without saying though, that blacklistd as an SSH intrusion prevention system is phenomenally simpler on FreeBSD with pf than Fail2ban is. How simple is simple though?

It's amazing how quick it is to get blacklistd with a simple enough configuration running on a FreeBSD system using pf. First, let's say that your /etc/pf.conf looks something like this:

set block-policy drop
set skip on lo0
scrub in

block in log on iface0 all
pass in quick log on iface0 proto tcp from any to any 22 keep state
pass in quick on iface0 inet proto icmp icmp-type { echoreq, unreach, timex } keep state
pass out quick log on iface0 proto { tcp, udp, icmp } from any to any keep state

So we've got our SSH port open in pf, but inherently that is not going to simply stop bruteforcing. This just means every other port is going to have packets sent to it dropped (since the block-policy is set to drop). We have to first add in the anchor for blacklistd into our /etc/pf.conf:

set block-policy drop
set skip on lo0
scrub in

anchor "blacklistd/*" in on iface0

block in log on iface0 all
pass in quick log on iface0 proto tcp from any to any 22 keep state
pass in quick on iface0 inet proto icmp icmp-type { echoreq, unreach, timex } keep state
pass out quick log on iface0 proto { tcp, udp, icmp } from any to any keep state

Great, now we can go setup our /etc/blacklistd.conf. There is surprisingly not a whole ton we need to do, because FreeBSD 14.3 out the box gives us a great starting point for our blacklistd rules, we can just change some of the ban lengths and attempts:

#
# Blacklist rule
# adr/mask:port type    proto   owner           name    nfail   disable
[local]
ssh             stream  tcp     *               *       3       24h

The nfail option is just specifying how many failed login attempts until blacklistd gets triggered, and the disable option is how long the ban lasts. Any rules set under [local] are in regards to locally ran services from the system and thus how they get handled by blacklistd.³ Now that we've created our blacklistd rules, let's go into /etc/ssh/sshd_config and set UseBlacklist yes to start using blacklistd, then add blacklistd_enable"YES"= to our /etc/rc.conf. Now restart sshd, start blacklistd, and run pfctl -f /etc/pf.conf to reload your new firewall rules and you've got blacklistd running! Surprisingly simpler than using Fail2ban, and trust us… we jumped through too many hoops with that…

I don't expect this to have been an eye-opening or profound entry, but it's a worthwhile writeup for any aspiring FreeBSD SysAdmins and Engineers to look at when using real FreeBSD systems in real production environments when weighing out how best to handle their security posture. Until then, stay safe on the internet and always better to be safe rather than sorry!