A couple of weeks ago I accidentally turned off one of my UPSes. Every morning it starts beeping a warning about the battery complete with yellow warning light. So I was inspecting it and wanted to silence the alarm. Well, hitting the big button is NOT the way to silence the alarm. Yes, it does silence the alarm, but it does so by turning OFF the UPS.

Oops. Killed three servers; my OpenBSD web server and firewall, my Solaris Tomcat box and my backup file server. I restored power, then did a ‘hot swap’ for new batteries, which sadly did not solve the beeping problem. I suspect it’s just old age on that UPS and it’s now cranky. Oh well.

Meantime, all three servers came back up without apparent incident. Except… my home info server running under Tomcat on the Solaris box was unavailable. I checked a local port, and it was working fine, but  not via the firewall server.

After much checking, a couple of reboots and some web reading, it became apparent that the OpenBSD firewall did NOT load the packet filter rules when it booted. As soon as I manually loaded them the Tomcat server was again available.

I searched and searched, but there is absolutely no reason I can find as to why a working OpenBSD server would fail to load the PF rules on boot. The rules are good; there were no error messages at all in the boot logs, and it’s always worked in the past.

For now I just made a note to check date and pf rules whenever that server gets rebooted, which fortunately is about once every several years. I also need to keep my fingers off the big UPS button!