One of the cool things Check_MK offers these days is the option for custom notifications. Email notifications are of course fine, but a lot of people are also interested in Pagerduty or their own SMS service or whatnot. Personally I was interested in an IRC based notification system where alerts would simply be sent as a message into a specific channel on my IRC server.
Let’s see how we can implement that 🙂
The biggest hurdle is to get a commandline IRC notification thing going on. For this I looked into the commandline client II, which could have worked but it didn’t seem to match my requirement of autojoining/autoreconnecting without having to bolt on monitoring for the irc client – I want the client to be constantly connected and joined to the channel of my choice. That way I know notifications will arrive (if the client leaves the channel you know you have problems to solve), instead of having the client rejoin/reconnect every time a notification needs to be sent and keeping fingers crossed that works every time.
Another option was to use my usual IRC client Irssi. This is a great IRC client with a small plugin to handle these notifications, but Irssi does not like to run without a proper terminal. Trust me, I’ve tried. Also: what does that “dummy terminal” startup flag do? It didn’t make any difference for me when I tried to get it up and running without decent TERM environment or without tty/pts in daemontools ;). The closest thing you can get for Irssi to run as a daemon is to start it in a detached screen, but that was flakey at best.
So I finally decided to look into building a small IRC bot (in perl obviously ;)). Fortunately the POE framework has this covered. I wrote a tiny bot to connect to the server/channel(s) of my choice, say “Hi!” and stay connected until a notification needs to be sent. The bot creates a named pipe which it reads notification messages from and once a message is read it will be sent to all connected channels. If kicked/disconnect it will try to rejoin/reconnect automatically after a few seconds. Simple enough, right?
In order to send the notifications I created a notification rule using the new WATO Notification rules system. If you’re using flexible notifications you can simply create a new user for IRC notifications and set that user up to be notified through the IRC plugin.
If you’re using system-wide notifications like I did be sure to change the contact selection to only notify 1 specific user so only 1 message is sent. (otherwise you’ll get the message repeated for every contact listed ;)).
“Yeah BenV, that sounds awesome, gief!”
Sure thing. Here’s how!
How to set up your own Check MK IRC notifications
First you’ll need to get the bot up and running.
The bot has a few perl depedencies, namely POE, POE::Component::IRC::State, POE::Component::IRC::Plugin::AutoJoin, POE::Component::IRC::Plugin::FollowTail and POE::Component::IRC::Plugin::Connector. Either install them through cpan or get them through your distro. In Pokemon OS this is all included in libpoe-component-perl-irc-perl.
(note that I also have a Check_MK Plugin package which also has this ircbot.pl file if you want to go that route, see further down below)
nagios@nagiosmachine:/home/nagios$ vim ircbot.pl
# Change settings, you probably want to editL
# - $readpipe location
# - $server / $port / $ssl / $ipv6
# - %channels hash, '#channel' => 'password' entries.
# - $nickname and $ircname
# - $debug if you want to see what's happening
# See if it connects
Personally I run this bot in DJB’s daemontools to make sure it stays up, but I’m sure it’ll work just as fine under upstart, systemd or whatever you use to make sure shit stays operational. The daemontools run script is trivial:
sleep 1 # prevent excessive respawning
exec setuidgid nagios /opt/nagios/ircbot.pl
Once the bot is up and running you should be able to send messages to your IRC channel with echo:
Next step is putting the IRC notification script for Check_MK in place.
Either do this manually (since it’s only 1 file):
nagios@nagiosmachine:/usr/share/check_mk/notifications$ chmod +x irc
Or you can install the Check_MK Package I made for these notifications, which also has the IRC bot included in the package as doc/treasures/ircbot.pl (seemed to be the best fitting place for something that shouldn’t really be in the check_mk tree):
nagios@nagiosmachine:~$ cmk -P install irc_notify-1.0.mkp
You should probably edit $CMK_DIR/notifications/irc to change the pipe location, message style, enable/disable colors (default enabled), etc. The source of that irc notifications script is based on the sms script and altered a bit to make the messages a bit fancier.
If you have an even better version I’m all ears 😉
Finally enable the notifications through WATO and you should be able to see something like this on irc:
This needs a better method of configuration for irc server/port/channel/colors etc, but for now this is fine.
Check out this post for the yadieyada on the latest version – Check_MK IRC_Notify new version