Check_MK Custom Notifications — IRC

by BenV on Oct.27, 2015, under Check MK

One of the cool things Check_MK offers these days is the option for custom notifications. Email notifications are of course fine, but a lot of people are also interested in Pagerduty or their own SMS service or whatnot. Personally I was interested in an IRC based notification system where alerts would simply be sent as a message into a specific channel on my IRC server.
Let’s see how we can implement that 🙂

The biggest hurdle is to get a commandline IRC notification thing going on. For this I looked into the commandline client II, which could have worked but it didn’t seem to match my requirement of autojoining/autoreconnecting without having to bolt on monitoring for the irc client – I want the client to be constantly connected and joined to the channel of my choice. That way I know notifications will arrive (if the client leaves the channel you know you have problems to solve), instead of having the client rejoin/reconnect every time a notification needs to be sent and keeping fingers crossed that works every time.

Another option was to use my usual IRC client Irssi. This is a great IRC client with a small plugin to handle these notifications, but Irssi does not like to run without a proper terminal. Trust me, I’ve tried. Also: what does that “dummy terminal” startup flag do? It didn’t make any difference for me when I tried to get it up and running without decent TERM environment or without tty/pts in daemontools ;). The closest thing you can get for Irssi to run as a daemon is to start it in a detached screen, but that was flakey at best.

So I finally decided to look into building a small IRC bot (in perl obviously ;)). Fortunately the POE framework has this covered. I wrote a tiny bot to connect to the server/channel(s) of my choice, say “Hi!” and stay connected until a notification needs to be sent. The bot creates a named pipe which it reads notification messages from and once a message is read it will be sent to all connected channels. If kicked/disconnect it will try to rejoin/reconnect automatically after a few seconds. Simple enough, right?

In order to send the notifications I created a notification rule using the new WATO Notification rules system. If you’re using flexible notifications you can simply create a new user for IRC notifications and set that user up to be notified through the IRC plugin.
If you’re using system-wide notifications like I did be sure to change the contact selection to only notify 1 specific user so only 1 message is sent. (otherwise you’ll get the message repeated for every contact listed ;)).

Setting up a custom notification rule for IRC

“Yeah BenV, that sounds awesome, gief!”
Sure thing. Here’s how!

How to set up your own Check MK IRC notifications

First you’ll need to get the bot up and running.
The bot has a few perl depedencies, namely POE, POE::Component::IRC::State, POE::Component::IRC::Plugin::AutoJoin, POE::Component::IRC::Plugin::FollowTail and POE::Component::IRC::Plugin::Connector. Either install them through cpan or get them through your distro. In Pokemon OS this is all included in libpoe-component-perl-irc-perl.
(note that I also have a Check_MK Plugin package which also has this ircbot.pl file if you want to go that route, see further down below)
nagios@nagiosmachine:/home/nagios$ wget http://notes.benv.junerules.com/download/1982 -O ircbot.pl nagios@nagiosmachine:/home/nagios$ vim ircbot.pl # Change settings, you probably want to editL # - $readpipe location # - $server / $port / $ssl / $ipv6 # - %channels hash, '#channel' => 'password' entries. # - $nickname and $ircname # - $debug if you want to see what's happening nagios@nagiosmachine:/home/nagios$ ./ircbot.pl # See if it connects
Personally I run this bot in DJB’s daemontools to make sure it stays up, but I’m sure it’ll work just as fine under upstart, systemd or whatever you use to make sure shit stays operational. The daemontools run script is trivial:
#!/bin/bash exec 2>&1 sleep 1 # prevent excessive respawning exec setuidgid nagios /opt/nagios/ircbot.pl

Once the bot is up and running you should be able to send messages to your IRC channel with echo:
nagios@nagiosmachine:/home/nagios$ echo "Hello, world!" > /opt/nagios/check_mk_msgs.fifo

Next step is putting the IRC notification script for Check_MK in place.
Either do this manually (since it’s only 1 file):
nagios@nagiosmachine:/usr/share/check_mk/notifications$ wget http://notes.benv.junerules.com/download/1984/ -O irc nagios@nagiosmachine:/usr/share/check_mk/notifications$ chmod +x irc
Or you can install the Check_MK Package I made for these notifications, which also has the IRC bot included in the package as doc/treasures/ircbot.pl (seemed to be the best fitting place for something that shouldn’t really be in the check_mk tree):
nagios@nagiosmachine:~$ wget http://notes.benv.junerules.com/download/1988/ -O irc_notify-1.0.mkp nagios@nagiosmachine:~$ cmk -P install irc_notify-1.0.mkp
You should probably edit $CMK_DIR/notifications/irc to change the pipe location, message style, enable/disable colors (default enabled), etc. The source of that irc notifications script is based on the sms script and altered a bit to make the messages a bit fancier.
If you have an even better version I’m all ears 😉

Finally enable the notifications through WATO and you should be able to see something like this on irc:

Check MK IRC Notifications in Irssi

Todo

This needs a better method of configuration for irc server/port/channel/colors etc, but for now this is fine.

Downloads

Here’s the stuff you might want:
irc_notify-1.3a.mkp (7517 downloads)      SHA1: 26efbb637c4b69adaec1418f5b3b8b0b8bb86927  MD5: 51779dac78d5efeb39315c2ef03be41b
ircbot.pl (1267 downloads)      SHA1: 74455498a3d1e177208d7272f45ea81bb4f2470e  MD5: b0741d3c991cc0997542ea916e7076b1
irc (1605 downloads)      SHA1: 5c867480f46545092531dddb3f4406cb6c52de1c  MD5: 2a670ccf4ec2db4b2937ff5ad9cad188
irc_notify-1.0.1.mkp (1533 downloads)      SHA1: 191c9735a93497fee378d04aff968c67acf797fc  MD5: f54045fd4d9132e67b69e572cec65ab8