BenV's notes

Check_MK plugin: MTR for pretty ping graphs

by on Dec.31, 2015, under Check MK

Another day, another Check_MK plugin!
This one is inspired by smokeping, but different because it doesn’t need smokeping. It does need the tool formerly known as Matt’s TraceRoute, aka mtr. It’s installed on all my machines by default and easily available in all distro’s that are worthy. Even pokemon OS has it 😉

The reason I wanted to build this plugin was first of all because of pretty graphs (of course!). The second reason was that my girlfriend had some network issues to figure out, but only ping and DNS resolve times don’t paint a complete picture. This plugin makes some graphs that hopefully fill that void a bit 🙂

Now that you’ve skipped the last 2 paragraphs, here are some example graphs that I made while testing the plugin:

Check_MK MTR plugin's perf-o-meter

Check_MK MTR plugin’s perf-o-meter


This is the plugin status per host on the service overview page of Check_MK. As you can see I configured multiple hosts.

Configuration

Configuration is simple, you install the agent plugin on the machine where you want to run the pings from. Next you copy the example mtr.cfg to your Check MK agent’s configuration directory, which in my case ends up as /etc/check_mk/mtr.cfg. In this file you make a section per host you want to ping, so for example:

[mathias-kettner.de]
# Options here, see the example for the default options
interval = 2

Execution

Once mtr.cfg is in place the plugin will automatically start mtr for every host the next time the agent is called. Because MTR takes a relatively long time to run (default it ‘only’ does 10 pings) they’re started in the background. Once the process is complete the report will be parsed as soon as check_mk_agent is called again, after which a new MTR run is started (unless you set time=300, which will wait until 5 minutes have passed since the last run).

Results

When the results come in the alarms will be generated if necessary and pnp4nagios creates some pretty graphs:

Check_MK MTR Plugin: destination host

Check_MK MTR Plugin: destination host


Above is the graph of the destination graph where you can see packet loss as black bars (scaled to match the height of your graph), and best, avg and worst ping times. I’ve tried to make it a bit ‘smokey’, but rrdgraph is quite a beast to time.
NOTE: I made these graphs for RRDTool 1.5 and higher since they use gradients, so if you have RRDTool 1.4 or older your graphs might look quite a bit less fancy or even fail completely. Sorry! 🙂
“But BenV! That graph looks like utter garbage!
Oh yeah?! Patches are welcome! 🙂
Check_MK MTR Plugin:  intermediate Hop

Check_MK MTR Plugin: intermediate Hop


Above graph is made for every hop in between the source and destination. They don’t have alarms, but sometimes it’s interesting to see that certain hops are slow all of a sudden.
Which is another interesting discussion: how relevant are these hops in between?
Well, since routing can give you different results for every packet they’re not THAT interesting, especially since often the hops will drop your icmp packets completely. On the other hand, if you see that all hops are suddenly slow you can draw certain conclusions from that about your own network/internet speed. (as opposed to only the destination being slow).
Just be aware that “hop 4” can be router A one time and router B the next time, just as the hopcount can vary.

Check_MK MTR Plugin - Hops Overview

Check_MK MTR Plugin – Hops Overview


This ‘hop overview’ shows all the average times for all hops including the destination host. Again, individual hops don’t mean much, but seeing all hops in a single graph together with the destination host can sometimes reveal interesting patterns. (such as a slow destination host)

WATO

Check_MK MTR Plugin: Wato settings

Check_MK MTR Plugin: Wato settings


As with my Check MK fail2ban plugin I’ve added the part so you can edit the parameters in WATO. Here you see the default values for the plugin.
If you click on the ‘MTR’ label under “Type of check” you can create custom rules to change the parameters for a specific host, like this:
Check_MK MTR Plugin: WATO rule parameters

Check_MK MTR Plugin: WATO rule parameters


Note that you can disable these alarms by setting them to 0.

Download

Of course now you want my plugin, so without wasting more bytes:
V0.5.2: mtr-0.5.2.mkp (54670 downloads)      SHA1: 0e6ea842de0965612fff1e9f37b3462c55ae7ae1  MD5: c1fba27991564523ad3381b41fbd5109 Agent updated to stop failing when mtr ran twice. Will try to fix this better in a future update, but at least it doesn’t crash anymore and will after 1 failed report return to properly parsing the specific host.
V0.5.1: — fsckup on my side, I patched an older version. Use 0.5.2 instead 😉
V0.5: mtr-0.5.mkp (1700 downloads)      SHA1: 75075fd2f4cfd0c74e04d492a03abd8bf344a6d2  MD5: faad8a2f0faf66ec1c07ed280fdea09b Alarms can now actually be disabled by setting levels to 0, better compatibility with older mtr / unpexpected mtr output and copes better with mtr returning multiple lines for a hop. (only first line is parsed)
V0.4: mtr-0.4.mkp (1594 downloads)      SHA1: efc201f28163678b9ab885f81cb94749f8469066  MD5: a3e42e5ba7d01365ce9e76c13917742e Initial release




:, , ,

25 Comments for this entry

  • fooc

    Hi,

    I followed your insctructions and installed the mkp, i put the config and plugin file in the right agent dirs on the server but when i rescan the localhost i cannot see the mtr check, can u help me?

  • BenV

    @fooc: when you run the check_mk_agent, can you see the <<>> section in the output? Do you have the mtr package installed on your machine? (i.e. can you run ‘mtr http://www.google.com‘ on your machine?)

    The mtr check will show up per configured host, so if no correct host sections are found it won’t show up 🙂

  • fooc

    mtr is working from command line.

    mtr.cfg:

    # Mtr Check_MK configuration

    # NOTE: your MTR report shouldn’t take longer than 15 minutes

    # [DEFAULTS]
    # type=icmp # icmp, tcp or udp
    # count=10 # number of pings per mtr report
    # force_ipv4=0 # force ipv4, exclusive with force_ipv6
    # force_ipv6=0 # force ipv6, exclusive with force_ipv4
    # size=64 # packet size
    # time=0 # minimum time between runs, 0 / default means run if mtr doesn’t run anymore
    # port=80 # UDP/TCP port to connect to
    # dns=0 # Use DNS resolution to lookup addresses
    # address= # Bind to source address
    # interval= # time MTR waits between sending pings
    # timeout= # ping Timeout, see mtr man page

    [www.google.com]
    type = icmp
    force_ipv4 = true

    [ipv6.google.com]
    type = icmp
    force_ipv6 = true

    this is agent output:

    <<>>
    Version: 1.2.6p16
    AgentOS: linux
    AgentDirectory: /etc/check_mk
    DataDirectory: /var/lib/check_mk_agent
    SpoolDirectory: /var/lib/check_mk_agent/spool
    PluginsDirectory: /usr/lib/check_mk_agent/plugins
    LocalDirectory: /usr/lib/check_mk_agent/local
    OnlyFrom:
    <<>>
    /dev/mapper/centos-root xfs 40456908 3208476 37248432 8% /
    devtmpfs devtmpfs 494956 0 494956 0% /dev
    tmpfs tmpfs 504684 0 504684 0% /dev/shm
    tmpfs tmpfs 504684 51044 453640 11% /run
    tmpfs tmpfs 504684 0 504684 0% /sys/fs/cgroup
    /dev/sda1 xfs 505580 167184 338396 34% /boot
    /dev/mapper/centos-home xfs 19748864 32976 19715888 1% /home
    tmpfs tmpfs 504684 3000 501684 1% /opt/omd/sites/oibsite01/tmp
    tmpfs tmpfs 100940 0 100940 0% /run/user/996
    tmpfs tmpfs 100940 0 100940 0% /run/user/0
    <<>>
    [df_inodes_start]
    /dev/mapper/centos-root xfs 40476672 61786 40414886 1% /
    devtmpfs devtmpfs 123739 343 123396 1% /dev
    tmpfs tmpfs 126171 1 126170 1% /dev/shm
    tmpfs tmpfs 126171 478 125693 1% /run
    tmpfs tmpfs 126171 13 126158 1% /sys/fs/cgroup
    /dev/sda1 xfs 512000 337 511663 1% /boot
    /dev/mapper/centos-home xfs 19759104 3 19759101 1% /home
    tmpfs tmpfs 126171 531 125640 1% /opt/omd/sites/oibsite01/tmp
    tmpfs tmpfs 126171 1 126170 1% /run/user/996
    tmpfs tmpfs 126171 1 126170 1% /run/user/0
    [df_inodes_end]
    <<>>
    <<>>
    <<>>
    /dev/mapper/centos-root / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
    /dev/sda1 /boot xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
    /dev/mapper/centos-home /home xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
    <<>>
    (root,46300,6744,00:01:13,1) /usr/lib/systemd/systemd –switched-root –system –deserialize 21
    (root,0,0,00:00:00,2) [kthreadd]
    (root,0,0,00:00:02,3) [ksoftirqd/0]
    (root,0,0,00:00:16,7) [migration/0]
    (root,0,0,00:00:00,8) [rcu_bh]
    (root,0,0,00:00:00,9) [rcuob/0]
    (root,0,0,00:00:00,10) [rcuob/1]
    (root,0,0,00:00:00,11) [rcuob/2]
    (root,0,0,00:00:00,12) [rcuob/3]
    (root,0,0,00:00:00,13) [rcuob/4]
    (root,0,0,00:00:00,14) [rcuob/5]
    (root,0,0,00:00:00,15) [rcuob/6]
    (root,0,0,00:00:00,16) [rcuob/7]
    (root,0,0,00:00:00,17) [rcuob/8]
    (root,0,0,00:00:00,18) [rcuob/9]
    (root,0,0,00:00:00,19) [rcuob/10]
    (root,0,0,00:00:00,20) [rcuob/11]
    (root,0,0,00:00:00,21) [rcuob/12]
    (root,0,0,00:00:00,22) [rcuob/13]
    (root,0,0,00:00:00,23) [rcuob/14]
    (root,0,0,00:00:00,24) [rcuob/15]
    (root,0,0,00:00:00,25) [rcuob/16]
    (root,0,0,00:00:00,26) [rcuob/17]
    (root,0,0,00:00:00,27) [rcuob/18]
    (root,0,0,00:00:00,28) [rcuob/19]
    (root,0,0,00:00:00,29) [rcuob/20]
    (root,0,0,00:00:00,30) [rcuob/21]
    (root,0,0,00:00:00,31) [rcuob/22]
    (root,0,0,00:00:00,32) [rcuob/23]
    (root,0,0,00:00:00,33) [rcuob/24]
    (root,0,0,00:00:00,34) [rcuob/25]
    (root,0,0,00:00:00,35) [rcuob/26]
    (root,0,0,00:00:00,36) [rcuob/27]
    (root,0,0,00:00:00,37) [rcuob/28]
    (root,0,0,00:00:00,38) [rcuob/29]
    (root,0,0,00:00:00,39) [rcuob/30]
    (root,0,0,00:00:00,40) [rcuob/31]
    (root,0,0,00:00:00,41) [rcuob/32]
    (root,0,0,00:00:00,42) [rcuob/33]
    (root,0,0,00:00:00,43) [rcuob/34]
    (root,0,0,00:00:00,44) [rcuob/35]
    (root,0,0,00:00:00,45) [rcuob/36]
    (root,0,0,00:00:00,46) [rcuob/37]
    (root,0,0,00:00:00,47) [rcuob/38]
    (root,0,0,00:00:00,48) [rcuob/39]
    (root,0,0,00:00:00,49) [rcuob/40]
    (root,0,0,00:00:00,50) [rcuob/41]
    (root,0,0,00:00:00,51) [rcuob/42]
    (root,0,0,00:00:00,52) [rcuob/43]
    (root,0,0,00:00:00,53) [rcuob/44]
    (root,0,0,00:00:00,54) [rcuob/45]
    (root,0,0,00:00:00,55) [rcuob/46]
    (root,0,0,00:00:00,56) [rcuob/47]
    (root,0,0,00:00:00,57) [rcuob/48]
    (root,0,0,00:00:00,58) [rcuob/49]
    (root,0,0,00:00:00,59) [rcuob/50]
    (root,0,0,00:00:00,60) [rcuob/51]
    (root,0,0,00:00:00,61) [rcuob/52]
    (root,0,0,00:00:00,62) [rcuob/53]
    (root,0,0,00:00:00,63) [rcuob/54]
    (root,0,0,00:00:00,64) [rcuob/55]
    (root,0,0,00:00:00,65) [rcuob/56]
    (root,0,0,00:00:00,66) [rcuob/57]
    (root,0,0,00:00:00,67) [rcuob/58]
    (root,0,0,00:00:00,68) [rcuob/59]
    (root,0,0,00:00:00,69) [rcuob/60]
    (root,0,0,00:00:00,70) [rcuob/61]
    (root,0,0,00:00:00,71) [rcuob/62]
    (root,0,0,00:00:00,72) [rcuob/63]
    (root,0,0,00:01:50,73) [rcu_sched]
    (root,0,0,00:01:08,74) [rcuos/0]
    (root,0,0,00:01:09,75) [rcuos/1]
    (root,0,0,00:00:00,76) [rcuos/2]
    (root,0,0,00:00:00,77) [rcuos/3]
    (root,0,0,00:00:00,78) [rcuos/4]
    (root,0,0,00:00:00,79) [rcuos/5]
    (root,0,0,00:00:00,80) [rcuos/6]
    (root,0,0,00:00:00,81) [rcuos/7]
    (root,0,0,00:00:00,82) [rcuos/8]
    (root,0,0,00:00:00,83) [rcuos/9]
    (root,0,0,00:00:00,84) [rcuos/10]
    (root,0,0,00:00:00,85) [rcuos/11]
    (root,0,0,00:00:00,86) [rcuos/12]
    (root,0,0,00:00:00,87) [rcuos/13]
    (root,0,0,00:00:00,88) [rcuos/14]
    (root,0,0,00:00:00,89) [rcuos/15]
    (root,0,0,00:00:00,90) [rcuos/16]
    (root,0,0,00:00:00,91) [rcuos/17]
    (root,0,0,00:00:00,92) [rcuos/18]
    (root,0,0,00:00:00,93) [rcuos/19]
    (root,0,0,00:00:00,94) [rcuos/20]
    (root,0,0,00:00:00,95) [rcuos/21]
    (root,0,0,00:00:00,96) [rcuos/22]
    (root,0,0,00:00:00,97) [rcuos/23]
    (root,0,0,00:00:00,98) [rcuos/24]
    (root,0,0,00:00:00,99) [rcuos/25]
    (root,0,0,00:00:00,100) [rcuos/26]
    (root,0,0,00:00:00,101) [rcuos/27]
    (root,0,0,00:00:00,102) [rcuos/28]
    (root,0,0,00:00:00,103) [rcuos/29]
    (root,0,0,00:00:00,104) [rcuos/30]
    (root,0,0,00:00:00,105) [rcuos/31]
    (root,0,0,00:00:00,106) [rcuos/32]
    (root,0,0,00:00:00,107) [rcuos/33]
    (root,0,0,00:00:00,108) [rcuos/34]
    (root,0,0,00:00:00,109) [rcuos/35]
    (root,0,0,00:00:00,110) [rcuos/36]
    (root,0,0,00:00:00,111) [rcuos/37]
    (root,0,0,00:00:00,112) [rcuos/38]
    (root,0,0,00:00:00,113) [rcuos/39]
    (root,0,0,00:00:00,114) [rcuos/40]
    (root,0,0,00:00:00,115) [rcuos/41]
    (root,0,0,00:00:00,116) [rcuos/42]
    (root,0,0,00:00:00,117) [rcuos/43]
    (root,0,0,00:00:00,118) [rcuos/44]
    (root,0,0,00:00:00,119) [rcuos/45]
    (root,0,0,00:00:00,120) [rcuos/46]
    (root,0,0,00:00:00,121) [rcuos/47]
    (root,0,0,00:00:00,122) [rcuos/48]
    (root,0,0,00:00:00,123) [rcuos/49]
    (root,0,0,00:00:00,124) [rcuos/50]
    (root,0,0,00:00:00,125) [rcuos/51]
    (root,0,0,00:00:00,126) [rcuos/52]
    (root,0,0,00:00:00,127) [rcuos/53]
    (root,0,0,00:00:00,128) [rcuos/54]
    (root,0,0,00:00:00,129) [rcuos/55]
    (root,0,0,00:00:00,130) [rcuos/56]
    (root,0,0,00:00:00,131) [rcuos/57]
    (root,0,0,00:00:00,132) [rcuos/58]
    (root,0,0,00:00:00,133) [rcuos/59]
    (root,0,0,00:00:00,134) [rcuos/60]
    (root,0,0,00:00:00,135) [rcuos/61]
    (root,0,0,00:00:00,136) [rcuos/62]
    (root,0,0,00:00:00,137) [rcuos/63]
    (root,0,0,00:00:00,138) [watchdog/0]
    (root,0,0,00:00:00,139) [watchdog/1]
    (root,0,0,00:00:16,140) [migration/1]
    (root,0,0,00:00:01,141) [ksoftirqd/1]
    (root,0,0,00:00:00,143) [kworker/1:0H]
    (root,0,0,00:00:00,144) [khelper]
    (root,0,0,00:00:00,145) [kdevtmpfs]
    (root,0,0,00:00:00,146) [netns]
    (root,0,0,00:00:00,147) [perf]
    (root,0,0,00:00:00,148) [writeback]
    (root,0,0,00:00:00,149) [kintegrityd]
    (root,0,0,00:00:00,150) [bioset]
    (root,0,0,00:00:00,151) [kblockd]
    (root,0,0,00:00:00,152) [md]
    (root,0,0,00:00:00,157) [khungtaskd]
    (root,0,0,00:00:02,158) [kswapd0]
    (root,0,0,00:00:00,159) [ksmd]
    (root,0,0,00:00:01,160) [khugepaged]
    (root,0,0,00:00:00,161) [fsnotify_mark]
    (root,0,0,00:00:00,162) [crypto]
    (root,0,0,00:00:00,170) [kthrotld]
    (root,0,0,00:00:00,172) [kmpath_rdacd]
    (root,0,0,00:00:00,173) [kpsmoused]
    (root,0,0,00:00:00,175) [ipv6_addrconf]
    (root,0,0,00:00:00,195) [deferwq]
    (root,0,0,00:00:00,226) [kauditd]
    (root,0,0,00:00:00,393) [hv_vmbus_con]
    (root,0,0,00:00:00,403) [ata_sff]
    (root,0,0,00:00:00,404) [scsi_eh_0]
    (root,0,0,00:00:00,405) [scsi_tmf_0]
    (root,0,0,00:00:00,406) [scsi_eh_1]
    (root,0,0,00:00:00,407) [scsi_tmf_1]
    (root,0,0,00:00:00,412) [scsi_eh_2]
    (root,0,0,00:00:00,415) [scsi_tmf_2]
    (root,0,0,00:00:00,419) [scsi_eh_3]
    (root,0,0,00:00:00,421) [scsi_tmf_3]
    (root,0,0,00:00:03,444) [kworker/1:1H]
    (root,0,0,00:00:00,496) [kdmflush]
    (root,0,0,00:00:00,497) [bioset]
    (root,0,0,00:00:00,507) [kdmflush]
    (root,0,0,00:00:00,508) [bioset]
    (root,0,0,00:00:00,522) [xfsalloc]
    (root,0,0,00:00:00,523) [xfs_mru_cache]
    (root,0,0,00:00:00,524) [xfs-buf/dm-0]
    (root,0,0,00:00:00,525) [xfs-data/dm-0]
    (root,0,0,00:00:00,526) [xfs-conv/dm-0]
    (root,0,0,00:00:00,527) [xfs-cil/dm-0]
    (root,0,0,00:00:51,528) [xfsaild/dm-0]
    (root,34936,2820,00:00:10,599) /usr/lib/systemd/systemd-journald
    (root,129128,696,00:00:00,617) /usr/sbin/lvmetad -f
    (root,46000,884,00:00:00,627) /usr/lib/systemd/systemd-udevd
    (root,0,0,00:00:00,663) [xfs-buf/sda1]
    (root,0,0,00:00:00,666) [xfs-data/sda1]
    (root,0,0,00:00:00,667) [kworker/0:1H]
    (root,0,0,00:00:00,668) [xfs-conv/sda1]
    (root,0,0,00:00:00,672) [xfs-cil/sda1]
    (root,0,0,00:00:00,677) [xfsaild/sda1]
    (root,0,0,00:00:00,707) [kdmflush]
    (root,0,0,00:00:00,708) [bioset]
    (root,0,0,00:00:00,715) [xfs-buf/dm-2]
    (root,0,0,00:00:00,716) [xfs-data/dm-2]
    (root,0,0,00:00:00,717) [xfs-conv/dm-2]
    (root,0,0,00:00:00,718) [xfs-cil/dm-2]
    (root,0,0,00:00:00,719) [xfsaild/dm-2]
    (root,116720,1348,00:00:02,734) /sbin/auditd -n
    (root,26396,1444,00:00:16,756) /usr/lib/systemd/systemd-logind
    (root,262056,3264,00:00:03,757) /usr/sbin/rsyslogd -n
    (dbus,34956,1496,00:00:41,759) /bin/dbus-daemon –system –address=systemd: –nofork –nopidfile –systemd-activation
    (root,508700,2672,00:00:13,768) /usr/sbin/NetworkManager –no-daemon
    (root,19176,860,00:00:10,769) /usr/sbin/irqbalance –foreground
    (root,126364,1012,00:00:03,773) /usr/sbin/crond -n
    (root,110032,644,00:00:00,775) /sbin/agetty –noclear tty1 linux
    (polkitd,527448,1812,00:00:08,782) /usr/lib/polkit-1/polkitd –no-debug
    (root,53060,1532,00:00:00,784) /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplicant.log -c /etc/wpa_supplicant/wpa_supplicant.conf -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
    (root,110520,1716,00:00:00,785) /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-55a03586-dc83-40c2-9d88-eac8c76f3ed6-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
    (root,553160,4144,00:00:20,984) /usr/bin/python -Es /usr/sbin/tuned -l -P
    (root,29300,724,00:00:01,986) /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
    (root,345492,4772,00:00:10,988) /usr/sbin/httpd -DFOREGROUND
    (root,82556,1820,00:00:00,989) /usr/sbin/sshd -D
    (root,90280,1112,00:00:05,1057) sendmail: accepting connections
    (smmsp,85720,560,00:00:00,1100) sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
    (apache,262452,1408,00:00:02,1151) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,71688,1516,00:01:18,1865) /usr/bin/python /omd/sites/oibsite01/bin/liveproxyd
    (oibsite+,795460,20376,00:04:55,2010) /omd/sites/oibsite01/bin/rrdcached -w 3600 -z 1800 -f 7200 -s oibsite01 -m 660 -l unix:/omd/sites/oibsite01/tmp/run/rrdcached.sock -p /omd/sites/oibsite01/tmp/rrdcached.pid -j /omd/sites/oibsite01/var/rrdcached
    (oibsite+,156072,784,00:00:05,2098) /omd/sites/oibsite01/bin/npcd -d -f /omd/sites/oibsite01/etc/pnp4nagios/npcd.cfg
    (oibsite+,109068,8488,00:00:07,2253) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (apache,345628,2812,00:00:00,2322) /usr/sbin/httpd -DFOREGROUND
    (root,0,0,00:00:00,2450) [kworker/0:2]
    (apache,345628,2984,00:00:00,2901) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,176024,6548,00:00:00,3237) /usr/bin/python /omd/sites/oibsite01/bin/liveproxyd 0
    (apache,345628,2984,00:00:00,3739) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,264952,13792,00:00:00,3931) /usr/bin/php-cgi -c /omd/sites/oibsite01/etc/apache/php.ini -d session.save_handler=files -d session.save_path=/omd/sites/oibsite01/tmp/php/session -d upload_tmp_dir=/omd/sites/oibsite01/tmp/php/upload -d soap.wsdl_cache_dir=/omd/sites/oibsite01/tmp/php/wsdl-cache -d safe_mode=Off -d mysql.default_socket=/omd/sites/oibsite01/tmp/run/mysqld/mysqld.sock
    (apache,345628,3472,00:00:00,3944) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,263284,56572,00:00:01,3963) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263608,12412,00:00:00,3964) /usr/bin/php-cgi -c /omd/sites/oibsite01/etc/apache/php.ini -d session.save_handler=files -d session.save_path=/omd/sites/oibsite01/tmp/php/session -d upload_tmp_dir=/omd/sites/oibsite01/tmp/php/upload -d soap.wsdl_cache_dir=/omd/sites/oibsite01/tmp/php/wsdl-cache -d safe_mode=Off -d mysql.default_socket=/omd/sites/oibsite01/tmp/run/mysqld/mysqld.sock
    (root,0,0,00:00:00,3987) [kworker/1:1]
    (oibsite+,1047048,4784,00:00:00,3995) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,1047048,4784,00:00:00,3996) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,1047048,4784,00:00:00,3997) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,1047048,4784,00:00:00,3998) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,1047048,4784,00:00:00,3999) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,82628,10832,00:00:00,4000) python /omd/sites/oibsite01/var/check_mk/precompiled/hypervsrv03
    (oibsite+,81312,9592,00:00:00,4002) python /omd/sites/oibsite01/var/check_mk/precompiled/Hypervsrv04testsnmp
    (oibsite+,81292,9376,00:00:00,4003) python /omd/sites/oibsite01/var/check_mk/precompiled/EIND-VEEAM01
    (oibsite+,82924,10188,00:00:00,4006) python /omd/sites/oibsite01/var/check_mk/precompiled/ILOHYPERV5
    (oibsite+,80932,9204,00:00:00,4009) python /omd/sites/oibsite01/var/check_mk/precompiled/ZYWALL110-01
    (oibsite+,102580,18652,00:00:00,4272) python /omd/sites/oibsite01/share/check_mk/modules/check_mk.py –defaults /omd/sites/oibsite01/etc/check_mk/defaults –automation diag-host — localhost ping 6556 1 5
    (oibsite+,102576,18632,00:00:00,4274) python /omd/sites/oibsite01/share/check_mk/modules/check_mk.py –defaults /omd/sites/oibsite01/etc/check_mk/defaults –automation diag-host — localhost agent 6556 1 5
    (oibsite+,102572,18652,00:00:00,4275) python /omd/sites/oibsite01/share/check_mk/modules/check_mk.py –defaults /omd/sites/oibsite01/etc/check_mk/defaults –automation diag-host — localhost snmpv1 6556 1 5
    (oibsite+,102700,18656,00:00:00,4276) python /omd/sites/oibsite01/share/check_mk/modules/check_mk.py –defaults /omd/sites/oibsite01/etc/check_mk/defaults –automation diag-host — localhost snmpv2 6556 1 5
    (oibsite+,102576,18652,00:00:00,4277) python /omd/sites/oibsite01/share/check_mk/modules/check_mk.py –defaults /omd/sites/oibsite01/etc/check_mk/defaults –automation diag-host — localhost snmpv2_nobulk 6556 1 5
    (apache,345628,2728,00:00:00,4288) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,109068,5128,00:00:00,4296) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,18512,2024,00:00:00,4304) snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.255.190.22 .1.3.6.1.4.1.232.11.1.3.0
    (oibsite+,109068,5128,00:00:00,4332) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,109068,5128,00:00:00,4354) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,18520,2024,00:00:00,4365) snmpwalk -v1 -c -m -M -Cc -OQ -OU -On -Ot 10.55.255.4 .1.3.6.1.2.1.2.2.1.8
    (oibsite+,11636,1364,00:00:00,4366) /bin/sh -c ping -A -i 0.2 -c 2 -W 5 127.0.0.1 2>&1
    (oibsite+,18516,2024,00:00:00,4368) snmpwalk -v1 -c -m -M -t 1.00 -r 5 -Cc -OQ -OU -On -Ot 127.0.0.1 .1.3.6.1.2.1.1.1.0
    (oibsite+,10800,688,00:00:00,4369) ping -A -i 0.2 -c 2 -W 5 127.0.0.1
    (oibsite+,18516,2024,00:00:00,4370) snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.255.255.216 .1.3.6.1.2.1.2.2.1.19
    (oibsite+,18516,2020,00:00:00,4371) snmpwalk -v2c -c -m -M -t 1.00 -r 5 -Cc -OQ -OU -On -Ot 127.0.0.1 .1.3.6.1.2.1.1.1.0
    (root,115248,1624,00:00:00,4372) /bin/bash /usr/bin/check_mk_agent
    (oibsite+,18512,2024,00:00:00,4373) snmpbulkwalk -v2c -c -m -M -t 1.00 -r 5 -Cc -OQ -OU -On -Ot 127.0.0.1 .1.3.6.1.2.1.1.1.0
    (root,47420,1624,00:00:00,4393) ps ax -o user,vsz,rss,cputime,pid,command –columns 10000
    (root,15460,744,00:00:00,4394) sed -e 1d -e s/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4,\5) /
    (oibsite+,18520,2024,00:00:00,4395) snmpwalk -v1 -c -m -M -Cc -OQ -OU -On -Ot 10.255.190.23 .1.3.6.1.2.1.2.2.1.10
    (apache,345628,3336,00:00:00,17654) /usr/sbin/httpd -DFOREGROUND
    (apache,345628,2960,00:00:00,18356) /usr/sbin/httpd -DFOREGROUND
    (apache,345628,2848,00:00:00,18949) /usr/sbin/httpd -DFOREGROUND
    (oibsite+,1047040,5868,00:02:25,18999) /omd/sites/oibsite01/bin/nagios -ud /omd/sites/oibsite01/tmp/nagios/nagios.cfg
    (oibsite+,109068,4908,00:00:00,21505) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263284,56044,00:00:03,21506) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263284,56024,00:00:03,21507) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263284,55984,00:00:04,21508) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263284,55988,00:00:03,21509) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (oibsite+,263284,56188,00:00:03,21510) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (root,0,0,00:00:29,21649) [kworker/u128:1]
    (oibsite+,263284,56040,00:00:03,22123) /usr/sbin/httpd -f /omd/sites/oibsite01/etc/apache/apache.conf
    (root,140972,3612,00:00:01,27231) sshd: root@notty
    (root,52696,1568,00:00:00,27235) /usr/libexec/openssh/sftp-server
    (root,0,0,00:00:00,30081) [kworker/0:2H]
    (root,0,0,00:00:00,50965) [kworker/1:0]
    (root,140944,5456,00:00:00,55988) sshd: root@pts/0
    (root,115380,2004,00:00:00,56151) -bash
    (root,185728,2412,00:00:00,56561) su oibsite01
    (oibsite+,115504,2164,00:00:00,56562) bash
    (root,0,0,00:00:10,57298) [kworker/u128:2]
    (root,0,0,00:00:00,58016) [kworker/1:2]
    (apache,345628,2996,00:00:00,58018) /usr/sbin/httpd -DFOREGROUND
    (apache,345628,2996,00:00:00,58029) /usr/sbin/httpd -DFOREGROUND
    (root,0,0,00:00:00,61130) [kworker/0:0]
    (apache,345628,2996,00:00:00,62652) /usr/sbin/httpd -DFOREGROUND
    (root,0,0,00:00:00,64438) [kworker/0:1]
    <<>>
    MemTotal: 1009368 kB
    MemFree: 93940 kB
    MemAvailable: 168520 kB
    Buffers: 0 kB
    Cached: 222096 kB
    SwapCached: 10032 kB
    Active: 374956 kB
    Inactive: 377744 kB
    Active(anon): 289472 kB
    Inactive(anon): 294576 kB
    Active(file): 85484 kB
    Inactive(file): 83168 kB
    Unevictable: 0 kB
    Mlocked: 0 kB
    SwapTotal: 2097148 kB
    SwapFree: 2039676 kB
    Dirty: 2304 kB
    Writeback: 0 kB
    AnonPages: 520704 kB
    Mapped: 25024 kB
    Shmem: 53444 kB
    Slab: 75412 kB
    SReclaimable: 37104 kB
    SUnreclaim: 38308 kB
    KernelStack: 6256 kB
    PageTables: 21716 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    WritebackTmp: 0 kB
    CommitLimit: 2601832 kB
    Committed_AS: 1202252 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 53912 kB
    VmallocChunk: 34359682048 kB
    HardwareCorrupted: 0 kB
    AnonHugePages: 241664 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0
    HugePages_Surp: 0
    Hugepagesize: 2048 kB
    DirectMap4k: 44992 kB
    DirectMap2M: 1003520 kB
    <<>>
    2.02 0.71 0.34 3/328 4399 2
    <<>>
    153266.20 285985.00
    <<>>
    eth0: 1248867570 8539574 0 0 0 0 0 0 765057458 8138431 0 0 0 0 0 0
    lo: 131774799 430234 0 0 0 0 0 0 131774799 430234 0 0 0 0 0 0
    [lo]
    Link detected: yes
    Address: 00:00:00:00:00:00

    [eth0]
    Link detected: yes
    Address: 00:15:5d:fe:19:0f

    <<>>
    01 18
    0A 6
    06 64
    <<>>
    1468402568
    8 0 sda 815569 2587 9141787 1076033 2298951 126913 33475984 2181403 0 839887 3255923
    253 0 dm-0 815017 0 9101945 1077715 2370855 0 33332664 2328137 0 837785 3406369
    253 1 dm-1 3070 0 25672 3935 16891 0 135128 279754 0 2895 283691
    253 2 dm-2 96 0 1441 78 16 0 4096 9 0 79 87
    [dmsetup_info]
    centos-home 253:2 centos home
    centos-swap 253:1 centos swap
    centos-root 253:0 centos root
    <<>>
    1468402568
    nr_free_pages 27276
    nr_alloc_batch 1099
    nr_inactive_anon 73644
    nr_active_anon 68625
    nr_inactive_file 20583
    nr_active_file 21580
    nr_unevictable 0
    nr_mlock 0
    nr_anon_pages 65998
    nr_mapped 6231
    nr_file_pages 58057
    nr_dirty 576
    nr_writeback 0
    nr_slab_reclaimable 9276
    nr_slab_unreclaimable 9577
    nr_page_table_pages 5361
    nr_kernel_stack 391
    nr_unstable 0
    nr_bounce 0
    nr_vmscan_write 16891
    nr_vmscan_immediate_reclaim 269
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem 13361
    nr_dirtied 3409195
    nr_written 3327541
    numa_hit 594809413
    numa_miss 0
    numa_foreign 0
    numa_interleave 12323
    numa_local 594809413
    numa_other 0
    workingset_refault 982782
    workingset_activate 141673
    workingset_nodereclaim 0
    nr_anon_transparent_hugepages 118
    nr_free_cma 0
    nr_dirty_threshold 12094
    nr_dirty_background_threshold 4031
    pgpgin 4570893
    pgpgout 16737992
    pswpin 2942
    pswpout 16891
    pgalloc_dma 7212350
    pgalloc_dma32 594260777
    pgalloc_normal 0
    pgalloc_movable 0
    pgfree 601518834
    pgactivate 533147
    pgdeactivate 574185
    pgfault 1886758020
    pgmajfault 705806
    pgrefill_dma 7760
    pgrefill_dma32 474431
    pgrefill_normal 0
    pgrefill_movable 0
    pgsteal_kswapd_dma 19311
    pgsteal_kswapd_dma32 1223400
    pgsteal_kswapd_normal 0
    pgsteal_kswapd_movable 0
    pgsteal_direct_dma 0
    pgsteal_direct_dma32 34455
    pgsteal_direct_normal 0
    pgsteal_direct_movable 0
    pgscan_kswapd_dma 20017
    pgscan_kswapd_dma32 1253522
    pgscan_kswapd_normal 0
    pgscan_kswapd_movable 0
    pgscan_direct_dma 0
    pgscan_direct_dma32 36382
    pgscan_direct_normal 0
    pgscan_direct_movable 0
    pgscan_direct_throttle 0
    zone_reclaim_failed 0
    pginodesteal 0
    slabs_scanned 176384
    kswapd_inodesteal 0
    kswapd_low_wmark_hit_quickly 16
    kswapd_high_wmark_hit_quickly 35
    pageoutrun 314
    allocstall 645
    pgrotated 16918
    drop_pagecache 0
    drop_slab 0
    numa_pte_updates 0
    numa_huge_pte_updates 0
    numa_hint_faults 0
    numa_hint_faults_local 0
    numa_pages_migrated 0
    pgmigrate_success 15536
    pgmigrate_fail 0
    compact_migrate_scanned 61595
    compact_free_scanned 262382
    compact_isolated 35555
    compact_stall 72
    compact_fail 44
    compact_success 26
    htlb_buddy_alloc_success 0
    htlb_buddy_alloc_fail 0
    unevictable_pgs_culled 6815
    unevictable_pgs_scanned 0
    unevictable_pgs_rescued 5741
    unevictable_pgs_mlocked 8463
    unevictable_pgs_munlocked 8463
    unevictable_pgs_cleared 0
    unevictable_pgs_stranded 0
    thp_fault_alloc 1527
    thp_fault_fallback 57
    thp_collapse_alloc 385
    thp_collapse_alloc_failed 0
    thp_split 56
    thp_zero_page_alloc 1
    thp_zero_page_alloc_failed 0
    cpu 1102530 19 853210 28469630 60121 0 26720 0 0 0
    cpu0 544198 10 429478 14228156 31774 0 13552 0 0 0
    cpu1 558332 9 423732 14241474 28346 0 13167 0 0 0
    intr 48668347 2790508 9 0 0 0 0 3 0 0 0 0 0 161 0 0 149767 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    ctxt 75336276
    btime 1468249302
    processes 1632214
    procs_running 2
    procs_blocked 0
    softirq 75384302 0 26751531 1 8817552 1570898 0 17547857 7695307 0 13001156
    <<>>
    Personalities :
    unused devices:
    <<>>
    <<>>
    Mail queue is empty
    <<>>
    [oibsite01]
    liveproxyd 0
    rrdcached 0
    npcd 0
    nagios 0
    apache 0
    crontab 0
    OVERALL 0
    <<>>
    [oibsite01]
    accept_passive_host_checks;accept_passive_service_checks;cached_log_messages;check_external_commands;check_host_freshness;check_service_freshness;connections;connections_rate;enable_event_handlers;enable_flap_detection;enable_notifications;execute_host_checks;execute_service_checks;external_command_buffer_max;external_command_buffer_slots;external_command_buffer_usage;external_commands;external_commands_rate;forks;forks_rate;host_checks;host_checks_rate;interval_length;last_command_check;last_log_rotation;livecheck_overflows;livecheck_overflows_rate;livechecks;livechecks_rate;livestatus_active_connections;livestatus_queued_connections;livestatus_threads;livestatus_version;log_messages;log_messages_rate;mk_inventory_last;nagios_pid;neb_callbacks;neb_callbacks_rate;num_hosts;num_services;obsess_over_hosts;obsess_over_services;process_performance_data;program_start;program_version;requests;requests_rate;service_checks;service_checks_rate
    1;1;60;1;0;1;2329;1.2067419162e-01;1;1;1;1;1;1;32768;0;1;6.3281250000e-02;42674;9.4375605149e-01;21058;3.5845328272e-01;60;1468402568;1468360800;0;0.0000000000e+00;0;0.0000000000e+00;1;0;20;1.2.6p16;743;2.0228296726e-20;0;18999;741516;8.6155502748e+00;14;563;0;0;1;1468310847;3.5.0;4275;3.4787955077e-01;843703;1.5852275571e+00
    <<>>
    <<>>

  • BenV

    @fooc: too bad the html ate half your comment, but I assume there was no mtr section.
    Where is your mtr.cfg, and what happens when you run the plugin manually? Does it give an error?
    Try running it from the plugin dir like this: ./mtr –debug

  • BenV

    @fooc: I forgot you also need to set your config dir:
    export MK_CONFDIR=”/etc/check_mk”
    and then the ‘./mtr –debug’

  • fooc

    cfg file is located in:
    /usr/lib/check_mk_agent
    /usr/lib/check_mk_agent/local
    /usr/lib/check_mk_agent/plugins

    I did the export as you asked. When i give the ./mtr –debug command it says:
    bash: ./mtr: Permission denied

    Under both the root and site user.

    Thanx for your effort

  • fooc

    I think i got the permissions problem:

    [root@localhost plugins]# chmod 755 mtr
    [root@localhost plugins]# ./mtr –debug
    <<>>

  • fooc

    Ah yes its working now!!!! thanx it was the permissions

  • mlist

    Hi BenV

    I’m writing “Check_MK beginner guide” and, as example of adding custom mkp, I chose to include your great plugin
    (obviously mentioning you and your website).
    I have been able to make it working on localhost (where check_MK is running) with the following steps:

    – mkp install /tmp/mtr-0.5.2.mkp
    – cp -pi /opt/omd/sites/mysite/local/share/check_mk/agents/cfg_examples/mtr.cfg /etc/check_mk/
    – cmk -II localhost

    After that I tried to make it working on another centos 7.2 64bit using the same mtr.cfg with:

    – scp /opt/omd/sites/mysite/local/share/check_mk/agents/mtr root@centos7tst1:/usr/lib/check_mk_agent/plugins/
    – scp /opt/omd/sites/mysite/local/share/check_mk/agents/cfg_examples/mtr.cfg centos7tst1:/etc/check_mk/

    The problem is that it doesn’t work and the check_MK gui display the following error:

    UNKNOWN – check failed – please submit a crash report!

    On the check_MK host, running “cmk -d centos7tst1”, the output is:

    <<>>
    **ERROR** Report file /var/lib/check_mk_agent/mtr.report.www-mysite-it has less than 3 lines, expecting at least 1 hop! Throwing away invalid report
    http://www.mysite.it|0|0

    and, indeed, on the centos7tst1 the file contains just one line:

    [root@centos7tst1 ~]# cat /var/lib/check_mk_agent/mtr.report.www-mysite-it
    1477302404

    while on localhost:

    [root@checkmktst1 ~]# cat /var/lib/check_mk_agent/mtr.report.www-mysite-it
    1477302399
    Start: Mon Oct 24 11:46:39 2016
    HOST: checkmktst1 Loss% Snt Last Avg Best Wrst StDev
    1.|– 10.79.160.3 0.0% 10 1.1 1.1 0.6 3.3 0.7
    2.|– 10.72.254.214 0.0% 10 0.8 1.1 0.7 2.9 0.5
    3.|– 10.72.254.229 0.0% 10 1.0 6.4 1.0 47.2 14.4
    4.|– 10.255.248.49 0.0% 10 0.7 1.1 0.6 3.0 0.6
    5.|– ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
    6.|– 10.39.11.138 0.0% 10 11.8 15.7 11.8 32.4 6.3
    7.|– mysite_PUB_IP 0.0% 10 11.9 12.1 11.8 12.7 0.0

    But manually running “mtr mysite_PUB_IP”, the output is the same on both machine.

    Do you have any suggestion?

    Best
    Marco
    Italy

  • BenV

    @mlist: Good luck with your guide, sounds like a good thing to have around for people new to Check_MK 🙂

    As to your problem, on the centos7tst1 machine try running the plugin manually:
    cd /etc/check_mk
    /usr/lib/check_mk_agent/plugins/mtr

    Maybe it’s a python error, but without output it’s hard for me to predict where the problem lies 🙂

  • mlist

    Hi BenV thanks for your kind reply.

    This is the required output:

    On localhost:
    [root@checkmktst1 check_mk]# cd /etc/check_mk/
    [root@checkmktst1 check_mk]# /usr/lib/check_mk_agent/plugins/mtr
    <<>>
    http://www.mysite.com|0|0

    On centos7tst1:
    [root@centos7tst1 ~]# cd /etc/check_mk
    [root@centos7tst1 check_mk]# /usr/lib/check_mk_agent/plugins/mtr
    <<>>
    http://www.mysite.com|0|0

    As you can see the output is the same.

  • BenV

    That output look good. First time it runs it’ll return zeroes because the run hasn’t finished yet.
    Run it again a few minutes later and you should see the zeroes turn into ping times and packet loss percentages etc.

    Can you confirm that this happens? 🙂

  • mlist

    I just found the problem.
    The problem was that check_mk_agent was running under systemd instead of xinetd. I know that the documentation recommend to use xinetd in fact I usually do in this way but because others checks were working correctly, I didn’t think that this could cause problems to some plugins.
    Reinstalling the agent solved the problem.

    Just a curiosity: why in your opinion it didn’t work using systemd?

  • BenV

    @mlist: I’m glad you found the problem.

    It’s hard to diagnose the problem when I can’t look around the system, so no idea why the systemd path didn’t work.
    In my opinion you don’t want to have any port open for check_mk to work, both systemd and xinetd are not necessary.

    The way I always have it running is through ssh.
    You can configure CheckMK to use specific “datasource programs”, which in my case is simply an ssh command with a specific key that’s in the authorized_keys file on the client with a forced command that always executes check_mk_agent.

    This way you don’t have any unecessary ports open and get encrypted data transfer for free without having to muck around with systemd/xinetd 🙂

  • mlist

    This is an interesting point of view since check_mk says that the choise to have an agent running checks locally, has also been done for security reasons and to avoid the use of nrpe or ssh:
    https://mathias-kettner.de/checkmk_security.html
    🙂

    Anyway I really thank you because I think your plugin is really useful. I searched for something like that in the past (using other monitoring solutions) but never found.
    I also thank you for your kindness, I will mention you in my guide; if you are registered in the check_mk english mailing list, you will receive updates about it anyway I’ll write you as soon as it will be available

    bye
    Marco
    Italy

  • BenV

    @mlist: As far as I’m concerned logging in on your clients with a monitoring specific ssh-key that can and will only execute the check_mk_agent program upon login while giving the results back over an encrypted channel is a lot safer than an open port that random people on your network can connect to without authentication to get that data. But maybe that’s just me :p

    Also, most linux systems run an ssh deamon by default anyway, so this attack vector is already present on most systems, suffering mostly from bots that scan for weak passwords.
    If you properly close down your system (i.e. set “AllowUsers” and “PermitRootLogin without-password” aka key-only and while you’re at it set PasswordAuthentication to “no” in sshd_config), you shouldn’t worry about this 🙂

    Will look forward to your guide, and yes I’m subscribed to that list 🙂

  • mlist

    I’m not a security expert but I think that your thought makes perfect sense, even for me.
    I know that this is “out of topic” but, because it is really interesting, I still kindly ask you to reply to these questions:

    a) Apart security aspects, check_MK asserts that, speaking about peformance, the agent based mechanism is its great strength. Using the the ssh method you just described, couldn’t lead to performance problems, in particular in large environments?

    b) Would you available to post some screenshot and steps about the ssh configuration? I could add it to my guide that just reached about 100 pages 🙂

  • BenV

    @mlist:
    a) While I agree that there’s some overhead for setting up the connection, it isn’t much. If you care about microsecond latency on checks that run every X minutes you’re probably looking in the wrong direction 😉
    At my current job we run a reasonably big Check_MK (OMD based) monitoring setup which uses gearman to distribute jobs to slave machines within our clients’ networks, those slaves use SSH to gather the check_mk_agent results. This setup currently has almost 700 hosts and over 10000 services, all based on ssh. Most of the overhead on the main Check_MK machine is due to Apache and Nagvis.

    b) I could do that, but I don’t have much time for this right now, I’ll see if I can make a post on this coming thurdsay.
    However, it’s not rocket science as I said 😉
    I assume you’re mainly interested in the ssh configuration through WATO.
    Short version:
    – WATO — Host and Service Parameters — DataSource Programs — Individual Program Call instead of agent access — Create Rule — Command line to execute: “ssh -l root -i /etc/check_mk/keys/yourmonitoringkey <IP>”

    Hope this helps 🙂

  • mlist

    a)
    Ok BenV, it sounds like you are a really skilled person, surely much more than me; I love to speak with people like you because I always learn something.
    Morover I just thought that this kind of configuration (ssh instead of tcp 6556) could help in situations where is difficult or even impossibile to open ports different that 22. You just did us a nice tip.

    b)
    Yes, I’m interested in the ssh configuration through WATO and a step by step post would be appreciated but you already driven me in the right direction thus if you have not time to write it, I’ll try by myself. Anyway I’ll keep my eyes on your website.

    Thanks again BenV!

    ciao
    Marco
    Italy

  • balroq

    You wouldn’t happen to have the script running without check_mk. I mean as simple command line script?

  • tintamar

    Hi BenV,

    I have just tested your plugin with success.

    Do you have some advice on where i should look to modify the display of the hop overview, since i am also interested in the IP of the hop number

    Thks

  • BenV

    @tintamar: Since those addresses often differ from one run to the next (different hop, hop with multiple ips, etc) I chose not to implement it that way.
    If you want to alter the graphs you can try editing check_mk-mtr.php in the pnp-templates, but you’ll also need to update the mtr check itself to put those IP addresses somewhere as performance data. IIRC it doesn’t do anything with this data right now, although the agent does provide the IPs in its output.

    BTW: If you get something fancy working I’m always interested in patches, feel free to email me 🙂

  • dheerugudi

    I have installed the plugin and placed the config file as mentioned above, but I get this error on checkMk dashboard
    “Invalid check parameter: Undefined key ‘rta’ in the dictionary. Allowed are avg, stddev, loss.! The parameter is: {‘loss’: (10, 25), ‘rta’: (150, 250), ‘rtstddev’: (150, 250), ‘stddev’: (150, 250), ‘avg’: (150, 250), ‘pl’: (10, 25)} The variable name is: checkgroup_parameters:mtr” did I miss anything?

  • BenV

    @dheerugudi:
    That’s fascinating, where the heck did those keys come from?

    Are you by chance running the enterprise version of Check_MK?
    If so: I know they reworked my plugin and included it (with my permission), but it’s a somewhat different version from the one available here, so it’s possible that the information on this page is not applicable to that version.
    If not: can you show your configuration? The keys ‘rta’ and ‘rtstddev’ and completely unknown to me, the plugin and this page never mention those — where do they come from?
    The only parameters my plugin supports are ‘loss’, ‘avg’, and ‘stddev’.

  • dheerugudi

    @BenV

    I am using free edition.
    Let me explain you the steps I followed to setup mtr plugin
    On checkmk agent
    1) Downloaded and installed V0.5.2: mtr-0.5.2.mkp from your website
    2) Created mtr folder and copied mtr.cfg under /usr/lib/check_mk_agent/plugins/mtr/mtr.cfg
    3) Placed mtr.cfg under /etc/check_mk/mtr.cfg

    On the check_mk dashboard for mtr config

    Hosts–>Manual Checks–>Traceroute with MTR–>(here I have created the rule) MTR destination=google.com saved it

    After that the gives that error.
    Let me know If I am missing anything.. Thanks

Leave a Reply

You must be logged in to post a comment.

Archives

  • 2018 (1)
  • 2016 (1)
  • 2015 (7)
  • 2014 (4)
  • 2013 (11)
  • 2012 (27)
  • 2011 (26)
  • 2010 (25)
  • 2009 (68)