February 16, 2014

How to Setup OMD in 1 Hour

By Hank Huang 2/16/2014 03:37:00 AM

What is OMD

OMD bundles Nagios together with many important add-ons and can easily be installed on every major Linux distribution. We provide pre-built packages for all enterprise Linux distributions and also for some other, such as Ubuntu.

Want to learn more about what OMD is about and what it can do for you? Check out my other post - Best Monitoring Solution - OMD (Nagios + Check_MK)

enter image description here

Install OMD Server on Ubuntu 12.04

Setup OMD Repository

Run the following on the Ubuntu server command line.

Stable

gpg --keyserver keys.gnupg.net --recv-keys F8C1CA08A57B9ED7
gpg --armor --export F8C1CA08A57B9ED7 | apt-key add -
echo 'deb http://labs.consol.de/repo/stable/ubuntu precise main' >> /etc/apt/sources.list.d/omd-stable.list
apt-get update

Testing-Release

gpg --keyserver keys.gnupg.net --recv-keys F8C1CA08A57B9ED7
gpg --armor --export F8C1CA08A57B9ED7 | apt-key add -
echo 'deb http://labs.consol.de/repo/testing/ubuntu precise main' >> /etc/apt/sources.list.d/omd-testing.list
apt-get update

Install OMD

apt-get install omd

And then just follow the instructions on the screen.

Setup Check_MK Agent on the Monitoring Target

You want to install the Check_MK Linux agent on the server you want to monitor - a remote host. The Check_Mk agent version on the repository is not up to date, so we will just download it from the website.

Install Check_MK Agent

Go to the check_mk download page. The Check_MK folks have removed the easy to install Ubuntu/CentOS packages since they also need to make a living. You can get those (and other cool features) by subscribing to their supported service. But for those of us who are just trying it out or staying on the open source version. You can find the agent by appending /agents at the end of your OMD server URL like so:

http://OMD-URL/SITE-NAME/check_mk/agents/

This is what you will see, and check_mk_agent.linux is what you want to use:

enter image description here

You want to save the check_mk_agent.linux file under /usr/bin/check_mk_agent on the target server. Make it executable by running chmod +x check_mk_agent

Setup xinetd Connection

First, make sure you have xinetd service installed on your target host. It is used to open a tcp port 6556 for the OMD server to connect to and execute the Check_MK agent this way. If xinetd is not yet installed, install it now:

apt-get install xinetd

By default, the xinetd connection configuration will be setup by the package itself and reloaded. Since the package is no longer available, We want to download the file from the same /agents page called xinetd.conf and save it under /etc/xinetd.d/check_mk. Secure the connection to allow only the OMD server to connect to the xinetd port. Edit /etc/xinetd.d/check_mk, uncomment the following line and add the OMD server’s IP besides 127.0.0.1

only_from      = 127.0.0.1 xx.xx.xx.xx

Now reload xinetd service

service xinetd reload

Patch for MegaCli Check (Optional)

You want to do this if you are running hardware RAID with LSI raid controller on your servers, otherwise you can skip this step.

MegaCli command currently doesn’t work with Linux kernel newer than 2.6 without a fix. Create a bash script using the following example but change the path of the original MegaCli command relative to your environment. Save the script under /usr/local/bin/ and name it MegaCli. Make it executable and it will work.

#!/bin/bash

setarch x86_64 --uname-2.6 /usr/local/sbin/MegaCli $@

change the path above to locate your MegaCli command accordingly.

Setup Firewall (Optional)

You can also skip this step if you don’t care about security. The following iptable rules will allow only the OMD server to contact this target host through tcp port 6556 and ping test. And this is assuming you have everything else locked down in the iptable rules, or else this is useless.

# Monitoring Agent communication
/sbin/iptables -A INPUT -p tcp -s 1.1.1.1 --dport 6556 -j ACCEPT
/sbin/iptables -A INPUT -p icmp -s 1.1.1.1 --icmp-type 8 -j ACCEPT
/sbin/iptables -A OUTPUT -p icmp -d 1.1.1.1 --icmp-type 0 -j ACCEPT

Change the 1.1.1.1 IP address to your OMD server IP address

Start Monitoring Target Hosts

Now that you have successfully installed and enabled the Check_MK agent on a target host. Let’s start monitoring this little pain in the butt without further ado.

Log in to Check_MK Multisite interface with admin privilege. Under the WATO - Configuration section on the left hand side panel. Click on Host & Folders, then on the right hand side of the page, click the New Host button. You will see the following:

enter image description here
Enter the host name or IP address for the Hostname field. If you entered FQDN instead of IP address, make sure it is able to translate by DNS server to an IP address that is pointing to the host you want to monitor. By default, the Agent Type for monitoring any new host is using Check_MK Agent (Server), so all you need to do is just enter Hostname information and click the Save & go to Services button at the end.

Your OMD server is now connecting to the check_mk agent on the target host. In a few seconds, you will see a list of available services to be monitored on the browser. Click on Activate missing button and you will see on the top of the page it tells you that you’ve made 1 change to the system.

enter image description here
You want to click on that button that tells you how many changes you have and then followed up by clicking Activate Changes! button. Now you should see the host and its services are now being monitored by the OMD server. Give it a few more minutes to populate some monitoring data to give you some visuals in the PNP4Nagios graphs.

enter image description here
Look how beautiful it is with all the basic health information you need in nice descriptive format and dynamic historical performance graph at your finger tips. This is just the tip of the iceberg of what OMD can do for you. I will be showing you the more advanced stuff from here on.

Now you know how to setup agents and add target hosts to be monitored, we are going to move on and find out how can we have the OMD system send notifications when things go south on any of your servers or services.

Add Custom Monitoring Script

This is a bigger topic which will be covered in future post. In short, there are 4 types of checks that sits on the target host instead of the OMD server.

  1. Check_MK local script
  2. Check_MK plug-in script
  3. Logwatch plug-in
  4. Nagios plug-in via MRPE

Setup Notification

enter image description here
Notifications are important if your company is providing services with 99.999% high availability. You want your system engineers to jump on the issues the moment they happen. The goal is to have incidents resolved before customer even notice it. So having a great notification methodology is vital for establishing a 24 on call framework. OMD(Check_MK) has just the right tools to help you acheive that.

Flexible Notification from Check_MK

Source Document
Require Check_MK version > 1.2.1i2

Each user can have multiple notification methods with separate filters. That way you can for example send always emails, but SMS only for certain services or in certain time periods - without creating multiple contacts per user which is a huge shortcoming in traditional Nagios!

For each notification the Nagios core will simply pass them to the Check_MK flexible notification feature while providing all necessary information about the alert. Then Check_MK will process those notification however you want via the configuration in the GUI (WATO).

Preparing Notifications in General

Setting up the notifications requires some care. If you miss one of the following steps then the most probable case is that simply nothing will happen. You need to do the following:

  1. Create a contact group. WATO users: this is done with the module Contact Groups.
  2. Important: put some hosts/services into that contact group. WATO: Host & Service Parameters -> Grouping -> Assignment of hosts/services to contact groups.
  3. Create a user and put him into that contact group. Enable notifications for that user.
  4. Enter an email address. WATO: Users & Contacts

  5. Make sure that your monitoring server is correctly setup so that it can send emails. Test this with:

    root@linux# echo "Mailbody" | mail -s "Testsubject" test@example.com
  6. Restart your core with cmk -R in Linux command line or activating changes in WATO.

Testing and Debugging

When dealing with notifications it’s important to know how to test and debug your setup. A very easy way to send test-notifications is to pick a random service that is contained in the upper contact group, open the Commands dialog and select Custom Notification:

enter image description here

  1. Check nagios log
    ~/var/log/nagios.log
  2. Check Check_MK notification log
    ~/var/check_mk/notify/notify.log

WATO users find this setting in Global Settings -> Notifications -> Debug notifications. The resulting log file is in the directory notify below Check_MK’s var directory. OMD users find the file in ~/var/check_mk/notify/notify.log. Isn’t it great to have fixed paths in OMD?

Send Notification through Email to SMS Gateway

enter image description here
Some countries’ mobile carriers support sending Email to their SMS gateway which will convert that Email message to a SMS(text) message to the destination mobile phone number. For example if you have a 5555555555 T-Mobile phone number in the USA, you can send SMS message to that number with Email using the following Email address:

5555555555@tmomail.net

Having notifications being sent to mobile devices are extremely helpful if you are looking to setup some kind of 24/7 on call monitoring. I wrote a custom notification script that will let you send notification through Email to SMS gateway with the bare minimum information you need from an alert.

Download the most updated script email-sms-custom.sh from Github.

Save my script into this directory on your OMD server and make it executable:
/omd/site/YOURSITE/local/share/check_mk/notifications/email-sms-custom.sh

Check_MK will consider every executable file in this directory as a notification script. You can use any programming language you want. If you use a scripting language (Shell, Perl, Python, etc.) then you can use a comment in the second line for giving the script a name. This name will appear in the UI for you to choose in a drop down list of different notification methods. I have named the script on the 2nd line to be SMS via Email to SMS Gateway.

Here is a list of all the Email to SMS Gateway provider in the world. Use them and test them at your own risk. All SMS message fees still applies.

OMD Maintenance

OMD Upgrade

http://www.mail-archive.com/checkmk-en@lists.mathias-kettner.de/msg07525.html

http://deranfangvomende.wordpress.com/2011/06/07/omd-update-0-42-to-0-49/
- OMD with lastest Check_MK version
- Download Demo Version

OMD Backup/Migrate

  1. Install the same version of OMD package on your new hardware
  2. Stop omd on the source server
  3. Copy /omd/sites/ to your new target server.
  4. Mind the permission! Create the same “Site” user on the target server
  5. Start OMD on your target server

High Resolution Graphing for PNP4Nagios for One Year

When you have longer period of graphing data on PNP4Nagios, zooming into year graph down to the day seems to have averaged out greatly. You can’t rely on the graph to dig up any historic event metrics this way.

To address the issue, edit rra.cfg in your pnp4nagios config. Documentation is on rrdtool and pnp4nagios websites.
http://serverfault.com/questions/534186/how-to-change-the-amount-of-data-stored-in-pnp4nagiosomd
http://apfelboymchen.net/gnu/rrd/create/