Thursday, November 21, 2013

This post will go over how to add graphs to nagios using the pnp4nagios plugin.

Step 1:
Install and/or compile the necessary applications.  pnp4nagios was installed as part of the command "yum install nagios*" performed in a previous post.

rpm -q pnp4nagios
pnp4nagios-0.6.20-1.el6.i686

Step 2:
Configure the pnp4nagios commands.  Since this is a small installation with only a few hosts being monitored, most of the defaults will be used.  However, the commands still need to be added to the nagios configuration:

define command {
command_name    process-service-perfdata-file
command_line    /usr/libexec/pnp4nagios/process_perfdata.pl --bulk=/tmp/service-perfdata
}

define command {
command_name    process-host-perfdata-file
command_line    /usr/libexec/pnp4nagios/process_perfdata.pl --bulk=/tmp/host-perfdata
}

Step 3:
Modify nagios.cfg.  The diff with the original cfg is shown below.

diff nagios.cfg nagios.cfg.bak
834,835c834
< 
< process_performance_data=1
---
> process_performance_data=0
857,858c856,857
< host_perfdata_file=/tmp/host-perfdata
< service_perfdata_file=/tmp/service-perfdata
---
> #host_perfdata_file=/tmp/host-perfdata
> #service_perfdata_file=/tmp/service-perfdata
872,873d870
< host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$
\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$
\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$ < service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$
\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$
\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$
\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$ 887,888d883 < host_perfdata_file_mode=a < service_perfdata_file_mode=a 900,901d894 < host_perfdata_file_processing_interval=15 < service_perfdata_file_processing_interval=15 912,913d904 < host_perfdata_file_processing_command=process-host-perfdata-file < service_perfdata_file_processing_command=process-service-perfdata-file

Step 4:
Restart nagios and verify the page shows up at http://<nagiosip>/pnp4nagios/


Step 5:
Add extended info in nagios to create links to graphs of the applicable host and service.  Add to templates.cfg:

define host {
name            host-pnp
action_url      /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_
register        0
}

define service {
name            service-pnp
action_url      /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=$SERVICEDESC$
register        0
}

Then add these new templates to the desired host and service definitions.  For example:

define host{
        use                     linux-server,host-pnp
        host_name               puppetmaster
        alias                   puppetmaster
        address                 192.168.1.15
}

define service{
        use generic-service,service-pnp
        host_name puppetmaster
        service_description PING
        check_command check_ping!100.0,20%!500.0,60%
}

Step 6:
Verify functionality.  Note the new graph icons available for the host/services.

Clicking on the icons gives links to the desired rrd graphs.  For example, the custom crond check shows the process has been running as desired.

pnp4nagios can also print out a nicely formatted report of the desired services/hosts to a pdf.

Tuesday, November 19, 2013

This post will go over how to add a custom check to a host being monitored by nagios.  In this case, nagios will check to make sure crond is running on the puppetmaster server, which is a centos machine.

Step 1:
Write the script that will check for the given condition, and verify its functionality.  This is a simple script that will check that the crond process is running. The data after the pipe is interpreted by nagios as performance data, and is being added in so that the status of the process can be graphed over a period of time in an rrd graph. Adding graphs to nagios will be covered in another post.

Note that the exit codes get interpreted by nagios as follows:
0 - OK
1 - WARNING
2 - CRITICAL
3 - UNKNOWN

#!/bin/bash

lineCount=`ps -eaf|grep -v grep|grep " crond"|wc -l`

if [ $lineCount -eq "0" ]; then
        echo "WARNING - crond is not running|proc=$lineCount"
        exit 1;
fi
if [ $lineCount -eq "1" ]; then
        echo "OK - crond is running|proc=$lineCount"
        exit 0;
fi
if [ $lineCount -gt "1" ]; then
        echo "UNKNOWN - crond process count > 1|proc=$lineCount"
        exit 3;
fi
echo "UNKNOWN - crond process count is unknown|proc=$lineCount"
exit 3;

Step 2:
Add the command to /etc/nagios/nrpe.cfg on the host.  The necessary line to add in this case is:

command[check_crond]=/usr/lib/nagios/plugins/check_crond

Step 3:
Add the service check to the nagios server.  Restart the nagios process.

define service{
        use generic-service
        host_name puppetmaster
        service_description Crond Process
        check_command check_nrpe!check_crond

}

service nagios restart

Step 4:
Verify functionality.

Thursday, November 14, 2013

This post will go over how to monitor a host in nagios using nrpe.  The nrpe is the "Nagios Remote Plugin Executor", and allows you to remotely execute commands on another machine and gather desired metrics.  The version of nrpe that was installed on the target centos machine does not allow for command line arguments from the nagios server, so all arguments and thresholds must be specified on the machine itself.  The puppetmaster server is the machine being added to nagios.

Step 1:
On the target machine, add the EPEL repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Step 2:
Install nagios and necessary packages.

yum install nagios* xinetd

Step 3:
Add the service to xinetd.

/etc/xinetd.d/nrpe
service nrpe
{
flags = REUSE
type = UNLISTED
port = 5666
socket_type = stream
wait = no
user = nagios
group = nagios
server = /usr/sbin/nrpe
server_args = -c /etc/nagios/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 192.168.1.16
}

Restart, the service

service xinetd restart
chkconfig xinetd on 

Step 4:
Make any changes or modifications to the data you want to monitor on the host in /etc/nagios/nrpe.cfg.  In this scenario, the root partition, number of users, current load, and number of processes are being monitored.

Step 5:
Configure the nagios server.
/etc/nagios/objects/commands.cfg
define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Add the host, service, and hostgroup definitions to the necessary config files in /etc/nagios.  The puppet server is being added to the "Linux Servers" hostgroup.

Restart the service
service nagios restart

Step 6:
Verify functionality.

Tuesday, November 12, 2013

This post will go over installing nagios on a centos machine that lives on a vmware esxi host.  Nagios is an open source monitoring and alerting system that is widely deployed as an infrastructure monitoring solution and can scale from one to thousands of hosts and services.  This post will go over a basic installation.

Step 1:
Clone a machine.

cd /vmfs/volumes/datastore1
mkdir "CentOS 6.4 - Nagios Server"
cd CentOS\ 6.4\ -\ Nagios\ Server/
cp ../Base CentOS\ 6.4/Base CentOS\ 6.4.vmx ./CentOS\ 6.4\ -\ Nagios\ Server.vmx
vmkfstools -i ../Base CentOS\ 6.4/Base CentOS\ 6.4.vmdk \
CentOS\ 6.4\ -\ Nagios\ Server.vmdk
vim-cmd solo/registervm \
/vmfs/volumes/datastore1/CentOS\ 6.4\ -\ Nagios\ Server/CentOS\ 6.4\ -\ Nagios\ Server.vmx
vim-cmd vmsvc/power.on 21

Step 2:
If necessary, reset the ip addresses and the interfaces in /etc/sysconfig/networ-scripts/ifcfg-*, modify udev rules in /etc/udev/rules.d/70-persistent-net.rules, reset the hostname in /etc/sysconfig/network, reset the root password.

Step 3:
Add the EPEL repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Step 4:
Install nagios and necessary packages.

yum install -y nagios* openssl gd gd-devel httpd php gcc glibc glibc-common httpd

Step 5:
Set up apache and enable services.

htpasswd /etc/nagios/passwd nagiosadmin
chkconfig httpd on
chkconfig nagios on
service httpd restart
service nagios restart

Step 6:
Verify the system is up and monitoring the localhost at http://<systemip>/nagios

A later post will go over adding new hosts and services to the setup.

Thursday, November 7, 2013

This post will go over how to set up a machine so that the newly installed puppet master can control the system.

Step 1:
Install the puppet application on the local machine.
rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-7.noarch.rpm
yum install puppet

Step 2:
Make necessary config changes to the system.

Edit the hosts file
192.168.1.10 node
192.168.1.15 puppetmaster

Edit the puppet.conf file
server = puppetmaster
report = true
pluginsync = true

chkconfig puppet on
puppet agent --daemonize

Step 3:
Add the certs.
puppet agent --server=puppetmaster -t --waitforcert 15
Notice: Did not receive certificate
Notice: Did not receive certificate
Notice: Did not receive certificate
Notice: Did not receive certificate
Notice: Did not receive certificate
Info: Caching certificate for server1.node
Info: Caching certificate_revocation_list for ca
Info: Retrieving plugin
Info: Caching catalog for server1.node
Info: Applying configuration version '1382998929'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.02 seconds

Sign the cert on the puppetmaster.
puppet cert list
  "server1.node" (SHA256) 8E:09:B0:E9:9C:76:99:4A:94:53:5C:39:FD:3A:32:DA:D3:FF:7C:64:F4:BF:6A:83:40:8F:97:E5:FA:5F:BF:87
puppet cert --sign server1.node
Notice: Signed certificate request for server1.node
Notice: Removing file Puppet::SSL::CertificateRequest server1.node at '/var/lib/puppet/ssl/ca/requests/server1.node.pem'

Step 4:
Create and test a manifest for the configured nodes
The canonical example, installing ntp.
Adding to site.pp
package { 'ntp':
        ensure => installed,
}
file { '/etc/ntp.conf':
        path       => '/etc/ntp.conf',
        ensure     => file,
        require    => Package['ntp'],
}
service { 'ntpd':
        name       => 'ntpd',
        ensure     => running,
        enable     => true,
        require    => Package['ntp'],
        subscribe  => File['/etc/ntp.conf'],
}

On the new node:

rpm -q ntp
package ntp is not installed
puppet agent --server=puppetmaster --test
Info: Retrieving plugin
Info: Caching catalog for server1.node
Info: Applying configuration version '1382999615'
Notice: /Stage[main]//Package[ntp]/ensure: created
Notice: Finished catalog run in 5.41 seconds
rpm -q ntp
ntp-4.2.4p8-3.el6.centos.i686


Tuesday, November 5, 2013

This post will go over how to install a puppet server and a puppet client on a CentOS 6.4 vm running on VMWare 5.1.

Step 1:
Clone a vm.

cd /vmfs/volumes/datastore1
mkdir "CentOS 6.4 - Puppet Server"
cd CentOS\ 6.4\ -\ Puppet\ Server/
cp ../Base\ CentOS\ 6.4/Base\ CentOS\ 6.4.vmx ./CentOS\ 6.4\ -\ Puppet\ Server.vmx
vmkfstools -i "/vmfs/volumes/datastore1/Base CentOS 6.4/Base CentOS 6.4.vmdk" \
"/vmfs/volumes/datastore1/CentOS 6.4 - Puppet Server/CentOS 6.4 - Puppet Server.vmdk"
vim-cmd solo/registervm "/vmfs/volumes/524734d7-f389d00a-4f68-b870f4dd73cf/CentOS 6.4 \
- Puppet Server/CentOS 6.4 - Puppet Server.vmx"
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/power.on 18


Step 2:
Install the puppet server.  Add the puppetlabs repository.

rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-7.noarch.rpm

Install the application.

yum install puppet-server

Step 3:
Enable services.  Start the puppet master.

/etc/init.d/puppetmaster start

Permanently enable services.

puppet resource service puppet ensure=running enable=true
puppet resource service puppetmaster ensure=running enable=true

Modify the config file.

/etc/puppet/puppet.conf
[master]
certname = puppetmaster
autosign = false

Step 4:
Install passenger.  First, install necessary packages:

yum install httpd httpd-devel mod_ssl ruby-devel rubygems gcc make gcc-c++ \
curl libcurl-devel openssl-devel

Install passenger.  The output of the second command will display how to configure the apache vhost.

gem install rack passenger
passenger-install-apache2-module

Install the puppet master rack application

mkdir -p /usr/share/puppet/rack/puppetmasterd
mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
cp /usr/share/puppet/ext/rack/files/config.ru /usr/share/puppet/rack/puppetmasterd/
chown puppet /usr/share/puppet/rack/puppetmasterd/config.ru

Step 5:
Sign certs of new machines.  Another post will go through how to add a node to the server, but the two commands needed are:

puppet cert list
puppet cert --sign