Setting Up nginx As a Proxy on CentOS

This post will go over how to set up nginx to proxy and load balance over multiple servers on CentOS.

Step 1:
Install nginx.  If you have not already done so, install the epel repository:

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
yum install nginx

The configuration file is stored in /etc/haproxy/haproxy.cfg.  You may want to back this file up as it will be modified in the next step.  You can also set up the service to start at boot:

chkconfig nginx on

Step 2:
Set up the server configuration.  The configuration file that will be modified is /etc/nginx/conf.d/default.conf.  You may want to back up this file before proceeding.  Some of the changes that will be made to this file are shown below.

server{
     listen 1.1.1.1:80;
     server_name example.com;

     location / {
          proxy_pass http://webservice;
     }
}

upstream webservice{
     server 10.0.0.2:80;
     server 10.0.0.3:80;
}

Note that the backend servers will be apache servers.

Step 3:
Set up the backend servers defined in haproxy.cfg.  In this example, just the apache test page is being used.  Apache can be installed with a simple

yum install httpd
service httpd on

Step 4:
Verify functionality.Checking the access logs on the nginx machine at /var/logs/nginx/access.log show the client making the connection.  Checking the apache logs at /var/logs/httpd/access.log on the servers behind the proxy show the requests being load balanced across both machines.  Taking down one of the apache servers results in the site staying up, but requests only being passed to the active server.

Setting Up HAProxy on CentOS

This post will go over installing and setting up HAProxy on CentOS.  HAProxy is a load balancer, and in this case, will have a public ip of 1.1.1.1 with 10.0.0.0/8 as the private network load balancing http.

Step 1:
Install HAProxy.  If you have not already done so, install the epel repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
yum install haproxy

The configuration file is stored in /etc/haproxy/haproxy.cfg.  You may want to back this file up as it will be modified in the next step.  You can also set up the service to start at boot:

chkconfig haproxy on

Step 2:
Set up the haproxy.cfg file.  There are a number of configuration options for haproxy, most of which can be found here.  The below configuration is how this system will be set up.

global
  maxconn 2000 # this system only has 512MB of memory
  daemon
  nbproc 1 # specify the number of processors to use
  log 127.0.0.1 local2
  user haproxy
  group haproxy
defaults
  log global
  mode http
  option httplog
  option dontlognull
  option http-server-close
  option forwardfor except 127.0.0.0/8
  option redispatch
  retries 3
  timeout http-request 10s
  timeout queue 1m
  timeout connect 10s
  timeout client 1m
  timeout server 1m
  timeout http-keep-alive 10s
  timeout check 10s
  maxconn 1900
listen webservice 0.0.0.0:80
  mode http
  status enable
  stats uri /haproxy?stats
  stats realm Private
  stats auth admin:password
  balance roundrobin
  option httpclose
  option forwardfor
  server server2 10.0.0.2 check
  server server3 10.0.0.3 check

Once the configuration is set up, start the service:

service haproxy start

Step 3:
Set up the backend servers defined in haproxy.cfg.  In this example, just the apache test page is being used.  Apache can be installed with a simple

yum install httpd
service httpd on

Step 4:
Verify functionality.

Upon the initial connection, the stats page already shows some data.

A couple of page refreshes shows the counters going up and the load being spread across the two servers.

To get a better sense of how the system will perform, the FreeBSD tool siege will be used with 1000 concurrent hits.

Shutting down apache on one of the servers shows that the web page remains reachable, but all of the connections go to the one remaining server.

Iptables Basics IV

This post is a continuation of the previous post on some of the basic functions of the linux firewall.

To match a specific set of tcp flags, execute:

iptables -A INPUT -p tcp --tcp-flags FIN,PSH,URG FIN,PSH,URG -j DROP

--tcp-flags - Match the specified tcp flags.  The first group of flags specifies the tcp flags to examine, in this case, FIN,PSH,URG.  The second group of flags specifies which flags must be set, in this case, FIN,PSH,URG.  This statement explicitly drops the Xmas tree scan from nmap.  If you wanted to make sure a specific flag was unset, you would leave it off of the second set of flags, e.g. --tcp-flags FIN,PSH,URG FIN,PSH.  This statement would examine FIN,PSH,URG, and match if FIN and PSH were set, and URG was unset.  You can use an exclamation point to reverse the meaning of the match, e.g.  ! --tcp-flags FIN,PSH,URG FIN,PSH,URG would match when FIN,PSH,URG are unset.

To limit the number of connections, execute:

iptables -A INPUT -p tcp --tcp-flags SYN SYN -m limit --limit 3/s --limit-burst 5 -j ACCEPT
iptables -A INPUT -p tcp --tcp-flags SYN SYN -j LOG --log-prefix SYN-ATTACK:
iptables -A INPUT -p tcp --tcp-flags SYN SYN -j DROP

-m limit - use the limit module
--limit - Specifiy the maximum average matching rate.  Can also specify /m (minutes), /h(hour), /d(day).
--limit-burst - Specify the maximum initial number of packets to match.  This value gets reset every time the --limit that is specified is not reached.

The statements following the limit statement log any packets that go above the threshold and drop them.

Iptables Basics III

This post is a continuation of the previous post on some of the basic functions of the linux firewall.

To block traffic at layer 2 by mac address, execute:

iptables -A INPUT -m mac --mac-source 00:11:22:33:44:55 -j DROP

-m - specifies an extension module to test for a specific property.  You can see a list of possible command line options for a specific module by adding on a -h after the "-m mac" option, in this case, --mac-source is the only available option for this module.  You can see a list of all possible modules in the man page of iptables-extensions.
--mac-source - source mac address of the ethernet frame to drop.

To block ping requests, execute:

iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

-p icmp - specifies the protocol of the packet to check
--icmp-type - the type of icmp packet to match.  A full list of possibilities can be seen by executing iptables -p icmp -h.  You can also add on --reject-with option if you want to provide more information to the source of the ping.

To block a range of addresses, execute:

iptables -A INPUT -m iprange --src-range 1.1.1.1-1.1.1.10 -j DROP

-m iprange - specifies an extension module to test for a specific property.  You can see a list of possible command line options for a specific module by adding on a -h after the "-m iprange" option, in this case, --src-range and --dst-range are the only available option for this module.

Iptables Basics II

This post is a continuation of the previous post on some of the basic functions of the linux firewall.

To block all incoming traffic, but allow outgoing traffic, execute:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -m state --state NEW,ESTABLISHED -j ACCEPT

-m - specifies an extension module to test for a specific property.  You can see a list of possible command line options for a specific module by adding on a -h after the "-m state" option, in this case, --state is the only available option for this module.
--state - specifies state to match, which can be [INVALID|ESTABLISHED|NEW|RELATED|UNTRACKED]
-j - jump to the specified target chain.

The last rule is saying to allow NEW and ESTABLISHED connections back in; in other words, any connection that originated from the host.

To block a specific source ip address, execute:

iptables -A INPUT -s 1.1.1.1 -j DROP

To block an entire source network, execute:

iptables -A INPUT -s 1.1.1.1/8 -j DROP

To block a specific port, execute:

iptables -A INPUT -p tcp --dport 80 -j DROP

-p - specify the protocol to inspect
--dport - specify the port to match

To block traffic to a specific destination host or destination subnet, execute:

iptables -A OUTPUT -d 1.1.1.1 -j DROP
iptables -A OUTPUT -d 1.1.1.1/8 -j DROP

Iptables Basics I

This post will go over some of the basic functionality of iptables, the linux firewall.

To reset the rules of the firewall, execute:

iptables -F // delete every rule in the chain for the table
iptables -X // delete every non-builtin chain in the table
iptables -t nat -F // default table is filter, need to specify other tables.
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

To display the rules of the firewall, as well as packet count/byte count, execute:

iptables -t <table> -L -n -v --line-numbers

-t - specify the table to act on, default is filter if nothing is specified.
-L - list the chains in the default table, which is the filter table.  Use -t to switch tables.
-n - list ip addresses and ports as numbers.
-v - verbose output.
--line-numbers - show the line number of the rule in the chain.

To dump the iptables rules to stdout, execute:

iptables-save > rules.txt

Rules that get printed out from the format above can be restored by executing:

iptables-restore < rules.txt

To delete a specific rule, use the output from above to get the line number of the rule, and execute:

iptables -D <chain> <line>
e.g. iptables -D INPUT 1

To insert a rule, execute:

iptables -I <chain> <line-number> <rule>
e.g. iptables -I INPUT 2 -s 1.1.1.1 -j DROP

To set the default policy for a chain, execute:

iptables -P <chain> <policy>
e.g. iptables -P INPUT DROP

To log a rule, execute:

iptables <rule> -j LOG --log-prefix "Log Rule: "
e.g. iptables -s 1.1.1.1 -j LOG --log-prefix "Log Rule: "

Note that --log-prefix can be up to 29 characters long, and is useful for grepping through logs.  You may want to also limit the number of logs generated.  This can be done by adding on:

-m limit --limit <average-limit> --limit-burst <burst-limit>
e.g. iptables -s 1.1.1.1 -m limit --limit 3/m --limit-burst 5 -j LOG \
--log-prefix "Log Rule: "

Note that this rule is non-terminating, meaning after executing, the next rule will be executed.

Booting into Rescue Mode in CentOS

This post will go over booting into rescue mode in CentOS.  To be able to boot into rescue mode, you will need to boot to the install media and select "Rescue installed system".

You will then need to pick a language, keyboard, and whether or not you want networking enabled.  Depending on the situation, you can then try to mount the system as read-write, read-only, or skip mounting the system.

Assuming the system has started successfully and the filesystems have been mounted successfully, you are now in runlevel 1, you now have a temporary root filesystem at /, and the actual system filesystems exist under /mnt.

You can then chroot to the filesystem to be able to execute commands as if you had booted to the actual os.  This will allow you to troubleshoot the system without the need for a full boot.

If all you need is root access to the local filesystem, you can boot into runlevel 1 from the grub boot screen by appending the word "single" at the end of the kernel line.

Using Grub From the Command Line in CentOS

This post will go over some basic commands you can use in the Grub CLI to try to boot a system that is having trouble.

When the system is coming up, be sure to press a key at the initial grub screen so the system does not try to boot automatically.  Once at the OS selection screen, press "c" to enter the grub cli.

In the grub cli, you can issue "help" for a list of commands.

To see what hard disks are available, you can use auto completion.  Start by typing in "root (hd" then tab to get a list of hard disks and partitions.  In this case, there is only one hard disk, and two partitions.

You can now do the same thing to view a list of files on the ext2 boot partition.  Type in "kernel (hd0,0)/" then tab to see a list of files.  In this case, vmlinuz-2.6.32-431.el6.i686 looks like a usable kernel.

If necessary, you can also add in the initial ramdisk.

You can now try booting the system with the "boot" command.

Creating Jail Flavours in FreeBSD

This post will go over how to create flavors in FreeBSD.  Flavours are directories that are copied over to a jail upon creation, and can make commonly created jails easier to deploy.

Step 1:
Create a directory for the flavour you want to have available.  This setup will have one "vanilla" jail that all other jails will be built from.  This vanilla jail is built from the example.local jail in the previous post.

cp -R /usr/jails/example.local /usr/jails/flavours/vanilla

Step 2:
Create a new jail using the "vanilla" flavour.  Be sure to add the necessary ip address to rc.conf, and restart the netif service.

echo 'ifconfig_em0_alias1="inet 10.0.2.21 netmask 255.0.0.0"' > > /etc/rc.conf
/etc/rc.d/netif restart
ezjail-admin create -f vanilla nginx01.local 10.0.2.21

Step 3:
Console in to the newly created jail and add the necessary packages and make any changes for the nginx flavour.

ezjail-admin console nginx01.local
pkg install nginx
echo 'nginx_enable="YES"' >> /etc/rc.conf'

Step 4:
Exit and stop the jail.  Copy the directory of the nginx01.local jail to a new directory under flavours, in this case, webserver-nginx.  Delete the original jail.  Note that 10.0.2.21 is now available for re-use in another jail.

ezjail-admin stop nginx01.local
cp -R nginx01.local flavours/webserver-nginx
ezjail-admin delete -w -f nginx01.local

Step 5:
You now have a "webserver-nginx" flavour available for easy creation.  The next time you need a jail for hosting nginx, be sure to have a free ip and execute

ezjail-admin create -f webserver-nginx nginx02.local 10.0.2.21
ezjail-admin start nginx02.local

Creating a Jail in FreeBSD

This post will go over how to create a jail in FreeBSD.  In this case, FreeBSD 10.0 is being used, and ezjail will be used to manage the jails.

Step 1:
Install ezjail-admin.  FreeBSD 10.0 uses pkgng for package management, so installation can be accomplished with

pkg install ezjail

Step 2:
Install the base jail.  In this example, the binary applications will be used.

ezjail-admin install

This creates the base jail in /usr/jails/basejail that all other jails will use.  The filesystem is mounted inside the jail as a read-only filesystem.  This creates a single point for base system management and saves disk space.

Step 3:
Create the jail.  In this case, "example.local" is the jail name, and 10.0.2.20 is the ip address.

ezjail-admin create example.local 10.0.2.20

The configuration file for the new jail will be /usr/local/etc/ezjail/example_local.  Be sure to add in the ip alias on the host system, and to verify the jail binds to the address.

echo 'ifconfig_em0_alias0="inet 10.0.2.20 netmask 255.0.0.0"' >> /etc/rc.conf 

Step 4:
Enable ezjail at boot and start the service.

echo 'ezjail_enable="YES"' >> /etc/rc.conf
service ezjail start

You can verify the jail is running with jls.

JID     IP Address     Hostname          Path
1       10.0.2.20      example.local     /usr/jails/example.local

Step 5:
To access a jail console, use

ezjail-admin console example.local

Any modifications to the jail outside of the base system will now be stored in the /usr/jails/example.local directory.  You may need to add a nameserver to the new jail to be able to add packages.  Something like "echo 'nameserver 8.8.8.8' > /etc/resolv.conf" from the jail console should do.  You can stop and start a particular jail with the command

service ezjail start <jailname>
service ezjail stop <jailname>

Digitally Signing an Encrypted File Using GnuPG

This post will go over digitally signing a file.  Combined with data encrypted with a secure symmetric cipher, this will provide reasonable assurance that an encrypted file was sent from the source that claims to have sent the file, and that the file was not modified during transit, as long as the private key used to sign the file has been kept safe.  As before, GnuPG will be used for both encryption and digital signatures.

Step 1:
Generate a GnuPG key pair.

gpg --gen-keys

There are a number of options that can be specified when generating a key pair.  For this example, the following values will be used:

Type of key: RSA and RSA (default)
Keysize: 4096
Valid for: Never expires

Real name: Test User
Email address: test@user.com
Comment: A test user.

Verify the key pair has been generated with:

gpg --list-keys

Step 2:
Send your public key, or make your public key available, to the party you want to send you digitally signed file to.  It is a good idea to also send the fingerprint of the public key through another means of communication, for example, over the phone.

To export the public key, execute:

gpg --armor --export "Test User"

To get the fingerprint of the public key, execute:

gpg --fingerprint "Test User"

Step 3:
Import the public key.  Once the public key has been sent and verified, the receiver needs to import the public key.

gpg --import publickey.key 

Step 4:
Encrypt and sign the file.

gpg --sign --symmetric --cipher-algo AES256 secret.txt

Step 5:
Decrypt the file.  Decrypting the file will automatically verify the digital signature.

gpg -d secret.txt.gpg
...gpg: Good signature from "Test User (A test user.) "...

Encrypting Data Using GnuPG

This post will go over how to encrypt and decrypt files using GnuPG with a symmetric cipher.  GnuPG stands for "Gnu Privacy Guard", and is an open implementation of the PGP standard.  Note that the method described below does not provide message integrity (this will be described in another post).

Step 1:
Install GnuPG.  GnuPG should be installed by default during a CentOS installation, but if necessary, execute:

yum install gnupg

Step 2:
Encrypt a file.  Upon first execution (or by using gpg --version), the application will print out a list of available ciphers, hashes, and key algorithms.

Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128, 
        CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

The default symmetric cipher used on this version of gpg was 3DES.  AES256 will be used instead.

gpg --cipher-algo AES256 -c secret.txt 

This will prompt for a password and produce the encrypted file "secret.txt.gpg".  Checking the file type should yield AES256:

file secret.txt.gpg 
secret.txt.gpg: GPG symmetrically encrypted data (AES256 cipher)

Step 3:
Decrypt a file.  Output goes into secret.txt.

gpg -o secret.txt -d secret.txt.gpg

Installing and Setting Up Fail2Ban

This post will go over how to install and use fail2ban, an extremely useful and versatile tool that can ban certain ip addresses that are showing malicious behaviour.  This post will go over how to monitor an ssh server for too many failed login attempts, and block the offending source ip address.

Step 1:
Install fail2ban.  If you have not already done so, install the epel repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

And install the application.

yum install fail2ban

Step 2:
Copy the configuration file.

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Step 3:
Set up the policy of the application.  There are a number of options that you can set in the newly created jail.local configuration file.  For the purposes of this post, only a few options will be modified:

#any ip that crosses the threshold will be banned for 24 hours, or 86400 seconds.
#bantime=600
bantime=86400

#an ip has 5 chances to log in successfully within the "findtime", defined below, before being banned for 24 hours.
#maxretry=3
maxretry=5

#an ip has 5 chances within one hour to log in successfully to a system before being banned for 24 hours.
#Note that after one hour, the threshold resets and the ip has another five attempts.
#findtime=600
findtime=3600

Modify the iptables rules to send mail to whatever address you want.  More importantly, ensure the "logpath" option specifies the log file to check for failed login attempts.

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=youraddress@yourdomain.com, sender=fail2ban@example.com]
logpath  = /var/log/secure
maxretry = 5

Step 4:
Enable fail2ban.

service fail2ban start
chkconfig fail2ban on

Verify the iptables chain is now active.  Execute "iptables -L".  The INPUT chain should have the line

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-SSH  tcp  --  anywhere             anywhere            tcp dpt:ssh

And the chain fail2ban should be available, although empty right now.

Chain fail2ban-SSH (1 references)

 target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Step 5:
Verify operation.  Attempting to log in from 192.168.1.15 to the server that now has fail2ban active (192.168.1.16) and failing five times results in the source ip being added to the fail2ban chain.

REJECT     all  --  192.168.1.15         anywhere            reject-with icmp-port-unreachable

And the source machine can not even make an attempt for the next 24 hours.

ssh root@192.168.1.16
ssh: connect to host 192.168.1.16 port 22: Connection refused

Manually Checking File Integrity

This post will go over a quick and dirty way to check file integrity on a *nix system.  This may be useful to check for things such as bit rot or malicious tampering, although, in this example, only backups are being checked for bit rot.  There are other tools that can do this better, and some of them will be the topic of future posts.

Step 1:
Specify the hashing algorithm you want to use.  While there are pros and cons for different hashing algorithms, in this example, md5 will be used for one main reason, it is faster than sha256.

Step 2:
Specify the files you want to ensure are still readable.  For example, the folder "Documents" and "Images", as well as all sub-folders are the current subject of scrutiny.  These files are respectively 1.7GB and 178MB.

Step 3:
Calculate the hashes of the files and store in a file.  This can be done with the find and xargs command.

find $DIR/Images -type f -print0 | xargs -0 md5sum > images.md5
find $DIR/Documents -type f -print0 | xargs -0 md5sum > docs.md5

Step 4:
Force the system to re-read the file from disk and calculate the hashes again.  This will notify you if a file has been tampered with, or is suffering from bit-rot.  In this example, the files have already been cached in main memory, and the checksums are being calculated shortly thereafter, thus not hitting the disk.  The whole point of this is to verify the on disk data is still good.

md5sum -c images.md5 | grep FAIL
real    0m0.542s
md5sum -c documents.md5 | grep FAIL
real    0m5.346s

To force the system to clear the cached files, first write any data buffered in memory to disk:

sync

Then, clear the cache:

echo 3 > /proc/sys/vm/drop_caches

Re-check the hashes.

md5sum -c images.md5 | grep FAIL
real    0m19.415s
md5sum -c documents.md5 | grep FAIL
real    1m6.774s

Logging in to a Host Over SSH Using Expect

This post will go over how to log in to a remote system over ssh using expect.  While it would be a better option to use key-based authentication for non-interactive logins, this is not always an option.

Step 1:
Automatically accept the host fingerprint to known_hosts.  Here is an expect script that will accept the remote host's fingerprint to the known_hosts file, otherwise, you will need to accept the key manually.

#!/usr/bin/expect
set timeout 5
set ip [lindex $argv 0]
spawn ssh user@$ip

expect "(yes/no)?"
send "yes\n"

Step 2:
Create a script to log in and do whatever task needs to be automated.  For example, this script logs in to a host, grabs the uptime, and stores the session in the file log.txt.

#!/usr/bin/expect

set timeout 5
set ip [lindex $argv 0]
set password [lindex $argv 1]
set output [open "log.txt" "a+"]

spawn ssh user@$ip

expect "Password:"
send "$password\n"

expect "#"
send "uptime\n"

expect "#"
send "exit\n"

expect eof

SSH Keys For Logging in to a Host Without a Password

This post will go over how to use keys to log in to a remote system over ssh.  Note that once this system is in place, it becomes critical to keep your private key safe and secure, and not give it out to anyone.  In this example, 10.0.0.3 is the client, and 10.0.0.2 is the server.

Step 1:
On the client, generate an ssh key pair.  This example is using rsa keys.

ssh-keygen -t rsa -b 4096

This will create a public key, id_rsa.pub, and a private key, id_rsa, in the users home folder in the .ssh directory.  During this command, you have the option to encrypt the private key with a password.  Although you can unlock the private key once and login to remote machines without a password afterwards (shown in step 4), you can also leave the private key in an unencrypted format.  It depends on the convenience/security trade-off you are willing to make.

Step 2:
Copy the public key to the server(s) you want to log in to.  You can use the ssh-copy-id command for convenience.  In this case, the local machine at 10.0.0.3 wants to log in to the remote server at 10.0.0.2.

ssh-copy-id -i ~/.ssh/id_rsa.pub <user>@10.0.0.2

The -i specifies the public key you want to transfer over.  The <user> is the account you want to log in to the server with.  Once this command completes, the public key of the client machine, 10.0.0.3, will have their public key added to the authorized_keys file of the server, 10.0.0.2.  Logging in to the console of the server at 10.0.0.2 as the user and cating the file ~/.ssh/authorized_keys reveals a new public key.

Step 3:
Modify the ssh server config file to only support key based logins.  Although not necessary for key based authentication, the server is still vulnerable to brute force attacks using username/password combinations.  There are tools to limit the number of login attempts and/or block repeated failures, but if possible, it would be best to just disable the option altogether.  In /etc/ssh/sshd_config, change "PasswordAuthentication yes" to "PasswordAuthentication no", and restart the service.

Step 4 (optional):
Unencrypt an encrypted private key once for use multiple times.  If you encrypted your private key, you can use an ssh-agent to unencrypt and load it into memory.

Run the "ssh-agent" command.

ssh-agent

This command will output some environment variables that the program needs.  Add these variables manually to your shell session, or execute the modified version of the command.

eval $(ssh-agent)

Once this is done, add your private key(s).

ssh-add

Adding Graphs to Nagios Using pnp4nagios

This post will go over how to add graphs to nagios using the pnp4nagios plugin.

Step 1:
Install and/or compile the necessary applications.  pnp4nagios was installed as part of the command "yum install nagios*" performed in a previous post.

rpm -q pnp4nagios
pnp4nagios-0.6.20-1.el6.i686

Step 2:
Configure the pnp4nagios commands.  Since this is a small installation with only a few hosts being monitored, most of the defaults will be used.  However, the commands still need to be added to the nagios configuration:

define command {
command_name    process-service-perfdata-file
command_line    /usr/libexec/pnp4nagios/process_perfdata.pl --bulk=/tmp/service-perfdata
}

define command {
command_name    process-host-perfdata-file
command_line    /usr/libexec/pnp4nagios/process_perfdata.pl --bulk=/tmp/host-perfdata
}

Step 3:
Modify nagios.cfg.  The diff with the original cfg is shown below.

diff nagios.cfg nagios.cfg.bak
834,835c834
< 
< process_performance_data=1
---
> process_performance_data=0
857,858c856,857
< host_perfdata_file=/tmp/host-perfdata
< service_perfdata_file=/tmp/service-perfdata
---
> #host_perfdata_file=/tmp/host-perfdata
> #service_perfdata_file=/tmp/service-perfdata
872,873d870
< host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$
\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$
\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
< service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$
\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$
\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$
\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
887,888d883
< host_perfdata_file_mode=a
< service_perfdata_file_mode=a
900,901d894
< host_perfdata_file_processing_interval=15
< service_perfdata_file_processing_interval=15
912,913d904
< host_perfdata_file_processing_command=process-host-perfdata-file
< service_perfdata_file_processing_command=process-service-perfdata-file

Step 4:
Restart nagios and verify the page shows up at http://<nagiosip>/pnp4nagios/

Step 5:
Add extended info in nagios to create links to graphs of the applicable host and service.  Add to templates.cfg:

define host {
name            host-pnp
action_url      /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_
register        0
}

define service {
name            service-pnp
action_url      /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=$SERVICEDESC$
register        0
}

Then add these new templates to the desired host and service definitions.  For example:

define host{
        use                     linux-server,host-pnp
        host_name               puppetmaster
        alias                   puppetmaster
        address                 192.168.1.15
}

define service{
        use generic-service,service-pnp
        host_name puppetmaster
        service_description PING
        check_command check_ping!100.0,20%!500.0,60%
}

Step 6:
Verify functionality.  Note the new graph icons available for the host/services.

Clicking on the icons gives links to the desired rrd graphs.  For example, the custom crond check shows the process has been running as desired.

pnp4nagios can also print out a nicely formatted report of the desired services/hosts to a pdf.

Adding a Custom Check/Creating Plugins for Nagios.

This post will go over how to add a custom check to a host being monitored by nagios.  In this case, nagios will check to make sure crond is running on the puppetmaster server, which is a centos machine.

Step 1:
Write the script that will check for the given condition, and verify its functionality.  This is a simple script that will check that the crond process is running. The data after the pipe is interpreted by nagios as performance data, and is being added in so that the status of the process can be graphed over a period of time in an rrd graph. Adding graphs to nagios will be covered in another post.

Note that the exit codes get interpreted by nagios as follows:
0 - OK
1 - WARNING
2 - CRITICAL
3 - UNKNOWN

#!/bin/bash

lineCount=`ps -eaf|grep -v grep|grep " crond"|wc -l`

if [ $lineCount -eq "0" ]; then
        echo "WARNING - crond is not running|proc=$lineCount"
        exit 1;
fi
if [ $lineCount -eq "1" ]; then
        echo "OK - crond is running|proc=$lineCount"
        exit 0;
fi
if [ $lineCount -gt "1" ]; then
        echo "UNKNOWN - crond process count > 1|proc=$lineCount"
        exit 3;
fi
echo "UNKNOWN - crond process count is unknown|proc=$lineCount"
exit 3;

Step 2:
Add the command to /etc/nagios/nrpe.cfg on the host.  The necessary line to add in this case is:

command[check_crond]=/usr/lib/nagios/plugins/check_crond

Step 3:
Add the service check to the nagios server.  Restart the nagios process.

define service{
        use generic-service
        host_name puppetmaster
        service_description Crond Process
        check_command check_nrpe!check_crond

}

service nagios restart

Step 4:
Verify functionality.

Monitoring a Host in Nagios Using NRPE

This post will go over how to monitor a host in nagios using nrpe.  The nrpe is the "Nagios Remote Plugin Executor", and allows you to remotely execute commands on another machine and gather desired metrics.  The version of nrpe that was installed on the target centos machine does not allow for command line arguments from the nagios server, so all arguments and thresholds must be specified on the machine itself.  The puppetmaster server is the machine being added to nagios.

Step 1:
On the target machine, add the EPEL repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Step 2:
Install nagios and necessary packages.

yum install nagios* xinetd

Step 3:
Add the service to xinetd.

/etc/xinetd.d/nrpe
service nrpe
{
flags = REUSE
type = UNLISTED
port = 5666
socket_type = stream
wait = no
user = nagios
group = nagios
server = /usr/sbin/nrpe
server_args = -c /etc/nagios/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 192.168.1.16
}

Restart, the service

service xinetd restart

chkconfig xinetd on 

Step 4:
Make any changes or modifications to the data you want to monitor on the host in /etc/nagios/nrpe.cfg.  In this scenario, the root partition, number of users, current load, and number of processes are being monitored.

Step 5:
Configure the nagios server.

/etc/nagios/objects/commands.cfg
define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Add the host, service, and hostgroup definitions to the necessary config files in /etc/nagios.  The puppet server is being added to the "Linux Servers" hostgroup.

Restart the service

service nagios restart

Step 6:
Verify functionality.

Installing Nagios to Monitor VM's

This post will go over installing nagios on a centos machine that lives on a vmware esxi host.  Nagios is an open source monitoring and alerting system that is widely deployed as an infrastructure monitoring solution and can scale from one to thousands of hosts and services.  This post will go over a basic installation.

Step 1:
Clone a machine.

cd /vmfs/volumes/datastore1
mkdir "CentOS 6.4 - Nagios Server"
cd CentOS\ 6.4\ -\ Nagios\ Server/
cp ../Base CentOS\ 6.4/Base CentOS\ 6.4.vmx ./CentOS\ 6.4\ -\ Nagios\ Server.vmx
vmkfstools -i ../Base CentOS\ 6.4/Base CentOS\ 6.4.vmdk \
CentOS\ 6.4\ -\ Nagios\ Server.vmdk
vim-cmd solo/registervm \
/vmfs/volumes/datastore1/CentOS\ 6.4\ -\ Nagios\ Server/CentOS\ 6.4\ -\ Nagios\ Server.vmx
vim-cmd vmsvc/power.on 21

Step 2:
If necessary, reset the ip addresses and the interfaces in /etc/sysconfig/networ-scripts/ifcfg-*, modify udev rules in /etc/udev/rules.d/70-persistent-net.rules, reset the hostname in /etc/sysconfig/network, reset the root password.

Step 3:
Add the EPEL repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Step 4:
Install nagios and necessary packages.

yum install -y nagios* openssl gd gd-devel httpd php gcc glibc glibc-common httpd

Step 5:
Set up apache and enable services.

htpasswd /etc/nagios/passwd nagiosadmin
chkconfig httpd on
chkconfig nagios on
service httpd restart
service nagios restart

Step 6:
Verify the system is up and monitoring the localhost at http://<systemip>/nagios

A later post will go over adding new hosts and services to the setup.