Using Grub From the Command Line in CentOS

This post will go over some basic commands you can use in the Grub CLI to try to boot a system that is having trouble.

When the system is coming up, be sure to press a key at the initial grub screen so the system does not try to boot automatically.  Once at the OS selection screen, press "c" to enter the grub cli.

In the grub cli, you can issue "help" for a list of commands.

To see what hard disks are available, you can use auto completion.  Start by typing in "root (hd" then tab to get a list of hard disks and partitions.  In this case, there is only one hard disk, and two partitions.

You can now do the same thing to view a list of files on the ext2 boot partition.  Type in "kernel (hd0,0)/" then tab to see a list of files.  In this case, vmlinuz-2.6.32-431.el6.i686 looks like a usable kernel.

If necessary, you can also add in the initial ramdisk.

You can now try booting the system with the "boot" command.

Creating Jail Flavours in FreeBSD

This post will go over how to create flavors in FreeBSD.  Flavours are directories that are copied over to a jail upon creation, and can make commonly created jails easier to deploy.

Step 1:
Create a directory for the flavour you want to have available.  This setup will have one "vanilla" jail that all other jails will be built from.  This vanilla jail is built from the example.local jail in the previous post.

cp -R /usr/jails/example.local /usr/jails/flavours/vanilla

Step 2:
Create a new jail using the "vanilla" flavour.  Be sure to add the necessary ip address to rc.conf, and restart the netif service.

echo 'ifconfig_em0_alias1="inet 10.0.2.21 netmask 255.0.0.0"' > > /etc/rc.conf
/etc/rc.d/netif restart
ezjail-admin create -f vanilla nginx01.local 10.0.2.21

Step 3:
Console in to the newly created jail and add the necessary packages and make any changes for the nginx flavour.

ezjail-admin console nginx01.local
pkg install nginx
echo 'nginx_enable="YES"' >> /etc/rc.conf'

Step 4:
Exit and stop the jail.  Copy the directory of the nginx01.local jail to a new directory under flavours, in this case, webserver-nginx.  Delete the original jail.  Note that 10.0.2.21 is now available for re-use in another jail.

ezjail-admin stop nginx01.local
cp -R nginx01.local flavours/webserver-nginx
ezjail-admin delete -w -f nginx01.local

Step 5:
You now have a "webserver-nginx" flavour available for easy creation.  The next time you need a jail for hosting nginx, be sure to have a free ip and execute

ezjail-admin create -f webserver-nginx nginx02.local 10.0.2.21
ezjail-admin start nginx02.local

Creating a Jail in FreeBSD

This post will go over how to create a jail in FreeBSD.  In this case, FreeBSD 10.0 is being used, and ezjail will be used to manage the jails.

Step 1:
Install ezjail-admin.  FreeBSD 10.0 uses pkgng for package management, so installation can be accomplished with

pkg install ezjail

Step 2:
Install the base jail.  In this example, the binary applications will be used.

ezjail-admin install

This creates the base jail in /usr/jails/basejail that all other jails will use.  The filesystem is mounted inside the jail as a read-only filesystem.  This creates a single point for base system management and saves disk space.

Step 3:
Create the jail.  In this case, "example.local" is the jail name, and 10.0.2.20 is the ip address.

ezjail-admin create example.local 10.0.2.20

The configuration file for the new jail will be /usr/local/etc/ezjail/example_local.  Be sure to add in the ip alias on the host system, and to verify the jail binds to the address.

echo 'ifconfig_em0_alias0="inet 10.0.2.20 netmask 255.0.0.0"' >> /etc/rc.conf 

Step 4:
Enable ezjail at boot and start the service.

echo 'ezjail_enable="YES"' >> /etc/rc.conf
service ezjail start

You can verify the jail is running with jls.

JID     IP Address     Hostname          Path
1       10.0.2.20      example.local     /usr/jails/example.local

Step 5:
To access a jail console, use

ezjail-admin console example.local

Any modifications to the jail outside of the base system will now be stored in the /usr/jails/example.local directory.  You may need to add a nameserver to the new jail to be able to add packages.  Something like "echo 'nameserver 8.8.8.8' > /etc/resolv.conf" from the jail console should do.  You can stop and start a particular jail with the command

service ezjail start <jailname>
service ezjail stop <jailname>

Digitally Signing an Encrypted File Using GnuPG

This post will go over digitally signing a file.  Combined with data encrypted with a secure symmetric cipher, this will provide reasonable assurance that an encrypted file was sent from the source that claims to have sent the file, and that the file was not modified during transit, as long as the private key used to sign the file has been kept safe.  As before, GnuPG will be used for both encryption and digital signatures.

Step 1:
Generate a GnuPG key pair.

gpg --gen-keys

There are a number of options that can be specified when generating a key pair.  For this example, the following values will be used:

Type of key: RSA and RSA (default)
Keysize: 4096
Valid for: Never expires

Real name: Test User
Email address: test@user.com
Comment: A test user.

Verify the key pair has been generated with:

gpg --list-keys

Step 2:
Send your public key, or make your public key available, to the party you want to send you digitally signed file to.  It is a good idea to also send the fingerprint of the public key through another means of communication, for example, over the phone.

To export the public key, execute:

gpg --armor --export "Test User"

To get the fingerprint of the public key, execute:

gpg --fingerprint "Test User"

Step 3:
Import the public key.  Once the public key has been sent and verified, the receiver needs to import the public key.

gpg --import publickey.key 

Step 4:
Encrypt and sign the file.

gpg --sign --symmetric --cipher-algo AES256 secret.txt

Step 5:
Decrypt the file.  Decrypting the file will automatically verify the digital signature.

gpg -d secret.txt.gpg
...gpg: Good signature from "Test User (A test user.) "...

Encrypting Data Using GnuPG

This post will go over how to encrypt and decrypt files using GnuPG with a symmetric cipher.  GnuPG stands for "Gnu Privacy Guard", and is an open implementation of the PGP standard.  Note that the method described below does not provide message integrity (this will be described in another post).

Step 1:
Install GnuPG.  GnuPG should be installed by default during a CentOS installation, but if necessary, execute:

yum install gnupg

Step 2:
Encrypt a file.  Upon first execution (or by using gpg --version), the application will print out a list of available ciphers, hashes, and key algorithms.

Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128, 
        CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

The default symmetric cipher used on this version of gpg was 3DES.  AES256 will be used instead.

gpg --cipher-algo AES256 -c secret.txt 

This will prompt for a password and produce the encrypted file "secret.txt.gpg".  Checking the file type should yield AES256:

file secret.txt.gpg 
secret.txt.gpg: GPG symmetrically encrypted data (AES256 cipher)

Step 3:
Decrypt a file.  Output goes into secret.txt.

gpg -o secret.txt -d secret.txt.gpg

Installing and Setting Up Fail2Ban

This post will go over how to install and use fail2ban, an extremely useful and versatile tool that can ban certain ip addresses that are showing malicious behaviour.  This post will go over how to monitor an ssh server for too many failed login attempts, and block the offending source ip address.

Step 1:
Install fail2ban.  If you have not already done so, install the epel repository.

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

And install the application.

yum install fail2ban

Step 2:
Copy the configuration file.

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Step 3:
Set up the policy of the application.  There are a number of options that you can set in the newly created jail.local configuration file.  For the purposes of this post, only a few options will be modified:

#any ip that crosses the threshold will be banned for 24 hours, or 86400 seconds.
#bantime=600
bantime=86400

#an ip has 5 chances to log in successfully within the "findtime", defined below, before being banned for 24 hours.
#maxretry=3
maxretry=5

#an ip has 5 chances within one hour to log in successfully to a system before being banned for 24 hours.
#Note that after one hour, the threshold resets and the ip has another five attempts.
#findtime=600
findtime=3600

Modify the iptables rules to send mail to whatever address you want.  More importantly, ensure the "logpath" option specifies the log file to check for failed login attempts.

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=youraddress@yourdomain.com, sender=fail2ban@example.com]
logpath  = /var/log/secure
maxretry = 5

Step 4:
Enable fail2ban.

service fail2ban start
chkconfig fail2ban on

Verify the iptables chain is now active.  Execute "iptables -L".  The INPUT chain should have the line

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-SSH  tcp  --  anywhere             anywhere            tcp dpt:ssh

And the chain fail2ban should be available, although empty right now.

Chain fail2ban-SSH (1 references)

 target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Step 5:
Verify operation.  Attempting to log in from 192.168.1.15 to the server that now has fail2ban active (192.168.1.16) and failing five times results in the source ip being added to the fail2ban chain.

REJECT     all  --  192.168.1.15         anywhere            reject-with icmp-port-unreachable

And the source machine can not even make an attempt for the next 24 hours.

ssh root@192.168.1.16
ssh: connect to host 192.168.1.16 port 22: Connection refused

Manually Checking File Integrity

This post will go over a quick and dirty way to check file integrity on a *nix system.  This may be useful to check for things such as bit rot or malicious tampering, although, in this example, only backups are being checked for bit rot.  There are other tools that can do this better, and some of them will be the topic of future posts.

Step 1:
Specify the hashing algorithm you want to use.  While there are pros and cons for different hashing algorithms, in this example, md5 will be used for one main reason, it is faster than sha256.

Step 2:
Specify the files you want to ensure are still readable.  For example, the folder "Documents" and "Images", as well as all sub-folders are the current subject of scrutiny.  These files are respectively 1.7GB and 178MB.

Step 3:
Calculate the hashes of the files and store in a file.  This can be done with the find and xargs command.

find $DIR/Images -type f -print0 | xargs -0 md5sum > images.md5
find $DIR/Documents -type f -print0 | xargs -0 md5sum > docs.md5

Step 4:
Force the system to re-read the file from disk and calculate the hashes again.  This will notify you if a file has been tampered with, or is suffering from bit-rot.  In this example, the files have already been cached in main memory, and the checksums are being calculated shortly thereafter, thus not hitting the disk.  The whole point of this is to verify the on disk data is still good.

md5sum -c images.md5 | grep FAIL
real    0m0.542s
md5sum -c documents.md5 | grep FAIL
real    0m5.346s

To force the system to clear the cached files, first write any data buffered in memory to disk:

sync

Then, clear the cache:

echo 3 > /proc/sys/vm/drop_caches

Re-check the hashes.

md5sum -c images.md5 | grep FAIL
real    0m19.415s
md5sum -c documents.md5 | grep FAIL
real    1m6.774s