keepalived, virtual IP plus alerting

KeepAliveD is a tool you should learn as soon as possible. You are going to need this wonderful instrument and its magnificent powers.

All starts with a a simple apt-get install keepalived and continues with a very short configuration needed.

requirements

Create two Debian 8 VM on the same network.

• debianA: 192.168.60.101/24
• debianB: 192.168.60.102/24

Your firewall must allow VRRP protocol communication between clients on the same subnet.

Install keepalived:

On both machines

apt-get install -y keepalived

sysctl

A little tune is required in order to make keepalived working

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
sysctl -p

keepalived.conf

location: /etc/keepalived/keepalived.conf

We are going to use the very same configuration on all machines despite what said on the official KeepAliveD installation guide, just because I don’t want to force a specific machine to always be the MASTER so I let the protocol choose what machine should be the MASTER and who the BACKUP, depending on the priority calculated (see next example).

keepalived.conf

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 100
    priority 100
    virtual_ipaddress {
        192.168.60.100/24 label eth0:1
    }
}

After you configured all the VMs, simply restart daemon: service keepalived restart or /etc/init.d/keepalived restart.

On both VMs, do a ip addr | grep 192.168 should show something similar to this. The machine with the keepalived ip should be only one, maybe the vm2 in your case

vm1

inet 192.168.60.101/24 ... eth0
ient 192.168.60.100/24 ... eth0:1

vm2

inet 192.168.60.102/24 ... eth0

priority based on what happens on the VM

Let’s say you want to drop the IP in case the service is no more operational, this is done lowering the priority programmatically on a script base. Lowering the priority means the IP will be assigned to the VM with a greater priority.

keepalived.conf

vrrp_script chk_proxy {
        script "[ `curl http://localhost:8080/check` = 'ok' ]"
        interval 2
        weight 2
}

vrrp_instance VI_1 {
        interface eth1
        state MASTER
        virtual_router_id 201
        priority 100
        virtual_ipaddress {
            192.168.60.100/24
        }
        track_script {
            chk_proxy
        }
}

The chk_proxy script is called every 2 seconds. If the exit code is not zero, the priority will be lowered by weight. This means if one of the machines has the web service down, will have a priority of 98. If every machine has the web service down, every one will have the same priority and the IP will be assigned randomly to a machine not able to handle the request. Pay attention to this.

script

You can also write an external executable script and set it in the config file. The exit code will count.

• script "/var/my.script"
• script "ifconfig tap0"
• script "pidof mysqld
• …

notification script

Being notified when a cluster changes its config is a good thing and we want to “ab”use this.

keepalived.conf

vrrp_script chk_proxy {
        script "[ `curl http://localhost:8080/check` = 'ok' ]"
        interval 2
        weight 2
}

vrrp_instance VI_1 {
        interface eth1
        state MASTER
        virtual_router_id 201
        priority 100
        virtual_ipaddress {
            192.168.60.100/24
        }
        track_script {
            chk_proxy
        }
        notify /usr/local/bin/keepalivednotify.sh
}

/usr/local/bin/keepalivednotify.sh

!/bin/bash

# $1 = “GROUP” or “INSTANCE”
# $2 = name of group or instance
# $3 = target state of transition (“MASTER”, “BACKUP”, “FAULT”)

TYPE=$1
NAME=$2
STATE=$3
HOSTNAME=`hostname`
TIME=`date`

# scripts doing specific actions, if needed.
MASTER_SH="/var/kad_master.sh"
BACKUP_SH="/var/kad_backup.sh"
FAULTY_SH="/var/kad_faulty.sh"

# notify syslog
logger -t kad.notify "vrrp $NAME ($TYPE) on $HOSTNAME changed his state to $STATE on $TIME"

# visit mailgun.net for a free account!
curl -ks --user "api:key-MAILGUN-KEY" \
    https://api.mailgun.net/v3/brugnara.me/messages \
    -F from="KeepAliveD <keepalived@brugnara.me>" \
    -F to=teamdev@brugnara.me \
    -F subject="[KeepAliveD] $HOSTNAME - $NAME is now $STATE" \
    -F text="vrrp $NAME ($TYPE) on $HOSTNAME changed his state to $STATE on $TIME"

case $STATE in
        "MASTER")
                  if [ -f $MASTER_SH ]; then
                    /bin/bash $MASTER_SH
                  fi
                  exit 0
                  ;;
        "BACKUP") if [ -f $BACKUP_SH ]; then
                    /bin/bash $BACKUP_SH
                  fi
                  exit 0
                  ;;
        "FAULT")  if [ -f $FAULTY_SH ]; then
                    /bin/bash $FAULTY_SH
                  fi
                  exit 0
                  ;;
        *)        echo "unknown state"
                  exit 1
                  ;;
esac

For me it’s very useful to take more actions regarding the KeepAliveD status, so here’s what they do, just as a share. May not be required for your server farm.

/var/kad_master.sh

ifconfig tap0 up
brctl addif br0 tap0 eth1
ifconfig br0 up

/var/kad_backup.sh

ifconfig tap0 down
brctl delif br0 tap0 eth1
ifconfig br0 down

/var/kad_faulty.sh

ifconfig tap0 down
brctl delif br0 tap0 eth1
ifconfig br0 down

Thanks to this blog post and also check this official manual for more details.