A SNMP Agent for Pacemaker

Abstract

The pacemaker program is the Cluster Resource Manager in Unix operation system cluster. It decides which resoure starts on what node in respect to the availability of nodes and the configuration constraints. Please see the pacemaker page for the description of the cluster software.

When you operate a cluster you want to know, what it's status is or get alerts if something happens. In short, enjoy all the benefits of a Network Monitoring System (NMS). The internet standard protocol for NMS to monitor devices and applications is the Simple Network Management Protocol (SNMP).

In this article I will describe, how you can set up a SNMP agent that publishes the state of your cluster nodes and resources. Every NMS (OpenNMS, Zabbix, nagios or the commercial alternatives) will be able to monitor the cluster properly.

A SNMP Agent for Pacemaker

The pacemaker program is the Cluster Resource Manager in Unix operation system cluster. It decides which resoure starts on what node in respect to the availability of nodes and the configuration constraints. Please see the pacemaker page for the description of the cluster software.

When you operate a cluster you want to know, what it's status is or get alerts if something happens. In short, enjoy all the benefits of a Network Monitoring System (NMS). The internet standard protocol for NMS to monitor devices and applications is the Simple Network Management Protocol (SNMP).

In this article I will describe, how you can set up a SNMP agent that publishes the state of your cluster nodes and resources. Every NMS (OpenNMS, Zabbix, nagios or the commercial alternatives) will be able to monitor the cluster properly.

The SNMP Agent

The standard SNMP agent for most of the Unix systems (including Linux) is net-snmp. So first of all you have to install that software. My software uses that software to publish the cluster state. In fact, it extends the net-snmp agent for the additional information of the cluster state.

Most Unix systems provide a packaging system, that makes the instalation of the net-snmp agent easy. Under Linux you use yum (RH), zypper (SUSE) oder aptitude (Debian, Ubuntu) to install the snmpd software.

The ages uses the configuration file /etc/snmp/snmpd.conf. Please edit this file to refect your needs. You have to specify, which version of SNMP you want to use (v1, v2c or v3), which community string (other than public!) or username and passphrases you want to use. You also can specify which NMS stations have access to the agent. Requests of other machines are dropped. Pleases see the man pages on snmpd.conf for further information.

If properly configured your management station should be able to retrieve information from that agent:

$ snmpwalk <IP of your host>
$
$ SNMPv2-MIB::sysDescr.0 = STRING: Linux node1 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
$ SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-TC::linux
$ DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (877774) 2:26:17.74
$ SNMPv2-MIB::sysContact.0 = STRING: Me <me@example.org>
$ SNMPv2-MIB::sysName.0 = STRING: node1
$ SNMPv2-MIB::sysLocation.0 = STRING: Sitting on the Dock of the Bay
$ (...)

Please make sure that your snmp agent runs a root. We need that user later, since on most systems only root is able to retrieve cluster information. Of course, you can add the snmp user to have access to the cluster data.

Getting the Files

Your can get the snmp agent software for pacemaker from github . You can download four files:

  • README: The file you should read first.
  • pcmk-subagent.pl: The agent
  • SYS4.mib and PACEMAKER.mib: The MIB files describing the structure of information provided by the agent.

The MIB files are needed on all machines, the cluster node and the mangement station. Just copy it into the directory where your system expects the MIBs. This might be /usr/share/snmp/mibs or /usr/share/mibs/.

Installation

On the cluster nodes, copy the pcmk-subagent.pl to a location of your choice. Most apppropiate would be /usr/local/sbin. You also have to install some perl libraries:

  • NetSNMP::agent::Support;
  • NetSNMP::ASN;
  • XML::Simple;
  • Switch;
  • POSIX;
  • List::Util;

With this file in place you can tell the net-snmp agent to include the script. You have to add the following line to your snmpd.conf file:

perl do '/usr/local/sbin/pcmk-subagent.pl'

and restart the service. Of course, enter a different location, if you saved the file unter a different location.

Testing

On the management station you can add the MIB files to enable it to use the human-readable names instead of the difficult numeric OIDs. Copy the two files to the directory of your system and tell it to use them. The quick and dirty way would be to copy it to the home directory and enter:

$ export MIBS=+ALL
$ export MIBDIRS=$HOME

on your shell.

Then you should be able to retrieve all information from your cluster node:

$ snmpwalk node1 sys4Pacemaker

PACEMAKER-MIB::sys4PcmkTotalNodes.0 = INTEGER: 2
PACEMAKER-MIB::sys4PcmkOnlineNodes.0 = INTEGER: 2
PACEMAKER-MIB::sys4PcmkNodeIndex.1 = INTEGER: 1
PACEMAKER-MIB::sys4PcmkNodeIndex.2 = INTEGER: 2
PACEMAKER-MIB::sys4PcmkNodeName.1 = STRING: "node2"
PACEMAKER-MIB::sys4PcmkNodeName.2 = STRING: "node1"
PACEMAKER-MIB::sys4PcmkNodeId.1 = INTEGER: 40151232
PACEMAKER-MIB::sys4PcmkNodeId.2 = INTEGER: 23374016
PACEMAKER-MIB::sys4PcmkNodeStatus.1 = INTEGER: online(1)
PACEMAKER-MIB::sys4PcmkNodeStatus.2 = INTEGER: online(1)
PACEMAKER-MIB::sys4PcmkResourcePrimitiveNumber.0 = INTEGER: 4
PACEMAKER-MIB::sys4PcmkResourceGroupNumber.0 = INTEGER: 1
PACEMAKER-MIB::sys4PcmkResourceCloneNumber.0 = INTEGER: 2
PACEMAKER-MIB::sys4PcmkResourceMasterNumber.0 = INTEGER: 1
PACEMAKER-MIB::sys4PcmkResourceFailures.0 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceIndex.1 = INTEGER: 1
PACEMAKER-MIB::sys4PcmkResourceIndex.2 = INTEGER: 2
PACEMAKER-MIB::sys4PcmkResourceIndex.3 = INTEGER: 3
PACEMAKER-MIB::sys4PcmkResourceIndex.4 = INTEGER: 4
PACEMAKER-MIB::sys4PcmkResourceIndex.5 = INTEGER: 5
PACEMAKER-MIB::sys4PcmkResourceIndex.6 = INTEGER: 6
PACEMAKER-MIB::sys4PcmkResourceIndex.7 = INTEGER: 7
PACEMAKER-MIB::sys4PcmkResourceIndex.8 = INTEGER: 8
PACEMAKER-MIB::sys4PcmkResourceIndex.9 = INTEGER: 9
PACEMAKER-MIB::sys4PcmkResourceIndex.10 = INTEGER: 10
PACEMAKER-MIB::sys4PcmkResourceIndex.11 = INTEGER: 11
PACEMAKER-MIB::sys4PcmkResourceIndex.12 = INTEGER: 12
PACEMAKER-MIB::sys4PcmkResourceName.1 = STRING: "resA"
PACEMAKER-MIB::sys4PcmkResourceName.2 = STRING: "resB"
PACEMAKER-MIB::sys4PcmkResourceName.3 = STRING: "resC"
PACEMAKER-MIB::sys4PcmkResourceName.4 = STRING: "resD"
PACEMAKER-MIB::sys4PcmkResourceName.5 = STRING: "groupWebserver"
PACEMAKER-MIB::sys4PcmkResourceName.6 = STRING: "resIP"
PACEMAKER-MIB::sys4PcmkResourceName.7 = STRING: "resFS"
PACEMAKER-MIB::sys4PcmkResourceName.8 = STRING: "resApache"
PACEMAKER-MIB::sys4PcmkResourceName.9 = STRING: "resMon"
PACEMAKER-MIB::sys4PcmkResourceName.10 = STRING: "cloneHealthCPU"
PACEMAKER-MIB::sys4PcmkResourceName.11 = STRING: "clonePing"
PACEMAKER-MIB::sys4PcmkResourceName.12 = STRING: "msDRBD"
PACEMAKER-MIB::sys4PcmkResourceStatus.1 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.2 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.3 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.4 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.5 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.6 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.7 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.8 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.9 = INTEGER: Stopped(0)
PACEMAKER-MIB::sys4PcmkResourceStatus.10 = INTEGER: Started(1)
PACEMAKER-MIB::sys4PcmkResourceStatus.11 = INTEGER: Started(1)
PACEMAKER-MIB::sys4PcmkResourceStatus.12 = INTEGER: Started(1)
PACEMAKER-MIB::sys4PcmkResourceFailure.1 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.2 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.3 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.4 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.5 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.6 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.7 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.8 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.9 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.10 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.11 = INTEGER: 0
PACEMAKER-MIB::sys4PcmkResourceFailure.12 = INTEGER:  0
Michael Schwartzkopff, 23. January 2014