Monitoring F5 BigIP with Zabbix

Abstract

One of the coolest features of Zabbix version 2 ist the low level discovery that works with the Zabbix agent and SNMP tables. But when I discovered that this feature not only works with INTEGER as indices in SNMP tables but also with DisplayString it was one of the WOW moments.

Monitoring BigIP with Zabbix

One of the coolest features of Zabbix version 2 is its low level discovery mechanism which works with the Zabbix agent and SNMP tables. Imagine the look on my face, when I discovered that low level discovery not only works with INTEGER as indices in SNMP tables, but also with DisplayString - it was one of those WOW moments.

DisplayString as Index

Originally implementations of SNMP indexed tables by plain INTEGER numbers. As an example, the interface table ifTable begins with the description on the interfaces and every interface has its own integer index:

IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1

But recent SNMP agent implementations not only use INTEGER data types for indexing, but also other data types like DisplayString. One of the vendors that use this technique extensively is F5. If you look into the table that describes a pool of real servers that can be attached to a virtual server it is indexed by the name of that pool. The table reads:

F5-BIGIP-LOCAL-MIB::ltmPoolName."webpool" = STRING: webpool
F5-BIGIP-LOCAL-MIB::ltmPoolName."mailpool" = STRING: mailpool

Internally these indices are converted to OIDs (of course). The OID of the table name (i.e. ltmPoolName or .1.3.6.1.4.1.3375.2.2.5.1.2.1.1) is followed by the length of the string in octets and the ASCII representation of every character in the string. So the OID of the webpool name would be

.1.3.6.1.4.1.3375.2.2.5.1.2.1.1.7.119.101.98.112.111.111.108

F5 BigIP

The BigIP loadbalancer of company F5 use this indexing technique extensively. All tables are indexed by names of the objects they are describing.

For monitoring the F5's first you need the F5-BIGIP-SYSTEM MIB, which describes the health of the appliance. But for collecting the performance data you need the data of the pool and the virtual server tables of the F5.BIGIP-LTM-MIB. The first describes the health of the pool members and the second gives you a good overview of the performance your your loadbalancer. The ltmPoolTable is indexed by the name of the pool and the ltmVirtualServStatTable by the names of the virtual servers.

If your do not want the add all table entries manually, you need Zabbix's low level discovery feature which also works for non-INTEGER indices, using the names as indices.

Health Monitoring of Pools

To monitor the health of my pools I use values of ltmPoolMemberCnt and ltmPoolActiveMemberCnt. I just compare both numbers. Both tables are indexed by the name of the pool. So first I configure the discovery of the ltmPool and use the ltmPoolName (.1.3.6.1.4.1.3375.2.2.5.1.2.1.1). After the discovery run the {#SNMPVALUE} contains the name of the pool and the {#SNMPINDEX} contains some kind of index we do not use. Since F5 uses the names as index we are happily can use the {#SNMPVALUE} macro in all our configurations.

With this information I can add prototype items, the number of members in the pool and the number of members that are really available. The configuration for the ltmPoolMemberCnt would be:

Name ltmPoolMemberCnt.$1
Type SNMP Agent
Key ltmPoolMemberCnt.["{#SNMPVALUE}"]
SNMP OID .1.3.6.1.4.1.3375.2.2.5.1.2.1.23."{#SNMPVALUE}"

Now I can create the according prototype triggers. If there are no active members left in the pool, a trigger is set to High. As you can see in the template I also have Warning triggers if both numbers are equal.

Please find the complete template here

Performance Monitoring of Virtual Servers

The virtual server table tells you about the perfomance of the virtual servers you configured in you LTM appliance. Interesting items are the number of bytes in and out per server and the number of connections that server has to handle.

All these numbers do exist only in the SNMPv2 MIB since all numbers are of the Counter64 data type. So your have to configure SNMPv2 everywhere.

The discovery runs on the ltmVirtualServStatName table and returns the names of the configured virtual servers in the {#SNMPVALUE} macro. I configure Zabbix to fetch

ltmVirtualServStatClientBytesIn."{#SNMPVALUE}"
ltmVirtualServStatClientBytesOut."{#SNMPVALUE}"
ltmVirtualServStatClientCurConns."{#SNMPVALUE}"

every 5 minutes. Please be sure to use the numeric representation of the OIDs in the configuration if the items. Otherwise you would have to take care that the Zabbix user has correct access to the F5 MIBs.

With the data Zabbix fetched you can create nice graphs displaying the bytes flowing in and out on the client side and the number of connections to that specific virtual server.

One virtual server (DNS) of a loadbalancer we are monitoring handles about 100.000 simultaneous connections with a throughput of 0.5 Mbit/s upload and 2 Mbit/s download on the client side. The rise in the number of concurrent connections happened when nscd was disabled on a complete class of hosts.

Michael Schwartzkopff, 25. December 2012