Memory Problems on a Cisco 2960 with Portsecurity

This article describes how we managed to exhaust the memory of switches of Cisco 2960 models and how we solved the problem.

Project

In a recent project the task was to assign dynamic access control lists (dACL) to access ports after successful recognizing the attached MAC address. The access switches in the offices were various models of the catalyst 2960 series of Cisco (2960, 2960S, 2960X). This project was set up in preparation of a wider portsecurity project with 802.1x.

The team set up an Identity Services Engine (ISE) as the central AAA infrastructure. On the switches MAC authentication bypass (MAB) was configured.

Access would be granted if the MAC address was stored in the backend database.

Additional the ISE was configiured to send information about an ACL the switch should apply to that port after authentication. The technology is described in the switch manuals

So far, everything was configured according to Cisco textbooks.

Problem

But suddenly some switches started to behave strange. They mixed up authentication sessions, did not apply ACLs, or, even worse, were not accessible at all.

The monitoring of the equipment quickly showed that the switches went out of memory, the switch logged warnings like:

%ACLMGR-4-UNLOADING: Unloading PACL input label 8 Fa0/2 IP feature
%ACLMGR-4-RELOADED: Reloading PACL input label 10 Fa0/2 IP feature

Memory utilization was high on all switches:

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    2705AEC    21988468    19287072     2701396     1799040     2507124
      I/O    2C00000     4194304     2395448     1798856     1333348     1797032
Driver te    1880000     1048576          44     1048532     1048532     1048532

The switch has 22 MByte RAM of which only 2.7 MByte were left free. In lowest the switch had left less than 10% free RAM.

The manufacturer recommended the devices should have at least 10% of free RAM, better 20%. We checked the number of authorized sessions, because we had found out that the memory utilization correlated to the number of sessions and the size of the ACLs.

Solution

We were able to gather enough data of session numbers and memory utilization since we had a large quantity of switches. For the specific models (2960, -S, or -X) we plotted the utilization against the number of sessions and did a linear fit.

RAM utilization of 2960S models
Figure 1. Sample plot of memory utilization against the number of authenticated sessions.

Following the recommendation from Cisco we were able to deduct that there should not be more than 80 concurrent sessions on this switch model.

Switch model 2960-X has a lot more RAM so we did not see this problem on these switches.

Michael Schwartzkopff, 13 Mar 2019