This article describes how we managed to exhaust the memory of switches of Cisco 2960 models and how we solved the problem.
In a recent project the task was to assign dynamic access control lists (dACL) to access ports after successful recognizing the attached MAC address. The access switches in the offices were various models of the catalyst 2960 series of Cisco (2960, 2960S, 2960X). This project was set up in preparation of a wider portsecurity project with 802.1x.
The team set up an Identity Services Engine (ISE) as the central AAA infrastructure. On the switches MAC authentication bypass (MAB) was configured.
Access would be granted if the MAC address was stored in the backend database.
Additional the ISE was configiured to send information about an ACL the switch should apply to that port after authentication. The technology is described in the switch manuals
So far, everything was configured according to Cisco textbooks.
But suddenly some switches started to behave strange. They mixed up authentication sessions, did not apply ACLs, or, even worse, were not accessible at all.
The monitoring of the equipment quickly showed that the switches went out of memory, the switch logged warnings like:
%ACLMGR-4-UNLOADING: Unloading PACL input label 8 Fa0/2 IP feature %ACLMGR-4-RELOADED: Reloading PACL input label 10 Fa0/2 IP feature
Memory utilization was high on all switches:
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 2705AEC 21988468 19287072 2701396 1799040 2507124 I/O 2C00000 4194304 2395448 1798856 1333348 1797032 Driver te 1880000 1048576 44 1048532 1048532 1048532
The switch has 22 MByte RAM of which only 2.7 MByte were left free. In lowest the switch had left less than 10% free RAM.
The manufacturer recommended the devices should have at least 10% of free RAM, better 20%. We checked the number of authorized sessions, because we had found out that the memory utilization correlated to the number of sessions and the size of the ACLs.
We were able to gather enough data of session numbers and memory utilization since we had a large quantity of switches. For the specific models (2960, -S, or -X) we plotted the utilization against the number of sessions and did a linear fit.
Following the recommendation from Cisco we were able to deduct that there should not be more than 80 concurrent sessions on this switch model.
Switch model 2960-X has a lot more RAM so we did not see this problem on these switches.