Making Resource Utilizations Dynamic

Abstract

With the placement strategy minimal the cluster resource manager (pacemaker) can move all resources to the minimum limit of real nodes that the resources need. Other nodes will run no resources.

Only a little change in the Resource Agent controlling the virtual hosts is needed to make the utilization allocation of resources dynamic every monitoring interval.

Dynamic Resource Utilization

Energy consumption is one of the biggest problems computing centers face today. Their racks are stacked with machines who's CPUs, HDs and other electric parts are more energy hungry than ever before. The consumption would be acceptable if the machines were under full load, but most of them idle at a load of 10%. Load or not, they are still in need of energy and climatisation - a waste on all levels.

Virtualisation is a useful approach to address the issue of energy consumption. Several virtual servers run on one hardware. Virtualized hardware gets better used than traditional servers. But there's still way room for optimization because utilization varies with time - especially in clusters.

This blog article deals with the dynamic measurement of system resource utilization of virtual machines in a Linux cluster. Dynamic utilizations of system resources like CPU and RAM of the host hardware is added as a parameter the cluster resource manager can take into account for its descisions. The next article will show a way how to run only as many hardware as nescessary.

The Cluster Software

With the placement strategy minimal the cluster resource manager (pacemaker) can move all resources to the minimum limit of real nodes that the resources need. Other nodes will remain unemployed.

Only a little change in the Resource Agent VirtualDomain controlling the virtual hosts is needed to make the utilization allocation of resources dynamic every monitoring interval.

Utilizations

Utilizations describe the ability of the cluster resource manager to factor the system resources like RAM or CPU into its placement algorithms. For an in depth read on Utilization check The Cluster Guy's article on Feature Spotlight: Utilization website.

Nodes in a cluster offer utilizations and resources consume it. The cluster manager places the resources on the nodes so that no node is unable to cope with the resources running on it. Special placement strategies describe the way pacemaker chooses the nodes for the resources. The strategy minimal i.e. tells the manager to allocate as few nodes possible to run the resources.

Tip

Why call a system resource a utilization? Resources in a cluster are the services the cluster takes care of. Reusing an already assigned term clearly calls for trouble. That is why we talk about utilizations.

Up to now, utilizations are static. When you create a cluster node you enter how many CPUs and RAM it offers. When you configure a resource, you enter how many system resources (CPU, RAM) it consumes. But this is not the way real resources behave. Resources in a cluster consume utilizations dynamic. Especially virtual hosts in a clustered enviroment need very few system resources in the night or on weekends. The utilization increases in the morning, when people start to work.

In this article I will describe the dynamic utilization with the CPU load. Other system resources like RAM can be implemented easily.

CPU Ticks

One important utilization of a virtual machine is the CPU load it needs. In my setup I use the CPU ticks (hundredth of seconds) to measure the load. This gives me a more granular control over the utilizations than integer number of CPU(s) a host needs. So every real CPU on provides 100 ticks per second. In the further setup I will use the tick utilization.

Integration into VirtualDomain

The Linux cluster resource manager pacemaker controls its resources with the resource agents. The resource agent to manage a virtual host in a clustered enviroment is called VirtualDomain. It uses libvirt.

I add some lines of code to measure the ticks the virtual host really consumes every time pacemaker monitors the virtual resources. My script checks for the existance of the virt-top program and uses it to grep the load from the output from the function VirtualDomain_Monitor():

if [ -x /usr/bin/virt-top ]; then
  CPUS=$(/usr/bin/virsh nodeinfo | awk '/CPU\(s/ {print $2}')
  CPU_TICKS=$(/usr/bin/virt-top -n 2 -d 5 --stream | grep $DOMAIN_NAME | tail -1 | awk '{printf "%.0f",$7}')
  OLD_TICKS=$(crm_resource -r $OCF_RESOURCE_INSTANCE -z -g ticks)
  NEW_TICKS=$(((OLD_TICKS + CPU_TICKS * CPUS) / 2))
  /usr/sbin/crm_resource -r $OCF_RESOURCE_INSTANCE -z -p ticks -v ${NEW_TICKS}
fi

Please also note that I included some hysteresis behaviour when calulating the new number of ticks. It is an average from the previous and the measured value. So any short peak in the load will not cause any problems.

With this patch the cluster is able to run the virtual hosts on the minimum set of real nodes according to their real utilizations. Since the cluster uses a live migration if configured the users on the virtual machines should not notice the re-location of their machine.

Michael Schwartzkopff, 08. May 2013