Dealing with power loss

Data centres are the second biggest global user of electricity. In Europe, current power consumption by data centres stands at 60TWh and by 2020 is projected to exceed 100TWh. The costs of that energy use are significant. The most efficient data centre spends around 30% of their revenue on energy while the average spend across European data centres on energy is 40% of revenue.

Although there have been a lot of efforts to bring renewable energy to the data centre, it is still heavily dependent upon the main electricity grids. Those grids, however, are under enormous strain and it is not unusual to see rolling brown-outs in some countries, where power is turned off for periods of time to even out the usage.

The strain on the grids in Europe is predicted to get much worse before it gets better. Although some European countries have begun to diversify their power grids, others are facing serious problems over the next decade.

The nuclear problem
The main reason for the strain on electricity grids across Europe is primarily the reaction to the meltdown of the Fukushima Daiichi nuclear plant. The impact of this disaster resulted in Switzerland and Germany announcing they would close down all their nuclear power stations.

They are not alone. Belgium decided on a nuclear shutdown as a result of cracks in two plants. Italy has shelved plans for nuclear plants while admitting that this could cause problems for its heavy industries such as car production and its burgeoning data centre market. Spain has banned all new nuclear plants while other countries such as the United Kingdom, France and Finland are all suffering big delays in construction of new nuclear capacity.

Solar and wind are beginning to replace the generating capacity that is being lost. They are supported by a wave of new gas, oil and even coal generating plants but it will take time before they come on line. The reality of the situation is that Europe may start to experience the sort of rolling power problems that many US states suffer in extremely hot and cold weather.

Data centres are not a priority
Should Europe find itself having to restrict power, data centres are likely to find themselves treated the same way as heavy industry. This means that they will have to accept a degree of reduced power and power outages. Preference will be given to hospitals, homes and other essential services.
Data centres will be forced to increasingly rely on power from their own generators. At present, generators are installed as a stop gap measure. In most cases they come online while power is restored but in extreme instances they are used to provide enough power to gracefully shutdown parts of the data centre.

As we move forward, that role may well change. This means that rather than a single yearly test, generators will need to be tested more frequently and more thoroughly. A higher level of spares will need to be kept and issues such as fuel treatment and refuelling have to be taken into account.

What does this mean for data centre operators?
 Power reduction programmes need to be accelerated and cut
deeper than before. A major part of this will be replacing equipment
more frequently to ensure that only the most power friendly
systems are installed.
 A detailed understanding of where and when power is used is
essential to ensure key systems are kept powered.
 Generating facilities must be overhauled, reviewed, regularly tested
and given a higher priority than many other systems.

Accurate measurement a priority
One of the first things that data centres need to do is assess how they are measuring the amount of power that they use. This might seem self evident but there are now so many ways to measure power that there is a real risk of ending up with too much data and too little information.

Where to monitor power is a subject of much discussion. Power is lost through transmission and through conversion. A lot of data centres do multiple AC/DC conversions inside their infrastructure. What they do not do is calculate how much power is lost during the conversion process. It is important to know how many conversions take place and the power loss to ensure it is factored into the generating requirements.

The most effective way to know how much power is being consumed by devices is through intelligent PDU strips such as those sold by vendors such as Cannon Technologies. These strips allow you to monitor either individual devices or groups of devices. The ability to monitor a group of devices is especially useful when dealing with a cluster of systems that need to be protected as a single unit. The operator can see how much data the group uses and ensure that there is sufficient power to keep all devices running.

Once you are getting usage data from devices, the next stage is to understand usage patterns. Not all devices pull data at the same rate 24x7. While storage devices and switches are generally a constant load, computational device power varies constantly. With virtualisation, predicting power requirements has become more art than science for many data centres.

One reason for this is that power monitoring is not linked to the automated systems that manage the virtualisation. This means that data centres need to develop better systems to capture historical data and to be able to restrict the automated movement of virtual machines when there is a power situation.

When the data centre is in a power restricted mode, it should be possible to start shutting down lightly used hardware and consolidating loads elsewhere. While this already happens to a limited degree, it is not driven by power availability. This will require a rethinking of how power and computational load work together.

There are solutions on the market that allow operators to see where power is being used, capture usage patterns and power down systems remotely. One such system is CannonView which is designed for remote operations of a data centre.

Don’t forget to measure power used by cooling
A major source of power consumption is the cooling systems in the data centre. When calculating the amount of power required to keep the data centre running, it is critical to understand the requirements of cooling systems.

If cooling systems are allowed to lose power, there is a significant risk of thermal runaway, especially in high density racks containing switches and blade servers. Once thermal runaway begins, it is extremely hard to stop and when running on limited power from generators, it may not be possible to deliver enough cooling to prevent serious damage to equipment.

Get generator spares and service right
With many data centres only checking their generators once a year when they run an emergency test, there is a serious risk of not providing the right level of spares and support.

In the aftermath of Hurricane Sandy, some data centre owners discovered that their fuel tanks had become contaminated. This led to stories of teams of workers using jerry cans to carry diesel to the top of building to refuel generators. Fuel contamination is not unusual but it is not the only fuel related issue around data centres.

Fuel for diesel generators needs to be properly conditioned. This helps to remove impurities which reduces damage to the engine. As well as the conditioning, filters and strainers need to be regularly checked and changed. Where a generator is not used regularly, these tend to get forgotten. A clogged filter can cause a generator to fail unexpectedly.

One of the main causes for clogged filters is contamination of the fuel supply. Another is a build up of muck at the bottom of fuel tanks. When the tank is run very low, muck is passed through the fuel lines and clogs the filters. Cleaning can be a long and costly process.

A major concern for some data centre owners has been theft of fuel. Data centres that do not have high levels of physical security are becoming attractive targets for thieves who can sell on stolen diesel for a lot of money.
Conclusion
Energy is the lifeblood for any data centre, it is also the biggest cost. As Europe adjusts to the shutdown of nuclear generating capacity, energy supplies 24x7x365 can no longer be taken for granted.

Data centres need to look at how they can cope with power outages. In future, these may not just be for minutes or a few hours but potentially for a day and on a regular basis.

Detailed understanding of where the power is consumed by both IT equipment and cooling is essential. It is also important to understand where power is lost through conversion to ensure that data centre owners have the most complete picture of power requirements in their facility.
Once this is done, time spent getting the right generating capacity and making sure that spares and fuel supplies are secure and fit for purpose are a critical investment.

Data centres need to act now and act fast.